Buy a hard drive enclosure for $20 and put your own hard drive in it. Doing so will cost half as much as buying something sold as an "external hard drive", but it will work exactly the same (connects using USB).
Also, solid state (no moving parts) USB devices are not faster than USB hard drives. Test this in Sandra if you don't believe me. Sandra's own reference benchmarks show this:
80GB external hard drive= ~27MB/s
1GB SD card = ~17MB/s
The only reason for buying USB storage with no moving parts is because it's less likely to fail. Then again, what is the chance of both your normal and backup drive failing at the same time? That has to be like one in a billion.