Rainbow has it exactly right -- in order to be useful as a performance enhancer the flash media needs to meet a minimum read, write and random access speed criteria before it will be used.
Many of the larger devices are meant to have data stream-written to them (ie high speed, high megapixel camera.) Readyboost needs a device with high read and write speeds, along with high speeds under fully random accesses. Most "big" flash media don't meed that criteria in the way they're designed.
My ancient Kingston 256mb econo-class flash USB stick is readyboost compliant, but a year-old 80x 4GB SD card isn't. Guess where it fails? Random access.
If you want to know why your device is rejected, Vista logs that data in the performance event logs. Go looking for readyboost and you'll find where it does the performance profiling and reports the data (and if that data meets the requirements.)