I work-arounded the problem by reducing the number of sectors to write at once to the device. I traced the problem by enabling all kinds of USB debugging in the kernel and always saw problems with large request after 6KB.
echo 8 > /sys/block/[your device]/device/max_sectors
This limits requests ...