[slow file copy] buffering issues / questions !

kpoman · Last edited by kpoman on Fri Aug 05, 2011 6:45 pm; edited 1 time in total

Hello to all,

Guys I am having a problem here. I try to copy a lot of big files from one machine to another (I am using samba).

The source machine has sata disks with about 80MB/s speed, with about 4GB of RAM.
The target machine has a SAN / FiberChannel DS4700 IBM storage, with about 8GB of RAM.

I start the copy and monitor it both sides with dstat.

The copy is extremely fast for the first about 8GB of data, then goes extremely slow.

In dstat, I do see more than 40MB/s in the network input/output on both sides. So the data is indeed being transfered.

Why on earth, given the superstorage I have on the target, the overall speed is slower than a couple of MB/s ?

I dont understand this at all:
- my read-device is about 80MB/s reading.
- my write device can handle more than 300MB/s (tested generating files with dd)
- the network is gigabit ethernet and performing about 40MB/s

The overall speed is about 3MB/s !

Please help me understand the issue.
_________________
please, help me, pity on me :'(

kpoman · Posted: Sun Aug 07, 2011 9:07 pm Post subject:

any ideas ????
_________________
please, help me, pity on me :'(

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

What kernel versions are you using on the two machines? Does it help if you use a better protocol? What if you stream a file from one to the other via netcat?

kpoman · Posted: Mon Aug 08, 2011 2:20 am Post subject:

I am on 2.6.39 gentoo on both.
I tried also copy from sda to external usb disk with similar results !
_________________
please, help me, pity on me :'(

kpoman · Posted: Thu Nov 03, 2011 9:52 pm Post subject:

do someone have an idea ?
can someone try to reproduce ? I dont understand why this happens.
right now with dstat I am copying lot of data, I do see the read speed stagned at 1MB/s while in the beginning it was doing 40MB/s !
_________________
please, help me, pity on me :'(

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

I have two ideas. Try transferring the data via a raw protocol like ad-hoc streaming over netcat, to cut out the complexity of SMB. Also, try doing the copy without the use of USB-attached storage. USB mass storage devices are usable, but limitations of USB 1.1/USB 2.0 mean that it will not be as fast as copying data across an internal bus.