HORRIBLE Disk to Disk copy/move performance

Message

taskara · Post by **taskara** » Wed Jul 02, 2003 2:47 am

you should not need tuning

just the other built directly in

edit: you're welcome

what's your output of

Code: Select all

hdparm -tT /dev/hdb[./code] ?

zakl · Post by **zakl** » Sun Jul 06, 2003 7:26 pm

I LIED!!! It was a complete fluke. The problem is not fixed! Maybe I just need to buy a new motherboard with a better chipset. This is pissing me off. I copy an average of 3-5GB at a time between drives, and these crappy 2-3MB speeds I'm getting in Linux are rediculous. It's getting frustrating now, I can copy from my FTP server across the network faster than I can copy between local ATA 100 hard drives. This is weak. Maybe the KT266a chipset just doesn't have good IDE support for Linux.

Ex.

5GB copy from FTP to local machine: 9-11MB/s solid.
5GB copy from hda to hdc: 4MB/s MAX, an average of 2-3MB/s.
Sometimes it spikes on different files, and I get 20MB/s for a couple seconds then it flops back down.

This is making me sick. It shouldn't take me 40 minutes to move 3GB of files to my other hard drive.

taskara · Post by **taskara** » Mon Jul 07, 2003 1:21 am

maybe one of the drives is faulty?

what if u ftp a file down to ur other drive, do u get the same speeds?

have u tried different hdd's?

kt266 shoudl ahve good linux support.

even if it doesn't (which it does) the generic ide chipset should have better performance!

is dma ON? I can't remember..

alec · Post by **alec** » Mon Jul 07, 2003 3:12 am

I had a similar problem a while back - grabbing from the LAN was quicker than grabbing from the hard drive - but just including every motherboard in the IDE section of my kernel and using hdparm did the trick - guess that won't help you. I have heard that 2.4.21 boasts signifigant IDE performance increases - perhaps using a 2.4.21 kernel instead of a 2.4.20 kernel would help? I'm running -ac right now, as it is the only kernel aside from the vanilla I saw that is 2.4.21.

Few more questions - have you used other linuxes or Windows with which you measured the performance you're looking for? If you boot up a LiveCD and do some copying, do you get better performance? Did the `/etc/init.d/hdparm` script not help?

zakl · Post by **zakl** » Tue Jul 08, 2003 7:26 am

taskara:
dma is on
32bit transfer on
active mode is udma5
And it can't be the drives because speeds in Win2K are on par... 20MB+/s drive to drive copy/move.

alec:
I will try a newer kernel. Probably the ac-sources or the pfeifer-sources.
Yes, I do/have used windows and other linux distro's, I don't have any tests from the linux's, but see above for the windows results.
hdparm is installed, and didn't improve performance at all.
I might try the live CD after a bit, but currently I have stuff compiling via ssh in Konsole, so I can't reboot right now.

zakl · Post by **zakl** » Fri Jul 11, 2003 10:06 am

This may sound a bit off to some of you, but I have been playing around and I just tried something new. I honestly think it is the filesystem on my 120GB drive. I formatted the whole drive as FAT32, which technically FAT32 is suppose to be able to handle much larger partitions, but I think there are bugs and flaws, being the reason that Microsoft no longer suggests it be used in todays windows installations.

Anyway, this is what I did. I started some rather large bit torrent downloads, each torrent in excess of 1.5GB. I decided to save them directly to my storage drive(the 120GB) this time, and when I saved them, it took forever in the space allocation. This leads me to believe that it is the file system giving me problems. Whenever I start large torrents like this in NTFS or reiserfs, the space is allocated almost instantly, as where it took almost 2 minutes to allocate on my FAT32 drive. Also, every time I boot to windows, windows wants to run a check on the partition. Although this could mean there are errors and they simply need to be fixed, it's not that simple, because even after fixing them it happens again. So, as soon as I get money to buy 80GB worth of CD's, I will be burning all my anime and formatting it as reiserfs.

I'll update this post when that happens to let yall know if it was in fact my problem.

MasterX · Post by **MasterX** » Fri Jul 11, 2003 3:02 pm

There was a discussion in another post about enabling the dma on a scsi. The solution was to look at the /proc/ide/hdX/settings. This is what I get for my hard disk

Code: Select all

name                    value           min             max             mode
----                    -----           ---             ---             ----
acoustic                0               0               254             rw
address                 0               0               2               rw
bios_cyl                1867            0               65535           rw
bios_head               255             0               255             rw
bios_sect               63              0               63              rw
breada_readahead        8               0               255             rw
bswap                   0               0               1               r
current_speed           69              0               70              rw
failures                0               0               65535           rw
file_readahead          124             0               16384           rw
init_speed              69              0               70              rw
io_32bit                0               0               3               rw
keepsettings            0               0               1               rw
lun                     0               0               7               rw
max_failures            1               0               65535           rw
max_kb_per_request      128             1               255             rw
multcount               16              0               16              rw
nice1                   1               0               1               rw
nowerr                  0               0               1               rw
number                  1               0               3               rw
pio_mode                write-only      0               255             w
slow                    0               0               1               rw
unmaskirq               0               0               1               rw
using_dma               1               0               1               rw
wcache                  0               0               1               rw

It seems to me that I can change the current speed to 70 and max_kb_per_request to 255 do you know if this will improve the performance? Moreover how can I change these with hdparm

zakl · Post by **zakl** » Sun Jul 13, 2003 12:11 am

I now am 100% sure it is a problem with the FAT32 file system. I downloaded from my ftp server to that partition, and it is downloading at 4MB/s. As to where when I download from the ftp server to any other partitions(W2K NTFS, or linux reiserfs) it goes at 9-11MB/s.

darkangael · Post by **darkangael** » Sat Oct 25, 2003 2:37 pm

I have the same problem.
I am using
Athlon 2400XP+
Epox 8k9a2+ motherboard (KT400)
512 meg DDR333
Pioneer DVD-ROM
Pioneer DVR-106 DVD Burner
Fujitsu MPE 3136AT 13 gig HDD(Slave)
MAXTOR 6L080J4 80 gig HDD(Master)

Copy between /home (ext3) and /mnt/winstor (fat32) is abominable. It is really slow, maxing at 20 meg/sec and averaging 3-8 meg. It also slows down the whole system while it does it (with no added CPU usage I should add).

I have DMA turned on, and the xfermode is whatever the default is (supposedly it's detected properly by default, and I am pretty sure at some earlier stage last time I tried to solve this I checked it out).

I agree totally that there is something wrong with the fat32 driver, or the filesystem itself. I have no troubles in windows with the same sort of copying (using explore2fs from my /home partition to fat32) EDIT: I just realised that it could also not be, I am no sure but I think compiling does similar things to performance, but not as bad).

By the way: The /homes and /mnt/winstor are on the same hard disk. I have the same problems copying from another partition on another disk which is also fat32.

I am using gentoo-sources 2.4.2(1?). I will test the theory on the fat32 when I boot back to linux after using my windows only programs.

pilla · Post by **pilla** » Sat Oct 25, 2003 5:21 pm

MasterX wrote:I can not find the PIIXn Tuning support. Is that a problem?

Not all kernel versions have it. You should be OK without it too.

darkangael · Post by **darkangael** » Sun Oct 26, 2003 6:50 pm

Definitely not FAT32 problem for me. It also sucks when copying /usr/src/linux to /home/darkangael

hdparm -tT gives

Code: Select all

/dev/hda:
 Timing buffer-cache reads:   128 MB in  0.52 seconds =246.15 MB/sec
 Timing buffered disk reads:  64 MB in  1.69 seconds = 37.87 MB/sec

hdparm -I gives

Code: Select all

/dev/hda:

ATA device, with non-removable media
        Model Number:       MAXTOR 6L080J4
        Serial Number:      664216412353
        Firmware Revision:  A93.0500
Standards:
        Used: ATA/ATAPI-5 T13 1321D revision 1
        Supported: 5 4 3 2 & some of 6
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  156355584
        device size with M = 1024*1024:       76345 MBytes
        device size with M = 1000*1000:       80054 MBytes (80 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        bytes avail on r/w long: 4      Queue depth: 1
        Standby timer values: spec'd by Vendor, no device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Recommended acoustic management value: 128, current value: 254
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    READ BUFFER cmd
           *    WRITE BUFFER cmd
           *    Host Protected Area feature set
           *    Look-ahead
           *    Write cache
           *    Power Management feature set
                Security Mode feature set
           *    SMART feature set
           *    Mandatory FLUSH CACHE command
           *    Device Configuration Overlay feature set
           *    Automatic Acoustic Management feature set
                SET MAX security extension
           *    DOWNLOAD MICROCODE cmd
           *    SMART self-test
           *    SMART error logging
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
        not     frozen
        not     expired: security count
        not     supported: enhanced erase
        40min for SECURITY ERASE UNIT.
HW reset results:
        CBLID- above Vih
        Device num = 0 determined by the jumper
Checksum: correct

It turns out I am using 2.4.20 kernel. Gonna test it out with the 2.6 kernel I built a while ago as my opengl performance is better with it (although nwn still crashes, and somewhat harder I think

)

darkangael · Post by **darkangael** » Sun Oct 26, 2003 7:13 pm

using 2.6.0-test6-mm1 I get really strange results. Everything appears to be quicker, but konqueror says it's still 3.6 meg/sec and gkrellm says 0. Gkrellm says that the CPU goes up to 100% as well, which is just wrong. top says that the cpu usage is "iowait". Anyone know what this is? It never used to use CPU. It's still using udma6 mode and DMA is on like before.

hdparm -tT gives

Code: Select all

/dev/hda:
 Timing buffer-cache reads:   128 MB in  0.45 seconds =285.76 MB/sec
 Timing buffered disk reads:  64 MB in  1.79 seconds = 35.74 MB/sec

ww9rivers · Post by **ww9rivers** » Wed Jan 14, 2004 7:55 am

I would also like to share my bit of expierence regarding disk performance issue.

I have 3 disks in a Dell 400SC box, using gs-sources-2.4.23_pre8. I got very bad disk performances. I tweaked a few "hdparm" settings but things went only slightly better. After reading this thread and went into my kernel settings, I found out that I had the PIIx support compiled as a module, which caused problems when I tried to turn on "using_dma" with hdparm.

After "modprobe piix", I was able to turn on "using_dma" which improved "hdparm -Tt" output of "buffered disk reads" from less than 5MB/sec to above 56MB/sec.

HORRIBLE Disk to Disk copy/move performance

Thanks for sharing