Need help on RAID5->LVM->ext3 throughput

RayOfLight

Hello,

I've been running a 6x2TB RAID5 setup for about many years, and all this time I have felt the speed of this setup is awfully slow, so I should be ashamed of having 6 HDDs working slower than one by itself

when I run hdparm I get:

Timing cache reads at about 11GB/s, and
Timing buffered disk reads, about 150MB/s for raw /dev/sd* drives

when I go up to /dev/md, the speed goes up to 390 MB/s

if I go to the lvm drive, /dev/dm-*, the speed stays at md speed: 390 MB/s or so.

so far so good!

now, dd timing, if I run dd if=(sd*,md*,dm-*), I get about 135MB/s, 450MB/s, and 450MB/s respectively, that's awesome!

But now the sad truth:

blopsalot · Apprentice Joined: 28 Jan 2017 Posts: 231

one of the main advantages of ext4 over ext3 is significantly faster fsck. or consider xfs.

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

RAID might improve linear speeds but seeks remain as slow as always. That is also why RAID0 to make system faster is mostly a hoax. That's also why SSD is so insanely fast... no seek times. That matters a lot more than linear speeds (which is the one thing HDDs don't completely stink at).

Your fsck is a task that relies heavily on seeking all over the place. If read-only, it should perform about the same as a single drive would. If there are writes, mdadm has to do additional reads to recalculate parity, additional writes to update parity, as well as mdadm metadata (update times, event counts, ...). Many small writes are kind of worst case for RAID as that's where the RAID overhead kicks in.

RayOfLight · Posted: Sun Jan 21, 2018 1:39 am Post subject:

Thanks for the replies, although my ext3 FSs (I built the array 8 years ago) are tuned with low inode count and a big bytes-per-inode amount, I'll check on changing my FS, I've just read that when XFS fails, it loses data and ext4 is better, I'll look if I go full-BTRFS... .

On the other comment, fsck is almost full read-only, I'm not complaining right now on write speeds, and about 600-1400 kB/s is... crazy, are you sure that's normal?

I have to be glad that LV is 1.5TB and not 10TB, I'll try moving its contents to another LV and creating a new LV with a different FS.

blopsalot · Apprentice Joined: 28 Jan 2017 Posts: 231

RayOfLight · Posted: Sun Jan 21, 2018 2:27 am Post subject:

I'm working on it, I have just discovered that my kernel has no ext3 support, only ext4.
fstab is set to ext3 and it has never complained very loudy:

Hu · Moderator Joined: 06 Mar 2007 Posts: 21589

If this is a first generation 4K drive, it is possible that you misaligned the partition. Some of the first generation drives did not adequately advertise their alignment requirements, so tools tended to align partitions for DOS compatibility. That can hurt throughput by causing unnecessary I/O, especially when writes are made (though you did say this is a read-mostly workload).

The kernel ext4 driver can be used for ext3 filesystems, and this is a very common approach. It will not enable any of the ext4-specific features that would prevent the filesystem later being interpreted as ext3. For that, you need to explicitly migrate the drive to ext4.

RayOfLight · Posted: Sun Jan 21, 2018 6:04 am Post subject:

ST2000DL003-9VT166

I remember being extra concious when building the array back then, but I still don't know how to check if it is aligned or not as smartmontools report:

mike155 · Posted: Sun Jan 21, 2018 11:04 am Post subject:

NeddySeagoon · Posted: Sun Jan 21, 2018 11:05 am Post subject:

RayOfLight,

Look at your partition tables in sectors. That's always 512B sectors.
Divide the start sector by 8. If you get remainder zero, you are correctly aligned.

Don't pay too much attention to what smartctl -i says. Many early 4k sector drives lied to the outside world about the physical sector size.

This drive is clearly not 4k aligned.