LVM RAID questions...

Zucca · Posted: Tue May 18, 2021 4:46 pm Post subject: LVM RAID questions...

So. First time for everything - my server's btrfs array has some errors. Data is there and I can copy the stuff from there using normal means or copying.

I'm now in process of removing a single 3TB disk from the array (btrfs is doing it's balance to ensure my files still have two copies).

I could format a new btrfs on to that 3TB disk. Then move another disk from the original btrfs disk array into the new one. Then copy files over from the old array to the new. Finally (reformat) move the rest of the disk from old array into the new. Bang. Straight forward and plain simple - that's what btrfs is; easy to use, but slow (although speed isn't an issue here).

Since the LVM(2) has the ability to use mdraid functionality under the hood...
I could on the other hand use LVM. Add the single 3TB disk as a base of new vg. Then remove another disk from btrfs array and add it too to the vg. Then create some lvs for /, swap, /home, maybe /var... Then copy files over and finally add the rest of the disks to the vg.

At this stage can I change the data ordering of my lvs? For example can I change / to RAID6, swap to RAID5, /home to RAID10... without copying files back and forth? Are these actions "online", meaning I can perform them while lvs are in use?

I think I have more questions too, but let's start with these.

Thank you.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

NeddySeagoon · Posted: Tue May 18, 2021 6:12 pm Post subject:

Zucca,

Zucca · Posted: Tue May 18, 2021 7:15 pm Post subject:

Zucca · Posted: Wed May 19, 2021 8:43 am Post subject:

Now that I would need to reformat the entire five disk array at once, I might as well go full lvm raid.

I'm thinking of using ext4 and xfs. Or do people use other filesystems on lvm?
Opinions and expariences?
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

Hu · Moderator Joined: 06 Mar 2007 Posts: 21631

I use ext4 on LVM because ext* is traditional. I have no complaints about it, but neither do I have any arguments against using other well known filesystems on LVM.

Zucca · Posted: Thu May 20, 2021 3:34 pm Post subject:

loads

Striped allocation - ensures that data allocations are aligned with the stripe unit. This increases throughput.

Does any of these well supported, single disk, filesystems have compression support? zstd would be nice. I think f2fs has, but I'm running spinning rust still on this box.

Right now I'm leaning against a mix of ext4 and xfs. Although sticking with single fs (at least with the directories that system needs to boot) would ease the job of constructing kernel and initramfs.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

Zucca · Posted: Sat May 22, 2021 6:44 am Post subject: jfs... hmmm.

i've been rftm'ing here and there and also looked at the tables on wikipedia (which maynot be up-to-date).

HAMMER(2) is one reason I'd like to try out DragonFly BSD someday.

Anyways. Looks like jfs is one option too. Although there seems to bee fragmentation issue. That is, there's no way to do online defragmentation of jfs on Linux, afaik.

Oh well.
I have few days off from work, starting tomorrow. I'll have then decided the layout of my filesystems.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

wwdev16 · n00b Joined: 29 Aug 2018 Posts: 52

I'm curious to know how things have worked out, and if you have used any encryption facilities.

I stopped using lvm because I couldn't get a server to not hang during shutdown with:

md raid -> luks on whole device -> lvm pv -> lvm group -> / as lv due to / still being mounted ro. So I am wondering if you get a clean shutdown or reboot.

As far as my experience:

xfs usually not a problem unless the kernel fs has a bug like in the recent
past. IIRC it might have been 5.11.
initramfs generators (e.g. dracut) become hard to use with more layers and w/
non-typical usage. Like having the luks device key be on flash drive that is only
present during boot. Or say having a gpt partition-table on top of a luks device.

Zucca · Posted: Wed May 26, 2021 8:47 am Post subject:

Zucca · Posted: Wed May 26, 2021 10:31 am Post subject:

I'm creating lvm raid6 by running:

Zucca · Posted: Thu May 27, 2021 11:28 am Post subject:

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

Is RAID a solution worth using for your use case? I'm not a fan of it in my home systems, since they lack some of the niceties that otherwise make it useful:

Zucca · Posted: Thu May 27, 2021 4:15 pm Post subject:

pjp,

I do have leds on the hotswap cage. Although those are activity leds. One can still conclude which drive is in question by:

wwdev16 · n00b Joined: 29 Aug 2018 Posts: 52

zucca,

If you end up building a raid again, you may want to look at md raid. It doesn't
need udev rules to get a stable device name and it can easily be used in an initramfs.

Zucca · Posted: Fri May 28, 2021 6:40 pm Post subject:

wwdev16,

device names

mapping

physical

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

Zucca,

I didn't want to derail the thread too much, but as it is at least somewhat related, what dock/cages are you using for hotswap? Some of the stuff I've looked at seemed of questionable quality.
_________________
Quis separabit? Quo animo?

Zucca · Posted: Sat May 29, 2021 6:26 am Post subject:

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

Thanks, I was probably being cheap when I looked at theirs (well, cheap without a recommendation).

I have my drives nicely spaced, so I'm not sure I want them that close together. But I'm wondering if I could use the single drive version internally. Nope, I'm thinking of a different case with easier side access. I guess it'll have to go on the shopping list if I ever find a suitable "NAS" case.
_________________
Quis separabit? Quo animo?

Zucca · Posted: Fri Jun 18, 2021 6:06 am Post subject:

I have now all the things running... Although i need to tune my initramfs init script a little more.

By moving from btrfs based storage I now need to have some way to do snapshots or something that'll save me from accidental "rm actions" etc...
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

wwdev16 · n00b Joined: 29 Aug 2018 Posts: 52

Does CONFIG_DM_SNAPSHOT help at all?

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

Zucca, something like LVM? Tends to work pretty well.
Until your thin pool overflows and explodes, which is unrecoverable. Honestly.... Why doesn't it just freeze when running out of space?

Anyway, full snapshots are an option too. If they overflow, you only lose that one snapshot and not the whole pool. Obviously, less efficient in terms of performance and storage used, but much safer.

Zucca · Posted: Fri Jun 18, 2021 3:57 pm Post subject:

Ewww. lvm snapshotting sounds half done. :E

I think I'll resort to rdiff-backup or similar which has "history" function.
_________________
..: Zucca :..
Gentoo IRC channels reside on Libera.Chat.
--

pjp · Administrator Joined: 16 Apr 2002 Posts: 20067

Not really half done, they're intended to be temporary. VMWare snapshots are short-term use as well.

Backup: snapshot, backup from the snapshot, delete snapshot.
Upgrade: snapshot, upgrade, verify upgrade, rollback or delete snapshot.

I've only tested the rollback part once in a VM. The only problem I recall was /boot was not part of LVM, so kernel changes would have needed to be handled outside of the rollback feature (I forget the command, it's called something else).
_________________
Quis separabit? Quo animo?

szatox · Advocate Joined: 27 Aug 2013 Posts: 3135

it's lvconvert --merge.
Anyway, yeah, I do use lvm and snapshots. Still, blowing all your data up just because you've ran out of extents is something I'd classify as a critical failure. It shouldn't happen on a persistent storage, regardless of how long you're going to keep that snap.
Full snaps are OK (as long as you can afford the performance hit).