Recovering resized partitions

slackline · Last edited by slackline on Fri Dec 15, 2017 7:43 pm; edited 1 time in total

Hi,

I cocked up the other day when I went to resize a LVM partition on my RAID array. Meant to add 50Gb to my /dev/vg0/music partition but stupidly unmounted /dev/vg0/pics and added 50Gb to that. Without thinking I went ahead and tried to reduce it before using resizefs, series of commands used was...

eccerr0r · Posted: Fri Dec 15, 2017 7:15 pm Post subject:

If I'm reading this correctly, I'd be readying the backup for restore... (BTW, the command list mentioned doesn't seem complete. )

I don't know how bad the fragmentation is on your LVM but if there's was fragmentation, reducing the LVM size will lose critical information on which blocks contain the extents that used to belong to the logical volume and it'd require some careful analysis to rebuild instead of just re-extending the volume. If you don't have fragmentation, then things are easier and re-extending will work.

Fortunately if you just extended it and chopping it back off the same amount, then you shouldn't lose any information.

It looks like your volume is still the smaller size compared to the ext4fs that was on that volume...
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

slackline · Posted: Fri Dec 15, 2017 7:43 pm Post subject:

What am I missing from the command list that you expect to see? I stupidly didn't resize2fs after adding 50Gb before I tried to remove it.

From memory I think there might have been ~25% fragmentation but since the partition hasn't been mounted I'd hope there hasn't been any read/write activity (although this could just demonstrate my ignorance of how LVM works).

Any idea how I might mount get an LVM partition mounted without recourse to the partition table (essentially forcing mount to ignore trying to read the partition table). Various searches suggested its possible e.g. here and NeddySeagoons post, but I'm unsure how to work out the sector on the LVM to specify as an offset option.
_________________
"Science is what we understand well enough to explain to a computer. Art is everything else we do." - Donald Knuth

eccerr0r · Posted: Fri Dec 15, 2017 8:04 pm Post subject:

From what I'm gathering, it should be perfectly fine to extend a partition 50GB and the shrink it 50GB, as long as the two amounts are exactly the same, as long as you didn't resize2fs. But if the filesystem was ever larger than the volume size, then it's danger...

Or did you do this:

unmount
lvextend -L +50G
resize2fs
lvreduce -L -50G

If this was done, this can be dangerous as you don't know what fragments were used for the 50G that you removed and can potentially be permanently lost though I would hope resize2fs did not move anything to the space chopped off (or hope lvextend will use the same allocation groups as before you reduced). You should lvextend the partition back to the full size and fsck it, and then resize it back to its original size and then lvreduce it.

The only assumption is that the start of the volume didn't change. I don't know what you did if you somehow destroyed the beginning of the partition?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

szatox · Advocate Joined: 27 Aug 2013 Posts: 3136

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

LVM is not partitions. It's logical volumes. Space for any given LV might be allocated anywhere. If you reduce and then grow, that grown space might be located elsewhere. If it's SSD and you have issue_discards = 1 in your lvm.conf, the data is already gone. On HDD or with issue_discards = 0 you have to refer to /etc/lvm/{archive,backup} or the circular ondisk metadata for a backup before your goofed change and go from there (tell lvextend specifically to use the old extents for allocation).

slackline · Posted: Sat Dec 16, 2017 8:02 am Post subject:

szatox · Advocate Joined: 27 Aug 2013 Posts: 3136

Slackline, what puzzles me about the commands you pasted at the beginning is that you omitted -L in the first one (hence "does is it even work") and the second one you pasted increases size of the volume when you said you reduced it.

Now, extending and then reducing a volume without expanding filesystem located on it _should_ be perfectly safe. The only sane - like in "not totally useless" - way to implement it is to mimic a regular partition (you add space at the end, and you remove it from the end). The only real danger comes from rounding errors, if you happen to remove more space than you added and as a result truncate your filesystem.
Removing space from a volume and then adding it back on the other hand produces not predictable results. LVM may allocate extents located somewhere else on a physical volume or even extents from another physical volume, so (unlike regular partitions) this is not recoverable, unless you restore metadata from backup and your data hasn't been discarded/overwritten.

Finally,

slackline · Posted: Sun Dec 17, 2017 8:14 am Post subject:

Thanks for your continued assistance szatox.

I'd grep'ed my history and copy and pasted the output (omitting the row numbers) suggesting I did indeed miss out the '-L' option in the first command (I'd been awake five minutes as I have little time to do computing maintenance due to daughter, probably not the best time to do something I should be paying attention to!).

eccerr0r · Posted: Sun Dec 17, 2017 8:28 am Post subject:

Yes, this looks a lot more like what commands that could have done something. I was concerned the commands you initially posted had an error so we have to guess what actually happened.

These are the danger commands:

slackline · Posted: Sun Dec 17, 2017 2:06 pm Post subject:

Thanks also for your assistance eccerr0r, curently lvdisplay suggests /dev/vg0/pics is 200Gb

NeddySeagoon · Posted: Sun Dec 17, 2017 5:46 pm Post subject:

slackline,

A logical volume is not a partition. A partition is a sequence of contiguous (to the user) blocks on a block device of sorts.
Usually a HDD. With bad sector remapping in rotating rust and wear levelling is SSDs the physical blocks may not actually be contiguous.
The operating can't tell. The HDD/SSD hides that level of detail.
A partition is fully described by the partition table.

A logical volume is a few layers of abstraction away from a partition.
The next layer up is the phyicsical volume. It comprises at least one partition. It may use several drives. The physical volume hides the joins between multiple partitions and even lets multiple partitions be arranged in different ways.
Think raid. With two partitions, you can do raid1, raid0 or join them end to end. With more partitions, higher raid levels are possible. Its defined at pvcreate time.

Logical Volumes themselves are the divisions of the physical volume. LVM hides all this from you. You have no idea where your data will go.
On a pristine physical volume, logical volumes are allocated in single segments.
On a fragmented physical volume, a new logical volume will fill in some of the gaps and may be several extents, even when it new.
LVM hides all this from the filesystem on top of the logical volume.

Now if you can unravel all these layers of abstraction and your physical volume is simple and the logical volume is only a single extent, you can mount the filesystem it contains with the offset= option.
That's a lot of ifs and buts. Once you have a more complex structure, LVM needs to be able to join up the pieces of the filesystem before you can mount it.
Calculating offset= is non trivial.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

eccerr0r · Posted: Sun Dec 17, 2017 5:57 pm Post subject:

This should be a recoverable situation in my opinion. Unless you ran out of time, don't rush to recreating the volume.

It looks like e2fsck and lvdisplay have conflicting size reports... Not exactly sure why they differ however.

What do other programs report as the size of the pics volume, like wc -c /dev/vg0/pics ?

Fragmentation of the logical volume within the volume group may preclude one from using offsets on the physical volume...hence the worry and needing the restores. I wish there was a way to "defragment" LVM, there seems to be some programs out there to do that, but very risky... backups are very important.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

NeddySeagoon · Posted: Sun Dec 17, 2017 6:40 pm Post subject:

slackline,

pvdisplay -m is useful.

szatox · Advocate Joined: 27 Aug 2013 Posts: 3136

eccerr0r · Posted: Mon Dec 18, 2017 1:35 am Post subject:

Except this is on a rust spinner...

Despite it seeming very plausible to be on the SSD, the chances of successfully extending a 200G partition to 250G on a LVM on a 250G SSD is unlikely with all the metadata and rounding inaccuracies. Or at least it's kind of strange to have a 250G PV on an SSD with all those mechanical HDDs lying around.

---

BTW, LVM2 defragmenter though not helpful for OP... https://bisqwit.iki.fi/source/lvm2defrag.html -- If all volumes were contiguous on the physical volumes, and does not span them, then mounting with an offset will work, hence the value in defragmentation.

If there is at least one free allocation extent, it seems like it should be possible to safely defragment LVM's minus the time it needs to rewrite the metadata. Hmm...this may be a neat tool to write, but dangerous and questionable value when a logical volume necessarily spans over a physical volume.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

slackline · Posted: Mon Dec 18, 2017 2:44 pm Post subject:

Thanks NeddySeagoon, eccerr0r and szatox for your continued advice.

This is indeed a HDD volume rather than SSD, albeit two disks setup in RAID1. LVM seems nice and flexible to an end user such as myself (to date, rapidly learning!) but horrendously complicated under the hood based on your initial description in this thread NeddySeagoon.

Taking things in turn, e2fsck still reports....

NeddySeagoon · Posted: Mon Dec 18, 2017 3:43 pm Post subject:

slackline,

/dev/vg0/pics appears to be all in one piece

slackline · Posted: Mon Dec 18, 2017 4:05 pm Post subject:

Hi Neddy Seagoon,

slackline · Posted: Wed Dec 20, 2017 12:44 pm Post subject:

Found the time to sit down and run testdisk (what a comprehensive tool) and have pasted the output here.

I've redacted most of the repetitive lines to leave the following which I think shows the start/finish of each of the partitions on one of the volumes of the RAID...

NeddySeagoon · Posted: Wed Dec 20, 2017 2:10 pm Post subject:

slackline,

In theory, that's correct. but given all those false positives, its going to take a long time.
There are a few things to do to try to narrow it down.

We know that

slackline · Posted: Sun Dec 24, 2017 7:59 am Post subject:

Thanks yet again NeddySeagoon.

After a reboot, which I thought I'd already tried prior to starting this thread, the partition has automounted.

Your time and effort patiently answering questions is very much appreciated.

slackline
_________________
"Science is what we understand well enough to explain to a computer. Art is everything else we do." - Donald Knuth

NeddySeagoon · Posted: Sun Dec 24, 2017 10:02 am Post subject:

slackline,

Its a logical volume. You will confuse your readers by referring to it as a partition.

I'm pleased its fixed :)
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.