ddrescue & btrfs [done]

[idella4]

avendesora

roger that. By rights I should could be satisfied with the distfiles being copied. You are suggesting to not attempt a kernel at all? I was going to stop.

This is an exercise so I'm quite happy to stop at this point and wrap up.
The kernels can be re-emerged other than most appear to have been retired from the portage tree.

I don't quite follow the

[avendesora]

Here's my opinion:

Trying to recover the distfiles is safe: portage keeps (multiple) hashes for each of those files. So if something you recovered was in fact corrupt, emerge will tell you and you can download that again.

Trying to recover anything else is risky, but could very well be worth it for valuable data - not the case here as you have stated.

Trying to recover something like kernel sources, or a compiled image, is not worth the risk involved - you have no way of determining if it is corrupt or not.


But if you want to continue this exercise in data recovery, please go ahead - I'm just warning you against using what you have recovered (except for the distfiles), and that forcibly creating I/O-crashed processes is not advisable. Follow NeddySeagoon's advice for dd, and try to come up with the tree walker yourself - the more you scratch your brain to make it work, the more you'll take out of the exercise.

Next step in the exercise would be to retry doing all that, but directly manipulating the raw image (to get rid of the I/O failures and yung dd's and all). Probably hard to real hard - best place to start looking at that would be the btrfsck source code, I think, to understand the filesystem layout. Would be easier if it was an "old" filesystem (ext2, fat or something like that). Modern filesystems tend to have more intricate on-disk structures.

[idella4]

avendesora

[NeddySeagoon]

idella4,

Copied no error, probably means different things to you and me.
No error means it was able to read the every block the damaged btrfs metadata pointed it to. That does not mean that was the right list of blocks comprising the file, nor does it mean that even if it was the right list of blocks that your data is intact inside them.

As avendesora says, if you use those files without veryfying the contents, you get to keep all the pieces.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[idella4]

Neddy,
sure. How do I verify a file? I take it you're suggesting to do so before trying to emerge a copied file. A stat? Peruse the two files via hexedit?
The btrfsck once used listed 3 lines of mis-aligned metadata.
That's way open for interpretation. Each flaw has an impact of what scale? How can you tell?
3 is a very low number?? I can only guess. Either way, this exercise is on the verge of being wrapped up.
_________________
idella4@aus

[NeddySeagoon]

idella4,

You need something like a md5sum for each file, or you need another known good file copy to do a diff against.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[idella4]

Neddy,

sure. ok. Modify my bash script to md5sum each source and destination file and run diff source destination and output it to a logfile.txt?
Are you going to tell me now the source files in gen2.img lack a standing of a known good file?
If so, I don't know if verifying them is achievable without re-fetching every one from portage.
Here is my final added extra.

[NeddySeagoon]

idella4,

You are missing the point. dd read something from somewhere in your filesystem but you don't know it was it the right something.
You need known good md5sums or known good files.

Comparing before and after the copy tells you nothing.

Think of the problem in vfat, where a corrupt FAT shows that one disk block belongs to several files. How do you tell which file it belongs to, if any?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[idella4]

ok. well, that implies I would need to re-fetch them from portage.
A few samples would be worth the effort, but not the 5,950 files.
Let's put it this way. If I were to re-emerge say the 2.6.30 kernel pretending for the moment that it were present in portage, without verifying it, I should take an image of my genny prior to emerging and attempting to compile it. If it breaks genny into pieces, to avoid the humiliation of picking up those pieces, I have genny backed up and can re-implant genny in the usual way. That is the "wrong way" to test a package file.

That said, the retrieval exercise is done. Those distfiles are in place to satisfy the exercise and any emerging I may find myself doing will likely be new packages and not draw upon those 3-4 gig.
To wrap up, testing a selected sample may suffice as a token test of a selected sample, and the remainder can remain an unknown powder keg that I can surpass with a system image backup in place intermittently updated.

The only real cost of my exercise was losing suse and the replaced state of /mnt/gentoo in a folder in suse's root.
_________________
idella4@aus

[avendesora]

I said the distfile part is safe because portage keeps hashes of known-good files, and those hashes are assumed to be sufficient.
If your recovery recovered corrupted distfiles, emerge will tell you just that and you'll have to re-download (only) the broken distfile(s).

[NeddySeagoon]

avendesora,

Thats true for things still in the tree. You can get old ebuilds from the attic but thats just the ebuilds.
Where do you recover the file hashes from?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[idella4]

what is a file hash?

[avendesora]

[idella4]

I know what md5sum and sha and the like. Is a hash or hash file just a type of an alternate md5sum style algorithm?
Making a tar of /mnt/gentoo in btrfs. I didn't realise tar was sooooo slow.
_________________
idella4@aus

[avendesora]

[NeddySeagoon]

avendesora, idella4,

Asking the gentoo infra guys, I was told the you can get the manifest/digest files along with the ebuilds in the attic.
These files will contain the hashes you need to validate the recovered source files for things no longer in the tree.
I've not tested.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[idella4]

Neddy,
you got me again. What is the attic? Never heard of that before.

[NeddySeagoon]

idella4,

Every ebuild that has ever been in portage is stored here
Looking in development-sources which is a completely dead branch now, you find nothing. it says

[idella4]

NeddySeagoon,

thanks for the explanation. Can you continue explaining?

[avendesora]

If you apply a hash function (aka digest function, aka checksum function) to the contents of a file, you get that file's hash (the hash is the output of the hash function, usually represented as a long hex string).

What is important with that hash is that:
- any file that has the same _contents_ will have the same hash
And with a good hash function:
- any alteration of that file contents will result in a different hash
(obviously not mathematically true, but close enough for usual purposes)

So at some point linux-2.6.30.tar.gz was an official distfile.
Before publishing the ebuild, someone calculated the sha1, sha256 and rmd160 hash values of that tarball (verifying with what is posted on kernel.org), and stored those hash values in the Manifest file in the sys-kernel/gentoo-sources portage directory.
(There were digest files before that, but the Manifest replaced those I think.)

The Manifest file, and all its past versions, are retrievable at the urls mentioned by NeddySeagoon and yourself.
If you can find a revision of that Manifest file old enough to include linux-2.6.30.tar.gz, assuming the CVS has not been hacked, then you know those hash values for the real, known good, kernel tarball.
You then compare those values with the hash values of the file you have recovered. If all three match, chances are really really good that the file you recovered is good. If any of them do not match, you are _certain_ that it is corrupted.

In the CVS, you'll also find the old .ebuild files, and everything that is necessary to emerge that version of the kernel.
(That is what you posted: an .ebuild.)

As for snapshots there is tons of information in search engines. Try "filesystem snapshots", and look at the homepages/wikis of btrfs, ZFS, nilfs, VxFS (if you want a commercial example), AdvFS (a historical (but still relevant) example), ... (at the very least, look at the wikipedia entries for those filesystems, and their relevant links).

If you don't understand what you are doing with those subvolumes and snapshots, please spend the time to _read_ about them.

[NeddySeagoon]

idella4,
There were 9 versions of the gentoo-sources-2.6.30-r8.ebuild, from reading the log, mostly for keyword changes on more architectures. The version of the ebuild did not change as the keywording (and other changes) had no effect on already installed users.

AMD64 users don't really want to rebuild their kernel just because its SPARC keyword was changed.

If you go into Manifest you will see all the Manifest fies for gentoo-sources. Looking inside Version 1.815 you find lines

[idella4]

[NeddySeagoon]

idella4,

The attic has always been there. Ebuilds are made available under the GPL, which says something about providing source code to anyone who asks.
There is some limit of time, or the word 'reasonable' may be used. I don't recall exactly.

Gentoo is not obliged to provide copies of files that the Gentoo only mirrors. Go to the source of those files for that. e.g. kernel.org for kernels.

[idella4]

Neddy,

yes, sure. Can only say you have delivered again. I am puzzled why pappy didn't bring up the attic in the making of the overlay.

[NeddySeagoon]

idella4,

The Manifest file changes every time there in a change to /usr/portage/<catagory>/package as it contains a list of files, their sizes and hashes in that directory and its subdirectories. Thats why there as so many versions of Manifest for gentoo-sources.