Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
re-creating a raid 1 on new system
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
seba_gentoo
n00b
n00b


Joined: 21 Jul 2016
Posts: 7
Location: Bavaria

PostPosted: Thu Jul 21, 2016 9:59 am    Post subject: re-creating a raid 1 on new system Reply with quote

Dear forum members,

I was forced to set up a new gentoo system. I had an old one with software (mdadm) raid 1 running.
I have issues with creating and mounting this two disks to a raid and make it mountable.
I, of course, don't want to lose my data stored on this drives.

#cat /proc/mdstat
Personalities : [raid1]
md127 : inactive sdb1[1](S)
976630488 blocks super 1.2

md0 : active raid1 sda1[0]
976761472 blocks [2/1] [U_]
bitmap: 1/8 pages [4KB], 65536KB chunk

unused devices: <none>

#mount /dev/md0 /mnt/md0
mount: frong fs type, bad option, bad superblock on /dev/md0
missing codepage or helper program, or other error

#dmesg| tail
[ 3421.400766] EXT3-fs error (device md0): ext3_check_descriptors: Block bitmap for group 1 not in group (block 0)!
[ 3421.454371] EXT3-fs (md0): error: group descriptors corrupted


Could someone help me out of this problem?
Thanks a lot!

sebastian
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Thu Jul 21, 2016 10:21 am    Post subject: Reply with quote

file -s /dev/md0?

mdadm --examine /dev/sd*?
Back to top
View user's profile Send private message
pilla
Bodhisattva
Bodhisattva


Joined: 07 Aug 2002
Posts: 7729
Location: Underworld

PostPosted: Thu Jul 21, 2016 10:58 am    Post subject: Reply with quote

Hello,

Looks like you got some nasty file system corruption in your RAID.

First, sdb1 seems to be a dead disk. So I would try first to fsck it directly, then try to mount it without using the RAID system.

Code:

fsck.ext3 -f -p -v /dev/sdb1
mkdir /mnt/sdb1
mount /dev/sdb1 /mnt/sdb1


There are other flags for fsck, but I think those are a good start (force even if clean, fix automatically, verbose).

If it does not work, you can try it with sda1. Software RAID should not have special stuff in the disk for RAID 1, so you should be able to mount the disks without it separately.
_________________
"I'm just very selective about the reality I choose to accept." -- Calvin
Back to top
View user's profile Send private message
seba_gentoo
n00b
n00b


Joined: 21 Jul 2016
Posts: 7
Location: Bavaria

PostPosted: Thu Jul 21, 2016 11:44 am    Post subject: Reply with quote

file -s:
/dev/md0: Linux rev 1.0 ext3 filesystem data, UUID=4fefcfa7-16d4-48fc-84d2-44d5adbc3694 (errors) (large files)

examine sda1:
/dev/sda1:
Magic : a92b4efc
Version : 0.90.00
UUID : 3836eeda:095a9bde:94d4a16f:27c4d8e2 (local to host seba_desktop)
Creation Time : Wed Jul 20 23:27:31 2016
Raid Level : raid1
Used Dev Size : 976761472 (931.51 GiB 1000.20 GB)
Array Size : 976761472 (931.51 GiB 1000.20 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 0

Update Time : Thu Jul 21 13:52:13 2016
State : clean
Internal Bitmap : present
Active Devices : 1
Working Devices : 1
Failed Devices : 1
Spare Devices : 0
Checksum : 90af2c78 - correct
Events : 27


Number Major Minor RaidDevice State
this 0 8 1 0 active sync /dev/sda1

0 0 8 1 0 active sync /dev/sda1
1 1 0 0 1 faulty removed

examine sdb1:
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ba924d1c:a85f2460:dc03db17:da419e2e
Name : seba_desktop:127 (local to host seba_desktop)
Creation Time : Wed Jul 13 10:58:39 2016
Raid Level : raid1
Raid Devices : 2

Avail Dev Size : 1953260976 (931.39 GiB 1000.07 GB)
Array Size : 976630464 (931.39 GiB 1000.07 GB)
Used Dev Size : 1953260928 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=48 sectors
State : clean
Device UUID : 5d939af1:fd090aec:6f6218c6:1c334139

Internal Bitmap : 8 sectors from superblock
Update Time : Wed Jul 20 23:23:12 2016
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 5d3aed23 - correct
Events : 2398


Device Role : Active device 1
Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

regards,
sebastian
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Thu Jul 21, 2016 12:10 pm    Post subject: Reply with quote

These don't really make much sense. One is 0.90 metadata, the other is 1.2 metadata. Both state they were created just a day/a week ago. So neither of them is "an old one". Did you mess around with mdadm --create?

sdb1 has a data offset of 262144 sectors, you can try and see if there is anything that makes sense at this offset.

Code:

losetup --find --show --read-only --offset=$((262144*512)) /dev/sdb1
file -s /dev/loop?
mount -o ro /dev/loop? /mnt/somewhere
Back to top
View user's profile Send private message
seba_gentoo
n00b
n00b


Joined: 21 Jul 2016
Posts: 7
Location: Bavaria

PostPosted: Thu Jul 21, 2016 6:39 pm    Post subject: Reply with quote

[quote="frostschutz"]These don't really make much sense. One is 0.90 metadata, the other is 1.2 metadata. Both state they were created just a day/a week ago. So neither of them is "an old one". Did you mess around with mdadm --create?

sdb1 has a data offset of 262144 sectors, you can try and see if there is anything that makes sense at this offset.

[code]
losetup --find --show --read-only --offset=$((262144*512)) /dev/sdb1
/dev/loop0
file -s /dev/loop?
/dev/loop0: data
mount -o ro /dev/loop? /mnt/somewhere
mount: unknown filesystem type 'linux_raid_member'
[/code][/quote]

Yes, I mess around with mdadm.
First I tried to create a new raid according to this manual:
https://wiki.gentoo.org/wiki/Complete_Handbook/Software_RAID
I didn't make the step with mkfs because I don't want to lose my data.
Than I found a post and tried to create a new raid with one missing disk.

How to (which mode) remove all the settings and start from scratch?

Thanks for your efforts
sebastian
Back to top
View user's profile Send private message
seba_gentoo
n00b
n00b


Joined: 21 Jul 2016
Posts: 7
Location: Bavaria

PostPosted: Thu Jul 21, 2016 6:57 pm    Post subject: Reply with quote

[quote="pilla"]Hello,

Looks like you got some nasty file system corruption in your RAID.

First, sdb1 seems to be a dead disk. So I would try first to fsck it directly, then try to mount it without using the RAID system.

[code]
fsck.ext3 -f -p -v /dev/sdb1

[b]/dev/sdb1: Note: if several inode or block bitmap blocks or part
of the inode table require relocation, you may wish to try
running e2fsck with the '-b 32768' option first. The problem
may lie only with the primary block group descriptors, and
the backup block group descriptors may be OK.

/dev/sdb1: Block bitmap for group 1 is not in group. (block 0)[/b]


mkdir /mnt/sdb1
mount /dev/sdb1 /mnt/sdb1
[/code]

There are other flags for fsck, but I think those are a good start (force even if clean, fix automatically, verbose).

If it does not work, you can try it with sda1. Software RAID should not have special stuff in the disk for RAID 1, so you should be able to mount the disks without it separately.[/quote]

Sorry, but mount is not possible.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2977
Location: Germany

PostPosted: Thu Jul 21, 2016 7:31 pm    Post subject: Reply with quote

seba_gentoo wrote:

Yes, I mess around with mdadm.
I didn't make the step with mkfs because I don't want to lose my data.


Unfortunately, mdadm --create already loses you data. You don't ever use mdadm --create if there is data you want to keep on it (unless you know really really really well what you're doing and even then you usually need a lot more options than a regular --create to make things work). If the RAID is already there all you need is --assemble, if that does not work you --examine and ask for help.

mdadm writes metadata to the start of the disk (1.1, 1.2 metadata) or to the end of the disk (0.90, 1.0 metadata) and either will overwrite your filesystem data in those places - depending on where that filesystem was before.

Unfortunately I'm not sure if/how this can be repaired in your particular case. fsck is not a tool that magically fixes things... fsck makes a lot of guesswork and then writes to the disk, if the guesses were wrong, fsck caused even more damage instead of fixing anything. That's why I sometimes refer to fsck as being a file system "crunch killer" instead of a "checker".

Quote:
How to (which mode) remove all the settings and start from scratch?


mdadm --zero-superblock would remove mdadm metadata, then you could try with whatever tools (testdisk/photorec, fsck with and without backup-superblocks, ...).

Before you experiment with fsck, consider putting an overlay/snapshot on it (or pulling a full copy) so you can undo any further damages if your experiments go wrong.

https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file (and overlay functions below that)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54030
Location: 56N 3W

PostPosted: Thu Jul 21, 2016 8:01 pm    Post subject: Reply with quote

seba_gentoo,

Tell us exactly what you did. Posting /root/.bash_history may be useful.
What sort of raid superblock did you have originally?

If you originally used version 0.9, that has the raid data at the end of the drive and the filesystem starting in the usual place, when you overwrite that with a version 1.2 raid superblock, you destroy the start of the filesystem.
In particular, the primary filesystem superblock is gone. However, it may still mount, if you tell mount to use an alternate superblock.

Whenever something bad happens to a HDD, the last thing to do is to run anything that writes to the disk. It can only make a bad situation worse.
Data recovery requires a bit of guesswork, so the first step is to make an image, so you have an undo facility in case you make a wrong guess.

Most of your data will still be there but you will not be able to recover any of it 'in place'.
If we get something to mount and if files are still there, you will need to copy them.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
seba_gentoo
n00b
n00b


Joined: 21 Jul 2016
Posts: 7
Location: Bavaria

PostPosted: Thu Jul 21, 2016 8:35 pm    Post subject: Reply with quote

[quote="NeddySeagoon"]seba_gentoo,

Tell us exactly what you did. Posting /root/.bash_history may be useful.
What sort of raid superblock did you have originally?

I did it really bad!: (some most important things from bash_history)

mdadm --create /dev/md0 --auto md --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
mount /dev/md0 /mnt/stick/
mdadm --assemble --run --level=1 /dev/md0 --raid-devices=2 /dev/sda1 /dev/sdb1
mdadm --assemble --run /dev/md0 --raid-devices=2 /dev/sda1 /dev/sdb1
mdadm --assemble --run /dev/md0
mdadm --assemble --run /dev/md0 /dev/sda1
mdadm --assemble --run /dev/md0 /dev/sdb1
mdadm --stop /dev/md127
mdadm --stop /dev/md0
mdadm --create /dev/md127 --auto md --level=1 --raid-devices=2 /dev/sda1
mdadm -A /dev/md127 -f --update-summaries /dev/sda1 /dev/sdb1
mdadm --examine /dev/sda1 /dev/sdb1
mdadm -S /dev/md127
mdadm -S /dev/md0
fdisk -l /dev/sda
mdadm -E /dev/sda1
mdadm -E /dev/sdb1
mdadm -A /dev/md0 /dev/sdb1 missing
mdadm --create /dev/md0 --metadata=0.9 --raid-devices=2 --level=1 /dev/sdb1
fsck.ext2 /dev/sda1
fsck.ext2 /dev/sdb1
e2fsck -b 32768 /dev/md127
e2fsck -b 32768 /dev/sda1
e2fsck -b 32768 /dev/sdb1

I'll try to secure some data with ddrescue before I do something with mdadm.
Thanks guys!
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 54030
Location: 56N 3W

PostPosted: Thu Jul 21, 2016 9:46 pm    Post subject: Reply with quote

seba_gentoo,

Code:
mdadm --create /dev/md0 --auto md --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1

This command would write new raid version 1.2 metadata to the end of both drives, destroying what was already there.
If it was already version 1.2 metadata no user data is destroyed. It also syncs the two parts of the mirror, making them identical.
I don't know how that is done.
You did not make any filesystem - thats good.
The you did
Code:
mount /dev/md0 /mnt/stick/

Did that command work?
If your original raid used version 1.2 metadata, it might. The filesystem may still have been there.
If you originally had 0.9 metadata, the --create destroyed the start of the filesystem and the mount will fail.

What to try ...

If your raid superblocks were originally 0.9 then
Code:
mount -o ro,sb=131072 /dev/sda1 /mnt/stick
will try to use the first backup fs superblock to mount the filesystem.
There are other backup superblocks too

Now, if you originally had 1.2 raid superblocks, the first backup superblock is 262144 blocks further down the drive.
Looking at those numbers 262144 is bigger than 131072 so the first backup fs superblock may be gone as well.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
seba_gentoo
n00b
n00b


Joined: 21 Jul 2016
Posts: 7
Location: Bavaria

PostPosted: Fri Jul 22, 2016 7:17 am    Post subject: Reply with quote

NeddySeagoon wrote:
seba_gentoo,

The you did
Code:
mount /dev/md0 /mnt/stick/

Did that command work?
No, the mount command never worked. There was always a problem with wrong fs type, bad option etc.

If your raid superblocks were originally 0.9 then
Code:
mount -o ro,sb=131072 /dev/sda1 /mnt/stick
will try to use the first backup fs superblock to mount the filesystem.
There are other backup superblocks too

In this case mount replies with:
Code:
mount: unknown filesystem type 'linux_raid_member'

Now, if you originally had 1.2 raid superblocks, the first backup superblock is 262144 blocks further down the drive.
Looking at those numbers 262144 is bigger than 131072 so the first backup fs superblock may be gone as well.


Thank you for your proposals.
sebastian

I am about to order a new drive and secure the image if this disk and to go to more risky things.
Back to top
View user's profile Send private message
seba_gentoo
n00b
n00b


Joined: 21 Jul 2016
Posts: 7
Location: Bavaria

PostPosted: Sat Aug 06, 2016 8:44 pm    Post subject: Thanks Reply with quote

Hello Forum Members.

I succeeded. I used testdisk to recover the data.
I created new raid and made new file system.

Thanks for all your proposals.

stay tuned,
sebastian
Back to top
View user's profile Send private message
pilla
Bodhisattva
Bodhisattva


Joined: 07 Aug 2002
Posts: 7729
Location: Underworld

PostPosted: Sun Aug 07, 2016 11:27 am    Post subject: Reply with quote

Good news!

Just remember that RAID is not a substitute for backup. You should backup your important data and store somewhere else. You could even use one of the many cloud services with encryption.
_________________
"I'm just very selective about the reality I choose to accept." -- Calvin
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum