Issues checking a disk [Solved]

iandoug · Last edited by iandoug on Mon Nov 08, 2021 6:36 pm; edited 1 time in total

Hi

We're are suffering loadshedding again which means frequent shut downs and reboots.

Yesterday at boot, it found and corrected some errors on sde1

Today it got stuck on sde1 again, then said UNEXPECTED INCONSISTENCIES, run frck manually.

Then it just sat there and looked at me.... eventually I pressed the reset button, on this reboot i got stuck there again and eventually printed message similar to below, then finished boot after a while.

After booting, I ran fsck.

NeddySeagoon · Posted: Sun Oct 10, 2021 4:37 pm Post subject:

iandoug,

Is the partition table still intact?

What does

iandoug · Posted: Sun Oct 10, 2021 5:13 pm Post subject:

Hi Neddy

fdisk finds nothing.

Forgot to mention that during boot it says that that drive was not registered with DBUS, "despite waiting 1000..000 ms".

Tried to find exact message in dmesg, it's not there, but all this is:
https://pastebin.com/hxi9bqU5

It's a 500GB Data disk (WD, FWIW), EXT4 I think. Don't think there was anything crucial on there. Have no backups, my tape drive is broken... :-(

I see one of the fans is also not running. But drive is sharing power with another drive which is ok.

Let me see if another SATA channel works ...
_________________
Asus X570-PRO, Ryzen 7 5800X, GeForce GTX 1650, 32 GB RAM | Asus Sabertooth P990, AMD FX-8150, GeForce GTX 560, 16GB Ram

mike155 · Posted: Sun Oct 10, 2021 5:26 pm Post subject:

iandoug, dmesg shows many hardware/bus errors.

Fix the hardware before running fsck - otherwise it is possible that fsck deletes your data...

iandoug · Posted: Sun Oct 10, 2021 5:48 pm Post subject:

NeddySeagoon · Posted: Sun Oct 10, 2021 8:08 pm Post subject:

iandoug,

figueroa · Posted: Tue Oct 12, 2021 3:40 am Post subject:

Also, don't ignore the power connection. Easy quick check is swap connections with other SATA device. A recent disk issue on our school's business office desktop turned up a cracked power connector.

What is loadshedding?
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

Hu · Moderator Joined: 06 Mar 2007 Posts: 21633

figueroa · Posted: Tue Oct 12, 2021 3:48 pm Post subject:

Thanks, Hu. When this was first posted, I tried an Internet search for the term and got nothing meaningful. Today, it's all about rolling blackouts, so I may have mistyped it. I have UPSs on every electronic device in the house out of abundance of caution, yet we enjoy just about the most stable power I could ever imagine, and yet it's a rural electric cooperative that buys power primarily from Georgia Power.

ADDED: At the school where I support the network and desktop PCs (500 miles remote from me), we also use UPSs on mission critical computers and network devices. We've been luck and have not suffered any physical damage that can be tracked to power outages. We do, however, have a lot of seriously old (over 8 years) hard drives that periodically swan dive out of service without warning. Fortunately, backups are automatic and redundant.
_________________
Andy Figueroa
hp pavilion hpe h8-1260t/2AB5; spinning rust x3
i7-2600 @ 3.40GHz; 16 gb; Radeon HD 7570
amd64/23.0/split-usr/desktop (stable), OpenRC, -systemd -pulseaudio -uefi

iandoug · Posted: Tue Oct 12, 2021 5:27 pm Post subject:

Load-shedding is the South African term.

Overseas they use rolling blackouts or brownouts.

We are currently enduring load-shedding since last Thursday, due to end this Thursday.

We're at "Level 2" but how that is implemented depends on where you are and who you get your power from.
I'm in Cape Town and we have a pumped storage hydro-electric scheme, so municipality pumps water to top dam at night, and then generates during the day, meaning that Cape Town reverts to Level 1 during day while rest of country is at level 2.

I have UPS in office and another for the NAS servers, Fibre box, etc, but they can't stay up for the 2 - 2.5 hours that we shed at a time.

So I need to keep shutting down the computers (except firewall) and restarting, which is a pain when they go off from 2AM to 4AM etc ... I have cron jobs that run in the night/early morning.

I think two of my older PCs suffered damage because neither will boot ... they were in another room sans UPS... it's the power surge when it comes on I think ... as the power network itself stabilises. Been reports of people losing fridges etc as well.

Cheers, Ian
_________________
Asus X570-PRO, Ryzen 7 5800X, GeForce GTX 1650, 32 GB RAM | Asus Sabertooth P990, AMD FX-8150, GeForce GTX 560, 16GB Ram

iandoug · Posted: Wed Oct 13, 2021 5:30 pm Post subject:

Hi guys

Okay, I transplanted the problematic drive from the Trooper box to the Fractal box. The Trooper box dates from 2013 and has basically been on 24/7 since then.

I no longer get those hardware errors when booting Trooper. Possibly the drives are now running at 6 instead of 1.5. The PC was "sluggish" before.

Installing in Fractal meant rearranging the internal structure, which was not without issues....

Anyway, the BIOS/Kernel/Whatever is now playing silly buggers with me... plugging in the drive breaks the boot process... either because it can't find sda3, or can't find the madm drives.

After much creative rearranging of cables to sockets, I have given up and decided to switch to UUID syntax in fstab. Why this is necessary is beyond me, drives should be allocated in order of hardware port number, not randomly.

Now I need some advice in fixing fstab so that it still boots and does not leave me looking for a rescue thumbdrive. :-)

I have:

NeddySeagoon · Posted: Wed Oct 13, 2021 7:04 pm Post subject:

iandoug,

UUID is a property of a filesystem. If you make a new filesystem over an old one, the UUID is changed.
PARTUUID is a property of a partition. It will not change regardless of what happens to any filesystem that partition may hold, even if the filesystem is changed.

When you give UUID (or PARTUUID) in fstab, mount looks at all known locations until it finds the UUID you specified and returns the kernel major:minor device numbers, and says to the kernel, its that one.

In your situation, the major:minor device numbers keep changing.
The relationship between sd* and the device numbers is fixed.

iandoug · Posted: Wed Oct 13, 2021 8:16 pm Post subject:

Hi Neddy

System is detecting old drive first, in preference to new ones :-)

So is this better then?

Hu · Moderator Joined: 06 Mar 2007 Posts: 21633

For filesystems other than root, you can test this live by unmounting the relevant filesystem, then using mount /home (for example). mount is then forced to use fstab to discover that you want a UUID based mount, and in turn forced to do the UUID-driven search to find /home.

The PARTUUID for boot looks a character short. Other than that, the identifiers seem to match with your earlier lsblk output, and you are correctly matching uuid to uuid and partuuid to partuuid. (Giving one where the other is needed would fail, but you got this part correct.)

NeddySeagoon · Posted: Wed Oct 13, 2021 8:39 pm Post subject:

iandoug,

In fstab, it makes no difference. To read fstab, root has to be mounted.

-- edit --

You may not umount /home if a normal user is logged in as it will be in use.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

iandoug · Posted: Wed Oct 13, 2021 8:48 pm Post subject:

Hi Guys

Thanks.

Do I need to fiddle with Grub at all? Or will this "just work" ?

Thanks, Ian
_________________
Asus X570-PRO, Ryzen 7 5800X, GeForce GTX 1650, 32 GB RAM | Asus Sabertooth P990, AMD FX-8150, GeForce GTX 560, 16GB Ram

NeddySeagoon · Posted: Wed Oct 13, 2021 8:53 pm Post subject:

iandoug,

What is on your kernel command line in grub.cfg?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

iandoug · Posted: Wed Oct 13, 2021 9:03 pm Post subject:

NeddySeagoon · vmlinuz-5.10.27-gentoo root=/dev/sda3

iandoug,

iandoug · Posted: Thu Oct 14, 2021 7:47 am Post subject:

Hi Neddy

Thanks. I have been using old grub on all my boxes, this is first time with grub2 and so am a bit lost as to where things get specified.

Do I change this

iandoug · Posted: Thu Oct 14, 2021 10:35 am Post subject:

Sigh.

Remember, ‘GRUB_DISABLE_LINUX_PARTUUID’ and ‘GRUB_DISABLE_LINUX_UUID’ are also considered to be set to ‘false’ when they are unset.
https://www.gnu.org/software/grub/manual/grub/html_node/Root-Identifcation-Heuristics.html

vs this, which says default is true:

GRUB_DISABLE_LINUX_PARTUUID true Since version 2.04. If false, and if there is either no initramfs or GRUB_DISABLE_LINUX_UUID is set to true, ${GRUB_DEVICE_PARTUUID} is passed in the root parameter on the kernel command line.
https://wiki.gentoo.org/wiki/GRUB2/Configuration_variables

Or am I reading it wrong?

Thanks, Ian
_________________
Asus X570-PRO, Ryzen 7 5800X, GeForce GTX 1650, 32 GB RAM | Asus Sabertooth P990, AMD FX-8150, GeForce GTX 560, 16GB Ram

iandoug · Posted: Tue Oct 26, 2021 7:33 am Post subject:

NeddySeagoon · Posted: Tue Oct 26, 2021 7:56 am Post subject:

iandoug,

If the drive worked elsewhere, its not the drive or data cable in use there.
That points to the SATA port on the motherboard (in the original system) being defective, or with a low probability, the power.

Answer N to fsck. It can be a filesystem destruction tool. Its OK to look but not let it make changes.

Run the long test with smartclf for a low level surface scan, without data being exported over the interface.
dd the entire drive to /dev/null to do the same thing but using the SATA interface.
The filesystem is the next layer up. Do not test that unless one or both of the above tests pass.

Was the original drive a member of the RAID set.
It will have an old event count and mdadm won't be happy. It won't be happy with two members of the RAID set in the same slot either.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

iandoug · Posted: Tue Oct 26, 2021 9:18 am Post subject:

NeddySeagoon · Posted: Tue Oct 26, 2021 12:33 pm Post subject:

iandoug,

fsck is a last resort, after you have validated backups or a ddrescue image of the drive.
While the interface with the drive is under a clould, you know nothing about the the content of the drive with any certainty.
That being the case, its not safe to let fsck make any changes.

The long test does a "copy all contents to nowhere" but the data never leaves the drive.
If that fails, the drive is probably scrap because it can't read its own writing but see later.

If the long test passes the internals of the drive are OK and the "copy all contents to nowhere" copies the entire contents of the drive over the interface.
That checks the interface separately from the rest of the drive.

I said above, that the drive is probably scrap if it can't read its own writing. That means the failing sector remapping hasn't work.
The idea is that drives move data from a failing sector to a spare sector when they detect read problems.
This remapping can be forced with a write to the 'failed' sector.
Either the write will succeed to the original sector, or the write will fail to the original sector and the drive will write it to a space sector.
It appears to succeed in either case.
You can tell from the smart data. Has the Pending Sector Count or the Reallocated Sector Count changed?

Long stot to explain the odd behaviour. HAve you ever dd imaged the drive or any of its partitions.
That duplicates UUIDs and having duplicate UUIDs in the same system is a verybadthing.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.