Prompted for / during bootup

phil_r · Posted: Mon Mar 13, 2017 4:29 pm Post subject: Prompted for / during bootup

Hi guys. I've built Gentoo many, many times but this is my first time with a NVMe PCI Flash Memory card. Everything is fine but I get this prompt during bootup:

https://postimg.org/image/bcx7wc0vn/

Basically I'm prompted for the root block device, because apparently it can't find the device uuid anywhere - except the one it displays is the correct one. If I just hit "Enter" it continues and boots just fine.

Any ideas?
Thanks!

PS [/url]I would attach grub.cfg and .config but I don't see the option to attach files anymore...?
_________________
Just when you think you know the answers, I change the questions.

NeddySeagoon · Posted: Mon Mar 13, 2017 7:09 pm Post subject:

phil_r,

It sounds like the kernel tries to mount root before everything is ready.
Does adding rootwait to the kernel command line in grub.cfg have any effect?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

phil_r · Posted: Mon Mar 13, 2017 7:18 pm Post subject:

@Neddy - I will check on that tonight and get back to you!
_________________
Just when you think you know the answers, I change the questions.

phil_r · Posted: Tue Mar 14, 2017 1:03 am Post subject:

No success with the rootwait, sadly.
_________________
Just when you think you know the answers, I change the questions.

NeddySeagoon · Posted: Tue Mar 14, 2017 9:07 am Post subject:

phil_r,

In a way, I'm pleased about that. rootwait makes the kernel check that the root filesystem is there before it tries to mount it.
It avoids the panic when root isn't ready and waits forever instead.
That it didn't fix it, shows that its not that.

Unfortunately, I'm out of ideas. I don't have a NVMe PCI Flash Memory card to experience the problem first hand.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

phil_r · Posted: Tue Mar 14, 2017 5:14 pm Post subject:

I've never had this issue before with a regular (/dev/sdX) type drive. But I also had to build an initramfs for this, which I've never had to do before either. So I'm sure I'm just missing a simple step somewhere - but it is weird that it identifies the UUID of the / filesystem, prompts for the same one and then simply works if I just let it continue...
_________________
Just when you think you know the answers, I change the questions.

NeddySeagoon · Posted: Tue Mar 14, 2017 6:03 pm Post subject:

phil_r,

That rootwait will be applied to the initrd, not the real root.

Why do you need an initrd?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

phil_r · Posted: Tue Mar 14, 2017 6:24 pm Post subject:

Without the initrd, it kernel panics about 2 seconds into the boot up process.
_________________
Just when you think you know the answers, I change the questions.

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

Have you tried using that UUID in /etc/fstab? Or perhaps you already have, and the "drive" isn't ready when fstab is processed.

"UUID=" goes in place of the "/dev/sd?" place, that is, it would be the first field in the fstab line.

Edit to add, you might get satisfaction with the kernel boot parm "root=" too.

From /usr/src/*/init/do_mounts.c :

phil_r

@cboldt:

I did have it in fstab, it didn't make a difference.

Setting:
root=PARTUUID=xxxxxxxxxxx (insert partuuid here)

had no effect, however when I did:
root=UUID=xxxxxxxxxxx (insert uuid here)

seems to of cracked the problem - it boots without stopping now.

Thank you all for the guidance!
_________________
Just when you think you know the answers, I change the questions.

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

Appreciate the feedback. Funny how the kernel docs can be ever so slightly misleading, and real world experience is mighty useful. You might be able to get rid of your initramfs too - I'm sure that it (initramfs startup) has to find rootfs on its own, without reference to /etc/fstab ... but I'm not clear on why you found the need for an initramfs in the first place. Not that it matters, once you have the thing booting, you are probably happy to "leave it alone"!

Oh, one other thought. The kernel boot parm structure may change over time. The instruction to use PARTUUID is from a 4.9.6 kernel, and from the source code, the kernel is looking for that exact string.

NeddySeagoon · Posted: Tue Mar 14, 2017 7:54 pm Post subject:

phil_r,

You only need an initrd if your kernel cannot boot without some help from user space.
You must configure everything needed to boot as <*>, not <M> since you can't load modules until root is mounted and if you need modules to mount root, well ... don't do that.

UUID and PARTUUID refer to different block structures.
PARTUUID is a partition reference. UUID is a filesystem reference.
The kernel can mount root by PARTUUID but not by UUID. root=UUID= needs the user space mount tool and an initrd.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

phil_r · Posted: Tue Mar 14, 2017 8:41 pm Post subject:

I don't know what to tell you...

All critical functions are compiled in, not modules. The NVM driver, the filesystem drivers, PCI related things all compiled in.

Without the intird, it didn't boot. With it, it does.

Using PARTUUID failed. UUID worked (and yes I got the right values from /dev/disk/by-uuid | by-partuuid)

I can play with it more and try to narrow it down some but for right now, at least it boots without help.
_________________
Just when you think you know the answers, I change the questions.

NeddySeagoon · Posted: Tue Mar 14, 2017 8:49 pm Post subject:

phil_r,

The non initrd panic message would be useful.
You cannot use root=UUID= without an initrd though.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

I also wonder what is in the initramfs.

Neddy - appreciate the heads up that UUID and PARTUUID are different. They indeed are. I ran blkid here, and UUID is associated with the filesystems. PARTUUID is 8 hex digits, followed by a hyphen and two more hex digits. My partition scheme is NOT GPT, and I don't use EFI.

Another "root=" parameter scheme that is supposed to work is to give the major/minor device as a four digit hex value, or major:minor separated by a colon, I assume in that case that major and minor are decimal.

NeddySeagoon · Posted: Tue Mar 14, 2017 9:45 pm Post subject:

cboldt,

That's the MSDOS DiskID, from the disk label followed by the partition number.
Look in fdisk.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

Yep, it is the same, "Disk identifier: 0x9af1d243" - get rid of the leading 0x hex identifier, and it becomes PARTUUID on (that device) on my system, after adding the hyphen and partition number.

phil_r · Posted: Wed Mar 15, 2017 2:10 pm Post subject:

I'm currently using a 4.9.13 kernel. I'll compile a 4.10 kernel with no initrd and boot that, however I'm at work so although I can compile the kernel I won't be able to snap the error for several hours.

Thank you for continuing to work with me on this.
_________________
Just when you think you know the answers, I change the questions.

NeddySeagoon · Posted: Wed Mar 15, 2017 2:18 pm Post subject:

phil_r,

Working on other broken Gentoos saves us breaking our own :)
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

phil_r · Posted: Thu Mar 16, 2017 4:04 am Post subject:

So this is what I get if I boot with a kernel and no initramfs generated:

http://pasteboard.co/JXK5iJjXm.jpg

Some snippets of /dev that might be useful to you:

NeddySeagoon · Posted: Thu Mar 16, 2017 10:08 am Post subject:

phil_r,

You can't just throw away the initrd. You need to change a few kernel options from <M> to <*> too.
The initrd is a temporary root filesystem in a file. It holds all the kernel modules so that they can be loaded before the real root is mounted.
The unknown-block(0,0) at the top of the image shows that the kernel cannot talk to your HDD at all, probably because it needs some modules from the initrd.

root=UUID= needs the initrd as it uses /bin/mount to mount root. You must either use the root=PARTUUID= or /dev/nvme0n...
That usually leads to different numbers in the panic statement unknown-block(0,0).

You don't say what filesystem you use on root.

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

Just to make the point pedantically (thinking more of other readers, not you), whatever the kernel needs to recognize the NVMe PCI Flash Memory card must be built into the kernel, for root=PARTUUID to work. That covers more than the filesystem!

Edit to add -- the opening screen tells us that "All critical functions are compiled in, not modules." isn't true. Something is missing from the kernel. The major,minor designation of 0,0 would be something different.

What do you get from `ls -l /dev/nvme0n1p4`?

phil_r · Posted: Thu Mar 16, 2017 3:03 pm Post subject:

@Neddy - root is ext4. /home is xfs. I learned that lesson a long time ago :-)

If you see something that should be compiled in rather than a module please let me know.

@cboldt - The NVMe driver is compiled in.

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

Yep, 259,3 is what I was looking for. Try "root=259:3" as a boot parameter. That is one of the forms that is spelled out in the kernel Documentation. Another form is a hex representation, but with major=259 (>0xff), this deviates from the four (hex) digit example. The hex representation would have five digits, "10403"

We can't tell what has to be compiled in, that isn't, but the fact that the kernel reports inability to find the root at 0,0 is a symptom that the device is itself unknown to the kernel.
.
Once it's up, show us the output of `lsmod`. Maybe there is something other than "the device driver itself," which you haven't specified yet. My kenel config is set like so ...

phil_r · Posted: Thu Mar 16, 2017 3:42 pm Post subject:

Don't forget, the original issue here is that without root=UUID in grub.cfg my system won't boot unless I supply the UUID during boot-up (which it defaults to so all I have to is hit ENTER). If I do that the system boots fine so there can't be too much wrong with the kernel. I'm not worried about needing an initrd or not or anything like that. I do have a bootable system but it's a tad kludgy. I don't mind having to have the extra parameter in grub.cfg, it's just strange to me that I never needed to do that with a rotational hard drive. Once everything is booted, it's all rock solid.
_________________
Just when you think you know the answers, I change the questions.