[Solved] Root Filesystem Read-only / Wrong boot

[phobos13013]

Wow, this one is super weird....

I was briefly on linux-4.0.9 then got the upgrade for linux-4.1.12. Did the usual, copied over my old kernel config, built it in the new kernel, copied it to boot and that was that. I did edit my fstab some but I see no major issues there. I continued to use the system but eventually it started lagging after a few days so i restarted.... Now when I start into my new kernel, the filesystem is readonly and I can do nothing as root or normal user. Also, my boot directory is jacked. None of my old kernels are there nor grub the only thing is some old kernel files from 3.1.0 which I havent used since I first turned this box back on about 1.5 years ago....

How did i overwrite my previous /boot with files from over a year and half ago and how can I get it back?
_________________
...and it should be known by now

[charles17]

Sounds like you forgot to mount /boot before make install modules_install?

[phobos13013]

Good thought, and I am not denying this is not a super basic issue... but I am still not so certain. That said, I can confirm that /boot is still alive and well just not properly mounted (mounting /dev/sda1 shows my correct kernel files). I did just notice that the whole netmount thing that was in the news awhile back was not done, so I added that to boot. But it was not the solution.

I was about to post my fstab but borked the syntax and now my fstab is dead... is there a place I get a template version or do I have to make it from scratch?
_________________
...and it should be known by now

[phobos13013]

I rebuilt the fstab (posted here: http://bpaste.net/show/020ca9f48633), assuming this is correct, but maybe I am not being clear on the primary issue here...., it is that my system is booting read-only and I do not know why. During the boot sequence, I am being told that:

[charles17]

[phobos13013]

Well I can comment out proc pretty easy, dont think that is my problem. And I can try to follow this guide for EFI... I hadnt been using it before although I was using LVM with no problems; why is this a problem all of a sudden? I am not really sure what the EFI settings mean, though so do not feel I understand this guide. For instance, what is this mapper directory for? Is there any other explanation for all this other EFI stuff? I am not using an initiramfs and do not really know how or why to set it up. So any guidance towards this is appreciated.
_________________
...and it should be known by now

[charles17]

Forget about EFI. It is LVM where I was trying to point your attention.
But maybe your fstab would also work for LVM. Did you set up for LVM at all?

[phobos13013]

Well, I have it in my kernel, I guess I am not sure what it is that I need to do to set up for LVM in the first place. The thing is, I have had this same configuration for boot on this system since at least July but upgraded a new kernel and now I am stuck with a non-mounting boot and a read-only root and there is no clear reason why. If you can point me to a way to configure LVM (which I have never done before), its appreciated.
_________________
...and it should be known by now

[krinn]

/dev/sda1 /boot ext2 defaults,noauto 0 2

the noauto mean you will have not /boot mount until you ask it to mount.
that's normal and you should keep it like that, your previous fstab wasn't using it if it was mounting /boot always.

as /boot partition isn't mount, what you see in /boot comes from your /boot subdirectory of your /
so what you see in it, is that times ago, you forget to mount /boot and install a kernel and it was copied there.

if your system works without initram, it mean your system doesn't specially need something to mount /
if your / is mount read only, you have some issue.

you can

[phobos13013]

This I think has gotten to the root of the matter, PUN INTENDED!

I had seen this remount command somehwere in the forums but hadnt actually tried it yet and unfortunately time is short for the rest of my American day so wont be able to test but I have faith in that temporary fix.

I will explore dmesg and do the basic diagnostic also, but I think you are right LVM complaints are a symptom and not a disease. The question is where is the disease, this I am afraid is above my paygrade... which for Linux is 0. Any other ideas or info is appreciated until my next post which is inevitably forthcoming.
_________________
...and it should be known by now

[phobos13013]

Well, I have finally gotten some time to do some testing.

You are correct in assuming that previously I mounted /boot unabashedly, I hope that hasnt gotten me in to trouble. From now on I will definitely be locking that down. You are also correct that I have a bum /boot from when I was trying to reinstall a kernel to it when I first turned this box back on a few years ago.

The mount -o command worked but none of my major services are working so this just won't do. One weird thing I will say is that when I run the desktop manager after the remount happens is a desktop that is typically setup for a sub instance of X I had setup for vnc and such activities. It is not my primary user desktop.

You actually may be right that I am not using lvm.... but then where did it come from? My understanding is that it is used to access partitions above sda3, am I incorrect here?

regarding /run and tmpfs, at the start of openrc, I get the following info, not sure if it means anything.

[Syl20]

[charles17]

[krinn]

[phobos13013]

Whoa! All good info folks, I appreciate everyone thinking this through. I have been looking at some or all of these issues.... Please see below for individual responses:

CneGroumF: I have evolved my kernel upgrade process over the years even within the last few years... My current process is to just do a cp of the .config from the old linux directory to the new kernel version then do an eselect to change the kernel version then run make and build the kernel again from the previous .config which I know is valid. I then copy the bzImage to /boot renaming it to linux-version convention. Then I remake grub.cfg. I have no doubt there are all sorts of issues and problems with doing it this way but really it has worked for me for the last few version upgrades (since at least the last few 3.X I was running); I have thought about make oldconfig but just have never run it. I do this, though, for exactly the reason you mention. Awhile ago I rebuit a kernel without copying the oldconfig and had a hell of a time trying to figure out why my X was bad my net was bad and all that.

charles17: Thanks for sticking with me here; yeh good point I do not see any mention of lvm... which doesnt surprise me since it couldnt work on boot and the complaint from openrc is that it cant find lvm. I do not know how to confirm but for some reason when I did the emerge that brought in the new kernel, i want to say that an lvm upgrade was in there... but it could just be priming bias from the current conundrum. I am musing at this point actually. Anyway, the lvm complaint is after grub, its during the drive mouning after openrc picks up the initscript.

krinn: I know what you are thinking and I had this same idea, since I did notice that... But from what I can tell, that is normal behavior for my system. First looking back at all my grub.cfg (from past kernels also), this is also the case on the boot script line. My understanding from the research i did after seeing that is the initial action of the boot script is to mount all drives ro until later in the script where it then makes them rw, it has something to do with timing of services and such... I am not an expert on any of this so may be talking out of my lower regions but they are probably more informed than my upper regions anyway. Whatever the made up excuse, the fact is I went into my grub.cfg and removed any mention of ro after the script line for the boot location and it had zero effect. That said, now that I think about it, maybe that line isnt the same as the grub.cfg line and there is somewhere else that is telling this thing to boot ro... I just do not understand why it would have happened all of a sudden.. and where for that matter the thing is that would say to do that.

Any other suggestions are appreciated.
_________________
...and it should be known by now

[charles17]

[phobos13013]

I am bumping this looking for some fresh ideas on this... I have an interactive boot set currently, but dont know what if any of the services are contributing to this. Not sure which services to add or deny to triage it. So far what I have tried comes out the same, read-only.
_________________
...and it should be known by now

[charles17]

[krinn]

[phobos13013]

[krinn]

[NeddySeagoon]

phobos13013,

post your dmesg after root comes up read only.
I suspect that when the root fs is checked, it may have some damage that cannot be fixed on the fly.

-- edit --

Logical Volume Manager is not used to access Logical Partitions. The naming is an unfortunate coincidence.
A Logical Volumes are like partitions but they can be dynamically resized, providing the filesystem supports the operation too.
Unlike a partition, where all the blocks are in one contiguious chunk on the disk, a logical volume need not be in one piece and can even be spread over several disks.
Logical Volume manager hides all this from you.

In short, if you didn't set up your install on logical volumes, you are not using logical volume manager.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

[phobos13013]

Still truding along here... but I think I had a memory flash that is the cause of what got here, more on that below but let me provide responses first:

krinn:
I am not getting rc.logged right now unless its in /var/log/messages; I cannot confirm this because it is bloated to GBs in size cannot open, will have to scrub it i guess. I am posting dmesg for Neddy below.

NeddySeagoon:
I have the old dmesg from when I first got this error towards the beginning of the thread (see the only other bpaste link). Go ahead and compare it to the most recent one here: https://bpaste.net/show/91c716edf624. But I think you are right, it is seeing some kind of error to the root drive which is triggering e2fsck but it cannot resolve it. Before it was getting through the whole check, but now it panics at like 18% and continues with the boot. Is there a way to fix this my transfering my root partition to another one and try to boot to that one?

I have changed some kernel and config settings, but nothing has helped. There is one last possiblity here. I recently installed a file manager that I have not used before as a test. I was transferring files to backup and then deleteing them from the hdd. There are some partitions on here that I used to not be able to modify contents of and I am wondering if the file manager was able to exceed its permissions which borked the drive. If this is the issue, is there anything that can be done to salvage this short of formatting, if that?
_________________
...and it should be known by now

[krinn]

About my query for rc.log, the file is alone and made up by openrc if you have enable the option.
That's in /etc/rc.conf

[steveL]