Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Can't emerge zfs (zfs-kmod, spl, zfs) after botched upgrade!
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5934

PostPosted: Sun Aug 13, 2017 10:49 am    Post subject: Reply with quote

Quote:
At this point I thought I had all my bases covered and I would be able to boot or at least see all my kernel entries and that at the very least I would have a kernel that would fail to load the OS but instead I get the welcome to grub and nothing more...I get no grub prompt and do not get to my grub menu either. What went wrong and how do I fix? I realize at this point I will have to boot a live cd to do anything but I am not sure what it is I need to fix...I suspect its my grub install that is messed but where do I start?


If "grub-probe /" doesn't return with "zfs" then something is still wrong.

Make sure you are using grub 2.0.2 with USE=libzfs

Feel free to pop onto freenode IRC #zfsonlinux if you need more zfs-specific support. :D
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Mon Aug 14, 2017 2:14 pm    Post subject: Reply with quote

bunder wrote:
Quote:
At this point I thought I had all my bases covered and I would be able to boot or at least see all my kernel entries and that at the very least I would have a kernel that would fail to load the OS but instead I get the welcome to grub and nothing more...I get no grub prompt and do not get to my grub menu either. What went wrong and how do I fix? I realize at this point I will have to boot a live cd to do anything but I am not sure what it is I need to fix...I suspect its my grub install that is messed but where do I start?


If "grub-probe /" doesn't return with "zfs" then something is still wrong.

Make sure you are using grub 2.0.2 with USE=libzfs

Feel free to pop onto freenode IRC #zfsonlinux if you need more zfs-specific support. :D



Not sure if this is the case or not....

I do not have 'libzfs' use flag for grub and did not have it before attempting to upgrade to a newer kernel and it booted on zfs just fine with the 3.6.11 kernel, maybe this is because I am doing it via initramfs I don't know. I did do a grub-probe / prior to rebooting and it reported 'zfs' so as far as I could tell grub was able to detect and speak zfs without the flag. I had considered enabling the flag but I could not do it easily as its hard masked within the profile...for a reason.

The version I have installed is 2.0.2, I am going to try to reinstall grub again first before I start enabling this flag. Are you able to confirm if it is strictly needed when using an initrams?
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5934

PostPosted: Tue Aug 15, 2017 5:57 am    Post subject: Reply with quote

grub loads before the kernel and initramfs. assuming your /boot is on zfs, if grub doesn't understand zfs, you'll get stuck in a grub rescue shell before the menu loads, with an error along the lines of "unsupported filesystem"
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Tue Aug 15, 2017 1:05 pm    Post subject: Reply with quote

bunder wrote:
grub loads before the kernel and initramfs. assuming your /boot is on zfs, if grub doesn't understand zfs, you'll get stuck in a grub rescue shell before the menu loads, with an error along the lines of "unsupported filesystem"


This is where I don't think grub really needs to understand zfs to boot as my /boot is on an ext2 partition and / is on zfs along with the rest of my partitions. So my understanding is that grub will see /boot just fine with my kernels and initramfs and when my initramfs loads it will take care of the part to be able to mount and read my / partition that is on zfs and finish loading the system. Is this right?
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5934

PostPosted: Thu Aug 17, 2017 2:22 pm    Post subject: Reply with quote

Sounds right to me
Back to top
View user's profile Send private message
mondjef
n00b
n00b


Joined: 13 Jun 2011
Posts: 71
Location: Ottawa, ON...Canada

PostPosted: Sun Sep 03, 2017 8:00 pm    Post subject: Reply with quote

WOW just over two weeks and I am happy to say I have my system back to normal and running the newest gentoo kernel, 4.12.5 with zfs 7.1!

However, I was met with challenges almost every step of the way...I will try to document all the issues I ran into and how I resolved them for myself and for anyone else that runs into similar situations in the hopes it proves useful and saves them time.

Moving forward I have a fairly simple method to keep me up to date and running on newer kernels as they become available instead of neglecting to upgrade in the hopes of avoiding so many problems.

So just to recap...I was upgrading from kernel 3.6.11 to 4.12.5 primarily because I could no longer benefit from ZFS updates as they would not compile against a kernel that old so I was caught between a rock and a hard place. The ZFS upgrades was important to me so I decided it was best I get with the times and upgrade my kernel.

Firstly, to rescue my machine a got a copy of SystemRescueCD which is based on gentoo and has the latest ZFS modules built in. I installed this to usb and booted from it so I could chroot into my machine and import my zfs pools to work with. This is where I ran into the first issue...I had some trouble running some commands and resolved it by running the following command.
Code:
unset path

for more info about this see this thread https://forums.gentoo.org/viewtopic-p-8107714.html#8107714

The second issue I faced was a result of my own doing, I screwed up the installation of grub...more specifically I messed up reinstalling grub to my disks. Just make sure you take your time and use common sense here and you will avoid this embarrassing mishap. I first mistakenly installed grub to a partition (/dev/sda2) instead of to the device (/dev/sda). I then realized this and proceeded to fix by installing to the device like I should have done in the first place. I have three disks in my machine for redundancy I have a boot partition on all three in case one gets hosed so I proceeded to install grub to those as well. When creating my grub.cfg file I think the way I installed grub to the three disks caused issues as grub root did not seem correct to me. After reboot I would get to the grub welcome screen and grub would then just hang there. I chroot'd back in and fixed grub by doing a clean install, making sure /boot was properly mounted at the time, then installing the boot loader to the device. This time around grub root looked correct to me.

Now that grub was fixed, I proceeded to reboot. This time around grub menu came up just fine and my machine proceeded to boot and this is where I was met with my next issue. Early on during boot sequence I got the error message "libgcc_s.so.1 must be installed for pthread_cancel to work" and the boot failed. After some digging and googling about this I came across the following thread that elaborates on this error which seems to be a packaging issues that affects initramfs creation tools...including genkernel which I used to create my initramfs.

https://forums.gentoo.org/viewtopic-t-1049468-start-0.html

Apparently the fix has already be implemented in future release of genkernel, however I went and installed the newest one available which was suppose to have the fix and I was faced with the same error after recreating initramfs and rebooting. After this I did more googling about this and came across FearedBliss inatramfs creator which was created as a simplified replacement to genkernel and which did not have the libgcc_s.so.1 issue. So I gave it a go and used it to create an initramfs and rebooted.

On first boot, I did not have much luck but this was an issue with how grub-makeconf created the menu entries. grub-makeconf specified the root= kernel option as root=ZFS=rpool/GENTOO/root which is what I had prior and had worked at that time but it was not the case now. Based on the usage examples for the feared bliss initramfs created it indicated that it should be specified as root=rpool/GENTOO/root triggers=zfs. So I manually changed it and rebooted.

To my surprise my machine booted and started loading my kernel and mounting my zfs filesystem..yahoo! For some reason though on first boot I saw that some of my zfs datasets had trouble being mounted because the mount points were not empty. Not sure how this happened but I rebooted with the SystemRescueCD again and went in made sure those mounts points were empty as they should be and then rebooted and everything was fine.

With the exception of the firewall! I use arnos-ip-firewall script to configure my firewall it would fail to load as some of the rules generated iptables errors, specifically to do with MASQUERADE. This I eventually fixed by using make menuconfig and ensuring I had the proper kernel options selected. I ended up selecting a fair number of items to do with network tracking and routing...to the point I am not sure what was or is actually needed as by the time I was done I was confused with all the 'legacy' options available and was not sure which ones should or should not be selected. Either way I eventually got the right one selected so that it now works however my kernel is a bit bloated I am sure but I am not too concerned as I am up and running now.

So that was my trials and tribulations with my upgrade. The only thing I have left to figure out or clean up is to duplicate my boot partition to my other two drives as backups. I think I can use dd for this and then manually alter the grub.cfg file to alter grub root so that it points to those drives instead of sda.

I will be sticking with FearedBliss initramfs creator for now as it was simple and just worked for me.

When new gentoo-sources are emerged then I will simply follow the following steps:
copy current kernel config to new kernel
Code:
cp /usr/src/linux/.config /usr/src/newGentooSources/.config

change /usr/src/linux system link to new kernel
Code:
eselect kernel set #

change into kernel sources directory
Code:
cd /usr/src/linux

make old .config file from current kernel compatible with new kernel
Code:
make olddefconfig

Code:
make && make modules_install

Code:
mount /boot

Code:
make install

Code:
make modules_prepare

rebuild zfs kernel module against new kernel sources
Code:
emerge --ask @module-rebuild

create initramfs for new kernel
Code:
bliss-initramfs -k kernel_name

move initramfs to /boot
recreate grub menu entries
Code:
grub-mkconfig -o /boot/grub/grub.cfg

manually edit /boot/grub/grub.cfg menu entries to fix the root=ZFS to root=rpool/GENTOO/root issue and add zfs trigger option
reboot
clean up old kernels manually if desired

When a new version of zfs modules are emerged I only have to create a new initramfs for currently installed kernel using the bliss-initramfs command and copying initramfs over to boot. For other older kernels I have to temporarily switch to those kernels, build zfs modules against them using the emerge --ask @module-rebuild command then creating a new initramfs one by one.

Note: to be able to build zfs against all installed kernels it is important to retain their sources...
Code:
emerge --noreplace sys-kernel/gentoo-sources:newsource.version.here


[/code]Old sources that are no longer supported can be removed from the world file by:
Code:
emerge --deselect sys-kernel/gentoo-sources:unwanted.version.here
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum