View previous topic :: View next topic |
Author |
Message |
pendor n00b
Joined: 23 Dec 2007 Posts: 10 Location: Upstate New York, USA
|
Posted: Thu Jul 14, 2011 3:09 am Post subject: Native ZFS root tutorials |
|
|
Greetings all,
As some might be aware, an effort to bring native ZFS (non-FUSE) to Linux has been underway for some time over at http://zfsonlinux.org/. As ZFS is something I'm rather enthusiastic about, I've been working to integrate native ZFS and booting into Gentoo and also to document the steps for building a Gentoo system on a ZFS root.
I've managed to put together tutorials for building a Catalyst based installcd with integrated ZFS support as well as for setting up a ZFS root Gentoo system, both by migrating an existing Gentoo install and by starting fresh with the livecd. Thus far, I'm still using ext2 /boot partitions, but I'm looking into the status of the GRUB 1.99 tree to see if I can get its ZFS support working.
The results of my work based on the new 0.6.0-rc5 ZFS release are here:
https://github.com/pendor/gentoo-zfs-install
I have managed to run through these instructions to setup working systems, but it's likely that I've glossed over at least a few parts as I've been running through them repeatedly over the last couple of weeks. If anyone else is feeling adventurous and wants to give these a try, I'd welcome any feedback & corrections.
Best regards,
Zac Bedell |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Thu Jul 14, 2011 1:46 pm Post subject: |
|
|
Nice guides! Been following your comments on the genkernel should support native ZFS on Linux bug with interest.
My home file server, currently a 4-disk hardware RAID-5, is just begging to be converted to RAID-Z. I think I'm going to attempt this over the weekend. (Yes, I have excellent backups.)
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Sat Jul 23, 2011 2:02 pm Post subject: |
|
|
Two things worth noting:- The Dracut initramfs init system used in the zfs overlay genkernel-9999 supports different kernel boot parameters than the stable genkernel init system. You just use
Code: | root=<root-file-system> | instead of genkernel's Code: | root=/dev/ram0 real_root=<root-file-system> | This is actually shown on the new genkernel output but it's easy to miss. Worth a note in the guide.
I had to make an additional change to the kernel .config in order to get the zfs module to load cleanly. I had to set CONFIG_CRYPTO_ZLIB=y.Probably more to report as I progress.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
pendor n00b
Joined: 23 Dec 2007 Posts: 10 Location: Upstate New York, USA
|
Posted: Mon Jul 25, 2011 8:37 pm Post subject: |
|
|
Thanks for taking a look & for the feedback on the kernel parameters. I've managed to get Grub2 working to boot from pure ZFS (see next post...), so that fortunately obviates the need to mess with the kernel parameters directly. I was pondering an additional option to Genkernel to signal that Grub2 is in use. That could omit the usual notes about root vs. real_root and also re-run grub-mkconfig after a new kernel is built. With any luck I can dive into those changes this week. |
|
Back to top |
|
|
pendor n00b
Joined: 23 Dec 2007 Posts: 10 Location: Upstate New York, USA
|
Posted: Mon Jul 25, 2011 8:55 pm Post subject: |
|
|
At long last (and with some helpful direction from dajhorn of the zfs-discuss group), I've got Gentoo booting from a ZFS mirror with no /boot MD mirror required. This is using the Grub 1.99 release along with patches ported over from Darik's Ubuntu work.
Using GPT disks, I have Grub installed to EFI utility partitions on each device and no need for MDADM at all. There are still "partitions" on each device for Grub; but since the contents of them are installed once by Grub and never edited, there's no need to put them in mirrored storage. This is in contrast to a traditional /boot partition where the kernel & initramfs are periodically updated and thus must be mirrored across all devices.
I've also moved swap onto a ZVOL instead of in mdadm which may not be the most performant of options; but if you're hitting swap, performance is probably out the window anyways. I *have* run a full Gentoo stage1 on a VM that was so-configured and assigned a too-small amount of physical RAM. The compile took ages and was heavily into swap for most of it, but it completed successfully without any panics or other problems. Swap on ZVOL doesn't appear to cause any issues.
My Layman overlay has a patched ebuild for grub (I've called it 1.99_rc99 just to avoid invoking the Bazaar logic inherent to using -9999 version). The GentooInstall and GentooMigration documents have been updated to match.
The patches to both grub and the Gentoo ebuilds aren't final at this point as I had to sledgehammer a few things a bit more than I'd like. They are functional, but (for example) the man pages for Grub aren't building… I'll be trying to clean those things up a bit before I submit patches to upstream. That said, I have a number of VM's and one physical server running fine with these changes and no drama to report on any of them. |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Thu Jul 28, 2011 2:16 pm Post subject: |
|
|
Typo level issue with the guides found. In GentooInstall.mdown, under Creating the zpool, 2nd code listing, Code: | zfs create mountpoint=legacy rpool/Root | should be, I believe Code: | zfs set mountpoint=legacy rpool/ROOT |
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
pendor n00b
Joined: 23 Dec 2007 Posts: 10 Location: Upstate New York, USA
|
Posted: Fri Jul 29, 2011 1:02 am Post subject: |
|
|
Thanks for those! Will fix.
One other thing I should mention for anyone who reads this and isn't following the zfs-discuss group:
Putting swap on a ZVOL has proven to be a BadIdea(tm). There's a high likelihood of dead locks that will bring the system down as soon as memory grows into swap. I need to revert the tutorials back to using md for swap for the time being. Or in the case of my main server, swap on separate SSD that also holds the ZIL and L2ARC.
-Zac |
|
Back to top |
|
|
orthanner n00b
Joined: 09 Aug 2011 Posts: 3
|
|
Back to top |
|
|
pendor n00b
Joined: 23 Dec 2007 Posts: 10 Location: Upstate New York, USA
|
Posted: Tue Aug 09, 2011 3:10 am Post subject: |
|
|
I'm assuming `ls /dev/zfs` shows nothing? Sounds like either udev isn't installed & running or the --rbind mount of /dev didn't get setup for some reason.
`/etc/init.d/udev start` <i>should</i> create /dev/zfs. If it doesn't check logs & `dmesg` to see if there's any complaints.
If none of that works, is the error actually thrown during the emerge of Grub or when you try to install it with `grub-install`?
If during the emerge, anything relevant-looking from the emerge output would be useful.
If it's during grub-install, then try running `grub-probe -vvvv /` and see if there's any complaints about the pool status in there. If there are, you may be running into bugs I've been fixing in Grub over the last week or so. Jumping to =sys-boot/grub-9999 from the overlay should get you patches to fix a number of issues, though I'm still trying to hunt down a few more. Also, an extra `layman -S` would help to get the latest/greatest from the overlay. I actually just committed the last of the patches late last night. (Ain't living on the bleeding edge great? =) Lastly, `zpool scrub` may help Grub get a handle on the pool. That shouldn't be necessary, but it's a bandaid to fix the last of the bugs I'm hunting.
-Zac |
|
Back to top |
|
|
orthanner n00b
Joined: 09 Aug 2011 Posts: 3
|
Posted: Tue Aug 09, 2011 5:18 am Post subject: |
|
|
`ls /dev/zfs` gives just one line output: /dev/zfs
However, --rbind mount of /dev seemed to be successful.
The error occurs during emerge. Complete log is available here: http://ul.to/zecrog75 |
|
Back to top |
|
|
digrouz Tux's lil' helper
Joined: 19 Feb 2005 Posts: 124 Location: Mouscron, Belgium
|
Posted: Tue Aug 09, 2011 9:56 am Post subject: |
|
|
Hello,
A noob question
is the actual implementation of zfs in linux able to send trim commands to ssd drives? _________________ There are 10 types of people in the world: those who understand binary, and those who don't. |
|
Back to top |
|
|
pendor n00b
Joined: 23 Dec 2007 Posts: 10 Location: Upstate New York, USA
|
|
Back to top |
|
|
pendor n00b
Joined: 23 Dec 2007 Posts: 10 Location: Upstate New York, USA
|
Posted: Tue Aug 09, 2011 12:16 pm Post subject: |
|
|
For the build problem, I'm mostly just guessing at this point, but try:
Code: | USE="-truetype" emerge -v =sys-boot/grub-1.99_rc99 |
I'll try to verify that when I have more than an iPhone connecting me to the world, but it looks like the deploy is blowing up around the font stuff somewhere. Definitely a bug in either Grub or the ebuild, but that might be enough to get you by for now.
-Zac |
|
Back to top |
|
|
orthanner n00b
Joined: 09 Aug 2011 Posts: 3
|
Posted: Tue Aug 09, 2011 10:56 pm Post subject: |
|
|
BIG thanks! That was just it!
H'm... however, grub-probe still complains about unknown filesystem. Reboot had no effect. |
|
Back to top |
|
|
Gregor Kopka n00b
Joined: 03 Nov 2004 Posts: 5
|
Posted: Thu Sep 01, 2011 4:10 am Post subject: |
|
|
pendor wrote: | For the build problem, I'm mostly just guessing at this point, but try:
Code: | USE="-truetype" emerge -v =sys-boot/grub-1.99_rc99 |
|
I solved the build problem by: emerge bison flex before emerging grub, since grub will bail when not having them.
Else your script for a fresh install works nicely,
only downside is the zpool.cache which gave me trouble on regular basis each time i change the boot pool (zfs-discuss post).
Then usually type in dracut:
zpool export boot
zpool import boot -f -d /dev/disk/by-id
zfs umount -a
^D
and back in the system:
genkernel --zfs --no-clean --no-mrproper initramfs
grub-install <devices>
init 6
maybe it would be better to leave /etc/zpool.cache from the initramfs and use the root pool name for an explicit import after zfs module is loaded?
orthanner wrote: | H'm... however, grub-probe still complains about unknown filesystem. Reboot had no effect. |
Try a zpool scrub.
Also you could try to wipe any leftover zfs signatures from now unused parts of the disk(s), eg. detached mirrors, or in case you had a whole-disk pool on the drive before
Gregor |
|
Back to top |
|
|
c0ba n00b
Joined: 29 Sep 2011 Posts: 20
|
Posted: Thu Sep 29, 2011 8:56 am Post subject: Dracut problems =( |
|
|
Hi!
Thanks for a great job. It is really great that we can now have ZFS under Gentoo.
But I'm experiencing a problem with booting into it.
I've installed Gentoo on an ext4 separate drive, updated kernel to 3.0.4, ran through all the instructions from the github repo and now have a working system with ZFS support inside. But that is on ext4 drive.
Inside that working installation I've created a pool of 3 gpt hdd partitions from 3 different drives and installed Gentoo onto it with masked spl, zfs and genkernel. Now I'm able to successfully chroot into it. I've installed grub-1.99_rc99 and created a config for it, generated initramfs.
My pool is simple pool without parity so I've set my root as a legacy point.
Code: | server ~ # zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 1.63G 1.03T 35K /
tank/home 30K 1.03T 30K /home
tank/rt 254M 1.03T 254M legacy
tank/tmp 34K 1.03T 34K /tmp
tank/usr 1.30G 1.03T 592M /usr
tank/usr/portage 533M 1.03T 533M /usr/portage
tank/usr/src 209M 1.03T 209M /usr/src
tank/var 43.1M 1.03T 35.7M /var
tank/var/log 95.5K 1.03T 95.5K /var/log
tank/var/tmp 7.26M 1.03T 7.26M /var/tmp
|
Code: |
server ~ # ls -la /boot/
total 8604
drwxr-xr-x 4 root root 4096 Sep 29 10:18 .
drwxr-xr-x 6 root root 4096 Sep 27 02:20 ..
-rw-r--r-- 1 root root 0 Sep 28 19:59 .keep
drwxr-xr-x 3 root root 4096 Sep 29 10:19 grub
-rw-r--r-- 1 root root 4610753 Sep 29 10:18 initramfs-genkernel-x86_64-3.0.4-gentoo
-rw-r--r-- 1 root root 4152272 Sep 28 13:10 kernel-x86_64-3.0.4-gentoo
drwx------ 2 root root 16384 Sep 28 02:28 lost+found
|
And here is what my problem is with:
When I boot into dracut-generated initramfs I get an error saying that it cannot find root on zfs:tank/rt
Quote: | Dracut Warning: No root device "zfs:tank/rt" found |
Here is my grub.cfg menuentry:
Code: |
menuentry 'Gentoo GNU/Linux, with Linux x86_64-3.0.4-gentoo' --class gentoo --class gnu-linux --class gnu --class os {
load_video
insmod gzio
insmod part_gpt
insmod ext2
#insmod zfs
set root='(hd0,gpt2)'
#search --no-floppy --fs-uuid --set=root b78195ba-b70f-49ba-bcdd-4d930590d181
echo 'Loading Linux x86_64-3.0.4-gentoo ...'
linux /kernel-x86_64-3.0.4-gentoo boot=zfs root=ZFS=tank/rt dozfs
echo 'Loading initial ramdisk ...'
initrd /initramfs-genkernel-x86_64-3.0.4-gentoo
}
|
I've tried using
Code: | linux /kernel-x86_64-3.0.4-gentoo boot=zfs root=ZFS=tank real_root=ZFS=tank/rt dozfs |
Code: | linux /kernel-x86_64-3.0.4-gentoo boot=zfs root=ZFS=tank/rt |
Code: | linux /kernel-x86_64-3.0.4-gentoo boot=zfs root=zfs:tank/rt |
Code: | linux /kernel-x86_64-3.0.4-gentoo root=zfs:tank/rt |
But still get the same error (with different root) every time.
Inside dracut shell I can mount my zfs pool and see all the contents. I've tried to copy zpool.cache (after import-export) mounting zfs root inside dracut (with success) but that didn't help.
also tried to generate initramfs with dracut running
Code: | dracut --force /boot/initramfs-genkernel-x86_64-3.0.4-gentoo 3.0.4-gentoo |
which also didn't help
What it seems to me is that I pass a wrong line to kernel and still cannot find out which is right. From here and here it looks like one of my linux root configs should work. But it doesn't
What am I missing? Please help!
Zfs is cool and I do not want to give up!
Thanks! |
|
Back to top |
|
|
c0ba n00b
Joined: 29 Sep 2011 Posts: 20
|
Posted: Thu Sep 29, 2011 9:21 pm Post subject: Re: Dracut problems =( |
|
|
Hi all!
I noticed a bug in dracut script on boot and submitted the patch to zfsonlinux git repo: https://github.com/zfsonlinux/zfs/pull/416
This patch fixed my problem.
The bug was hard to be noticed since a bash error notice is printed along with lots of other stuff while booting and is scrolled away from the screen. I used shift+pageUp to watch the full event log.
https://picasaweb.google.com/lh/photo/3qfldN8OBimKpH5-xQmsuQ
By the way, if anybody is curious - the working kernel root option for me is:
Code: | linux /kernel-x86_64-3.0.4-gentoo root=ZFS=tank/rt |
And if you want to use the newer kernel like me - you need to modify the ebuild files of the zfs overlay by pendor. Since it was decided to get rid of versioning in linux-sources - you need to change the >=virtual/linux-sources-2.6 to >=virtual/linux-sources-0 in zfs-9999.ebuild, zfs-0.6.0_rc5.ebuild, spl-9999.ebuild, spl-0.6.0_rc5.ebuild and run Code: | ebuild <filepath> manifest | on those 4 files
I'll try to submit a patch to pendor git repo later.
BTW hostid on working system now returnes not the same as written in /etc/hostid (30333033 vs. 30303030). Is that ok? |
|
Back to top |
|
|
c0ba n00b
Joined: 29 Sep 2011 Posts: 20
|
Posted: Mon Oct 03, 2011 8:52 am Post subject: Re: Dracut problems =( |
|
|
Hello everybody! (am I talking to anyone? =)
Since pendor didn't update his clone of the zfsonlinux repo yet, to get the things working you need to follow the techique described here in the bottom: https://github.com/pendor/gentoo-zfs-overlay to use behlendorf's branch which now is fixed |
|
Back to top |
|
|
Simba7 l33t
Joined: 22 Jan 2007 Posts: 706 Location: Billings, MT, USA
|
Posted: Thu Oct 06, 2011 3:07 am Post subject: |
|
|
Hey Pendor, could you build an x86 ISO so I can build my systems with it?
I know you are against using x86, but I have a couple systems here with a decent amount of memory (4+Gb) and hard disk space. |
|
Back to top |
|
|
Hypnos Advocate
Joined: 18 Jul 2002 Posts: 2889 Location: Omnipresent
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10587 Location: Somewhere over Atlanta, Georgia
|
Posted: Thu Oct 06, 2011 5:05 pm Post subject: |
|
|
Simba7 wrote: | I know you are against using x86, but I have a couple systems here with a decent amount of memory (4+Gb) and hard disk space. | It's not that he's against it, it's that the kernel will barely do it for reasons of allocatable kernel memory space. I've actually been unsuccessful in building an x86-based native ZFS system; the kernel module would crash. I don't have a definitive diagnosis, but I got through the Catalyst build of a minimal install CD with native ZFS support but could never get the filesystem to behave properly. I'll eventually experiment with x86_64.
zfs-fuse has been running my /home and /usr directories flawlessly for months now, though.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
c0ba n00b
Joined: 29 Sep 2011 Posts: 20
|
Posted: Mon Oct 31, 2011 8:39 am Post subject: |
|
|
Hi!
After some time playing with ZFS on Gentoo I'm done. Seems like it's not that ready for being used under gentoo (linux at all?).
After several days of work the whole system hanged without any response. This happened on a regular basis.
I researched the internet for any solution and made some evil tuning but nothing helped me. I did about 3-4 reinstalls of the whole system (3 hhds involved), trying to install zfs-9999 and zfs-0.6.0_rc5 but wasn't successful in getting it stable.
I switched to btrfs (it's in kernel!) and am happy now =). |
|
Back to top |
|
|
EatMeerkats Apprentice
Joined: 15 Mar 2006 Posts: 234
|
|
Back to top |
|
|
disi Veteran
Joined: 28 Nov 2003 Posts: 1354 Location: Out There ...
|
Posted: Wed May 23, 2012 7:28 am Post subject: |
|
|
Hi Pendor,
thanks for the guide
I tried to follow your guide yesterday, but got stuck with grub2-mkconfig, which refused to run the drive probes, because it couldn't load the zfs module (insmod fails with the kernel on the cd).
I guess the reason was, that I used zfs-fuse of a systemrescuecd. Do you know any boot media with zfsonlinux on it or which one do you use to install?
thanks _________________ Gentoo on Uptime Project - Larry is a cow |
|
Back to top |
|
|
kernelOfTruth Watchman
Joined: 20 Dec 2005 Posts: 6111 Location: Vienna, Austria; Germany; hello world :)
|
|
Back to top |
|
|
|