Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Native ZFS root tutorials
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
pendor
n00b
n00b


Joined: 23 Dec 2007
Posts: 10
Location: Upstate New York, USA

PostPosted: Thu Jul 14, 2011 3:09 am    Post subject: Native ZFS root tutorials Reply with quote

Greetings all,

As some might be aware, an effort to bring native ZFS (non-FUSE) to Linux has been underway for some time over at http://zfsonlinux.org/. As ZFS is something I'm rather enthusiastic about, I've been working to integrate native ZFS and booting into Gentoo and also to document the steps for building a Gentoo system on a ZFS root.

I've managed to put together tutorials for building a Catalyst based installcd with integrated ZFS support as well as for setting up a ZFS root Gentoo system, both by migrating an existing Gentoo install and by starting fresh with the livecd. Thus far, I'm still using ext2 /boot partitions, but I'm looking into the status of the GRUB 1.99 tree to see if I can get its ZFS support working.

The results of my work based on the new 0.6.0-rc5 ZFS release are here:

https://github.com/pendor/gentoo-zfs-install

I have managed to run through these instructions to setup working systems, but it's likely that I've glossed over at least a few parts as I've been running through them repeatedly over the last couple of weeks. If anyone else is feeling adventurous and wants to give these a try, I'd welcome any feedback & corrections.

Best regards,
Zac Bedell
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 7668
Location: Somewhere over Atlanta, Georgia

PostPosted: Thu Jul 14, 2011 1:46 pm    Post subject: Reply with quote

Nice guides! Been following your comments on the genkernel should support native ZFS on Linux bug with interest.

My home file server, currently a 4-disk hardware RAID-5, is just begging to be converted to RAID-Z. I think I'm going to attempt this over the weekend. (Yes, I have excellent backups.)

- John
_________________
This space intentionally left blank.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 7668
Location: Somewhere over Atlanta, Georgia

PostPosted: Sat Jul 23, 2011 2:02 pm    Post subject: Reply with quote

Two things worth noting:
  • The Dracut initramfs init system used in the zfs overlay genkernel-9999 supports different kernel boot parameters than the stable genkernel init system. You just use
    Code:
    root=<root-file-system>
    instead of genkernel's
    Code:
    root=/dev/ram0 real_root=<root-file-system>
    This is actually shown on the new genkernel output but it's easy to miss. Worth a note in the guide.
  • I had to make an additional change to the kernel .config in order to get the zfs module to load cleanly. I had to set CONFIG_CRYPTO_ZLIB=y.
Probably more to report as I progress.

- John
_________________
This space intentionally left blank.
Back to top
View user's profile Send private message
pendor
n00b
n00b


Joined: 23 Dec 2007
Posts: 10
Location: Upstate New York, USA

PostPosted: Mon Jul 25, 2011 8:37 pm    Post subject: Reply with quote

Thanks for taking a look & for the feedback on the kernel parameters. I've managed to get Grub2 working to boot from pure ZFS (see next post...), so that fortunately obviates the need to mess with the kernel parameters directly. I was pondering an additional option to Genkernel to signal that Grub2 is in use. That could omit the usual notes about root vs. real_root and also re-run grub-mkconfig after a new kernel is built. With any luck I can dive into those changes this week.
Back to top
View user's profile Send private message
pendor
n00b
n00b


Joined: 23 Dec 2007
Posts: 10
Location: Upstate New York, USA

PostPosted: Mon Jul 25, 2011 8:55 pm    Post subject: Reply with quote

At long last (and with some helpful direction from dajhorn of the zfs-discuss group), I've got Gentoo booting from a ZFS mirror with no /boot MD mirror required. This is using the Grub 1.99 release along with patches ported over from Darik's Ubuntu work.

Using GPT disks, I have Grub installed to EFI utility partitions on each device and no need for MDADM at all. There are still "partitions" on each device for Grub; but since the contents of them are installed once by Grub and never edited, there's no need to put them in mirrored storage. This is in contrast to a traditional /boot partition where the kernel & initramfs are periodically updated and thus must be mirrored across all devices.

I've also moved swap onto a ZVOL instead of in mdadm which may not be the most performant of options; but if you're hitting swap, performance is probably out the window anyways. I *have* run a full Gentoo stage1 on a VM that was so-configured and assigned a too-small amount of physical RAM. The compile took ages and was heavily into swap for most of it, but it completed successfully without any panics or other problems. Swap on ZVOL doesn't appear to cause any issues.

My Layman overlay has a patched ebuild for grub (I've called it 1.99_rc99 just to avoid invoking the Bazaar logic inherent to using -9999 version). The GentooInstall and GentooMigration documents have been updated to match.

The patches to both grub and the Gentoo ebuilds aren't final at this point as I had to sledgehammer a few things a bit more than I'd like. They are functional, but (for example) the man pages for Grub aren't building… I'll be trying to clean those things up a bit before I submit patches to upstream. That said, I have a number of VM's and one physical server running fine with these changes and no drama to report on any of them.
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 7668
Location: Somewhere over Atlanta, Georgia

PostPosted: Thu Jul 28, 2011 2:16 pm    Post subject: Reply with quote

Typo level issue with the guides found. In GentooInstall.mdown, under Creating the zpool, 2nd code listing,
Code:
zfs create mountpoint=legacy rpool/Root
should be, I believe
Code:
zfs set mountpoint=legacy rpool/ROOT
:)

- John
_________________
This space intentionally left blank.
Back to top
View user's profile Send private message
pendor
n00b
n00b


Joined: 23 Dec 2007
Posts: 10
Location: Upstate New York, USA

PostPosted: Fri Jul 29, 2011 1:02 am    Post subject: Reply with quote

Thanks for those! Will fix.

One other thing I should mention for anyone who reads this and isn't following the zfs-discuss group:

Putting swap on a ZVOL has proven to be a BadIdea(tm). There's a high likelihood of dead locks that will bring the system down as soon as memory grows into swap. I need to revert the tutorials back to using md for swap for the time being. Or in the case of my main server, swap on separate SSD that also holds the ZIL and L2ARC.

-Zac
Back to top
View user's profile Send private message
orthanner
n00b
n00b


Joined: 09 Aug 2011
Posts: 3

PostPosted: Tue Aug 09, 2011 1:14 am    Post subject: Reply with quote

I am trying to perform a fresh install according to this: https://github.com/pendor/gentoo-zfs-install/blob/master/install/GentooInstall.mdown

However, emerging GRUB2 (=sys-boot/grub-1.99_rc99) results in error:
ACCESS DENIED: open_wr: /dev/zfs
Back to top
View user's profile Send private message
pendor
n00b
n00b


Joined: 23 Dec 2007
Posts: 10
Location: Upstate New York, USA

PostPosted: Tue Aug 09, 2011 3:10 am    Post subject: Reply with quote

I'm assuming `ls /dev/zfs` shows nothing? Sounds like either udev isn't installed & running or the --rbind mount of /dev didn't get setup for some reason.

`/etc/init.d/udev start` <i>should</i> create /dev/zfs. If it doesn't check logs & `dmesg` to see if there's any complaints.

If none of that works, is the error actually thrown during the emerge of Grub or when you try to install it with `grub-install`?

If during the emerge, anything relevant-looking from the emerge output would be useful.

If it's during grub-install, then try running `grub-probe -vvvv /` and see if there's any complaints about the pool status in there. If there are, you may be running into bugs I've been fixing in Grub over the last week or so. Jumping to =sys-boot/grub-9999 from the overlay should get you patches to fix a number of issues, though I'm still trying to hunt down a few more. Also, an extra `layman -S` would help to get the latest/greatest from the overlay. I actually just committed the last of the patches late last night. (Ain't living on the bleeding edge great? =) Lastly, `zpool scrub` may help Grub get a handle on the pool. That shouldn't be necessary, but it's a bandaid to fix the last of the bugs I'm hunting.

-Zac
Back to top
View user's profile Send private message
orthanner
n00b
n00b


Joined: 09 Aug 2011
Posts: 3

PostPosted: Tue Aug 09, 2011 5:18 am    Post subject: Reply with quote

`ls /dev/zfs` gives just one line output: /dev/zfs
However, --rbind mount of /dev seemed to be successful.

The error occurs during emerge. Complete log is available here: http://ul.to/zecrog75
Back to top
View user's profile Send private message
digrouz
Tux's lil' helper
Tux's lil' helper


Joined: 19 Feb 2005
Posts: 124
Location: Mouscron, Belgium

PostPosted: Tue Aug 09, 2011 9:56 am    Post subject: Reply with quote

Hello,

A noob question :roll:

is the actual implementation of zfs in linux able to send trim commands to ssd drives?
_________________
There are 10 types of people in the world: those who understand binary, and those who don't.
Back to top
View user's profile Send private message
pendor
n00b
n00b


Joined: 23 Dec 2007
Posts: 10
Location: Upstate New York, USA

PostPosted: Tue Aug 09, 2011 11:21 am    Post subject: Reply with quote

Re: Trim, it doesn't look like it:

http://comments.gmane.org/gmane.os.solaris.opensolaris.zfs/44855
Back to top
View user's profile Send private message
pendor
n00b
n00b


Joined: 23 Dec 2007
Posts: 10
Location: Upstate New York, USA

PostPosted: Tue Aug 09, 2011 12:16 pm    Post subject: Reply with quote

For the build problem, I'm mostly just guessing at this point, but try:
Code:
USE="-truetype" emerge -v =sys-boot/grub-1.99_rc99


I'll try to verify that when I have more than an iPhone connecting me to the world, but it looks like the deploy is blowing up around the font stuff somewhere. Definitely a bug in either Grub or the ebuild, but that might be enough to get you by for now.

-Zac
Back to top
View user's profile Send private message
orthanner
n00b
n00b


Joined: 09 Aug 2011
Posts: 3

PostPosted: Tue Aug 09, 2011 10:56 pm    Post subject: Reply with quote

BIG thanks! That was just it!

H'm... however, grub-probe still complains about unknown filesystem. Reboot had no effect.
Back to top
View user's profile Send private message
Gregor Kopka
n00b
n00b


Joined: 03 Nov 2004
Posts: 5

PostPosted: Thu Sep 01, 2011 4:10 am    Post subject: Reply with quote

pendor wrote:
For the build problem, I'm mostly just guessing at this point, but try:
Code:
USE="-truetype" emerge -v =sys-boot/grub-1.99_rc99


I solved the build problem by: emerge bison flex before emerging grub, since grub will bail when not having them.

Else your script for a fresh install works nicely,
only downside is the zpool.cache which gave me trouble on regular basis each time i change the boot pool (zfs-discuss post).

Then usually type in dracut:
zpool export boot
zpool import boot -f -d /dev/disk/by-id
zfs umount -a
^D

and back in the system:
genkernel --zfs --no-clean --no-mrproper initramfs
grub-install <devices>
init 6

maybe it would be better to leave /etc/zpool.cache from the initramfs and use the root pool name for an explicit import after zfs module is loaded?

orthanner wrote:
H'm... however, grub-probe still complains about unknown filesystem. Reboot had no effect.

Try a zpool scrub.

Also you could try to wipe any leftover zfs signatures from now unused parts of the disk(s), eg. detached mirrors, or in case you had a whole-disk pool on the drive before

Gregor
Back to top
View user's profile Send private message
c0ba
n00b
n00b


Joined: 29 Sep 2011
Posts: 20

PostPosted: Thu Sep 29, 2011 8:56 am    Post subject: Dracut problems =( Reply with quote

Hi!

Thanks for a great job. It is really great that we can now have ZFS under Gentoo.

But I'm experiencing a problem with booting into it.
I've installed Gentoo on an ext4 separate drive, updated kernel to 3.0.4, ran through all the instructions from the github repo and now have a working system with ZFS support inside. But that is on ext4 drive.

Inside that working installation I've created a pool of 3 gpt hdd partitions from 3 different drives and installed Gentoo onto it with masked spl, zfs and genkernel. Now I'm able to successfully chroot into it. I've installed grub-1.99_rc99 and created a config for it, generated initramfs.

My pool is simple pool without parity so I've set my root as a legacy point.

Code:
server ~ # zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
tank              1.63G  1.03T    35K  /
tank/home           30K  1.03T    30K  /home
tank/rt            254M  1.03T   254M  legacy
tank/tmp            34K  1.03T    34K  /tmp
tank/usr          1.30G  1.03T   592M  /usr
tank/usr/portage   533M  1.03T   533M  /usr/portage
tank/usr/src       209M  1.03T   209M  /usr/src
tank/var          43.1M  1.03T  35.7M  /var
tank/var/log      95.5K  1.03T  95.5K  /var/log
tank/var/tmp      7.26M  1.03T  7.26M  /var/tmp


Code:

server ~ # ls -la /boot/
total 8604
drwxr-xr-x 4 root root    4096 Sep 29 10:18 .
drwxr-xr-x 6 root root    4096 Sep 27 02:20 ..
-rw-r--r-- 1 root root       0 Sep 28 19:59 .keep
drwxr-xr-x 3 root root    4096 Sep 29 10:19 grub
-rw-r--r-- 1 root root 4610753 Sep 29 10:18 initramfs-genkernel-x86_64-3.0.4-gentoo
-rw-r--r-- 1 root root 4152272 Sep 28 13:10 kernel-x86_64-3.0.4-gentoo
drwx------ 2 root root   16384 Sep 28 02:28 lost+found


And here is what my problem is with:
When I boot into dracut-generated initramfs I get an error saying that it cannot find root on zfs:tank/rt :(

Quote:
Dracut Warning: No root device "zfs:tank/rt" found


Here is my grub.cfg menuentry:
Code:

menuentry 'Gentoo GNU/Linux, with Linux x86_64-3.0.4-gentoo' --class gentoo --class gnu-linux --class gnu --class os {
        load_video
        insmod gzio
        insmod part_gpt
        insmod ext2
        #insmod zfs
        set root='(hd0,gpt2)'
        #search --no-floppy --fs-uuid --set=root b78195ba-b70f-49ba-bcdd-4d930590d181
        echo    'Loading Linux x86_64-3.0.4-gentoo ...'
        linux   /kernel-x86_64-3.0.4-gentoo boot=zfs root=ZFS=tank/rt dozfs
        echo    'Loading initial ramdisk ...'
        initrd  /initramfs-genkernel-x86_64-3.0.4-gentoo
}


I've tried using
Code:
 linux   /kernel-x86_64-3.0.4-gentoo boot=zfs root=ZFS=tank real_root=ZFS=tank/rt dozfs

Code:
 linux   /kernel-x86_64-3.0.4-gentoo boot=zfs root=ZFS=tank/rt

Code:
 linux   /kernel-x86_64-3.0.4-gentoo boot=zfs root=zfs:tank/rt

Code:
 linux   /kernel-x86_64-3.0.4-gentoo root=zfs:tank/rt


But still get the same error (with different root) every time.
Inside dracut shell I can mount my zfs pool and see all the contents. I've tried to copy zpool.cache (after import-export) mounting zfs root inside dracut (with success) but that didn't help.

also tried to generate initramfs with dracut running
Code:
dracut --force /boot/initramfs-genkernel-x86_64-3.0.4-gentoo 3.0.4-gentoo

which also didn't help

What it seems to me is that I pass a wrong line to kernel and still cannot find out which is right. From here and here it looks like one of my linux root configs should work. But it doesn't :cry:

What am I missing? Please help!

Zfs is cool and I do not want to give up! ;)

Thanks!
Back to top
View user's profile Send private message
c0ba
n00b
n00b


Joined: 29 Sep 2011
Posts: 20

PostPosted: Thu Sep 29, 2011 9:21 pm    Post subject: Re: Dracut problems =( Reply with quote

Hi all!

I noticed a bug in dracut script on boot and submitted the patch to zfsonlinux git repo: https://github.com/zfsonlinux/zfs/pull/416
This patch fixed my problem. :D

The bug was hard to be noticed since a bash error notice is printed along with lots of other stuff while booting and is scrolled away from the screen. I used shift+pageUp to watch the full event log.
https://picasaweb.google.com/lh/photo/3qfldN8OBimKpH5-xQmsuQ

By the way, if anybody is curious - the working kernel root option for me is:
Code:
 linux   /kernel-x86_64-3.0.4-gentoo root=ZFS=tank/rt


And if you want to use the newer kernel like me - you need to modify the ebuild files of the zfs overlay by pendor. Since it was decided to get rid of versioning in linux-sources - you need to change the >=virtual/linux-sources-2.6 to >=virtual/linux-sources-0 in zfs-9999.ebuild, zfs-0.6.0_rc5.ebuild, spl-9999.ebuild, spl-0.6.0_rc5.ebuild and run
Code:
ebuild <filepath> manifest
on those 4 files

I'll try to submit a patch to pendor git repo later.

BTW hostid on working system now returnes not the same as written in /etc/hostid (30333033 vs. 30303030). Is that ok?
Back to top
View user's profile Send private message
c0ba
n00b
n00b


Joined: 29 Sep 2011
Posts: 20

PostPosted: Mon Oct 03, 2011 8:52 am    Post subject: Re: Dracut problems =( Reply with quote

Hello everybody! (am I talking to anyone? =)

Since pendor didn't update his clone of the zfsonlinux repo yet, to get the things working you need to follow the techique described here in the bottom: https://github.com/pendor/gentoo-zfs-overlay to use behlendorf's branch which now is fixed
Back to top
View user's profile Send private message
Simba7
l33t
l33t


Joined: 22 Jan 2007
Posts: 684
Location: Billings, MT, USA

PostPosted: Thu Oct 06, 2011 3:07 am    Post subject: Reply with quote

Hey Pendor, could you build an x86 ISO so I can build my systems with it?

I know you are against using x86, but I have a couple systems here with a decent amount of memory (4+Gb) and hard disk space.
_________________
Router(PE1750,4GB RAM,3x36GB HDDs,2xIntel Pro/1000+GT Quad Port) | MyDT(PhmIIx3 720@3.3GHz,8GB RAM,2TB+2TB+1.5TB HDDs,GFGTX560Ti,LG BD-RE,Win7)
MyLT(Asus G53SX,32GB RAM,2x2TB HDDs,BD-RE,Intel 6230,Win7) | Wife(C2D@2.4GHz,4GB RAM,250GB HDD,DVDRW,Win7)
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2867
Location: Omnipresent

PostPosted: Thu Oct 06, 2011 2:04 pm    Post subject: Reply with quote

These HOWTOs would like awfully nice in the Gentoo Wiki (more here). There's already an introductory page on ZFS.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 7668
Location: Somewhere over Atlanta, Georgia

PostPosted: Thu Oct 06, 2011 5:05 pm    Post subject: Reply with quote

Simba7 wrote:
I know you are against using x86, but I have a couple systems here with a decent amount of memory (4+Gb) and hard disk space.
It's not that he's against it, it's that the kernel will barely do it for reasons of allocatable kernel memory space. I've actually been unsuccessful in building an x86-based native ZFS system; the kernel module would crash. I don't have a definitive diagnosis, but I got through the Catalyst build of a minimal install CD with native ZFS support but could never get the filesystem to behave properly. I'll eventually experiment with x86_64.

zfs-fuse has been running my /home and /usr directories flawlessly for months now, though.

- John
_________________
This space intentionally left blank.
Back to top
View user's profile Send private message
c0ba
n00b
n00b


Joined: 29 Sep 2011
Posts: 20

PostPosted: Mon Oct 31, 2011 8:39 am    Post subject: Reply with quote

Hi!

After some time playing with ZFS on Gentoo I'm done. Seems like it's not that ready for being used under gentoo (linux at all?).

After several days of work the whole system hanged without any response. This happened on a regular basis.
I researched the internet for any solution and made some evil tuning but nothing helped me. I did about 3-4 reinstalls of the whole system (3 hhds involved), trying to install zfs-9999 and zfs-0.6.0_rc5 but wasn't successful in getting it stable.

I switched to btrfs (it's in kernel!) and am happy now =).
Back to top
View user's profile Send private message
EatMeerkats
Apprentice
Apprentice


Joined: 15 Mar 2006
Posts: 234

PostPosted: Thu Nov 03, 2011 2:47 am    Post subject: Reply with quote

FYI: There was an important bugfix made yesterday -- https://github.com/zfsonlinux/zfs/issues/287#issuecomment-2595689
Back to top
View user's profile Send private message
disi
Veteran
Veteran


Joined: 28 Nov 2003
Posts: 1354
Location: Out There ...

PostPosted: Wed May 23, 2012 7:28 am    Post subject: Reply with quote

Hi Pendor,

thanks for the guide :)

I tried to follow your guide yesterday, but got stuck with grub2-mkconfig, which refused to run the drive probes, because it couldn't load the zfs module (insmod fails with the kernel on the cd).
I guess the reason was, that I used zfs-fuse of a systemrescuecd. Do you know any boot media with zfsonlinux on it or which one do you use to install?

thanks
_________________
Gentoo on Uptime Project - Larry is a cow
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 5675
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Wed May 23, 2012 9:43 pm    Post subject: Reply with quote

disi wrote:
Hi Pendor,

thanks for the guide :)

I tried to follow your guide yesterday, but got stuck with grub2-mkconfig, which refused to run the drive probes, because it couldn't load the zfs module (insmod fails with the kernel on the cd).
I guess the reason was, that I used zfs-fuse of a systemrescuecd. Do you know any boot media with zfsonlinux on it or which one do you use to install?

thanks


afaik the Gentoo live-DVD (12.1) has zfsonlinux included
_________________
Unofficial minimal livecd x86/amd64 w/reiser4+truecrypt (by Neo2)
2.6.37.2_plus_v1: BFS, CFS,THP,compaction, zcache or TOI
Hardcore Linux user since 2004 :D
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum