Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
NFS4 diskless boot [Solved]
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
eackad
n00b
n00b


Joined: 16 Jul 2014
Posts: 5

PostPosted: Wed Jul 16, 2014 4:37 pm    Post subject: NFS4 diskless boot [Solved] Reply with quote

I am trying to setup a diskless boot for nodes attached to a gentoo headnode. I followed the diskless gentoo page (diskless nodes).

I get the following error on the booting node:
Code:
Please append a correct "root=" boot option; here are the available partitions:
kernel panic - not syncing: vfs: Unable to mount fs on unknown-block(2,0)


There is a root= option in my /nodes/pxelinux.cfg/default
Code:
DEFAULT diskless
DISPLAY message.txt
PROMPT 10
TIMEOUT 50

LABEL memtest
KERNEL memtest

LABEL diskless
KERNEL bzImage
APPEND ip=dhcp root=/dev/nfs nfsroot=10.0.0.1:/nodes/base raid=noautodetect

LABEL diskless.old
KERNEL bzImage.orig
APPEND ip=dhcp root=/dev/nfs nfsroot=10.0.0.1:/nodes/base


The folders are exported (/etc/exports)
Code:
/nodes/base   10.0.0.0/23(rw,fsid=0,async,no_root_squash,no_all_squash,no_subtree_check,insecure)
/opt   10.0.0.0/23(async,ro,no_root_squash,no_all_squash,no_subtree_check,insecure)
/usr   10.0.0.0/23(async,ro,no_root_squash,no_all_squash,no_subtree_check,insecure)
/home  10.0.0.0/23(async,rw,no_root_squash,no_all_squash,no_subtree_check,insecure)

and exportfs shows them:
Code:
/nodes/base      10.0.0.0/23
/opt             10.0.0.0/23
/usr             10.0.0.0/23
/home            10.0.0.0/23


Further the headnode seems to say it has allowed the node to mount the location (from headnode /var/log/messages):
Code:
Jul 15 14:21:38 deepthought in.tftpd[3293]: RRQ from 10.0.0.3 filename /pxelinux.cfg/default
Jul 15 14:21:38 deepthought in.tftpd[3294]: RRQ from 10.0.0.3 filename /message.txt
Jul 15 14:21:40 deepthought in.tftpd[3295]: RRQ from 10.0.0.3 filename /bzImage
Jul 15 09:21:53 deepthought dhcpd: DHCPDISCOVER from 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought dhcpd: DHCPOFFER on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought dhcpd: DHCPREQUEST for 10.0.0.3 (10.0.0.1) from 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought dhcpd: DHCPACK on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:907 for /nodes/base (/nodes/base)
Jul 15 09:21:58 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:670 for /nodes/base (/nodes/base)
Jul 15 09:22:08 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:1003 for /nodes/base (/nodes/base)
Jul 15 09:22:28 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:848 for /nodes/base (/nodes/base)
Jul 15 09:22:58 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:785 for /nodes/base (/nodes/base)
Jul 15 09:23:28 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:972 for /nodes/base (/nodes/base)


The node's fstab is (/nodes/base/etc/)
Code:
deepthought:/nodes/base /               nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0
#deepthought:/opt       /opt            nfs4            sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
deepthought:/opt        /opt            nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0
#deepthought:/usr       /usr            nfs4            sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
deepthought:/usr        /usr            nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0
#deepthought:/home      /home           nfs4            sync,hard,intr,rw,nolock,rsize=8192,wsize=8192    0 0
deepthought:/home       /home           nfs4            intr,proto=tcp,wsize=32768,rsize=32768            0 0
none                    /proc           proc            defaults                                          0 0


Anyone have any ideas? Thanks for the help in advance!
_________________
Eddie


Last edited by eackad on Wed Aug 06, 2014 9:50 pm; edited 1 time in total
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 737

PostPosted: Wed Jul 16, 2014 6:25 pm    Post subject: Reply with quote

Here you go

Quote:
LABEL Gentoo
MENU LABEL Gentoo Live on PXE
LINUX gentoo/kernel-genkernel-x86-3.7.10-gentoo
APPEND ip=dhcp root=/dev/ram0 cdroot=1 real_root=/dev/nfs nfsroot=10.0.1.1:/mnt/linux.images/tftp/gentoo initrd=gentoo/initramfs-genkernel-x86-3.7.10-gentoo loop=gx86.sqfs looptype=squashfs


Quote:

# ls /mnt/linux.images/tftp/gentoo
System.map-genkernel-x86-3.7.10-gentoo excludes gx86.sqfs initramfs-genkernel-x86-3.7.10-gentoo kernel-genkernel-x86-3.7.10-gentoo

Quote:
# cat /etc/conf.d/atftp
# Config file for tftp server
rc_net_vn0_need="!net"
TFTPD_ROOT="/mnt/linux.images/tftp"


I hope you can make some sense of bold fragments.
Be carefull with paths, kernel and initramfs are relative to different directory than gx86.sqfs (containing netboot root filesystem)

And answering questions you're likely to ask next:
1) make sure you have ethernet driver compiled in kernel (builtin, not a module)
2) make sure your kernel supports root over NFS
3) you must either make a LiveCD-fashioned FS image or patch your initramfs to let you mount root directly over NFS
4) Unless you know it's a bad idea, use LiveCD-fashioned squashfs image rather than hacking initramfs. Your LAN is going to take it much better.
Back to top
View user's profile Send private message
eackad
n00b
n00b


Joined: 16 Jul 2014
Posts: 5

PostPosted: Thu Jul 17, 2014 9:34 pm    Post subject: NFS4 diskless boot Reply with quote

I think I have the kernel compiled with all that is needed:
Quote:
CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
CONFIG_NFS_V4_SECURITY_LABEL=y
CONFIG_ROOT_NFS=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y


The ethernet drivers are compiled, not modules and I think working correctly since the headnode gives a dhcp ip address successfully to the node.

However, your third comment is a possible problem. I followed the instructions for diskless gentoo here (http://wiki.gentoo.org/wiki/Diskless_nodes). There was no mention of constructing a initrd (let alone an initramfs). Will the method given in the wiki not work without a initrd? I want the structure to be as simple as possible.
_________________
Eddie
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 737

PostPosted: Fri Jul 18, 2014 8:38 pm    Post subject: Reply with quote

I don't know if it can work without initramfs. Honestly, I doubt it though.
Good news is you can use genkernel to build kernel and initramfs, and since you seem to have already configured kernel you wouldn't have to do anything more than run 'genkernel --oldconfig all' (perhaps adding as well --no-clean to speed up the process). Or you can fairly easily build your own one and tailor it to your wants.

To make liveCD you just have to run 'mksquashfs <source root directory> <output filesystem image.sqfs>' - pretty straight-forward.

You might also try block level access to filesystem. Linux has a nice support for iSCSI, unfortunately I think that would require initramfs too, since you need some user-space tools to configure it, and it is less popular than NFS.
Anyway, don't be afraid of that initramfs thing, it takes a few MB and allows you run "weird" setups ;) so why not to take advantage of this.
Back to top
View user's profile Send private message
eackad
n00b
n00b


Joined: 16 Jul 2014
Posts: 5

PostPosted: Mon Jul 21, 2014 9:33 pm    Post subject: mounting the remote directories Reply with quote

Thanks for the help! I built the system files using genkernel (and added kernel support for squashfs) as you suggested and added the commands to the pxelinux as follows
pxlinux.cfg/default
Quote:
DEFAULT diskless
DISPLAY message.txt
PROMPT 10
TIMEOUT 50
LABEL diskless
KERNEL base/kernel-genkernel-x86_64-3.12.20-gentoo-nb
APPEND ip=dhcp root=/dev/ram0 cdroot=1 real_root=/dev/nfs nfsroot=10.0.0.1:/nodes/base initrd=base/initramfs-genkernel-x86_64-3.12.20-gentoo-nb raid=noautodetect loop=nodes.sqfs looptype=squashfs


I made the squashfs using: mksquashfs /nodeimage/ nodes.sqfs
The /nodes/base has the following files
Quote:
drwxr-xr-x 2 root root 4.0K Jun 30 11:42 bin
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 dev
drwxr-xr-x 52 root root 4.0K Jul 8 10:40 etc
drwxr-xr-x 2 root root 4.0K Jun 30 07:53 home
drwxr-xr-x 3 root root 4.0K May 9 10:15 lib32
drwxr-xr-x 12 root root 4.0K Jun 30 11:43 lib64
drwxr-xr-x 4 root root 4.0K Jul 21 09:32 mnt
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 opt
drwxr-xr-x 2 root root 4.0K Jun 30 07:53 proc
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 root
drwxr-xr-x 2 root root 4.0K Jun 30 11:45 sbin
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 sys
drwxrwxrwx 2 root root 4.0K Jun 30 07:53 tmp
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 usr
drwxr-xr-x 7 root root 4.0K Jun 30 07:54 var
lrwxrwxrwx 1 root root 5 Apr 30 22:52 lib -> lib64

While /nodes/base/ has
Quote:
-rw-r--r-- 1 root root 1.3M Jul 21 09:41 initramfs-genkernel-x86_64-3.12.20-gentoo-nb
-rw-r--r-- 1 root root 5.4M Jul 21 09:41 kernel-genkernel-x86_64-3.12.20-gentoo-nb
-rw-r--r-- 1 root root 33M Jul 21 11:06 nodes.sqfs
-rw-r--r-- 1 root root 2.8M Jul 21 09:41 System.map-genkernel-x86_64-3.12.20-gentoo-nb



The node finds the files displaying:
Quote:
Loading base/kernel-genkernel-x86_64-3.12.20-gentoo-nb........
Loading base/initramfs-genkernel-x86_64-3.12.20-gentoo-nb....ready.


But it then complains:
Quote:
>> Attempting to mount NFS CD image on 10.0.0.1:/nodes/base with options ro,nolock, risze=1024,wsize=1024
>> Determining looptype ...
>> mounting squashfs filesystem
>> Copying read-write image contents to tmpfs
>> Mounting 10.0.0.1:/usr as /usr: mount -t nfs4 -o intr,proto=tcp,wsize=32760,rsize=32768, ro 10.0.0.1:/usr on /newroot/usr failed: No such file or directory
!! unable to mount 10.0.0.1:/usr for /usr


This is from the fstab on the nodes (/nodeimage/etc/fstab) shown in the first post. The /var/log/messgaes on the headnode remains:
Quote:
Jul 21 11:17:53 deepthought dhcpd: DHCPACK on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0
Jul 21 11:17:55 deepthought rpc.mountd[2713]: authenticated mount request from 10.0.0.3:836 for /nodes/base (/nodes/base)
the only real difference is that there is only one call to the rpc.mountd (instead of multiple as before)

There are few issues I do not understand about the above.
1) Should the /nodes/base be mounted ro or should it be rw?
2) Why does it mount the root system onto /newroot when I want it to be the /?
3) Why is /opt not mounted before /usr since it comes second in the fstab file and why doesn't it show up in /var/log/messages?

Any ideas what I am doing incorrectly?
_________________
Eddie
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 737

PostPosted: Tue Jul 22, 2014 8:03 pm    Post subject: Reply with quote

When you boot, initramfs is your root. You provide arguments that tell it what device is supposed to be your root, it mounts it in /newroot, prepairs /newroot to be ran, and when it's done with stuff you need before you init your system, it triggers pivot_root which rotates your active filesystem. /newroot becomes /, / becomes /oldroot (or something like that) and then /oldroot is umounted and memery freed.

So, if your initramfs mounts /newroot, it's good. Well, actually it's good when it mounts your real root in /newroot.

Now, bootloader obviously finds kernel and initramfs -> dchp and tftp are configured properly.
params you provided to kernel at boot seem correct.


Quote:
> Mounting 10.0.0.1:/usr as /usr: mount -t nfs4 -o intr,proto=tcp,wsize=32760,rsize=32768, ro 10.0.0.1:/usr on /newroot/usr failed: No such file or directory
!! unable to mount 10.0.0.1:/usr for /usr

hold on a minute, why would you mount 10.0.0.1:/usr on /newroot? Don't you have /usr inside your squashfs? This is something that should never happen, /usr is not /, if you have separate /usr, mount it over /nerwoor/usr. Still... why it attempts to mount /usr while you are still on initramfs? It shouldn't even see /etc/fstab yet. Was there and attempt to improve initramfs? (well, you can improve it adding support for persistend changes with aufs, but I suppose getting it running is higher priority)
Back to top
View user's profile Send private message
eackad
n00b
n00b


Joined: 16 Jul 2014
Posts: 5

PostPosted: Tue Jul 22, 2014 8:31 pm    Post subject: still a problem Reply with quote

prior to that error it says:
Quote:
>>Making tmpfs for /newroot
>> determining root device...
>> mounting /dev/nfs as root...

then it gives the stuff I posted previously. So I thought the error was coming after the pivot_root. Is there at least a way to make the directory for newroot to avoid that error?

I have the directory for the /usr in the squashfs and I have not modified the initramfs. I simply want the node's /usr to be mounted from the headnode's /usr.

Thanks again for all your help, it seems like it is so close to working, but I have no idea how to change any of the newroot setup.
_________________
Eddie
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 737

PostPosted: Thu Jul 24, 2014 7:04 pm    Post subject: Reply with quote

ok, let's take a step back, and look around.

Mount that squashfs image, chroot into it and show output of those 2 things:
Code:
grep -v '^[[:space:]]*#\|^[[:space:]]*$' /etc/fstab
rc-config --show all
Back to top
View user's profile Send private message
eackad
n00b
n00b


Joined: 16 Jul 2014
Posts: 5

PostPosted: Thu Jul 24, 2014 9:26 pm    Post subject: kernel issue Reply with quote

It seems that error was not the problem, the system was actually hanging on waiting for uevents. I fixed it with a new kernel that included the vga_16 frame buffer and vesa frame buffer since the node has nvidia cards that are not used for display.
Thanks for all your help! [solved]
_________________
Eddie
Back to top
View user's profile Send private message
szatox
l33t
l33t


Joined: 27 Aug 2013
Posts: 737

PostPosted: Fri Jul 25, 2014 7:07 pm    Post subject: Reply with quote

Great, good to know you made it work.
I'm curious what that mounting to /nerwoot was though. It still seems weird the way you described it
Back to top
View user's profile Send private message
toralf
Advocate
Advocate


Joined: 01 Feb 2004
Posts: 2975
Location: Hamburg/Germany

PostPosted: Fri Jul 25, 2014 7:21 pm    Post subject: Re: kernel issue Reply with quote

eackad wrote:
It seems that error was not the problem, the system was actually hanging on waiting for uevents. I fixed it with a new kernel that included the vga_16 frame buffer and vesa frame buffer since the node has nvidia cards that are not used for display.
Thanks for all your help! [solved]
pls put the [solved] in the topic
Back to top
View user's profile Send private message
dobbs
Tux's lil' helper
Tux's lil' helper


Joined: 20 Aug 2005
Posts: 100
Location: Wenatchee, WA

PostPosted: Thu Jul 31, 2014 11:54 pm    Post subject: Reply with quote

szatox wrote:
I don't know if it can work without initramfs. Honestly, I doubt it though.

It worked very well prior to upgrading to kernel 3.12.x (up to 3.10.25 at least). I had a nfsroot diskless cluster running for quite some time that suddenly ceased to boot after upgrading the server kernel. Surprisingly, the systems already running when the server restarted continued to function until they were rebooted, which hid the issue for several days. I have so far failed to boot any of my diskless nodes (or even a NetBSD nfsroot) in three months. I can only assume the kernel devs broke nfsroot on the server side somehow. It's extremely frustrating.

Addendum:
Got nfsroot working again without initrd. My similar topic is here: https://forums.gentoo.org/viewtopic.php?p=7592928#7592928
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum