| View previous topic :: View next topic |
| Author |
Message |
eackad
n00b
Joined: 16 Jul 2014
Posts: 5
|
 Posted: Wed Jul 16, 2014 4:37 pm Post subject: NFS4 diskless boot [Solved] |
 |
|
I am trying to setup a diskless boot for nodes attached to a gentoo headnode. I followed the diskless gentoo page (diskless nodes).
I get the following error on the booting node:
| Code: | Please append a correct "root=" boot option; here are the available partitions:
kernel panic - not syncing: vfs: Unable to mount fs on unknown-block(2,0)
|
There is a root= option in my /nodes/pxelinux.cfg/default
| Code: | DEFAULT diskless
DISPLAY message.txt
PROMPT 10
TIMEOUT 50
LABEL memtest
KERNEL memtest
LABEL diskless
KERNEL bzImage
APPEND ip=dhcp root=/dev/nfs nfsroot=10.0.0.1:/nodes/base raid=noautodetect
LABEL diskless.old
KERNEL bzImage.orig
APPEND ip=dhcp root=/dev/nfs nfsroot=10.0.0.1:/nodes/base
|
The folders are exported (/etc/exports)
| Code: | /nodes/base 10.0.0.0/23(rw,fsid=0,async,no_root_squash,no_all_squash,no_subtree_check,insecure)
/opt 10.0.0.0/23(async,ro,no_root_squash,no_all_squash,no_subtree_check,insecure)
/usr 10.0.0.0/23(async,ro,no_root_squash,no_all_squash,no_subtree_check,insecure)
/home 10.0.0.0/23(async,rw,no_root_squash,no_all_squash,no_subtree_check,insecure)
|
and exportfs shows them:
| Code: | /nodes/base 10.0.0.0/23
/opt 10.0.0.0/23
/usr 10.0.0.0/23
/home 10.0.0.0/23
|
Further the headnode seems to say it has allowed the node to mount the location (from headnode /var/log/messages):
| Code: | Jul 15 14:21:38 deepthought in.tftpd[3293]: RRQ from 10.0.0.3 filename /pxelinux.cfg/default
Jul 15 14:21:38 deepthought in.tftpd[3294]: RRQ from 10.0.0.3 filename /message.txt
Jul 15 14:21:40 deepthought in.tftpd[3295]: RRQ from 10.0.0.3 filename /bzImage
Jul 15 09:21:53 deepthought dhcpd: DHCPDISCOVER from 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought dhcpd: DHCPOFFER on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought dhcpd: DHCPREQUEST for 10.0.0.3 (10.0.0.1) from 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought dhcpd: DHCPACK on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0
Jul 15 09:21:53 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:907 for /nodes/base (/nodes/base)
Jul 15 09:21:58 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:670 for /nodes/base (/nodes/base)
Jul 15 09:22:08 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:1003 for /nodes/base (/nodes/base)
Jul 15 09:22:28 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:848 for /nodes/base (/nodes/base)
Jul 15 09:22:58 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:785 for /nodes/base (/nodes/base)
Jul 15 09:23:28 deepthought rpc.mountd[3204]: authenticated mount request from 10.0.0.3:972 for /nodes/base (/nodes/base)
|
The node's fstab is (/nodes/base/etc/)
| Code: | deepthought:/nodes/base / nfs4 intr,proto=tcp,wsize=32768,rsize=32768 0 0
#deepthought:/opt /opt nfs4 sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
deepthought:/opt /opt nfs4 intr,proto=tcp,wsize=32768,rsize=32768 0 0
#deepthought:/usr /usr nfs4 sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
deepthought:/usr /usr nfs4 intr,proto=tcp,wsize=32768,rsize=32768 0 0
#deepthought:/home /home nfs4 sync,hard,intr,rw,nolock,rsize=8192,wsize=8192 0 0
deepthought:/home /home nfs4 intr,proto=tcp,wsize=32768,rsize=32768 0 0
none /proc proc defaults 0 0
|
Anyone have any ideas? Thanks for the help in advance!
_________________
Eddie
Last edited by eackad on Wed Aug 06, 2014 9:50 pm; edited 1 time in total |
|
| Back to top |
|
 |
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3129
|
| Posted: Wed Jul 16, 2014 6:25 pm Post subject: |
|
|
Here you go
| Quote: | LABEL Gentoo
MENU LABEL Gentoo Live on PXE
LINUX gentoo/kernel-genkernel-x86-3.7.10-gentoo
APPEND ip=dhcp root=/dev/ram0 cdroot=1 real_root=/dev/nfs nfsroot=10.0.1.1:/mnt/linux.images/tftp/gentoo initrd=gentoo/initramfs-genkernel-x86-3.7.10-gentoo loop=gx86.sqfs looptype=squashfs |
| Quote: |
# ls /mnt/linux.images/tftp/gentoo
System.map-genkernel-x86-3.7.10-gentoo excludes gx86.sqfs initramfs-genkernel-x86-3.7.10-gentoo kernel-genkernel-x86-3.7.10-gentoo
|
| Quote: | # cat /etc/conf.d/atftp
# Config file for tftp server
rc_net_vn0_need="!net"
TFTPD_ROOT="/mnt/linux.images/tftp" |
I hope you can make some sense of bold fragments.
Be carefull with paths, kernel and initramfs are relative to different directory than gx86.sqfs (containing netboot root filesystem)
And answering questions you're likely to ask next:
1) make sure you have ethernet driver compiled in kernel (builtin, not a module)
2) make sure your kernel supports root over NFS
3) you must either make a LiveCD-fashioned FS image or patch your initramfs to let you mount root directly over NFS
4) Unless you know it's a bad idea, use LiveCD-fashioned squashfs image rather than hacking initramfs. Your LAN is going to take it much better. |
|
| Back to top |
|
|
eackad
n00b
Joined: 16 Jul 2014
Posts: 5
|
| Posted: Thu Jul 17, 2014 9:34 pm Post subject: NFS4 diskless boot |
|
|
I think I have the kernel compiled with all that is needed:
| Quote: | CONFIG_NFS_FS=y
CONFIG_NFS_V2=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
CONFIG_NFS_V4_SECURITY_LABEL=y
CONFIG_ROOT_NFS=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
|
The ethernet drivers are compiled, not modules and I think working correctly since the headnode gives a dhcp ip address successfully to the node.
However, your third comment is a possible problem. I followed the instructions for diskless gentoo here (http://wiki.gentoo.org/wiki/Diskless_nodes). There was no mention of constructing a initrd (let alone an initramfs). Will the method given in the wiki not work without a initrd? I want the structure to be as simple as possible.
_________________
Eddie |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3129
|
| Posted: Fri Jul 18, 2014 8:38 pm Post subject: |
|
|
I don't know if it can work without initramfs. Honestly, I doubt it though.
Good news is you can use genkernel to build kernel and initramfs, and since you seem to have already configured kernel you wouldn't have to do anything more than run 'genkernel --oldconfig all' (perhaps adding as well --no-clean to speed up the process). Or you can fairly easily build your own one and tailor it to your wants.
To make liveCD you just have to run 'mksquashfs <source root directory> <output filesystem image.sqfs>' - pretty straight-forward.
You might also try block level access to filesystem. Linux has a nice support for iSCSI, unfortunately I think that would require initramfs too, since you need some user-space tools to configure it, and it is less popular than NFS.
Anyway, don't be afraid of that initramfs thing, it takes a few MB and allows you run "weird" setups  so why not to take advantage of this. |
|
| Back to top |
|
|
eackad
n00b
Joined: 16 Jul 2014
Posts: 5
|
| Posted: Mon Jul 21, 2014 9:33 pm Post subject: mounting the remote directories |
|
|
Thanks for the help! I built the system files using genkernel (and added kernel support for squashfs) as you suggested and added the commands to the pxelinux as follows
pxlinux.cfg/default
| Quote: | DEFAULT diskless
DISPLAY message.txt
PROMPT 10
TIMEOUT 50
LABEL diskless
KERNEL base/kernel-genkernel-x86_64-3.12.20-gentoo-nb
APPEND ip=dhcp root=/dev/ram0 cdroot=1 real_root=/dev/nfs nfsroot=10.0.0.1:/nodes/base initrd=base/initramfs-genkernel-x86_64-3.12.20-gentoo-nb raid=noautodetect loop=nodes.sqfs looptype=squashfs
|
I made the squashfs using: mksquashfs /nodeimage/ nodes.sqfs
The /nodes/base has the following files
| Quote: | drwxr-xr-x 2 root root 4.0K Jun 30 11:42 bin
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 dev
drwxr-xr-x 52 root root 4.0K Jul 8 10:40 etc
drwxr-xr-x 2 root root 4.0K Jun 30 07:53 home
drwxr-xr-x 3 root root 4.0K May 9 10:15 lib32
drwxr-xr-x 12 root root 4.0K Jun 30 11:43 lib64
drwxr-xr-x 4 root root 4.0K Jul 21 09:32 mnt
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 opt
drwxr-xr-x 2 root root 4.0K Jun 30 07:53 proc
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 root
drwxr-xr-x 2 root root 4.0K Jun 30 11:45 sbin
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 sys
drwxrwxrwx 2 root root 4.0K Jun 30 07:53 tmp
drwxr-xr-x 2 root root 4.0K Jun 30 07:54 usr
drwxr-xr-x 7 root root 4.0K Jun 30 07:54 var
lrwxrwxrwx 1 root root 5 Apr 30 22:52 lib -> lib64
|
While /nodes/base/ has
| Quote: | -rw-r--r-- 1 root root 1.3M Jul 21 09:41 initramfs-genkernel-x86_64-3.12.20-gentoo-nb
-rw-r--r-- 1 root root 5.4M Jul 21 09:41 kernel-genkernel-x86_64-3.12.20-gentoo-nb
-rw-r--r-- 1 root root 33M Jul 21 11:06 nodes.sqfs
-rw-r--r-- 1 root root 2.8M Jul 21 09:41 System.map-genkernel-x86_64-3.12.20-gentoo-nb
|
The node finds the files displaying:
| Quote: | Loading base/kernel-genkernel-x86_64-3.12.20-gentoo-nb........
Loading base/initramfs-genkernel-x86_64-3.12.20-gentoo-nb....ready.
|
But it then complains:
| Quote: | >> Attempting to mount NFS CD image on 10.0.0.1:/nodes/base with options ro,nolock, risze=1024,wsize=1024
>> Determining looptype ...
>> mounting squashfs filesystem
>> Copying read-write image contents to tmpfs
>> Mounting 10.0.0.1:/usr as /usr: mount -t nfs4 -o intr,proto=tcp,wsize=32760,rsize=32768, ro 10.0.0.1:/usr on /newroot/usr failed: No such file or directory
!! unable to mount 10.0.0.1:/usr for /usr
|
This is from the fstab on the nodes (/nodeimage/etc/fstab) shown in the first post. The /var/log/messgaes on the headnode remains:
| Quote: | Jul 21 11:17:53 deepthought dhcpd: DHCPACK on 10.0.0.3 to 00:25:90:2c:ff:06 via enp2s0f0
Jul 21 11:17:55 deepthought rpc.mountd[2713]: authenticated mount request from 10.0.0.3:836 for /nodes/base (/nodes/base)
|
the only real difference is that there is only one call to the rpc.mountd (instead of multiple as before)
There are few issues I do not understand about the above.
1) Should the /nodes/base be mounted ro or should it be rw?
2) Why does it mount the root system onto /newroot when I want it to be the /?
3) Why is /opt not mounted before /usr since it comes second in the fstab file and why doesn't it show up in /var/log/messages?
Any ideas what I am doing incorrectly?
_________________
Eddie |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3129
|
| Posted: Tue Jul 22, 2014 8:03 pm Post subject: |
|
|
When you boot, initramfs is your root. You provide arguments that tell it what device is supposed to be your root, it mounts it in /newroot, prepairs /newroot to be ran, and when it's done with stuff you need before you init your system, it triggers pivot_root which rotates your active filesystem. /newroot becomes /, / becomes /oldroot (or something like that) and then /oldroot is umounted and memery freed.
So, if your initramfs mounts /newroot, it's good. Well, actually it's good when it mounts your real root in /newroot.
Now, bootloader obviously finds kernel and initramfs -> dchp and tftp are configured properly.
params you provided to kernel at boot seem correct.
| Quote: | > Mounting 10.0.0.1:/usr as /usr: mount -t nfs4 -o intr,proto=tcp,wsize=32760,rsize=32768, ro 10.0.0.1:/usr on /newroot/usr failed: No such file or directory
!! unable to mount 10.0.0.1:/usr for /usr |
hold on a minute, why would you mount 10.0.0.1:/usr on /newroot? Don't you have /usr inside your squashfs? This is something that should never happen, /usr is not /, if you have separate /usr, mount it over /nerwoor/usr. Still... why it attempts to mount /usr while you are still on initramfs? It shouldn't even see /etc/fstab yet. Was there and attempt to improve initramfs? (well, you can improve it adding support for persistend changes with aufs, but I suppose getting it running is higher priority) |
|
| Back to top |
|
|
eackad
n00b
Joined: 16 Jul 2014
Posts: 5
|
| Posted: Tue Jul 22, 2014 8:31 pm Post subject: still a problem |
|
|
prior to that error it says:
| Quote: | >>Making tmpfs for /newroot
>> determining root device...
>> mounting /dev/nfs as root...
|
then it gives the stuff I posted previously. So I thought the error was coming after the pivot_root. Is there at least a way to make the directory for newroot to avoid that error?
I have the directory for the /usr in the squashfs and I have not modified the initramfs. I simply want the node's /usr to be mounted from the headnode's /usr.
Thanks again for all your help, it seems like it is so close to working, but I have no idea how to change any of the newroot setup.
_________________
Eddie |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3129
|
| Posted: Thu Jul 24, 2014 7:04 pm Post subject: |
|
|
ok, let's take a step back, and look around.
Mount that squashfs image, chroot into it and show output of those 2 things:
| Code: | grep -v '^[[:space:]]*#\|^[[:space:]]*$' /etc/fstab
rc-config --show all
|
|
|
| Back to top |
|
|
eackad
n00b
Joined: 16 Jul 2014
Posts: 5
|
| Posted: Thu Jul 24, 2014 9:26 pm Post subject: kernel issue |
|
|
It seems that error was not the problem, the system was actually hanging on waiting for uevents. I fixed it with a new kernel that included the vga_16 frame buffer and vesa frame buffer since the node has nvidia cards that are not used for display.
Thanks for all your help! [solved]
_________________
Eddie |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3129
|
| Posted: Fri Jul 25, 2014 7:07 pm Post subject: |
|
|
Great, good to know you made it work.
I'm curious what that mounting to /nerwoot was though. It still seems weird the way you described it |
|
| Back to top |
|
|
toralf
Developer
Joined: 01 Feb 2004
Posts: 3922
Location: Hamburg
|
| Posted: Fri Jul 25, 2014 7:21 pm Post subject: Re: kernel issue |
|
|
| eackad wrote: | It seems that error was not the problem, the system was actually hanging on waiting for uevents. I fixed it with a new kernel that included the vga_16 frame buffer and vesa frame buffer since the node has nvidia cards that are not used for display.
Thanks for all your help! [solved] |
pls put the [solved] in the topic |
|
| Back to top |
|
|
dobbs
Tux's lil' helper
Joined: 20 Aug 2005
Posts: 105
Location: Wenatchee, WA
|
| Posted: Thu Jul 31, 2014 11:54 pm Post subject: |
|
|
| szatox wrote: | | I don't know if it can work without initramfs. Honestly, I doubt it though. |
It worked very well prior to upgrading to kernel 3.12.x (up to 3.10.25 at least). I had a nfsroot diskless cluster running for quite some time that suddenly ceased to boot after upgrading the server kernel. Surprisingly, the systems already running when the server restarted continued to function until they were rebooted, which hid the issue for several days. I have so far failed to boot any of my diskless nodes (or even a NetBSD nfsroot) in three months. I can only assume the kernel devs broke nfsroot on the server side somehow. It's extremely frustrating.
Addendum:
Got nfsroot working again without initrd. My similar topic is here: https://forums.gentoo.org/viewtopic.php?p=7592928#7592928 |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
