Root-NFS: Server returned error -101 while mounting

[Wizumwalt]

Trying to boot a diskless node via NFS and getting the following error.

Code: Select all

...
[13.56] Looking up port of RPC 100003/2 on 10.0.0.10
[43.56] rpcbind: server 10.0.100.10 not responding, timed out
[43.56] Root-NFS: Unable to get nfsd port number from server, using default
[43.56] Looking up port of RPC 100005/1 on 10.0.0.10
[73.56] rpcbind: server 10.0.0.10 not responding, timed out
[73.56] Root-NFS: Unable to get mountd port number from server, using default
[73.56] Root-NFS: Server returned error -101 while mounting /diskless/gentoo-ssi
[73.56] VFS: Unable to mount root fs via NFS, trying floppy.
[73.56] List of all partitions:
[73.56] 0b00       148575 sr0 driver: sr
[73.56] No filesystem could mount root, tried: nfs
[73.56] Kernel panic = not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
[73.56] Pid: 1, comm: swapper Not tained 2.6.34-gentoo-r6 #2
[73.56] Call Trace:
[73.56] [<ffffffff81543ada>] panic+0x63/0xe8
...

The error code -101 means "OS error code 101: Network is unreachable", yet I can't see why. The client node is configured via PXE and I can ping it. When it boots, it's obviously pulled the kernel image from the TFTP server and I have the MAC address tied to this static ip in the dhcp.conf file. I can confirm this w/ "tcpdump port 69" and /var/log/messages. Yet, at the point of mounting root via NFS, it seems that there's no network comm.

Here's the /etc/fstab from the client. 10.0.0.10 is the master node running dhcp, tftpd, nfs, etc...

Code: Select all

10.0.0.10:/diskless/gentoo-ssi  /   nfs     sync,hard,intr,ro 0 0
10.0.0.10:/opt      /opt        nfs     sync,hard,intr,rw 0 0
10.0.0.10:/home     /home       nfs     sync,hard,intr,rw 0 0
10.0.0.10:/root     /root       nfs     sync,hard,intr,rw 0 0

shm         /dev/shm    tmpfs       nodev,nosuid,noexec 0 0

pxelinux.cfg/default file on master node.

Code: Select all

DEFAULT /boot/kernel-2.6.34-gentoo-r6

APPEND ip=dhcp ro rootfstype=nfs root=/dev/nfs nfsroot=10.0.0.10:/diskless/gentoo-ssi init=/linuxrc raid=noautodetect

/etc/exports on master node.

Code: Select all

/diskless/gentoo-ssi    10.0.0.0/16(sync,ro,no_subtree_check,no_root_squash,no_all_squash)
/opt    10.0.0.0/16(sync,rw,no_subtree_check,no_root_squash,no_all_squash)
/home   10.0.0.0/16(sync,rw,no_subtree_check,no_root_squash,no_all_squash)
/root   10.0.0.0/16(sync,rw,no_subtree_check,no_root_squash,no_all_squash)

Client node /etc/conf.d/net

Code: Select all

config_eth0=( "noop" )
routes_eth0=(
    "default via 10.0.0.1"
    "10.0.0.0/16 via 10.0.0.1"
)

config_eth1=( "null" )

Any ideas much appreciated.

[Hu]

What IP address is assigned to the diskless client? What routes does the DHCP server push?

[Wizumwalt]

The IP of my diskless client I'm testing with is 192.168.1.1.

This is run on the master node (10.0.0.10).

Code: Select all

$ netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.0.0.0        0.0.0.0         255.255.0.0     U         0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         10.0.0.1        0.0.0.0         UG        0 0          0 eth0

Here's my dhcpd.conf file from the master.

Code: Select all

# dhcpd.conf
#

ddns-update-style none;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;
allow bootp;

subnet 10.0.0.0 netmask 255.255.0.0 {
    default-lease-time 86400;
    max-lease-time 86400;

    option routers 10.0.0.1;
    option broadcast-address 10.0.0.255;
    option subnet-mask 255.255.0.0;
    option domain-name-servers aaa.bbb.cc.dd, aaa.bbb.ccc.ddd;
    option domain-name "myplace.org";

    next-server 10.0.0.10;

    group {
        filename "pxelinux.0";

        host node-1 {
            hardware ethernet 00:35:e3:ba:22:38;
            fixed-address 10.0.1.1;
        }
        
        host node-2 {
            hardware ethernet 00:13:AB:C2:23:43;
            fixed-address 10.0.1.2;
        }

        # more hosts ...
    }
}

[nativemad]

You need to have the kernel level auto configuration enabled!

Code: Select all

Networking support -->
    Networking options -->
         [*] IP: kernel level autoconfiguration
         [*]    IP: DHCP support

It really fetches an ip for a second time! (At least on these setups that i use)

Cheers

[Hu]

The IP of my diskless client I'm testing with is 192.168.1.1.

Here's my dhcpd.conf file from the master.

Code: Select all

subnet 10.0.0.0 netmask 255.255.0.0 {
    option routers 10.0.0.1;
    next-server 10.0.0.10;
    group {
        filename "pxelinux.0";
        host node-1 {
            hardware ethernet 00:35:e3:ba:22:38;
            fixed-address 10.0.1.1;
        }
    }
}

Placing the client on a wholly different subnet may work, but makes the design more complex. The dhcpd configuration file you showed seems to contradict your statement that you are using 192.168.1.1 for the diskless client.

[nativemad]

Placing the client on a wholly different subnet may work, but makes the design more complex. The dhcpd configuration file you showed seems to contradict your statement that you are using 192.168.1.1 for the diskless client.

I think that this was a typo... Otherwise he would also need the subnet declaration for 192.168....

[pratyeka]

I just went through this.

The cause was that portmap was not running on the required interface on the NFS server.

Go to your server and check using lsof |grep udp and lsof | grep tcp that you have rpc.mountd and rpc.statd and portmap accessible.

Another way to test (just portmap) is to boot another client on the same network and do rpcinfo -p <server-ip>

Code: Select all

$ rpcinfo -p 192.168.8.1
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  41168  status
    100024    1   tcp  41168  status
    100005    1   udp  41167  mountd
    100005    1   tcp  41167  mountd
    100005    2   udp  41167  mountd
    100005    2   tcp  41167  mountd
    100005    3   udp  41167  mountd
    100005    3   tcp  41167  mountd
    100021    1   udp  52885  nlockmgr
    100021    3   udp  52885  nlockmgr
    100021    4   udp  52885  nlockmgr
    100021    1   tcp  35076  nlockmgr
    100021    3   tcp  35076  nlockmgr
    100021    4   tcp  35076  nlockmgr
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs

As you can see, portmap lists the rpc services that are running, in this case statd (not really required unless your nfs server is going to reboot with active clients), mountd, nlockmgr and nfs.

Diskless gentoo rocks!

The next step is likely to be running a custom init script ... I have my clients mount the nfs root read-only, then the init script remounts areas requiring write support as tmpfs.

This means you can turn off a diskless client at any time with no fear of problems.