Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
pNFS - parallel NFS setup howto
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
Gatak
Apprentice
Apprentice


Joined: 04 Jan 2004
Posts: 168

PostPosted: Wed Sep 18, 2019 6:37 pm    Post subject: pNFS - parallel NFS setup howto Reply with quote

Hi!

Trying to set up pNFS with the file or flexfile layouts (what's the difference?). However, even though I export with pnfs, mountstats show pnfs=notconfigured (see below).

I am really struggling to find any information on how to set up a pNFS file server. Most information is really old and refers to obsolete spnfsd packages and other things. Can anyone here help?

kernel-5.3.0 on both client and server. Same NFS config too.
net-fs/nfs-utils-2.4.1-r1 on both client and server

SERVER setup:
Kernel config:
Code:

 ┌───────────────────────────────────────────────────────────────────────────────────┐
 │ │            --- Network File Systems                                           │ │
 │ │            <*>   NFS client support                                           │ │
 │ │            < >     NFS client support for NFS version 2                       │ │
 │ │            < >     NFS client support for NFS version 3                       │ │
 │ │            <*>     NFS client support for NFS version 4                       │ │
 │ │            [*]     Provide swap over NFS support                              │ │
 │ │            [*]   NFS client support for NFSv4.1                               │ │
 │ │            [*]     NFS client support for NFSv4.2                             │ │
 │ │            (kernel.org) NFSv4.1 Implementation ID Domain                      │ │
 │ │            [*]     NFSv4.1 client support for migration                       │ │
 │ │            [ ]   Use the legacy NFS DNS resolver                              │ │
 │ │            <*>   NFS server support                                           │ │
 │ │            -*-     NFS server support for NFS version 3                       │ │
 │ │            [*]       NFS server support for the NFSv3 ACL protocol extension  │ │
 │ │            [*]     NFS server support for NFS version 4                       │ │
 │ │            [*]   NFSv4.1 server support for pNFS block layouts                │ │
 │ │            [*]   NFSv4.1 server support for pNFS SCSI layouts                 │ │
 │ │            [*]   NFSv4.1 server support for pNFS Flex File layouts            │ │
 │ │            <M>   Secure RPC: Kerberos V mechanism                             │ │
 │ │            [ ]     Secure RPC: Disable insecure Kerberos encryption types     │ │
 │ │            [*]   RPC: Enable dprintk debugging                                │ │
 │ │            < >   Ceph distributed file system                                 │ │
 │ │            <M>   SMB3 and CIFS support (advanced network filesystem)          │ │
 │ │            [ ]     Extended statistics                                        │ │
 │ │            [*]     Support legacy servers which use less secure dialects      │ │
 │ │            [ ]       Support legacy servers which use weaker LANMAN security  │ │
 │ │            [*]     Kerberos/SPNEGO advanced session setup                     │ │
 │ │            [*]     CIFS extended attributes                                   │ │
 │ │            [*]       CIFS POSIX Extensions                                    │ │
 │ │            [ ]     Enable CIFS debugging routines                             │ │
 │ │            [*]     DFS feature support                                        │ │
 │ └────────────────────────────.(+)───────────────────────────────────────────────┘ │
 ├───────────────────────────────────────────────────────────────────────────────────┤
 │     <Select>    < Exit >    < Help >    < Save >    < Load >                      │
 └───────────────────────────────────────────────────────────────────────────────────┘



/etc/exports:
Code:
/mnt/storageTemp/portage/distfiles  192.168.0.50(rw,no_subtree_check,no_root_squash,async,pnfs)
/usr/portage/   192.168.0.50(rw,no_subtree_check,no_root_squash,async,pnfs)
[/quote]


dmesg output:
Code:
# dmesg |grep -i nfs
[    1.085275] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.953873] NFS: Registering the id_resolver key type
[    1.953912] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.953922] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[   75.592931] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[   75.593034] NFSD: Using legacy client tracking operations.
[   75.593039] NFSD: starting 90-second grace period (net f0000098)



Client config:
/etc/fstab
Code:
192.168.0.1:/usr/portage                                /usr/portage/           nfs     nconnect=2,soft,noatime,rw,tcp,vers=4.2,ac,async,nofail 0 0
192.168.0.1:/mnt/storageTemp/portage/distfiles          /usr/portage/distfiles  nfs     nconnect=2,soft,noatime,rw,tcp,vers=4.2,ac,async,x-systemd.requires=/usr/portage,nofail 0 0


mountstats /usr/portage
Code:
# mountstats /usr/portage/
Stats for 192.168.0.1:/usr/portage mounted on /usr/portage:

  NFS mount options: rw,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,proto=tcp,nconnect=2,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.50,local_lock=none
  NFS server capabilities: caps=0x7fbffff,wtmult=512,dtsize=32768,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x28803,acl=0x3,sessions,pnfs=notconfigured,lease_time=90,lease_expired=0
  NFS security flavor: 1  pseudoflavor: 0

NFS byte counts:
  applications read 19224685 bytes via read(2)
  applications wrote 0 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 7128010 bytes via NFS READ
  client wrote 0 bytes via NFS WRITE
RPC statistics:
  22279 RPC requests sent, 22279 RPC replies received (0 XIDs not found)
  average backlog queue length: 0

GETATTR:
        22671 ops (101%)
        avg bytes sent per op: 227      avg bytes received per op: 240
        backlog wait: 0.002470  RTT: 0.513122   total execute time: 0.534912 (milliseconds)
CLOSE:
        4530 ops (20%)
        avg bytes sent per op: 235      avg bytes received per op: 112
        backlog wait: 0.019868  RTT: 0.539735   total execute time: 0.577704 (milliseconds)
DELEGRETURN:
        4530 ops (20%)
        avg bytes sent per op: 247      avg bytes received per op: 160
        backlog wait: 62.107064         RTT: 2.022737   total execute time: 64.153863 (milliseconds)
OPEN_NOATTR:
        3513 ops (15%)
        avg bytes sent per op: 295      avg bytes received per op: 352
        backlog wait: 0.021349  RTT: 0.596356   total execute time: 0.637062 (milliseconds)
LOOKUP:
        3163 ops (14%)  258 errors (8%)
        avg bytes sent per op: 255      avg bytes received per op: 290
        backlog wait: 0.002529  RTT: 0.734745   total execute time: 0.759406 (milliseconds)
ACCESS:
        2342 ops (10%)
        avg bytes sent per op: 235      avg bytes received per op: 167
        backlog wait: 0.002989  RTT: 0.584543   total execute time: 0.608881 (milliseconds)
READ:
        2303 ops (10%)
        avg bytes sent per op: 243      avg bytes received per op: 3200
        backlog wait: 0.020408  RTT: 0.869301   total execute time: 0.911420 (milliseconds)
OPEN:
        1056 ops (4%)   39 errors (3%)
        avg bytes sent per op: 321      avg bytes received per op: 400
        backlog wait: 0.026515  RTT: 1.165720   total execute time: 1.214962 (milliseconds)
READDIR:
        370 ops (1%)
        avg bytes sent per op: 247      avg bytes received per op: 15366
        backlog wait: 0.002703  RTT: 17.191892  total execute time: 17.235135 (milliseconds)
SEQUENCE:
        37 ops (0%)
        avg bytes sent per op: 116      avg bytes received per op: 80
        backlog wait: 0.027027  RTT: 0.891892   total execute time: 0.972973 (milliseconds)
SEEK:
        3 ops (0%)      1 errors (33%)
        avg bytes sent per op: 244      avg bytes received per op: 104
        backlog wait: 0.000000  RTT: 0.333333   total execute time: 0.333333 (milliseconds)
SERVER_CAPS:
        2 ops (0%)
        avg bytes sent per op: 172      avg bytes received per op: 164
        backlog wait: 0.000000  RTT: 0.000000   total execute time: 0.000000 (milliseconds)
EXCHANGE_ID:
        2 ops (0%)
        avg bytes sent per op: 232      avg bytes received per op: 92
        backlog wait: 0.000000  RTT: 0.500000   total execute time: 0.500000 (milliseconds)
NULL:
        1 ops (0%)
        avg bytes sent per op: 44       avg bytes received per op: 24
        backlog wait: 2.000000  RTT: 0.000000   total execute time: 3.000000 (milliseconds)
FSINFO:
        1 ops (0%)
        avg bytes sent per op: 172      avg bytes received per op: 168
        backlog wait: 0.000000  RTT: 0.000000   total execute time: 0.000000 (milliseconds)
PATHCONF:
        1 ops (0%)
        avg bytes sent per op: 164      avg bytes received per op: 116
        backlog wait: 0.000000  RTT: 0.000000   total execute time: 0.000000 (milliseconds)
CREATE_SESSION:
        1 ops (0%)
        avg bytes sent per op: 192      avg bytes received per op: 124
        backlog wait: 0.000000  RTT: 0.000000   total execute time: 0.000000 (milliseconds)
RECLAIM_COMPLETE:
        1 ops (0%)
        avg bytes sent per op: 124      avg bytes received per op: 88
        backlog wait: 0.000000  RTT: 6.000000   total execute time: 6.000000 (milliseconds)
Back to top
View user's profile Send private message
mike155
Advocate
Advocate


Joined: 17 Sep 2010
Posts: 2200
Location: Frankfurt, Germany

PostPosted: Wed Sep 25, 2019 12:05 am    Post subject: Reply with quote

I don't know anything about pnfs and I've never tried it.

But I'm surprised by your '/etc/exports' file:
Code:
/mnt/storageTemp/portage/distfiles  192.168.0.50(rw,no_subtree_check,no_root_squash,async,pnfs)
/usr/portage/                       192.168.0.50(rw,no_subtree_check,no_root_squash,async,pnfs)

This looks like a NFSv3 export file. As far as I can see, you want to use NFSv4. Under NFSv4, you are supposed to create an export directory (say '/exports') and bind mount all directories you want to export into that directory. Hence entries in your exports file should all start with that export directory.
Back to top
View user's profile Send private message
Gatak
Apprentice
Apprentice


Joined: 04 Jan 2004
Posts: 168

PostPosted: Thu Sep 26, 2019 7:13 pm    Post subject: Reply with quote

I've seen the use of /exports folder structure but nothing that says it must be that way. In any case I connect as vers=4.2 (look at the mountstats output) so it seems to be working there.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6579

PostPosted: Fri Sep 27, 2019 4:46 am    Post subject: Reply with quote

Requiring a virtual root for all NFS4 exports is cargo-culted advice that hasn't been true for over a decade. It'll work fine without it, unless you're trying to export filesystems with no UUID (squashfs/tmpfs)
Back to top
View user's profile Send private message
Gatak
Apprentice
Apprentice


Joined: 04 Jan 2004
Posts: 168

PostPosted: Fri Sep 27, 2019 12:20 pm    Post subject: Reply with quote

Parallel NFS is the real thing though. I have seen several docs in various about pNFS clients, but so far nothing for how to set up the server correctly :(
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum