slow NFS access because of other connections

Message

squanto · Post by **squanto** » Fri Sep 27, 2002 5:26 am

I am curious... I wrote all that is shown below, cause my nfs access to server sucks... Then I unplugged my 10Mb/s internet connection and nfs suddenly rocks.
How do I set up server / client to not try to send the data out over the internet connection, but just to eachother? Routing tables or something?
I am mounting via ip addresses, anything else I can do to improve performance?

Thanks

I know that NFS is kind of iffy when it comes to working and whatnot, but here is my situation:

Got brand spankin new case today for K6-2 server, set it up using 1.4 stage3 this afternoon and got an nfs server up on it right now. I can access the nfs share no problem now. When copying files across from my main desktop to nfs server I get about 11Mb/sec transfer rate.
Both computers are on 100Mb/sec eithernet cards attached to 100Mb/s switch, both registering as 100Mb/sec as per the led's on the cards. I have been playing with rsize and wsize options and even mountver options.

According to my little gnome tray system monitor, I am only using the network in spurts rather than saturating the connection. I transfer like 4.5MB of info every 30 seconds or so, regardless of my settings.

Is there a way, either server side or client side to make my connection saturate the line? With yellow dog linux on my G4 running NFS server, I could get around 27Mb/sec, which although not great, is better than my current situation.

If I have missed another post about nfs performance, please advise me to it, as I have been searching and reading for the past hour.

Thanks in advance!

ps: the only other thing plugged into the hub is my 10Mb/s internet connection to school, I thought it might be that, but since I get over average of 11Mb/sec, I ruled it out.

de4d · Post by **de4d** » Fri Sep 27, 2002 7:33 am

tell something more about your network layout.
i (maybe some1 else) have no idea how to 'plug in a 10Mbit internet connection into a hub'

squanto · Post by **squanto** » Fri Sep 27, 2002 1:03 pm

OK, no prob. My client and server are both connected to a 100Mb switch, running full duplex at 100Mb. I have a connection that goes from the switch to my wall, which is a 10Mb connection to the rest of campus, and thus a 10Mb connection to the internet. Thats how I can unplug it.

Somehow all my NFS (upd I believe) packets go the long way, and head out the 10Mb/s connection and then come back in over it, rather than going straight through the switch to their destination.

Does that help at all?

de4d · Post by **de4d** » Fri Sep 27, 2002 1:11 pm

what about ip/subnet configuration?
if ne packets are sent to the next campus server and back, this is caused by wrong routing configuration...
that your machines are all plugged into a hub in some way is nothing astonishing.... (imho)

Does that help at all?

not yet... :)

squanto · Post by **squanto** » Fri Sep 27, 2002 1:35 pm

All the machines on this side of campus are on subnet 255.255.248.0.
The ips of my server and client are right next to each other ie: x.x.142.33 and x.x.142.32, static ips.

wrong routing info as in gateway setup? or dns?
the machines are using different dns servers right now, but that can be fixed easy.

Thanks

EDIT: I now have a better understanding of my problem. With both machines booted (with campus network detached) and not attached to the rest of the campus network, they can ping eachother, but can't traceroute, mount nfs, ssh, or anything else between the two of them. I can ping with ips no problem, but can't even traceroute with the ips.
Once I put them back onto the network I can ssh in out and all around.
Is this something I messed up with my network config?

An idea I have is to alias an eth1 on both machines, give them 192.168.0.1 (and 2) and then make them mount over that? Is that even possible? I don't know much about aliasing network cards, let alone networking in general.

Thanks again!

de4d · Post by **de4d** » Sat Sep 28, 2002 11:34 am

... let alone networking in general.

maybe this is your main problem.
take a TCP/IP crashcourse ;)

try to find out (and post here) your routing configuration (route -n) of both machines. guessing of some typically wrong configurations doesnt make much sense ... imho

your dns doesnt matter in anyway. dns is _only_ supposed to resolve names and uses itself the tcp/ip protocol. forget about that.

you *could* use a second ip for each interface to have local traffic. but i think this is nothing more than a quick and dirty solution - and takes approx. the same time as doing things right from beginning...

another idea (maybe already mentioned):
does your hub switch back to 10 when pluggin in a 10Mbit device?

squanto · Post by **squanto** » Sat Sep 28, 2002 8:24 pm

de4d wrote:maybe this is your main problem.
take a TCP/IP crashcourse

try to find out (and post here) your routing configuration (route -n) of both machines. guessing of some typically wrong configurations doesnt make much sense ... imho

your dns doesnt matter in anyway. dns is _only_ supposed to resolve names and uses itself the tcp/ip protocol. forget about that.

you *could* use a second ip for each interface to have local traffic. but i think this is nothing more than a quick and dirty solution - and takes approx. the same time as doing things right from beginning...

another idea (maybe already mentioned):
does your hub switch back to 10 when pluggin in a 10Mbit device?

It is a switch, and thus 10Mbit connection runs 10Mbit and my computers run 100Mbit. So that isn't the problem.
As for crash course, I need one

that is what college is for, going to take a networking and operating systems course probably sometime soon.

route -n does this on client:

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
128.113.136.0   0.0.0.0         255.255.248.0   U     0      0        0 eth0
0.0.0.0         128.113.136.254 0.0.0.0         UG    1      0        0 eth0

and this on server:

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
128.113.136.0   0.0.0.0         255.255.248.0   U     0      0        0 eth0
0.0.0.0         128.113.136.254 0.0.0.0         UG    1      0        0 eth0

They look pretty similar to me. Any advice?

Thanks!

kirill · Post by **kirill** » Sat Sep 28, 2002 8:50 pm

squanto wrote: route -n does this on client:

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
128.113.136.0   0.0.0.0         255.255.248.0   U     0      0        0 eth0
0.0.0.0         128.113.136.254 0.0.0.0         UG    1      0        0 eth0

and this on server:

Code: Select all

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
128.113.136.0   0.0.0.0         255.255.248.0   U     0      0        0 eth0
0.0.0.0         128.113.136.254 0.0.0.0         UG    1      0        0 eth0

They look pretty similar to me. Any advice?

This looks like they are using your campus' router (128.113.136.254) as a gateway between each other.

When no gateway is present (==you "detached" from the campus network) they still can see+ping each other, but as you mentioned, they can't traceroute. That's because the first hop, the gateway, is missing.

de4d wrote:you *could* use a second ip for each interface to have local traffic. but i think this is nothing more than a quick and dirty solution - and takes approx. the same time as doing things right from beginning...

Now as de4d said, you'll need to setup some private ip addresses for both of your machines using "IP Aliasing". I would do it like this:

/etc/conf.d/net:

Code: Select all

# For adding aliases to a interface
# THE SERVER?
alias_eth0="192.168.0.33"

# NB:  The next is only used for aliases.
#
# To add a custom netmask/broadcast address to created aliases,
# uncomment and change accordingly.  Leave commented to assign
# defaults for that interface.
#
broadcast_eth0="192.168.0.255 192.168.0.255"
netmask_eth0="255.255.255.0 255.255.255.0"

Code: Select all

# For adding aliases to a interface
# THE CLIENT?
alias_eth0="192.168.0.32"

# NB:  The next is only used for aliases.
#
# To add a custom netmask/broadcast address to created aliases,
# uncomment and change accordingly.  Leave commented to assign
# defaults for that interface.
#
broadcast_eth0="192.168.0.255 192.168.0.255"
netmask_eth0="255.255.255.0 255.255.255.0"

There shouldn't be any need to remove your default gateway. When using private addresses machines should be able to talk with each other without bothering about the GW.

Of course, you'll also need to use your new ip-addresses in /etc/exports on the server and /etc/fstab on the client.

Good luck.
Please let us know if you get this working

squanto · Post by **squanto** » Mon Sep 30, 2002 12:58 am

Good luck.
Please let us know if you get this working

I will try this out tonight / tomorrow after class.
I will let you know the outcome.

Does this pose a security risk? Using private ips and all?
Cause I don't have the nfs mounted all the time, as the server is a RTCW server and I only need nfs for updates and what not, but I am going to be setting up another gentoo box, and would enjoy having 100Mbit connection to it to work on it.
This other box is going to be mail / web server, and I would like to make it semi secure.

Would there be any way to not make the first hop be to the school's router?
I know that with smb I can get about 80Mbit/sec , cause last year at school I could. I know smb uses different protocols, but it seems wierd that nfs would interact with the gateway differently than smb, but then again smb is only used mostly on lans, and not over the internet like nfs.

Thanks!

de4d · Post by **de4d** » Mon Sep 30, 2002 6:57 am

This looks like they are using your campus' router (128.113.136.254) as a gateway between each other.

When no gateway is present (==you "detached" from the campus network) they still can see+ping each other, but as you mentioned, they can't traceroute. That's because the first hop, the gateway, is missing.

sorry - but this is rather wrong.

theoreticcally, the first row in the routing table determines the route which is in case u want to send from client A to client B (w/o any closer look!!) the first one because 128.113.142.32 is in 128.113.128.0/22 as well as .33.

but look @ this

Code: Select all

1111 1111.1111 1111.1111 1100.0000 0000 <- this is your netmask
1000 0000.0111 0001.1000 1110.0010 0000 <- client A (32)
1000 0000.0111 0001.1000 1110.0010 0001 <- client B (33)
1000 0000.0111 0001.1000 1000.1000 1110 <- gateway(!)
   notice: this is a ZERO ^

now as u can easily see:
the gateway and your clients are NOT in the same subnet.
usually two hosts *cannot* talk to each other w/o gateway in between when they are in different subnets. why it works in spite: i dont know. but i usually dont try to configure a network which is basically not cleanly layouted.
consider asking one of your admins about your *real* subnet...

as this stuff may not help u in any case switch over to use a private (correctly layouted) subnet. but make sure you are not physically connected to other subnets with same adress, this may screw up your switch.

note: maybe some network guru could help to understand this configuration...[/quote]

squanto · Post by **squanto** » Mon Sep 30, 2002 2:11 pm

the gateway and your clients are NOT in the same subnet.
usually two hosts *cannot* talk to each other w/o gateway in between when they are in different subnets. why it works in spite: i dont know. but i usually dont try to configure a network which is basically not cleanly layouted.
consider asking one of your admins about your *real* subnet...

Well, I didn't try out the previous idea yet, but I will stop by the computing center today and ask them. I know that on campus there are about 4 different subnets, cause they have one for "freshman hill" one for the classrooms and one for the appartments I live in on campus, and there is one other for the other appartments. (I think now that I understand what is quoted, it now makes sense why I can't see lan games that are put up on campus, being that they are on other subnets.)

I would have thought that all of campus would be on the same subnet and everyone would be routed through one router, we have dual OC3 connections, so everyone gets preety good throughput.

Thanks for the input, I will check the math out and make sure I don't look like an idiot before I go talk with the computer ppl. I didn't even think that my problem could be cause the gateway is on a diff subnet.

Thanks

EDIT:

1111 1111.1111 1111.1111 1100.0000 0000 <- this is your netmask

, I think you have my netmask being 252.0 here, and not 248.0, cause when I do the math, I am on the correct subnet.
my subnet would be 1111 1111.1111 1111.1111 1000.0000 0000 (255.255.248.0) and then that zero you pointed out would be fine.

Any ideas? Or am I wrong?

EDIT again:
I went to the computer "help" desk and asked them. The first guy I talked with asked me what OS I was using, I told him Linux, and I was having trouble mounting an nfs drive, he gave me this blank look.
Then I asked another guy, and he seemed to think that I could get it to work with routing tables, but I don't think he was very confident in that idea.
The third person I asked told me it was my switch's fault and that it was acting like a hub and not like a switch. I think this guy is wierd, he is the head guy of the "help" desk and has fed me disinformation before. Do you think he is correct? that my switch is to blame? It is a D-Link DSS-8+ switch.
My computers are running 100Mbit full duplex, and my connection to campus is 10Mbit half duplex, as verified by numerous computers on campus, and by flashy(tm) leds that are on network cards and switches.

Any ideas?

Thanks for sticking with my huge post with multi edits, I just didn't feel it was worth replying, when this entire post is just one huge train of thought, and has all happened in the process of me sitting in physics class and then coming back to dorm room.

de4d · Post by **de4d** » Mon Sep 30, 2002 4:18 pm

, I think you have my netmask being 252.0 here, and not 248.0, cause when I do the math, I am

u are right...
its been monday morning :P

by now, u *must* blame your switch cause your routing tables are absolutely ok
try another one

squanto · Post by **squanto** » Mon Sep 30, 2002 4:28 pm

OK, will do, my roommate has an older d-link one. I will try that.

Thanks

de4d · Post by **de4d** » Mon Sep 30, 2002 4:34 pm

to know exactly try pluggin in your 10Mbit cable and dont let your 2 clients know about the gateway (or just use different adresses).
or plug in ne other 10mbit halfduplex device.
gl

squanto · Post by **squanto** » Mon Sep 30, 2002 5:10 pm

de4d wrote:to know exactly try pluggin in your 10Mbit cable and dont let your 2 clients know about the gateway (or just use different adresses).
or plug in ne other 10mbit halfduplex device.
gl

Well, I used my roommates switch, and that didn't help and then I tried the not letting them know about the gateway idea, and that didn't help either.
Which is wierd, cause if the computers don't know about the gateway, there is no reason that they should try to send data outside of the subnet to the outside world, they shouldn't even realize that there is an outside world, they should just see my subnet, and..... then it would be the switch's fault. arg, this is annoying. So my conclusion is that D-link switches suck with UDP NFS.

Anyone give me a link to a guide on using tcp NFS? I assume that I am using udp for all my nfs traffic since:

Code: Select all

rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  32768  status
    100024    1   tcp  32768  status
    100011    1   udp    856  rquotad
    100011    2   udp    856  rquotad
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100021    1   udp  32770  nlockmgr
    100021    3   udp  32770  nlockmgr
    100021    4   udp  32770  nlockmgr
    100005    1   udp  32771  mountd
    100005    1   tcp  32769  mountd
    100005    2   udp  32771  mountd
    100005    2   tcp  32769  mountd
    100005    3   udp  32771  mountd
    100005    3   tcp  32769  mountd

The linuxdoc.org howto is helpful to a point but they don't go indepth for tcp nfs.

Thanks for your time and help de4d! It is very much apreciated!

de4d · Post by **de4d** » Mon Sep 30, 2002 6:53 pm

hmm
this is absolutely weird
crazy this switch is differentiating between udp and tcp.... this might be a result of a broken arp table or a udp specific rwin but who really knows?
nfs over tcp should be no problem
just compile nfsovertcp support (exp) into the kernel and mount w/ some -p tcp option (or similar).
best reference is man i'd say.

Thanks for your time and help de4d! It is very much apreciated!

n/p @ all... im just a network fanatic seeking new problems && solutions :P

squanto · Post by **squanto** » Tue Oct 01, 2002 12:19 am

SCHWEEEEET!!!!

TCP kicks butt! and it works too. I averaged 9 MB/sec (yes, megabytes) transfer rate sustained with nfs over tcp.
I like TCP/IP. whoot, now to get automount to work

Thanks for all the help!
I had to use the gentoo-sources on my server box, but no big deal, cause I have like nothing enabled so kernel still is fine and small.

Thanks!

EdIt, I wasn't sure so I figured I would clarify. I am happy

de4d · Post by **de4d** » Tue Oct 01, 2002 2:27 pm

hmmm
this was quite what could have been expected.
but tell me why (and how!) your switch handles udp and tcp differently.
if you tell me how to measure filesystem speed (usually i do not bother about it), i will run several test with my switch. afair its got 4 100mbit and 2 10mbit connections.

squanto · Post by **squanto** » Tue Oct 01, 2002 5:16 pm

I have no idea why it does it, but to compare speeds I copy a 300MB file across between server and client, used to get around 1 MB /sec now get sustained 9MB/sec. I think that is an improvement. Also, I have network monitor in my gnome taskbar, and that is fairly accurate as to how much data is moving thru the network card, although I don't base anything off of it.

Code: Select all

time cp /nfsserver/home/reallybigfile /home/mydir

also can use /dev/zero and dev/null as the big file and the copy to directory as like they do in the linuxdoc.org howto for nfs.

I will do some reading on udp vs tcp and how they deal with routing and switches, but if you go with upd, I would suggest from my own experience not to use D-Link switches.
I will give an update if I find anything interesting.

-Andrew

EDIT: Pretty much the only thing I could find was the basic definitions of UDP and TCP and that UDP is not connection based and it has no retransmit ability while TCP can retransmit one packet at a time and works better for large pipe connections. This doesn't help me out much as to why my switch switches differently between udp and tcp traffic. Oh well.
Also, RTCW uses UDP, so when I play on my own server, which is the same server used for nfs, I send packets out and around campus and then back to my machine, so sometime ppl on campus have lower pings to my server than I do, and I am sitting right next to the server. Wierd.