Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Wifi randomly quits but still shows as connected (NetMngr)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
jroth
n00b
n00b


Joined: 08 Aug 2017
Posts: 66

PostPosted: Tue Mar 17, 2020 3:35 am    Post subject: Wifi randomly quits but still shows as connected (NetMngr) Reply with quote

I'm using NetworkManager ontop of systemd to manage my WiFi connection on a Dell XPS 13 model 9370. It was working pretty well for like a year, but recently I'm having a problem on my home wifi network - about once an hour, the wifi connection just... completely locks up. I can't send packets anywhere or ping anything; I can't even reach the router interface on 192.168.0.1. The only way to get it working again is to restart the NetworkManager service.

The weird thing is that when I'm having this problem, NetworkManager and the rest of the system still seem to think they're connected - "ip a" shows the same output when it's locked up as when it's working normally, and at the time that the problem starts there's no
associated messages in dmesg or journalctl, except for maybe this one:
Code:

Mar 09 22:03:26 hekate wpa_supplicant[2712]: wlp2s0: WPA: Group rekeying completed with c8:54:4b:09:c6:69 [GTK=TKIP]

that "group rekeying" message shows up from time to time in journalctl, and it
seems to often appear slightly before the WiFi quits, but correlation is not
causation.

Even stranger, I've only had this problem when connected to my home router (a
Zyxel C3000Z), and my laptop is the only device in the house that has this
problem. I even have a Gentoo desktop which is also using NetworkManager and
systemd and it never has this problem.

Anyone have any sense of what's going on here or how I could better diagnose
it?
Back to top
View user's profile Send private message
Juippisi
Developer
Developer


Joined: 30 Sep 2005
Posts: 425
Location: /home

PostPosted: Tue Mar 17, 2020 6:32 am    Post subject: Reply with quote

Few things come to mind,

1: Are you using 'iwd'? I noticed similar behaviour with iwd in systemd and when I switched it off, my net hasn't disconnected.
2: Maybe https://gitweb.gentoo.org/repo/gentoo.git/commit/net-misc/networkmanager?id=2587f0225c6aaa23fcef6a09f4e92c3b2fda3769 these changes broke it for you and you want to back out from them.

Sorry, no further clues. You might also want to try latest networkmanager-1.22.10 that needs unmasking.
Back to top
View user's profile Send private message
jroth
n00b
n00b


Joined: 08 Aug 2017
Posts: 66

PostPosted: Tue Mar 17, 2020 7:42 pm    Post subject: Reply with quote

Juippisi wrote:
Few things come to mind,

1: Are you using 'iwd'? I noticed similar behaviour with iwd in systemd and when I switched it off, my net hasn't disconnected.
2: Maybe https://gitweb.gentoo.org/repo/gentoo.git/commit/net-misc/networkmanager?id=2587f0225c6aaa23fcef6a09f4e92c3b2fda3769 these changes broke it for you and you want to back out from them.

Sorry, no further clues. You might also want to try latest networkmanager-1.22.10 that needs unmasking.
https://unix.stackexchange.com/questions/312280/split-string-by-delimiter-and-get-n-th-element

1. Nope, iwd isn't installed
2. That patch does seem to be from about the time (early this year) that the problem first occurred

I'll start emerging the the 1.22.10 version and see if that helps.
Back to top
View user's profile Send private message
jroth
n00b
n00b


Joined: 08 Aug 2017
Posts: 66

PostPosted: Wed Mar 18, 2020 5:11 am    Post subject: Reply with quote

jroth wrote:

I'll start emerging the the 1.22.10 version and see if that helps.


Upon emerging networkmanager 1.22.10 and rebooting, my wifi didn't work at all. Is there something I have to rebuild/reconfigure to get this version to run?
Back to top
View user's profile Send private message
Juippisi
Developer
Developer


Joined: 30 Sep 2005
Posts: 425
Location: /home

PostPosted: Wed Mar 18, 2020 5:57 am    Post subject: Reply with quote

jroth wrote:

Upon emerging networkmanager 1.22.10 and rebooting, my wifi didn't work at all. Is there something I have to rebuild/reconfigure to get this version to run?


It should work out of the box, it's not much different from the stable version... may I ask what USE flags you have enabled?

Here's what you can do to get the latest working version:
Make a local overlay, https://wiki.gentoo.org/wiki/Handbook:AMD64/Portage/CustomTree#Defining_a_custom_repository
Code:

cd path/to/your/local/repo
mkdir net-misc
cp -r /var/db/repos/gentoo/net-mics/networkmanager net-misc
cd net-misc/networkmanager
wget https://gitweb.gentoo.org/repo/gentoo.git/plain/net-misc/networkmanager/networkmanager-1.18.4-r1.ebuild?id=cc4abb64572ec522fa4904b9092e65cc3e3bd7c1 -O networkmanager-1.18.4-r5


Remove the unmask for 1.22 and try updating networkmanager. Your -r5 will be older than the current -r3 in tree.

Although I'd like to find a reason why 1.22 doesnt work at all ;)
Back to top
View user's profile Send private message
jroth
n00b
n00b


Joined: 08 Aug 2017
Posts: 66

PostPosted: Thu Mar 19, 2020 5:15 am    Post subject: Reply with quote

Juippisi wrote:

It should work out of the box, it's not much different from the stable version... may I ask what USE flags you have enabled?

Although I'd like to find a reason why 1.22 doesnt work at all ;)


I've got:

Code:
jacob@hekate ~ $ equery uses net-misc/networkmanager
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for net-misc/networkmanager-1.18.4-r3:
 U I
 + + abi_x86_32         : 32-bit (x86) libraries
 - - audit              : Enable support for Linux audit subsystem using sys-process/audit
 + + bluetooth          : Enable Bluetooth Support
 - - connection-sharing : Use net-dns/dnsmasq and net-firewall/iptables for connection sharing
 + + dhclient           : Use dhclient from net-misc/dhcp for getting ip
 - - dhcpcd             : Use net-misc/dhcpcd for getting ip
 - - gnutls             : Prefer net-libs/gnutls as SSL/TLS provider (ineffective with USE=-ssl)
 + + introspection      : Add support for GObject based introspection
 - - iwd                : Use net-wireless/iwd instead of net-wireless/wpa_supplicant for wifi support
                          by default
 - - json               : Enable JSON validation via dev-libs/jansson in libnm.
 + + modemmanager       : Enable support for mobile broadband devices using net-misc/modemmanager
 + + ncurses            : Add ncurses support (console display library)
 + + nss                : Use dev-libs/nss for cryptography
 - - ofono              : Use net-misc/ofono for telephony support.
 - - ovs                : Enable OpenVSwitch support
 - - policykit          : Enable PolicyKit authentication support
 + + ppp                : Enable support for mobile broadband and PPPoE connections using
                          net-dialup/ppp
 - - resolvconf         : Use net-dns/openresolv for managing DNS information in /etc/resolv.conf.
                          Generally, a symlink to /run/NetworkManager/resolv.conf is simpler. On
                          systems running systemd-resolved, disable this flag and create a symlink to
                          /run/systemd/resolve/stub-resolv.conf.
 + + systemd            : Enable use of systemd-specific libraries and features like socket activation
                          or session tracking
 - - teamd              : Enable Teamd control support
 - - test               : Enable dependencies and/or preparations necessary to run tests (usually
                          controlled by FEATURES=test but can be toggled independently)
 - - vala               : Enable bindings for dev-lang/vala
 + + wext               : Enable support for the deprecated Wext (Wireless Extensions) API; needed for
                          some older drivers (e.g. ipw2200, ndiswrapper)
 + + wifi               : Enable support for wifi and 802.1x security
jacob@hekate ~ $
Back to top
View user's profile Send private message
deagol
n00b
n00b


Joined: 12 Jul 2014
Posts: 30

PostPosted: Thu Mar 19, 2020 7:21 am    Post subject: Reply with quote

About the original problem:

This is very likely caused by PTK rekeying. Quite some cards (seems to be >50%) have some problems with replacing the unicast key in a running association. This is (with at least hostapd) by default the case when you use WPA-EAP. In that case any traffic at the time the connection is rekeyed is dangerous. (When there are no packets at all in a few ms window it works but even light traffic can trigger the bug.) When you wait a full hour it should start working again, but the ways below are a much better way to confirm you have that issue.
When your wifi card is using mac80211 and a kernel >=4.20 you will get a warning in the log when it (used to) freeze:
Code:

Rekeying PTK for STA XX:XX:XX:XX:XX:XX but driver can't safely do that.

Alternatively you can also enable debugging in wpa_supplicant and check if you get a log message like that when it freezes:
(You must see that line at the initial connect. If not something is wrong with your debug settings.)
Code:

WPA: Key negotiation completed with XX:XX:XX:XX:XX:XX [PTK=XXXX GTK=XXXX]

Depending what exactly is the problem for you a kernel >= 4.20 can even solve the issue. You have a very good chance when you are using ath9k on the system you update the kernel on.

Once you confirm you have a PTK rekey problem there are multiple options.
1) Stop rekeying PTK key. (In hostapd it can be disabled. Commercial Routers may be a bit more tricky.)
2) Try to get rekeying working (Can be hard without replacing HW, the problem can be the Router, the Client or even both)
2) Or a very recent workaround: Use wpa_supplicant from git and enable wpa_deny_ptk0_rekey (May need a compile time change)
Back to top
View user's profile Send private message
jroth
n00b
n00b


Joined: 08 Aug 2017
Posts: 66

PostPosted: Sat Mar 21, 2020 3:59 am    Post subject: Reply with quote

deagol wrote:

Alternatively you can also enable debugging in wpa_supplicant and check if you get a log message like that when it freezes:
(You must see that line at the initial connect. If not something is wrong with your debug settings.)
Code:

WPA: Key negotiation completed with XX:XX:XX:XX:XX:XX [PTK=XXXX GTK=XXXX]

I've managed to put my wpa_supplicant into debug mode and I see:
Code:
wpa_supplicant[8795]: wlp2s0: WPA: Key negotiation completed with c8:54:4b:09:c6:69 [PTK=CCMP GTK=TKIP]


I'll see if any of those rekeying messages show up.
Quote:

Depending what exactly is the problem for you a kernel >= 4.20 can even solve the issue. You have a very good chance when you are using ath9k on the system you update the kernel on.

This system is using ath10k, does that have the same issue?
Back to top
View user's profile Send private message
deagol
n00b
n00b


Joined: 12 Jul 2014
Posts: 30

PostPosted: Sat Mar 21, 2020 9:40 am    Post subject: Reply with quote

jroth wrote:

This system is using ath10k, does that have the same issue?
To my best knowledge ath10k is doing everything perfectly fine. It's doing everything critical in HW and bypass the problematic parts in mac80211.
It's also not a simple bug, more a oversight that there is a special case and not handling it. This results in card/driver internals playing a mayor role how it plays out. (The oversight is not limited to Linux drivers, a sizeable fraction of driver authors missed it.)

When we can link it to PTK rekeys - and your problem description is dead on how that would look like - it must be your AP.
The only potential derivation is that you claim that only restarting the NetworkManager helps. In my cases manual reconnects are also working. (Basically anything what forces wpa_supplicat to replace the PTK must help.)

Even when other clients are working fine with the AP it's still possible and even likely - depending what the other STAs are and which cards they have. Especially windows clients and at least some android devices have additional bugs which are accidentally mitigating the issue for most cards I tested. They are doing things from a standard point of view even worse, causing a much more obvious error and trigger a reconnect (But I only have a comprehensive understandig for mac80211 cards using ath9k, ath10k and iwlwifi.)

You can btw. trigger the PTK issue(s) quite reliable when you have network load at the time. Downloads/uploads or flood pings is sufficient to trigger it for me every time. Generally rule is: The closer you are to the AP (and the higher the connection rate) the better the chances to trigger the issue. Not using A-MPDU drastically reduces the chances but it also depends on the card/driver with the bug. ath9k simply is the worst-case scenario and affected by all isuses. iwlwifi had more "luck" and sides step most of them when not using A-MPDU. Ath10k is the winner so far which side steps all...
Back to top
View user's profile Send private message
jroth
n00b
n00b


Joined: 08 Aug 2017
Posts: 66

PostPosted: Mon Mar 23, 2020 1:49 am    Post subject: Reply with quote

deagol wrote:

When we can link it to PTK rekeys - and your problem description is dead on how that would look like - it must be your AP.


so, if all the evidence comes back the way you think it will, I will have to somehow fix my router? It's a Zyxel C3000Z, provided by my ISP.
Back to top
View user's profile Send private message
deagol
n00b
n00b


Joined: 12 Jul 2014
Posts: 30

PostPosted: Mon Mar 23, 2020 6:17 am    Post subject: Reply with quote

jroth wrote:
so, if all the evidence comes back the way you think it will, I will have to somehow fix my router? It's a Zyxel C3000Z, provided by my ISP.

Wenn you have an accesible setting to disable PTK rekey on the router that`s the obvious way to go.
Missing that you probably will have to use the new feature in wpa_supplicant to work around the problems in your client, at least when you need a fix now. But I really would test that first... After all there are other ways to get similar synthoms, they just seem less likely.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum