| View previous topic :: View next topic |
| Author |
Message |
musv
Advocate
Joined: 01 Dec 2002
Posts: 3372
Location: de
|
 Posted: Sat Mar 05, 2022 12:21 pm Post subject: systemd-resolved switches to wrong DNS |
 |
|
Hi there.
tldr:
Since my last system update systemd-resolved is doing some crazy stuff. Every few minutes systemd-resolved switches to the wrong DNS server.
Long story:
192.168.109.1: My Router, which provides a DNS service to the local network. DHCP is deactivated. Should be only the fallback option.
192.168.109.50: My NAS: provides DNS, DHCP, NFS, Webservice and so on. The DHCP server tells the clients to use 192.168.109.50 as primary DNS server, which works quite well.
After booting the client:
| systemd-resolved --status: |
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: uplink
Current DNS Server: 192.168.109.50
DNS Servers: 192.168.109.50
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google 2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google
Link 2 (eth0)
Current Scopes: DNS LLMNR/IPv4
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.109.50
DNS Servers: 192.168.109.50 192.168.109.1
DNS Domain: erz szb
|
Some minutes later:
| systemd-resolved --status: |
Global
Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: uplink
Current DNS Server: 192.168.109.50
DNS Servers: 192.168.109.50
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google 2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google
Link 2 (eth0)
Current Scopes: DNS LLMNR/IPv4
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.109.1
DNS Servers: 192.168.109.50 192.168.109.1
DNS Domain: erz szb
|
Why is systemd-resolved switching to the secondary DNS? This happens always since the last update (Systemd-250.3). With the previous versions I've never had this problem.
To show this daemon the right direction, just for trying I put the main DNS into the config:
| /etc/systemd/resolved.conf: |
[Resolve]
DNS=192.168.109.50
|
As the consequence systemd-resolved puts this entry in the first line of the /etc/resolv.conf. So the first 2 entries are: 192.168.109.50, the 3rd line is: 192.168.109.1. But ignorantly after some minutes it switches to 192.168.109.1 as the used DNS server. |
|
| Back to top |
|
 |
ozzy468
n00b
Joined: 27 Jan 2022
Posts: 17
|
| Posted: Fri Mar 11, 2022 8:25 pm Post subject: |
|
|
hi,
I can't really answer to your post, basically read it, because of 0 replies.
I use pihole as my dhcp/dns server in my network, and i was stunned how complicated dns can actually be. But from your post, 2 questions derive.
1st. Why do you have 2 dns servers in your network, guessing that both use the same dns servers in the background(8.8.8.8 or so)? and then, from a networking perspective
2nd. I guess on any "elaborated" network protocol setup, within your network, the router will always be 1 hop faster than the other dns server. So it might not be a systemd problem?
Both are more or less questions.
So why use the router's dns at all? |
|
| Back to top |
|
|
fred0
Tux's lil' helper
Joined: 17 Apr 2006
Posts: 81
Location: France
|
| Posted: Sat Apr 02, 2022 9:51 am Post subject: |
|
|
Hi,
Same for me. Any idea ? |
|
| Back to top |
|
|
AJM
Apprentice
Joined: 25 Sep 2002
Posts: 195
Location: Aberdeen, Scotland
|
| Posted: Mon Apr 04, 2022 10:15 am Post subject: |
|
|
| This is not intended to be an unhelpful reply, but my consistent experience over the years with SystemD is that you can expect things that never, ever gave any problems in the past to suddenly become unreliable / unpredictable. It's just the way it is I'm afraid - badly thought out, overreaching, buggy software. At least with Gentoo you have the choice of avoiding it - for now, at least. |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23357
|
| Posted: Mon Apr 04, 2022 3:09 pm Post subject: |
|
|
| When systemd-resolved changes the DNS servers, are there any log messages written? If so, what does it say? If not, can you enable more verbose logging and repeat the test? |
|
| Back to top |
|
|
fred0
Tux's lil' helper
Joined: 17 Apr 2006
Posts: 81
Location: France
|
| Posted: Fri Apr 08, 2022 1:01 pm Post subject: |
|
|
Hi,
I have local LXC container with net-dns/dnsmasq that use /etc/dnsmasq-hosts.conf for local network resolution.
On other LXC container (ubuntu, fedora, gentoo + openrc) everything work fine.
One QEMU VM with gentoo + systemd is also OK.
Every systems are up to date.
So long as I ask for a name pointing local network every thing work fine also.
But when I ask for external domain fall-back take place.
Example :
| Code: | | systemctl restart systemd-resolved.service |
| Code: | | systemd-resolve --status |
| Quote: | | Current DNS Server: 192.168.1.104 |
| Quote: | | 64 bytes from ce68f1cfc269 (192.168.1.111): icmp_seq=1 ttl=64 time=0.127 ms |
| Code: | | systemd-resolve --status |
| Quote: | | Current DNS Server: 192.168.1.104 |
| Quote: | | 64 bytes from 151.101.2.137 (151.101.2.137): icmp_seq=1 ttl=55 time=13.2 ms |
| Code: | | systemd-resolve --status |
| Quote: | | Current DNS Server: 1.1.1.1 |
| Quote: | | ping: cloud: Name or service not known |
Actually I have put | Code: | | systemctl restart systemd-resolved.service |
in a cron job every minute but that work not every time. |
|
| Back to top |
|
|
fred0
Tux's lil' helper
Joined: 17 Apr 2006
Posts: 81
Location: France
|
| Posted: Fri Apr 08, 2022 1:10 pm Post subject: |
|
|
| fred0 wrote: | Hi,
I have local LXC container 192.168.1.104 with net-dns/dnsmasq that use /etc/dnsmasq-hosts.conf for local network resolution.
On other LXC container (ubuntu, fedora, gentoo + openrc) everything work fine.
One QEMU VM with gentoo + systemd is also OK.
Every systems are up to date.
So long as I ask for a name pointing local network every thing work fine also.
But when I ask for external domain fall-back take place.
Example :
| Code: | | systemctl restart systemd-resolved.service |
| Code: | | systemd-resolve --status |
| Quote: | | Current DNS Server: 192.168.1.104 |
| Quote: | | 64 bytes from ce68f1cfc269 (192.168.1.111): icmp_seq=1 ttl=64 time=0.127 ms |
| Code: | | systemd-resolve --status |
| Quote: | | Current DNS Server: 192.168.1.104 |
| Quote: | | 64 bytes from 151.101.2.137 (151.101.2.137): icmp_seq=1 ttl=55 time=13.2 ms |
| Code: | | systemd-resolve --status |
| Quote: | | Current DNS Server: 1.1.1.1 |
| Quote: | | ping: cloud: Name or service not known |
Actually I have put | Code: | | systemctl restart systemd-resolved.service |
in a cron job every minute but that work not every time. |
|
|
| Back to top |
|
|
ozzy468
n00b
Joined: 27 Jan 2022
Posts: 17
|
| Posted: Fri Apr 08, 2022 6:16 pm Post subject: |
|
|
hi,
as a workaround, not a fix. Have you tried to set up a local domain? There is an option "Domains=" in systemd.network, and a "routing-only" domain "~."
So something like
DNS=your.dns.ip
Domains=~yourdomain
might do the trick. |
|
| Back to top |
|
|
fred0
Tux's lil' helper
Joined: 17 Apr 2006
Posts: 81
Location: France
|
| Posted: Fri Apr 08, 2022 7:45 pm Post subject: |
|
|
It is a Proxmox LXC container.
Maybe a link between Proxmox kernel and Gentoo systemd ?
Proxmox host kernel is 5.13.19-6-pve and gentoo QEMU VM with 5.15.26-gentoo kernel works.
I have this in /etc/resolv.conf
| Code: | # --- BEGIN PVE ---
search my_local_domain.local
nameserver 192.168.1.104
nameserver 1.1.1.1
# --- END PVE --- |
This in /etc/hosts
| Code: | # --- BEGIN PVE ---
127.0.0.1 localhost.localnet localhost
::1 localhost.localnet localhost
127.0.1.1 my_host.my_local_domain.local my_host
# --- END PVE --- |
And this in /etc/systemd/resolved.conf
| Code: | [Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google: 8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9: 9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google 2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google
#Domains=
#DNSSEC=allow-downgrade
#DNSOverTLS=no
#MulticastDNS=yes
#LLMNR=yes
#Cache=yes
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no |
Should I change this file ?
Normally no because DNS change are do by Proxmox (between BEGIN PVE and END PVE)
As musv said this happen after last update. I have updated all VMs and Proxmox together last Friday. |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3610
|
| Posted: Fri Apr 08, 2022 9:04 pm Post subject: |
|
|
Do you have a reason not to kill it with fire?
Resolved was a common source of problems on many systems I saw, ended up disabling and masking it in the installation script just to make sure it won't break things down the line.
I think you can change /etc/network/interfaces (since proxmox is based on debian) to make the update to DNS settings permanent. Obviously, update /etc/resolv.conf too, to make it effective immediately. |
|
| Back to top |
|
|
ozzy468
n00b
Joined: 27 Jan 2022
Posts: 17
|
| Posted: Fri Apr 08, 2022 10:41 pm Post subject: |
|
|
hi,
what I meant, was the "client" side. - I really have no xp concerning containers.
the original post, said, systemd falls back to the router/alternative dns server. now, from your post, systemd will also fall back to a dns server, outside your network. on first glance, this could be anything, network optimization, a new feature, that's been missed, or just a bug. we don't know.
So the idea was just, if you tell the client "there is a local domain"; that it would not switch/or switch and still use this domain for local dns - in a way that it's useless to look up local domains on an internet server. And - that was just a guess - that this behavior is implemented, no matter of buginess or intention.
But from your post, it looks like this has already been done. |
|
| Back to top |
|
|
fred0
Tux's lil' helper
Joined: 17 Apr 2006
Posts: 81
Location: France
|
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3610
|
| Posted: Tue Apr 19, 2022 12:01 am Post subject: |
|
|
| Mask it too, or it will come back to bite you in the ass later on. |
|
| Back to top |
|
|
fred0
Tux's lil' helper
Joined: 17 Apr 2006
Posts: 81
Location: France
|
| Posted: Tue Apr 19, 2022 6:59 am Post subject: |
|
|
| szatox wrote: | | Mask it too, or it will come back to bite you in the ass later on. |
Do you mean a thing like putting | Code: | | sys-apps/systemd -resolvconf |
in /etc/portage/package.use/sys ??? |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3610
|
| Posted: Tue Apr 19, 2022 11:58 am Post subject: |
|
|
No, I mean | Code: | | systemctl mask systemd-resolved |
Systemd keeps unit files in several locations with various priorities, the command above creates a link to /dev/null in the top priority location, which prevents systemd from looking in other locations and the service will not run even if something else attempts to pull it as a dependency.
Systemd-resolved is probably not an issue on non-systemd systems as they don't use it (at least for now). If you do want to have a local DNS cache, you can still use dnsmasq (for simplicity) or bind (for advanced, non-standard setup). Most people don't need it, but if you happened to find yourself an exception, do it yourself so it's done right. |
|
| Back to top |
|
|
fred0
Tux's lil' helper
Joined: 17 Apr 2006
Posts: 81
Location: France
|
| Posted: Thu Apr 21, 2022 6:17 am Post subject: |
|
|
Now, without resolved, dns resolution works. I don't know how but there is most likely a dns software on this system.
Is there perhaps a conflict with resolved ?
Could it be possible to remove the other dns software and restore resolved ?
How to find this other piece of code ? |
|
| Back to top |
|
|
|
Networking & Security
I am out of trouble thanks to 