alienjon
Veteran
Joined: 09 Feb 2005
Posts: 1732
|
 Posted: Sun Mar 23, 2025 3:41 am Post subject: Docker Network Problems |
 |
|
I currently run a few docker containers on a Gentoo server I have at home. Among them are frigate and mosquitto. I've had problems with mosquitto for a while (I have it as I understand that it's helpful for frigate, but I'm not using it directly) along the lines I'm about to describe, but recently frigate stopped recording clips and when I went to investigate found that the docker container was listed as 'unhealthy'. Down a bit of a rabbit hole I think there's a problem with how docker is getting access to my network. To admit up front, I'm not very good with firewalls. It's always been a difficult concept for me to fully get my head around, so I created this script (Based off of one I found online a while back) to help track and document what and why I have certain rules in place. I use iptables and will run this when I want to make changes:
| Code: | #!/bin/bash
set -e
IPTABLES=/sbin/iptables
echo " **********************"
echo " * Clearing old rules *"
echo " * flushing old rules"
${IPTABLES} --flush
${IPTABLES} --delete-chain
${IPTABLES} --table nat --flush
${IPTABLES} --table nat --delete-chain
echo ""
echo " *****************"
echo " * Initial setup *"
echo " * setting default policies"
${IPTABLES} -P INPUT DROP
${IPTABLES} -P FORWARD DROP
${IPTABLES} -P OUTPUT ACCEPT
echo " * allowing loopback devices"
${IPTABLES} -A INPUT -i lo -j ACCEPT
${IPTABLES} -A OUTPUT -o lo -j ACCEPT
${IPTABLES} -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
${IPTABLES} -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
echo ""
echo " *****************"
echo " * Blacklist IPs *"
## BLOCK ABUSING IPs HERE ##
#echo " * BLACKLIST"
#${IPTABLES} -A INPUT -s _ABUSIVE_IP_ -j DROP
#${IPTABLES} -A INPUT -s _ABUSIVE_IP2_ -j DROP
echo ""
echo " *****************"
echo " * Whitelist IPs *"
echo " * allowing all LAN ports from 192.168.1.0/24 and 192.168.2.0/24"
${IPTABLES} -A INPUT -s 192.168.1.0/24 -j ACCEPT
${IPTABLES} -A INPUT -s 192.168.2.0/24 -j ACCEPT
echo " * allowing Gentoo-specific addresses"
${IPTABLES} -A INPUT -s 200.236.31.1 -j ACCEPT
echo ""
echo " *****************"
echo " * Opening Ports *"
echo " * allowing ping responses"
${IPTABLES} -A INPUT -p ICMP --icmp-type 8 -j ACCEPT
echo " * allowing ftp and sftp, but only locally"
${IPTABLES} -A INPUT -p tcp -s 192.168.1.0/24 --dport ftp -j ACCEPT
${IPTABLES} -A INPUT -p tcp -s 192.168.1.0/24 --dport sftp -j ACCEPT
${IPTABLES} -A INPUT -p tcp -s 192.168.2.0/24 --dport ftp -j ACCEPT
${IPTABLES} -A INPUT -p tcp -s 192.168.2.0/24 --dport sftp -j ACCEPT
echo " * allowing ssh on port 22"
${IPTABLES} -A INPUT -p tcp --dport ssh -m state --state NEW -j ACCEPT
echo " * allowing DNS on port 53 (udp and tcp)"
${IPTABLES} -A INPUT -p udp -m udp --dport 53 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
echo " * allowing DHCPCD on ports 67 (v4), 68 (v4), and 546 (v6)"
${IPTABLES} -A INPUT -p tcp -m tcp --dport 67 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 68 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 546 -j ACCEPT
echo " * allowing Samba server ports 137, 138, 139, and 445"
${IPTABLES} -A INPUT -p udp -m udp --dport 137 -j ACCEPT
${IPTABLES} -A INPUT -p udp -m udp --dport 138 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 139 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 445 -j ACCEPT
echo " * allowing connections to Home Assistant on port 8123"
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8123 -j ACCEPT
echo " * allowing http on port 80"
${IPTABLES} -A INPUT -p tcp --dport 80 -m state --state NEW -j ACCEPT
echo " * allowing https on port 443"
${IPTABLES} -A INPUT -p tcp --dport 443 -m state --state NEW -j ACCEPT
echo " * allowing OpenVPN on port 1194"
${IPTABLES} -A INPUT -p tcp --dport 1194 -m state --state NEW -j ACCEPT
echo " * allowing PLEX ports (1900, 3005, 5353, 8324, 32400, 32410, 32412, 32413, 32414, and 32469"
${IPTABLES} -A INPUT -p tcp -m tcp --dport 1900 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 3005 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 5353 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8324 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 32400 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 32410 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 32412 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 32413 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 32414 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 32469 -j ACCEPT
echo " * allowing Monitorix on ports 7070 and 8080"
${IPTABLES} -A INPUT -p tcp -m tcp --dport 7070 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8080 -j ACCEPT
echo " * allowing Unifi on ports 8443, 8843, and 8880"
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8443 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8843 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8880 -j ACCEPT
echo " * allowing MongoDB on ports 27017 and 27117"
${IPTABLES} -A INPUT -p tcp -m tcp --dport 27017 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 27117 -j ACCEPT
echo " * allowing ports for frigate"
${IPTABLES} -A INPUT -p tcp -m tcp --dport 5000 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8554 -j ACCEPT
${IPTABLES} -A INPUT -p tcp -m tcp --dport 8971 -j ACCEPT
#echo " * allowing ports for docker (mosquitto)"
#${IPTABLES} --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 9001 -j DNAT --to-destination 172.18.0.2:9001 ! -i br-4876afc89013
#${IPTABLES} --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8555 -j DNAT --to-destination 172.20.0.2:8555 ! -i br-5cd6d96fcd08
# DROP everything else and Log it
#${IPTABLES} -A INPUT -j LOG
echo " * Dropping everything else"
${IPTABLES} -A INPUT -j DROP
#
# Save settings
#
echo ""
echo " ****************"
echo " * SAVING RULES *"
/etc/init.d/iptables save |
With this in place, if I try to start mosquitto I get:
| docker compose up -d: |
[+] Running 0/1
⠴ Container mosquitto Starting 1.6s
Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint mosquitto (76ebc2b03b12c3f2e5c2b5ce660a1d80eeb45f9bdf8979e0991d0b2530aa4941): Unable to enable DNAT rule: (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 1883 -j DNAT --to-destination 172.18.0.2:1883 ! -i br-4876afc89013: iptables: No chain/target/match by that name.
(exit status 1)) |
You'll notice that I have this rule in my script as well (though commented out) and get the same error if I uncomment it. Frigate was working okay, but recently started giving me a similar error:
| docker compose up -d: | [+] Running 1/1
✔ Container frigate Created 0.0s
Attaching to frigate
Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint frigate (bdeeb4c7b08b2162d9665c098da691751a5196e122ae00ef769a67263c95f2a3): Unable to enable DNAT rule: (iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 5000 -j DNAT --to-destination 172.20.0.2:5000 ! -i br-5cd6d96fcd08: iptables: No chain/target/match by that name.
(exit status 1)) |
I have had a recent kernel upgrade recently, but it's not clear to me what may have changed (and the config should have copied over, so I wouldn't assume that something was dropped, but am not even quite sure what to look for in the kernel config - I'm running 6.12.16). Dmesg has related info:
| dmesg: | [ 13.561927] intel_tcc_cooling: Programmable TCC Offset detected
[ 13.578224] mei_me 0000:00:16.0: enabling device (0000 -> 0002)
[ 13.609209] alx 0000:07:00.0 enp7s0: renamed from eth0
[ 13.758166] intel_rapl_common: Found RAPL domain package
[ 13.758170] intel_rapl_common: Found RAPL domain core
[ 13.758172] intel_rapl_common: Found RAPL domain dram
[ 13.942509] ACPI: bus type drm_connector registered
[ 16.683014] nvidia: loading out-of-tree module taints kernel.
[ 16.683026] nvidia: module license 'NVIDIA' taints kernel.
[ 16.683028] Disabling lock debugging due to kernel taint
[ 16.683033] nvidia: module license taints kernel.
[ 16.706372] nvidia-nvlink: Nvlink Core is being initialized, major device number 245
[ 16.707683] nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=io+mem
[ 16.824050] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 570.124.04 Tue Feb 25 04:12:12 UTC 2025
[ 16.940905] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 570.124.04 Tue Feb 25 03:39:21 UTC 2025
[ 17.076607] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[ 17.076618] [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0
[ 17.960073] EXT4-fs (sda4): re-mounted c9ff1a6e-2ae7-4c5b-983b-df07589728a6 r/w. Quota mode: none.
[ 18.033997] EXT4-fs (sda4): re-mounted c9ff1a6e-2ae7-4c5b-983b-df07589728a6 r/w. Quota mode: none.
[ 18.514963] Adding 524284k swap on /dev/sda3. Priority:-2 extents:1 across:524284k
[ 18.691725] XFS (sde1): Mounting V5 Filesystem cce5aa43-a261-44e0-bfea-21c761a386d7
[ 18.899215] XFS (sde1): Ending clean mount
[ 18.994401] fuse: init (API version 7.41)
[ 19.529980] XFS (sdc1): Mounting V5 Filesystem 766da839-f098-42e9-b585-d6640a30ee8e
[ 19.714329] XFS (sdc1): Ending clean mount
[ 53.120253] alx 0000:07:00.0 enp7s0: NIC Up: 1 Gbps Full
[ 85.193657] br-5e71065cab30: port 1(veth39f2e1b) entered blocking state
[ 85.193674] br-5e71065cab30: port 1(veth39f2e1b) entered disabled state
[ 85.193695] veth39f2e1b: entered allmulticast mode
[ 85.193890] veth39f2e1b: entered promiscuous mode
[ 85.579094] nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
[ 85.620563] nvidia-uvm: Loaded the UVM driver, major device number 243.
[ 86.207588] eth0: renamed from veth68dc020
[ 86.208880] br-5e71065cab30: port 1(veth39f2e1b) entered blocking state
[ 86.208893] br-5e71065cab30: port 1(veth39f2e1b) entered forwarding state
[ 116.206149] br-5cd6d96fcd08: port 1(veth0e00b48) entered blocking state
[ 116.206152] br-5cd6d96fcd08: port 1(veth0e00b48) entered disabled state
[ 116.206156] veth0e00b48: entered allmulticast mode
[ 116.206180] veth0e00b48: entered promiscuous mode
[ 116.886528] eth0: renamed from vethace7036
[ 116.887034] br-5cd6d96fcd08: port 1(veth0e00b48) entered blocking state
[ 116.887038] br-5cd6d96fcd08: port 1(veth0e00b48) entered forwarding state
[ 389.717315] Bluetooth: Core ver 2.22
[ 389.717363] NET: Registered PF_BLUETOOTH protocol family
[ 389.717368] Bluetooth: HCI device and connection manager initialized
[ 389.717377] Bluetooth: HCI socket layer initialized
[ 389.717383] Bluetooth: L2CAP socket layer initialized
[ 389.717397] Bluetooth: SCO socket layer initialized
[ 1119.626286] TCP: request_sock_TCP: Possible SYN flooding on port 192.168.1.4:5357. Sending cookies.
[ 3148.299124] br-4876afc89013: port 1(veth27553fd) entered blocking state
[ 3148.299128] br-4876afc89013: port 1(veth27553fd) entered disabled state
[ 3148.299132] veth27553fd: entered allmulticast mode
[ 3148.299161] veth27553fd: entered promiscuous mode
[ 3148.398983] eth0: renamed from veth198b178
[ 3148.399425] br-4876afc89013: port 1(veth27553fd) entered blocking state
[ 3148.399428] br-4876afc89013: port 1(veth27553fd) entered forwarding state
[ 3149.466472] br-4876afc89013: port 1(veth27553fd) entered disabled state
[ 3149.466706] veth27553fd (unregistering): left allmulticast mode
[ 3149.466709] veth27553fd (unregistering): left promiscuous mode
[ 3149.466710] br-4876afc89013: port 1(veth27553fd) entered disabled state
[ 3441.979138] br-4876afc89013: port 1(veth22b226f) entered blocking state
[ 3441.979142] br-4876afc89013: port 1(veth22b226f) entered disabled state
[ 3441.979146] veth22b226f: entered allmulticast mode
[ 3441.979186] veth22b226f: entered promiscuous mode
[ 3442.067908] eth0: renamed from veth5910543
[ 3442.069553] br-4876afc89013: port 1(veth22b226f) entered blocking state
[ 3442.069563] br-4876afc89013: port 1(veth22b226f) entered forwarding state
[ 3443.113259] br-4876afc89013: port 1(veth22b226f) entered disabled state
[ 3443.113479] veth22b226f (unregistering): left allmulticast mode
[ 3443.113482] veth22b226f (unregistering): left promiscuous mode
[ 3443.113485] br-4876afc89013: port 1(veth22b226f) entered disabled state
[ 3543.161885] br-5cd6d96fcd08: port 1(veth0e00b48) entered disabled state
[ 3543.162135] vethace7036: renamed from eth0
[ 3543.220931] br-5cd6d96fcd08: port 1(veth0e00b48) entered disabled state
[ 3543.221511] veth0e00b48 (unregistering): left allmulticast mode
[ 3543.221521] veth0e00b48 (unregistering): left promiscuous mode
[ 3543.221528] br-5cd6d96fcd08: port 1(veth0e00b48) entered disabled state
[ 3551.250515] br-5cd6d96fcd08: port 1(veth41fc1fc) entered blocking state
[ 3551.250532] br-5cd6d96fcd08: port 1(veth41fc1fc) entered disabled state
[ 3551.250553] veth41fc1fc: entered allmulticast mode
[ 3551.250731] veth41fc1fc: entered promiscuous mode
[ 3551.380033] eth0: renamed from veth25ce8e6
[ 3551.381482] br-5cd6d96fcd08: port 1(veth41fc1fc) entered blocking state
[ 3551.381497] br-5cd6d96fcd08: port 1(veth41fc1fc) entered forwarding state
[ 3552.443305] br-5cd6d96fcd08: port 1(veth41fc1fc) entered disabled state
[ 3552.444172] veth41fc1fc (unregistering): left allmulticast mode
[ 3552.444187] veth41fc1fc (unregistering): left promiscuous mode
[ 3552.444196] br-5cd6d96fcd08: port 1(veth41fc1fc) entered disabled state
[ 3574.951612] br-5cd6d96fcd08: port 1(veth60c8c24) entered blocking state
[ 3574.951625] br-5cd6d96fcd08: port 1(veth60c8c24) entered disabled state
[ 3574.951642] veth60c8c24: entered allmulticast mode
[ 3574.951757] veth60c8c24: entered promiscuous mode
[ 3575.047433] eth0: renamed from vethaed6c9e
[ 3575.048475] br-5cd6d96fcd08: port 1(veth60c8c24) entered blocking state
[ 3575.048482] br-5cd6d96fcd08: port 1(veth60c8c24) entered forwarding state
[ 3576.103309] br-5cd6d96fcd08: port 1(veth60c8c24) entered disabled state
[ 3576.104451] veth60c8c24 (unregistering): left allmulticast mode
[ 3576.104464] veth60c8c24 (unregistering): left promiscuous mode
[ 3576.104472] br-5cd6d96fcd08: port 1(veth60c8c24) entered disabled state |
Any thoughts as to where I should look to troubleshoot? The br-* devices seem to go in and out of different states, which feels related, but I'm not sure what that means int he larger scheme of things. |
|
Networking & Security
