Docker Ping & Network Issue [SOLVED]

Message

p2k2 · Post by **p2k2** » Wed Jan 25, 2017 9:08 am

Hi,

I've installed docker on my Gentoo workstation. The basic tests work as expected:

docker run --rm hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.
...

The docker info command gives the following result:

Code: Select all

Containers: 35
 Running: 0
 Paused: 0
 Stopped: 35
Images: 5
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-8:4-4993607-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.075 GB
 Data Space Total: 107.4 GB
 Data Space Available: 106.3 GB
 Metadata Space Used: 3.101 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93 (2015-01-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge overlay null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.39-gentoo
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.49 GiB
Name: workstation
ID: MT34:QL5I:GRYN:QMAN:H7XM:JTQR:2TBL:FY4Q:65YU:DSWD:OXO6:TZS6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://10.60.1.1:3128/
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Afterwards I tried some other images with network support. If I start a Centos container and ping
my workstation I receive a valid reply:

Code: Select all

docker run -it centos bash
[root@f81bc3cd3a2b /]# ping 10.60.1.52
PING 10.60.1.52 (10.60.1.52) 56(84) bytes of data.
64 bytes from 10.60.1.52: icmp_seq=1 ttl=64 time=0.039 ms
64 bytes from 10.60.1.52: icmp_seq=2 ttl=64 time=0.031 ms
64 bytes from 10.60.1.52: icmp_seq=3 ttl=64 time=0.032 ms
64 bytes from 10.60.1.52: icmp_seq=4 ttl=64 time=0.031 ms
^C
--- 10.60.1.52 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.031/0.033/0.039/0.005 ms
[root@f81bc3cd3a2b /]#

But if I try to ping a host on the same network I get no response from the host in the container:

Code: Select all

[root@f81bc3cd3a2b /]# ping 10.60.1.1 
PING 10.60.1.1 (10.60.1.1) 56(84) bytes of data.
^C
--- 10.60.1.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3000ms

[root@f81bc3cd3a2b /]#

First I checked if all kernel requirements are fulfilled:

Code: Select all

info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_LEGACY_VSYSCALL_EMULATE: enabled
- CONFIG_MEMCG_KMEM: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: enabled
- CONFIG_EXT3_FS: enabled
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: enabled
- CONFIG_EXT3_FS_SECURITY: enabled
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: missing
    - CONFIG_NF_CONNTRACK_FTP: missing
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: missing
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

Some optional storage drivers are missing but the core network config is complete.

The docker network configuration seems OK:

Code: Select all

docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
c475f7a639de        bridge              bridge              local               
33f73e26069a        host                host                local               
074bd9333101        none                null                local
docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "c475f7a639deed53d542ae8b3b762756c2ad130c6503271b868932da69ea1438",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "f81bc3cd3a2b6a0d919b06722920995210a965fb98565439cd420136ea73443e": {
                "Name": "evil_poitras",
                "EndpointID": "a943a44ab880d53e56f2594dde22db7db725ad97292090e701d1adc101b1681f",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

The docker daemon added the iptables rules:

Code: Select all

Chain INPUT (policy ACCEPT 1983 packets, 1072K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   14   973 ACCEPT     all  --  docker0 any     anywhere             anywhere            

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    4   336 DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    4   336 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 1185 packets, 222K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   26  1453 ACCEPT     all  --  any    docker0  anywhere             anywhere            

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    4   336 RETURN     all  --  any    any     anywhere             anywhere

And I also checked that the IP forwarding is active:

Code: Select all

cat /proc/sys/net/ipv4/ip_forward
1

I started wireshark and monitored the docker0 and enp0s31f6 interfaces to see if there are any packets coming back from the ping. If I start a ping from the container I receive the following packets:

Code: Select all

623	13.751429497	172.17.0.2	10.60.1.1	ICMP	98	Echo (ping) request  id=0x0011, seq=1/256, ttl=64 (no response found!)
624	13.751448342	10.60.1.52	10.60.1.1	ICMP	98	Echo (ping) request  id=0x0011, seq=1/256, ttl=63 (reply in 625)
625	13.751710387	10.60.1.1	10.60.1.52	ICMP	98	Echo (ping) reply    id=0x0011, seq=1/256, ttl=64 (request in 624)

If I got this correctly the container with the IP 172.17.0.2 starts the ping. My host machine transmits the ping request and the other host also sends a response but the response is not forwarded to the container.

I added two new rules to force a forwarding between the eth0 and docker0 :

Code: Select all

iptables -A FORWARD -i enp0s31f6 -o docker0 -j ACCEPT
iptables -A FORWARD -o enp0s31f6 -i docker0 -j ACCEPT

iptables -L -v
Chain INPUT (policy ACCEPT 67 packets, 13704 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   20  1303 ACCEPT     all  --  docker0 any     anywhere             anywhere            

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    7   588 DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    7   588 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  enp0s31f6 docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 enp0s31f6  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 21 packets, 4165 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   36  1943 ACCEPT     all  --  any    docker0  anywhere             anywhere            

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    7   588 RETURN     all  --  any    any     anywhere             anywhere

But this did not change the behaviour. I'm also using a Ubuntu 16.04 notebook with docker installed. On this machine the networking and ping work out of the box. I only noticed some differences in the iptables chains. This is the iptables -L -v from the notebook:

Code: Select all

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    24   2016 DOCKER-ISOLATION  all  --  any    any     anywhere             anywhere            
    12     1008 DOCKER     all  --  any    docker0  anywhere             anywhere            
    12     1008 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    12   1008 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere

The rules with cstate RELATED,ESTABLISHED and docker0 anywhere also receive some packets.
Which are zero on my Gentoo workstation.

Does somebody had some similar problems or could help me solve this issue.

Thank you in advance....

p2k2

pilla · Post by **pilla** » Wed Jan 25, 2017 9:35 am

Howdy,

Can you ping 10.60.1.1 from its own docker container?

p2k2 · Post by **p2k2** » Wed Jan 25, 2017 9:40 am

10.60.1.1 is a real machine on my network. I could ping it directly from my workstation which is my docker host:

Code: Select all

ping 10.60.1.1
PING 10.60.1.1 (10.60.1.1) 56(84) bytes of data.
64 bytes from 10.60.1.1: icmp_seq=1 ttl=64 time=0.353 ms
64 bytes from 10.60.1.1: icmp_seq=2 ttl=64 time=0.410 ms
64 bytes from 10.60.1.1: icmp_seq=3 ttl=64 time=0.404 ms
^C
--- 10.60.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.353/0.389/0.410/0.025 ms

Pinging from the Centos docker container still gets no response.

pilla · Post by **pilla** » Wed Jan 25, 2017 11:06 am

Maybe that's the problem, I am not sure bridge mode can properly deal with machines in the same network outside of docker. Can you try to use different networks for docker and your machines?

p2k2 · Post by **p2k2** » Wed Jan 25, 2017 12:02 pm

Hi pilla,

I think bridge mode should be able to deal with this situation. I started a container on the docker service on my notebook.

The network is a class B : 10.60.0.0/16

notebook (ubuntu 16.04) : 10.60.10.64
workstation (gentoo) : 10.60.1.52
external host : 10.60.1.1

Here are the results so far:

Code: Select all

uname -a
Linux notebook 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Code: Select all

systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─dns.conf, http-proxy.conf
   Active: active (running) since Mi 2017-01-25 08:44:33 CET; 3h 50min ago
     Docs: https://docs.docker.com
 Main PID: 28570 (dockerd)
   CGroup: /system.slice/docker.service
           ├─28570 dockerd --dns 10.60.1.129 -H fd://
           ├─28580 containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim containerd-shim --metrics-interval=0 --start-timeout
           └─29287 containerd-shim f2c16d505d3bbe43db7e219f7e4078348fe138523adc986f332b3cff012c3179 /var/run/docker/libcontainerd/f2c16d505d3bbe43db7e21

Docker is up and running

Code: Select all

docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 3
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host overlay bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-57-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.58 GiB
Name: notebook
ID: DQFC:4UML:BBZD:YPBJ:D76O:TLH4:QFFQ:IHN6:5KKZ:2YKD:PAK6:22QC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy: http://10.60.1.1:3128/
Https Proxy: http://10.60.1.1:3128/
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

The Centos container was started on the notebook in just the same way as on the Gentoo workstation:

Code: Select all

docker run -it --rm centos bash
[root@7407dd74db0e /]# ping 10.60.10.64
PING 10.60.10.64 (10.60.10.64) 56(84) bytes of data.
64 bytes from 10.60.10.64: icmp_seq=1 ttl=64 time=0.073 ms
64 bytes from 10.60.10.64: icmp_seq=2 ttl=64 time=0.046 ms
64 bytes from 10.60.10.64: icmp_seq=3 ttl=64 time=0.042 ms
64 bytes from 10.60.10.64: icmp_seq=4 ttl=64 time=0.042 ms
^C
--- 10.60.10.64 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.042/0.050/0.073/0.015 ms
[root@7407dd74db0e /]#

When I now try to ping the 10.60.1.1 from the container on the notwbook I get the following result:

Code: Select all


docker run -it centos bash
[root@c28a33640af1 /]# ping 10.60.1.1 
PING 10.60.1.1 (10.60.1.1) 56(84) bytes of data.
64 bytes from 10.60.1.1: icmp_seq=1 ttl=63 time=0.375 ms
64 bytes from 10.60.1.1: icmp_seq=2 ttl=63 time=0.467 ms
64 bytes from 10.60.1.1: icmp_seq=3 ttl=63 time=0.459 ms
^C
--- 10.60.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.375/0.433/0.467/0.048 ms

[root@c28a33640af1 /]# ping 10.60.1.52
PING 10.60.1.52 (10.60.1.52) 56(84) bytes of data.
64 bytes from 10.60.1.52: icmp_seq=1 ttl=63 time=0.583 ms
64 bytes from 10.60.1.52: icmp_seq=2 ttl=63 time=0.595 ms
64 bytes from 10.60.1.52: icmp_seq=3 ttl=63 time=0.618 ms
^C
--- 10.60.1.52 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.583/0.598/0.618/0.031 ms
[root@c28a33640af1 /]#

The network configuration seems to be identical between the two systems:

Code: Select all

docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
b1adfbdfa3b7        bridge              bridge              local               
e071bd21a4f3        host                host                local               
59ed570e9e1e        none                null                local

The network is also set to bridge. Now I compared the network settings for the bridge mode between the two docker installations:

Code: Select all

4c4
<         "Id": "c475f7a639deed53d542ae8b3b762756c2ad130c6503271b868932da69ea1438",
---
>         "Id": "b1adfbdfa3b773fe545402929bbe6e3becbcff2014262ff9f96247b6e780f604",
20,24c20,24
<             "f81bc3cd3a2b6a0d919b06722920995210a965fb98565439cd420136ea73443e": {
<                 "Name": "evil_poitras",
<                 "EndpointID": "a943a44ab880d53e56f2594dde22db7db725ad97292090e701d1adc101b1681f",
<                 "MacAddress": "02:42:ac:11:00:02",
<                 "IPv4Address": "172.17.0.2/16",
---
>             "7407dd74db0e7f3b31274c95e965ae60dd44c878c5540c5f1050ac20eafe4f0b": {
>                 "Name": "suspicious_mayer",
>                 "EndpointID": "7f495f61547ad6415ab1976aba558d4fc2124849ed0e5ff5603d671c30c3b4c0",
>                 "MacAddress": "02:42:ac:11:00:03",
>                 "IPv4Address": "172.17.0.3/16",

Both settings were obtained by executing docker network inspect bridge on both installations. Beside the different ids the only difference are the settings for the MAC and IP which does not explain my issues.

p2k2 · Post by **p2k2** » Thu Jan 26, 2017 6:58 am

I found my configuration error. I'm using systemd on my gentoo workstation and I had to explicitly activate the IP forwarding.

I had to adjust the /etc/systemd/network/50-static.network

Code: Select all

[Match]
Name=enp0s31f6

[Network]
DHCP=no
Address=10.60.1.52/16
Gateway=10.60.1.1
DNS=10.60.1.129
IPForward=1

The last line was missing.

Thanks....

pilla · Post by **pilla** » Thu Jan 26, 2017 10:56 am

Glad you got it working. I had a lot of interesting issues with networking in Docker (but on a Ubuntu server).

dblaci · Post by **dblaci** » Thu Mar 30, 2017 6:20 pm

I use openrc but as I found this thread, maybe I help others.

My problem was I did not enable the IP_VS_RR kernel config.

See:
https://bugs.gentoo.org/show_bug.cgi?id=614306