Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] Docker stopped starting containers
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
r7l
n00b
n00b


Joined: 16 Feb 2019
Posts: 22

PostPosted: Sat Feb 16, 2019 6:33 pm    Post subject: [Solved] Docker stopped starting containers Reply with quote

Hello,

i was running Docker just fine on a few machines but it just stopped working after the most recent update. I don't know what it is but i can't start any containers anymore. When starting container i get the following error message:

Code:
docker: Error response from daemon: ttrpc: client shutting down: read unix @->@/containerd-shim/moby/565c3b0f77ecd3eac83e5f448311c764dbc720a6bf072eec50caf50f71a0cc30/shim.sock: read: connection reset by peer: unknown.


It seems like as if containerd is trying to delete a non existing container. I've deleted everything and uninstalled Docker to reinstall it again. I've also downgraded to the old version but the error remains. I have this issue in 2 of 4 machines i've updated Docker.

This is the error in the log file:

Code:
time="2019-02-16T15:11:58.055921019+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/97a4a24fe6d8532f19b1415218e7afe2823c941e4a97c99784977d955bf06922/shim.sock" debug=false pid=26281
time="2019-02-16T15:11:58.057308149+01:00" level=error msg="failed to kill shim" error="ttrpc: client shutting down: read unix @->@/containerd-shim/moby/97a4a24fe6d8532f19b1415218e7afe2823c941e4a97c99784977d955bf06922/shim.sock: read: connection reset by peer: unknown"
time="2019-02-16T15:11:58.057536205+01:00" level=info msg="shim reaped" id=97a4a24fe6d8532f19b1415218e7afe2823c941e4a97c99784977d955bf06922
time="2019-02-16T15:11:58.057694715+01:00" level=error msg="stream copy error: reading from a closed fifo"
time="2019-02-16T15:11:58.101606438+01:00" level=error msg="97a4a24fe6d8532f19b1415218e7afe2823c941e4a97c99784977d955bf06922 cleanup: failed to delete container from containerd: no such container"
time="2019-02-16T15:11:58.101750561+01:00" level=error msg="Handler for POST /v1.39/containers/97a4a24fe6d8532f19b1415218e7afe2823c941e4a97c99784977d955bf06922/start returned error: ttrpc: client shutting down: read unix @->@/containerd-shim/moby/97a4a24fe6d8532f19b1415218e7afe2823c941e4a97c99784977d955bf06922/shim.sock: read: connection reset by peer: unknown"


Docker Version

Code:
Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.11.5
 Git commit:        6247962
 Built:             Sat Feb 16 13:15:59 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.5
  Git commit:       6247962
  Built:            Sat Feb 16 13:15:10 2019
  OS/Arch:          linux/amd64
  Experimental:     false


Docker Info

Code:
Containers: 8
 Running: 0
 Paused: 0
 Stopped: 8
Images: 3
Server Version: 18.09.2
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683b971d9c3ef73f284f176672c44b448662
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.83-gentoo
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 11.73GiB
Name: r42
ID: A4S5:PBU2:NNTX:K6UM:ZUUA:VZBQ:VKYS:INPH:C3MT:2K7Y:JIJV:6SJO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false


Thanks for any help in advance!


Last edited by r7l on Mon Feb 18, 2019 12:41 pm; edited 1 time in total
Back to top
View user's profile Send private message
r7l
n00b
n00b


Joined: 16 Feb 2019
Posts: 22

PostPosted: Sat Feb 16, 2019 11:54 pm    Post subject: Reply with quote

I've did some further testing on a working and a non working KVM machine. I've made sure both systems have the exact same versions of the Kernel and Docker. Kernel config is the same as well.

I've also enabled debug mode on both Docker instances and tried running this command:

docker run -it --rm busybox

Output on the machine with working Docker:

Code:
time="2019-02-17T00:35:27.560907855+01:00" level=debug msg="Calling GET /_ping"
time="2019-02-17T00:35:27.562313130+01:00" level=debug msg="Calling POST /v1.39/containers/create"
time="2019-02-17T00:35:27.562609894+01:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":true,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":[],\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"busybox\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":true,\"StdinOnce\":true,\"Tty\":true,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
time="2019-02-17T00:35:27.582813329+01:00" level=debug msg="container mounted via layerStore: &{/var/lib/docker/overlay2/fc74cdfc4181eedc67994fbd7b359c44a989b7fe3fdac98744620b159eb2169b/merged 0x564813022300 0x564813022300}"
time="2019-02-17T00:35:27.597272206+01:00" level=debug msg="Calling POST /v1.39/containers/53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8/attach?stderr=1&stdin=1&stdout=1&stream=1"
time="2019-02-17T00:35:27.597507683+01:00" level=debug msg="attach: stdin: begin"
time="2019-02-17T00:35:27.597541557+01:00" level=debug msg="attach: stderr: begin"
time="2019-02-17T00:35:27.597540238+01:00" level=debug msg="attach: stdout: begin"
time="2019-02-17T00:35:27.598031195+01:00" level=debug msg="Calling POST /v1.39/containers/53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8/wait?condition=removed"
time="2019-02-17T00:35:27.598621210+01:00" level=debug msg="Calling POST /v1.39/containers/53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8/start"
time="2019-02-17T00:35:27.599697849+01:00" level=debug msg="container mounted via layerStore: &{/var/lib/docker/overlay2/fc74cdfc4181eedc67994fbd7b359c44a989b7fe3fdac98744620b159eb2169b/merged 0x564813022300 0x564813022300}"
time="2019-02-17T00:35:27.600064638+01:00" level=debug msg="Assigning addresses for endpoint eager_nightingale's interface on network bridge"
time="2019-02-17T00:35:27.600099437+01:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
time="2019-02-17T00:35:27.600130261+01:00" level=debug msg="Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3 Serial:false PrefAddress:<nil> "
time="2019-02-17T00:35:27.602770740+01:00" level=debug msg="Assigning addresses for endpoint eager_nightingale's interface on network bridge"
time="2019-02-17T00:35:27.605590268+01:00" level=debug msg="Programming external connectivity on endpoint eager_nightingale (311c1dac5466dac0b5e61308a26c596a784b6f8878e19e03b3448228a554cc10)"
time="2019-02-17T00:35:27.606322064+01:00" level=debug msg="EnableService 53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8 START"
time="2019-02-17T00:35:27.606364697+01:00" level=debug msg="EnableService 53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8 DONE"
time="2019-02-17T00:35:27.608675882+01:00" level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8 module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/fc74cdfc4181eedc67994fbd7b359c44a989b7fe3fdac98744620b159eb2169b/merged
time="2019-02-17T00:35:27.611437198+01:00" level=debug msg="event published" ns=moby topic="/containers/create" type=containerd.events.ContainerCreate
time="2019-02-17T00:35:27.615019023+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8/shim.sock" debug=true pid=6471
time="2019-02-17T00:35:27+01:00" level=debug msg="registering ttrpc server"
time="2019-02-17T00:35:27+01:00" level=debug msg="serving api on unix socket" socket="[inherited from parent]"
time="2019-02-17T00:35:27.847804996+01:00" level=debug msg="sandbox set key processing took 72.946182ms for container 53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8"
time="2019-02-17T00:35:27.915116828+01:00" level=debug msg="event published" ns=moby topic="/tasks/create" type=containerd.events.TaskCreate
time="2019-02-17T00:35:27.915577326+01:00" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/create
time="2019-02-17T00:35:27.941254843+01:00" level=debug msg="event published" ns=moby topic="/tasks/start" type=containerd.events.TaskStart
time="2019-02-17T00:35:27.941659888+01:00" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/start
time="2019-02-17T00:35:27.958813529+01:00" level=debug msg="Calling POST /v1.39/containers/53cf510f7fdd33a7299786a3bf341987ac537b366e603b2aa6c1d8e01c8710e8/resize?h=93&w=319"


Output of the machine with broken Docker:

Code:
time="2019-02-17T00:35:41.783667227+01:00" level=debug msg="Calling GET /_ping"
time="2019-02-17T00:35:41.784870279+01:00" level=debug msg="Calling POST /v1.39/containers/create"
time="2019-02-17T00:35:41.785210376+01:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[],\"HostConfig\":{\"AutoRemove\":true,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":[],\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"Devices\":[],\"DiskQuota\":0,\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":null,\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"KernelMemory\":0,\"Links\":null,\"LogConfig\":{\"Config\":{},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":-1,\"NanoCpus\":0,\"NetworkMode\":\"default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":0,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"no\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"busybox\",\"Labels\":{},\"NetworkingConfig\":{\"EndpointsConfig\":{}},\"OnBuild\":null,\"OpenStdin\":true,\"StdinOnce\":true,\"Tty\":true,\"User\":\"\",\"Volumes\":{},\"WorkingDir\":\"\"}"
time="2019-02-17T00:35:41.804498040+01:00" level=debug msg="container mounted via layerStore: &{/var/lib/docker/overlay2/77f8d9e45f138c12d2d6c581aca0804f3266d8dc6602f556491ef8b4e8cec7e4/merged 0x563b7dbb0300 0x563b7dbb0300}"
time="2019-02-17T00:35:41.821412369+01:00" level=debug msg="Calling POST /v1.39/containers/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4/attach?stderr=1&stdin=1&stdout=1&stream=1"
time="2019-02-17T00:35:41.821614706+01:00" level=debug msg="attach: stdin: begin"
time="2019-02-17T00:35:41.821657513+01:00" level=debug msg="attach: stdout: begin"
time="2019-02-17T00:35:41.821704242+01:00" level=debug msg="attach: stderr: begin"
time="2019-02-17T00:35:41.822166598+01:00" level=debug msg="Calling POST /v1.39/containers/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4/wait?condition=removed"
time="2019-02-17T00:35:41.823655335+01:00" level=debug msg="Calling POST /v1.39/containers/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4/start"
time="2019-02-17T00:35:41.824950550+01:00" level=debug msg="container mounted via layerStore: &{/var/lib/docker/overlay2/77f8d9e45f138c12d2d6c581aca0804f3266d8dc6602f556491ef8b4e8cec7e4/merged 0x563b7dbb0300 0x563b7dbb0300}"
time="2019-02-17T00:35:41.825439026+01:00" level=debug msg="Assigning addresses for endpoint xenodochial_bartik's interface on network bridge"
time="2019-02-17T00:35:41.825479324+01:00" level=debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
time="2019-02-17T00:35:41.825507738+01:00" level=debug msg="Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:<nil> "
time="2019-02-17T00:35:41.828975610+01:00" level=debug msg="Assigning addresses for endpoint xenodochial_bartik's interface on network bridge"
time="2019-02-17T00:35:41.832521115+01:00" level=debug msg="Programming external connectivity on endpoint xenodochial_bartik (73b310ef449cfdc0bd68a4949523b330fa87e62002579c54e360c790e5cfb1db)"
time="2019-02-17T00:35:41.833331893+01:00" level=debug msg="EnableService 36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4 START"
time="2019-02-17T00:35:41.833378491+01:00" level=debug msg="EnableService 36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4 DONE"
time="2019-02-17T00:35:41.837042105+01:00" level=debug msg="bundle dir created" bundle=/var/run/docker/containerd/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4 module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/77f8d9e45f138c12d2d6c581aca0804f3266d8dc6602f556491ef8b4e8cec7e4/merged
time="2019-02-17T00:35:41.843635160+01:00" level=debug msg="event published" ns=moby topic="/containers/create" type=containerd.events.ContainerCreate
time="2019-02-17T00:35:41.849763495+01:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4/shim.sock" debug=true pid=21507
time="2019-02-17T00:35:41.850341722+01:00" level=info msg="shim reaped" id=36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4
time="2019-02-17T00:35:41.850346433+01:00" level=error msg="failed to kill shim" error="ttrpc: client shutting down: read unix @->@/containerd-shim/moby/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4/shim.sock: read: connection reset by peer: unknown"
time="2019-02-17T00:35:41.851012600+01:00" level=error msg="stream copy error: reading from a closed fifo"
time="2019-02-17T00:35:41.854719329+01:00" level=debug msg="attach: stdout: end"
time="2019-02-17T00:35:41.854736957+01:00" level=debug msg="attach: stderr: end"
time="2019-02-17T00:35:41.854747472+01:00" level=debug msg="attach: stdin: end"
time="2019-02-17T00:35:41.854908351+01:00" level=debug msg="attach done"
time="2019-02-17T00:35:41.855019709+01:00" level=debug msg="Closing buffered stdin pipe"
time="2019-02-17T00:35:41.856454492+01:00" level=debug msg="event published" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
time="2019-02-17T00:35:41.858998417+01:00" level=debug msg="Revoking external connectivity on endpoint xenodochial_bartik (73b310ef449cfdc0bd68a4949523b330fa87e62002579c54e360c790e5cfb1db)"
time="2019-02-17T00:35:41.860546929+01:00" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
time="2019-02-17T00:35:41.873720773+01:00" level=debug msg="Releasing addresses for endpoint xenodochial_bartik's interface on network bridge"
time="2019-02-17T00:35:41.873774167+01:00" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
time="2019-02-17T00:35:41.873808407+01:00" level=debug msg="Released address PoolID:LocalDefault/172.17.0.0/16, Address:172.17.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"
time="2019-02-17T00:35:41.889671294+01:00" level=error msg="36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4 cleanup: failed to delete container from containerd: no such container"
time="2019-02-17T00:35:41.893339661+01:00" level=error msg="Handler for POST /v1.39/containers/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4/start returned error: ttrpc: client shutting down: read unix @->@/containerd-shim/moby/36da583168b1e115ae142686e744b8fd5c707803330ba58403a45feb00091bf4/shim.sock: read: connection reset by peer: unknown"


Both machines running Gentoo Linux with ext4 file system on KVM and the versions as shown in the initial post and it was working fine prior to the lasted update on both machines.
Back to top
View user's profile Send private message
holm74
n00b
n00b


Joined: 21 Nov 2007
Posts: 23

PostPosted: Mon Feb 18, 2019 11:10 am    Post subject: Reply with quote

Hi R71,

not really my problem but similar: have a look here: https://forums.gentoo.org/viewtopic-p-8310714.html#8310714

Do you have the same behaviour?


Thanks for any reply,

Holm
Back to top
View user's profile Send private message
r7l
n00b
n00b


Joined: 16 Feb 2019
Posts: 22

PostPosted: Mon Feb 18, 2019 12:41 pm    Post subject: Reply with quote

I was able to resolve the issue. I had an overlay that had dev-lang/go which was pulled in by Portage instead of the one from Portage itself. It wasn't an issue with the last version but the current version of Docker does not work with it. Using dev-lang/go from Portage does fix the issue.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum