Bug 1266533 - Containers not starting/have no logs
Containers not starting/have no logs
Status: CLOSED WORKSFORME
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
3.x
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Andy Goldstein
chaoyang
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-25 10:20 EDT by Dean Peterson
Modified: 2015-10-21 11:22 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-21 11:22:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dean Peterson 2015-09-25 10:20:23 EDT
Description of problem:
I can take the exact same image created by Openshift and manually run it outside openshift with the same CMD and I can see logs using the "docker logs..." command. However, the container created by Openshift and Kubernetes does not print anything with the "docker logs..." command.

For example, when I run docker ps -a, I can see: 
a77de0586930 172.30.239.80:5000/test/tradeservice@sha256:29246b264315d43632bdd119cca7a283b7c7b716f1b3b1de284a41e177772223 "/bin/sh -c '/opt/jb About an hour ago k8s_wildfly-tradeservice.79fba1ed_tradeservice-2-s8iu1_test_0d7703c7-62e6-11e5-938b-0023ae6f50aa_0dfe605f

When I run docker logs k8s_wildfly-tradeservice.79fba1ed_tradeservice-2-s8iu1_test_0d7703c7-62e6-11e5-938b-0023ae6f50aa_0dfe605f 
or docker logs a77de0586930, absolutely nothing is returned. It just moves to a new line on the terminal.

When I manually run a container with the same image: docker run 172.30.239.80:5000/test/tradeservice@sha256:29246b264315d43632bdd119cca7a283b7c7b716f1b3b1de284a41e177772223 /bin/sh -c '/opt/jboss/wildfly/bin/standalone.sh -b 0.0.0.0 -bmanagement=0.0.0.0 --properties=/opt/jboss/wildfly/tradeapp.properties'

Then, I call docker logs for that container; I see the wildfly server starting as I would expect. Somehow, the container being started by Kubernetes is different even though it looks like it is calling /bin/sh -c '/opt/jboss/wildfly/bin/standalone.sh -b 0.0.0.0 -bmanagement=0.0.0.0 --properties=/opt/jboss/wildfly/tradeapp.properties' by looking at what is printed out on docker ps -a.

There is an associated error in the event logs:

failed
Failed to start with docker id 99a811606b51 with error: API error (500): Cannot start container 99a811606b51cdbeddbea14af474f0df432278ac9f73baea5d8ecaf5453f521e: cannot join network of a non running container: c8a4a648e63d5b18dcd8fad6fd2d70f466584f2a749bb4245bb86a6da1ceea55

I am only trying to start one container, why is it complaining about not being able to start one container because another is not running?

Version-Release number of selected component (if applicable):
Origin Binary 1.0.6 running on RHEL 7.1 with Docker 1.7.1

How reproducible:
I have reproduced this on multiple machines.

Steps to Reproduce:
1. Create Application using a Docker strategy template (my template excluding source urls - https://gist.github.com/deanpeterson/4c00d5665dfc2851d98a) 
2. Start Deployment
3. Pods never start because the containers never begin to run

Actual results:


Expected results:


Additional info:
Comment 1 Dan Mace 2015-09-25 10:43:08 EDT
Moving this to the Node component since it's related to Kube/Docker, not the deployment system.
Comment 2 Andy Goldstein 2015-09-25 12:02:40 EDT
When you run a pod, it always creates 1 container to create a network namespace that all the other "real" containers share. So a pod with 1 container (e.g. wildfly) will actually have 2.

Can you run 'docker inspect c8a4a648e63d5b18dcd8fad6fd2d70f466584f2a749bb4245bb86a6da1ceea55' and provide the output?

Also, how are you starting OpenShift? What is your exact command line?
Comment 3 Dean Peterson 2015-09-25 12:51:31 EDT
docker inspect - https://gist.github.com/deanpeterson/1e8b9a71c195d633e3e2


I downloaded the 1.0.6 binary and ran "./openshift start"
Comment 4 Andy Goldstein 2015-09-25 12:54:02 EDT
Did you run 'openshift start' as root?

Can you 'oc describe dc/<your deployment config name>' (if you have one) and 'oc describe pod/<your pod name>'?

Can you also share 'oc get events'?
Comment 5 Dean Peterson 2015-09-25 14:02:08 EDT
Yes, I run 'openshift start' as root.

./oc describe dc/tradeservice - https://gist.github.com/deanpeterson/0aac795d98ae6e9e23fd

./oc describe pod - https://gist.github.com/deanpeterson/74b43ebcf06004be1ebc

./oc get events - https://gist.github.com/deanpeterson/46cdbb4007c93b92b4a9
Comment 6 Dean Peterson 2015-09-25 15:06:10 EDT
(In reply to Andy Goldstein from comment #4)
> Did you run 'openshift start' as root?
> 
> Can you 'oc describe dc/<your deployment config name>' (if you have one) and
> 'oc describe pod/<your pod name>'?
> 
> Can you also share 'oc get events'?

Shoot, I hope I didn't miss my window.  Of course I had to be out of the office today running errands; that is why I did not respond right away.  I am desperate to get past this though.
Comment 7 Dean Peterson 2015-09-25 21:58:44 EDT
Any updates on this?
Comment 8 Dean Peterson 2015-09-26 01:30:24 EDT
I figured out the problem.  I vaguely remembered seeing this in the documentation and didn't think much of it at the time: https://docs.openshift.com/enterprise/3.0/admin_guide/manage_scc.html#enable-images-to-run-with-user-in-the-dockerfile

I have the line "USER root" in my Dockerfile.  Apparently that was preventing the container from starting.  After I followed the instructions to modify my scc to RunAsAny, the deployment was successful.

Granted, I should have read the documentation better, but an error message telling me my containers do not have the privilege to run would have helped immensely.
Comment 9 Michal Fojtik 2015-09-29 15:27:23 EDT
I'm re-opening this bug as I just got the same error but with the 'router' Pod.

This is what I can see in openshift log:

 E0929 19:04:50.685548    2462 manager.go:1783] Error running pod "router-1-g03tu_default" container "router": API error (500): Cannot start container 
                 a01a37cd249e628c81bdbf06d5837cae97a45cc8c9e95dfad3ae3478cce881d8: cannot join network of a non 
                 running container: 2884e47b31250b12dd8788fd2b8de3807ab01cd4ceed7baf9503d71284c98c9b


[root@openshiftdev openshift]# docker inspect a01a37cd249e628c81bdbf06d5837cae97a45cc8c9e95dfad3ae3478cce881d8
[
{
    "Id": "a01a37cd249e628c81bdbf06d5837cae97a45cc8c9e95dfad3ae3478cce881d8",
    "Created": "2015-09-29T19:04:49.733869072Z",
    "Path": "/usr/bin/openshift-router",
    "Args": [],
    "State": {
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 128,
        "Error": "cannot join network of a non running container: 2884e47b31250b12dd8788fd2b8de3807ab01cd4ceed7baf9503d71284c98c9b",
        "StartedAt": "0001-01-01T00:00:00Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "91065df9c3cd978f2f5363e4211035dd5de737991a79d8850c69604e32246b49",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "NetworkID": "",
        "PortMapping": null,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "ResolvConfPath": "",
    "HostnamePath": "",
    "HostsPath": "",
    "LogPath": "",
    "Name": "/k8s_router.49c21c4d_router-1-g03tu_default_dbcf8ef2-66dc-11e5-ac00-525400dec257_507faeb7",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c16,c612",
    "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c16,c612",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": [
            "/var/lib/openshift/volumes/pods/dbcf8ef2-66dc-11e5-ac00-525400dec257/volumes/kubernetes.io~secret/router-token-d7hn1:/var/run/secrets/kubernetes.io/serviceaccount:ro",
            "/var/lib/openshift/volumes/pods/dbcf8ef2-66dc-11e5-ac00-525400dec257/containers/router/a01a37cd249e628c81bdbf06d5837cae97a45cc8c9e95dfad3ae3478cce881d8:/dev/termination-log"
        ],
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemorySwap": -1,
        "CpuShares": 2,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "MemorySwappiness": null,
        "Privileged": false,
        "PortBindings": {
            "1936/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "1936"
                }
            ],
            "443/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "443"
                }
            ],
            "80/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "80"
                }
            ]
        },
        "Links": null,
        "PublishAllPorts": false,
        "Dns": [
            "192.168.122.151",
            "192.168.122.1"
        ],
        "DnsSearch": [
            "default.svc.cluster.local",
            "svc.cluster.local",
            "cluster.local",
            "local"
        ],
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": null,
        "NetworkMode": "container:2884e47b31250b12dd8788fd2b8de3807ab01cd4ceed7baf9503d71284c98c9b",
        "IpcMode": "container:2884e47b31250b12dd8788fd2b8de3807ab01cd4ceed7baf9503d71284c98c9b",
        "PidMode": "",
        "UTSMode": "host",
        "CapAdd": null,
        "CapDrop": null,
        "GroupAdd": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": "",
        "ConsoleSize": [
            0,
            0
        ]
    },
    "GraphDriver": {
        "Name": "devicemapper",
        "Data": {
            "DeviceId": "1380",
            "DeviceName": "docker-253:1-1439172-a01a37cd249e628c81bdbf06d5837cae97a45cc8c9e95dfad3ae3478cce881d8",
            "DeviceSize": "10737418240"
        }
    },
    "Mounts": [
        {
            "Source": "/var/lib/openshift/volumes/pods/dbcf8ef2-66dc-11e5-ac00-525400dec257/containers/router/a01a37cd249e628c81bdbf06d5837cae97a45cc8c9e95dfad3ae3478cce881d8",
            "Destination": "/dev/termination-log",
            "Mode": "",
            "RW": true
        },
        {
            "Source": "/var/lib/openshift/volumes/pods/dbcf8ef2-66dc-11e5-ac00-525400dec257/volumes/kubernetes.io~secret/router-token-d7hn1",
            "Destination": "/var/run/secrets/kubernetes.io/serviceaccount",
            "Mode": "ro",
            "RW": false
        }
    ],
    "Config": {
        "Hostname": "router-1-g03tu",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "ExposedPorts": {
            "1936/tcp": {},
            "443/tcp": {},
            "80/tcp": {}
        },
        "PublishService": "",
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "DEFAULT_CERTIFICATE=",
            "OPENSHIFT_CA_DATA=-----BEGIN CERTIFICATE-----\nMIIC5jCCAdCgAwIBAgIBATALBgkqhkiG9w0BAQswJjEkMCIGA1UEAwwbb3BlbnNo\naWZ0LXNpZ25lckAxNDQzNTUzNDEwMB4XDTE1MDkyOTE5MDMzMFoXDTE2MDkyODE5\nMDMzMVowJjEkMCIGA1UEAwwbb3BlbnNoaWZ0LXNpZ25lckAxNDQzNTUzNDEwMIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6SAvVTewJgXQxT9IucGVTBtc\n6yfFmgFn0DPO97Q59aijSIMquiWMRR35PyKJI5mcTWCRj7lrI1xPXkDZfLbTCgzm\nFcZqqEs+/Uhto+qY10nPFGMyC0UORKobjJGQV/U+q3APHGP7wQuqwIOl7O1KG5VF\nk46xGF6v7//5rmaD0ZSROIz/WTAYtEqz+4BzN7cxxGa9rhYc6ZNCqOt11qn+C2QW\nKmFmKxEksj6jNWr06f9wDN3l9UCiyeyy/IJ7po8AI42zeG+PPrjFeMRcAKONZAFl\nme8C+zVmmhfKAP+0SLzpuCJNePY59IGGLQNVy2MzAhWVmFjAx1kp7Ha0dy+EHQID\nAQABoyMwITAOBgNVHQ8BAf8EBAMCAKQwDwYDVR0TAQH/BAUwAwEB/zALBgkqhkiG\n9w0BAQsDggEBAD/ctPgUBIPjSZambQn1SdZGYktO7gyxq4n/GB+PnS3dFGsuhcPz\nVFWkHZw0/xjLPtVMZmRhtvmrscOFv7v4dzjXtyEkuh0eRxKiWdRdA0rxMtau4igp\n2nRwpGMvCKgjvaU4iOKlJ57wislk8KlT2f84gkq6xsAGNsMlsTaxsI/bb7fNLYw9\ntNjjrS6yTGxAg1dG4qNsLr3J0DwiCZ6qkj/N0c7rspRQcem9K9PvLO1f8Zl/qxEu\niY+WePHiJyKPk9mntoCu+4gliyE8LkUWva5MUM7zpcSYOe4wmiUJkuhKJKv68Rr8\ni/Ul5MKelwJ2F/Foh9ho4p8bKVHTAVFe1mU=\n-----END CERTIFICATE-----\n",
            "OPENSHIFT_CERT_DATA=-----BEGIN CERTIFICATE-----\nMIIDDTCCAfegAwIBAgIBCDALBgkqhkiG9w0BAQswJjEkMCIGA1UEAwwbb3BlbnNo\naWZ0LXNpZ25lckAxNDQzNTUzNDEwMB4XDTE1MDkyOTE5MDMzNloXDTE2MDkyODE5\nMDMzN1owOzEXMBUGA1UEChMOc3lzdGVtOnJvdXRlcnMxIDAeBgNVBAMTF3N5c3Rl\nbTpvcGVuc2hpZnQtcm91dGVyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC\nAQEAqMmQZIVLdovtLaYNOxeYlFO94xXuKOwi2/f/R55kbPO8jEMSVFDZ7OyJSf6r\nnRf5p5QLAUj01OZH9CwMVEDcNChalS0DO8Ap+gt1XP6P+WwD09jk41Ut/jMMgBa/\ndYVUZ1A7FzJ6aQgNTJa00bABERQ5CZ5xGUPvkR2c7lZiWkRI5Uw1wThYHOJpXqNw\ni+RYa2ArIWXR6+k1mXK9j+zPIcGPqmsjXhn0kJMmuvIFawSfmWfea2FhAuAJcV0G\nbdYiTveZKS1GWsNQRo6YCjfXT6MEloe+WTeA40mwK9bnK9uUldU0WsWwJ/afC3NP\n57LhT/KhQpvWXeL/hR3ZJtsQgwIDAQABozUwMzAOBgNVHQ8BAf8EBAMCAKAwEwYD\nVR0lBAwwCgYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADALBgkqhkiG9w0BAQsDggEB\nAL0F3tryUp/wuS7/3JV6r3VoC4zGIIgii+GXZ+HhkRGRgXKyydxofR/WCK3Qgt68\nevR6Gr1AXulwOOoVWuOqL5lAyoy9XcYgLhgu9tVqg0PoyaylUFBjxIPfHzC2hrN6\nBlnqpm6LrQVCkIBO3qxclqJm36uB2hcxqZZiE1siS7Uy8a5UsNcYD5k8IgHGEpQg\nY46k8M6tYRB6CpWblKroJuOhuvfT01mhhRM8/3L/xyWwj1twHKBwjUgyxqhmk9sa\nbPlr+0cBnH6VyZ0IKodWtAd43PRjTykoCbX2xfAfUpwcyqD2kbAfcZUepRnqQw0Q\ndAS2NabMjp9mOt6DuhU03J0=\n-----END CERTIFICATE-----\n",
            "OPENSHIFT_INSECURE=false",
            "OPENSHIFT_KEY_DATA=-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAqMmQZIVLdovtLaYNOxeYlFO94xXuKOwi2/f/R55kbPO8jEMS\nVFDZ7OyJSf6rnRf5p5QLAUj01OZH9CwMVEDcNChalS0DO8Ap+gt1XP6P+WwD09jk\n41Ut/jMMgBa/dYVUZ1A7FzJ6aQgNTJa00bABERQ5CZ5xGUPvkR2c7lZiWkRI5Uw1\nwThYHOJpXqNwi+RYa2ArIWXR6+k1mXK9j+zPIcGPqmsjXhn0kJMmuvIFawSfmWfe\na2FhAuAJcV0GbdYiTveZKS1GWsNQRo6YCjfXT6MEloe+WTeA40mwK9bnK9uUldU0\nWsWwJ/afC3NP57LhT/KhQpvWXeL/hR3ZJtsQgwIDAQABAoIBAGLBdItr9ihJXJ9t\n3aXqe2qlyJ96cFtfNlIvuxxjT2zz+yiyAaJ1n9xg1T8HE9AZfAgRtZKt5ho/FnCF\n6TzOqkiWJ9HKgW7PU4AqfiPICeMs0Nx9tgZRC1UE91ZmeOVA+o5sQfajoT2c6XMb\nZnadnTIp++ll29SljCEDC5Pj1S3sfGZsxCNhTWP6+E+ZBmniNHPFJxjCwYHDr+xq\nNtewbyoFB4079PFMDLAeGEcvxP4Re8wLQNeu6E1/BxUdqkXqQLOGEq+dGPY26Ff6\nNuZJ9IWSvujkiDXKMkVweRF3FfCJ5idwBFIm07LVy9Ou3A/RbyUJS/QGaXNlMDUT\nBlXfAzkCgYEAw1cJxvintdGUoU22G6R5tpthbhxXc4apC84At7eVv7vkiYj/AW55\n6boHKMnIdZZnb15WksWN/fNwk3Iif60KTMz4uYnPHoH9H+Ch13Fb9vhFxQXK1k01\nDbGxhuDQ+fZAOH93SyBA01UCqLLwIfYB/uIzY4lyFttVDmpmNpUFSTUCgYEA3TOr\nLB7OOUS9Gw316RjKL8vHQMu1U3n1gUyi3UmiQx2ZPINrW2Bp1o/kzZ4cGD2FZuKv\n8z3WpSHaV+7XFNN5Cblc8mXvAzt01eHyBUx/RYk4dYS2CTE2pqgSi+oriatZlLTK\n6KWJt5s3bYbOt02Q30mouOXHIxLpCPGESscn4dcCgYAtDs2djMu6n85F6tP/yk+c\n9Hhj2ru9VSl3JJx7xCgRoviTVPtlTCNo61UtbmYD4Z6YTM1xa4pWMqocCK62fqaq\nknko6wXQhrFpgfoy5N4OuLP3P1/LUOHU99D4SLFLhybHIXfV52mslaZYXXYQj1tk\nfK7X3eNoLE2RVq+fFX6zpQKBgQCyc4eJPYTzUJAa7mVP2o7qhkRF8tFi+cXkvPW6\n/jdoW82Osxr+Ccb4sPdOJlrV93PSr9hwrJeoeqoSPH3JsF06Sg9BApkTC3lTP0en\nnLIkX/JrZFgTDF/hjUABHjZnhxPq0vvLOhjjRdYuhw3ndS0F5f+2wrCnjA7yBx3J\n0edL2wKBgDVCP/9g2xdwWuN++CHftxcpfGsvKjhDOjyHRuPndF5j7yA/mw1R7ocJ\nTKoPv0f+BW65iUWpCcWtp8n+XrQK9D8YaDw6CLuio1/2xRwDGjeWiBOv17eQ0XHW\n4/+A2MQ2dwJYXqahb42ocaLVDwOKLzkc7kLamaihdmD/FqfxttW+\n-----END RSA PRIVATE KEY-----\n",
            "OPENSHIFT_MASTER=https://192.168.122.151:8443",
            "ROUTER_EXTERNAL_HOST_HOSTNAME=",
            "ROUTER_EXTERNAL_HOST_HTTPS_VSERVER=",
            "ROUTER_EXTERNAL_HOST_HTTP_VSERVER=",
            "ROUTER_EXTERNAL_HOST_INSECURE=false",
            "ROUTER_EXTERNAL_HOST_PASSWORD=",
            "ROUTER_EXTERNAL_HOST_PRIVKEY=/etc/secret-volume/router.pem",
            "ROUTER_EXTERNAL_HOST_USERNAME=",
            "ROUTER_SERVICE_NAME=router",
            "ROUTER_SERVICE_NAMESPACE=default",
            "STATS_PASSWORD=Lq1mF9maBu",
            "STATS_PORT=1936",
            "STATS_USERNAME=admin",
            "KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1",
            "ROUTER_PORT_80_TCP_PROTO=tcp",
            "DOCKER_REGISTRY_SERVICE_PORT_5000_TCP=5000",
            "DOCKER_REGISTRY_PORT_5000_TCP_PORT=5000",
            "KUBERNETES_SERVICE_HOST=172.30.0.1",
            "KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443",
            "KUBERNETES_PORT_443_TCP_PROTO=tcp",
            "DOCKER_REGISTRY_SERVICE_PORT=5000",
            "KUBERNETES_PORT=tcp://172.30.0.1:443",
            "ROUTER_PORT_80_TCP_ADDR=172.30.202.72",
            "ROUTER_SERVICE_PORT_80_TCP=80",
            "ROUTER_PORT_80_TCP=tcp://172.30.202.72:80",
            "DOCKER_REGISTRY_SERVICE_HOST=172.30.125.175",
            "DOCKER_REGISTRY_PORT_5000_TCP=tcp://172.30.125.175:5000",
            "DOCKER_REGISTRY_PORT_5000_TCP_ADDR=172.30.125.175",
            "KUBERNETES_SERVICE_PORT=443",
            "ROUTER_SERVICE_HOST=172.30.202.72",
            "ROUTER_PORT_80_TCP_PORT=80",
            "DOCKER_REGISTRY_PORT=tcp://172.30.125.175:5000",
            "DOCKER_REGISTRY_PORT_5000_TCP_PROTO=tcp",
            "KUBERNETES_PORT_443_TCP_PORT=443",
            "ROUTER_SERVICE_PORT=80",
            "ROUTER_PORT=tcp://172.30.202.72:80",
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "TEMPLATE_FILE=/var/lib/haproxy/conf/haproxy-config.template",
            "RELOAD_SCRIPT=/var/lib/haproxy/reload-haproxy"
        ],
        "Cmd": null,
        "Image": "openshift/origin-haproxy-router:latest",
        "Volumes": null,
        "VolumeDriver": "",
        "WorkingDir": "/var/lib/haproxy/conf",
        "Entrypoint": [
            "/usr/bin/openshift-router"
        ],
        "NetworkDisabled": false,
        "MacAddress": "",
        "OnBuild": null,
        "Labels": {
            "io.kubernetes.pod.name": "default/router-1-g03tu",
            "io.kubernetes.pod.terminationGracePeriod": "30"
        }
    }
}
]

There are no logs/errors regarding this in Docker.
Comment 10 Michal Fojtik 2015-09-29 15:32:21 EDT
Also I see another container that died same way:

8f9b4a39d765        openshift/origin-deployer:latest                                                                                "/usr/bin/openshift-d"   24 minutes ago      Created                                           k8s_deployment.c598a79_docker-registry-1-deploy_default_d205c218-66dc-11e5-ac00-525400dec257_33b5e516

But this container gets re-created, so the deployment succeeded after all. Dunno why the router was not redeployed automatically...
Comment 11 Andy Goldstein 2015-09-29 16:34:05 EDT
Michal, can you 'docker inspect', 'docker logs' the network container? i.e. from

Cannot start container 
                 a01a37cd249e628c81bdbf06d5837cae97a45cc8c9e95dfad3ae3478cce881d8: cannot join network of a non 
                 running container: 2884e47b31250b12dd8788fd2b8de3807ab01cd4ceed7baf9503d71284c98c9b

we need info about 2884e47b31250b12dd8788fd2b8de3807ab01cd4ceed7baf9503d71284c98c9b.
Comment 12 Michal Fojtik 2015-09-30 03:43:42 EDT
Andy: That container does not exists :(
Comment 13 Andy Goldstein 2015-10-02 13:20:16 EDT
I can't debug this until it happens again and the system in question stops being used so we can begin to figure out what's going on.
Comment 14 Michal Fojtik 2015-10-21 11:21:01 EDT
Andy, I think this is now fix or at least i'm not experiencing this anymore.
Comment 15 Andy Goldstein 2015-10-21 11:22:16 EDT
Closing due to inability to reproduce. Please reopen if it happens again.

Note You need to log in before you can comment on or make changes to this bug.