Description of problem: opendaylight_api container is unhealthy on a fresh deployment. 25532225dc1b 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-07-13.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) opendaylight_api Version-Release number of selected component (if applicable): OSP 13 Pudddle: 2018-07-13.1 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP13 with ODL 2. 3. Actual results: opendaylight_api container is unhealthy Expected results: Container should b e healthy Additional info:
I do see this occasionally. Restarting it helps.
I think restarting helps too. I did not manually restart but due to load the containers on each controller got restarted at some point and then they showed status as healthy.
@Sai, please post the output of "docker inspect opendaylight_api" showing the healthcheck "Log" section like below: $ sudo docker inspect opendaylight_api [ { "Id": "49b7939c04180a5dc5ec0634d9fd2f51f6101cc82a0f4c248ad56760a7860c4d", ..... "State": { "Status": "running", ...... "Health": { "Status": "unhealthy", "FailingStreak": 7428, "Log": [ { "Start": "2018-08-06T05:43:25.167736014Z", "End": "2018-08-06T05:43:25.424213779Z", "ExitCode": 1, "Output": "\n000 :0 0.000 seconds\ncurl: (3) IPv6 numerical address used in URL without brackets\n" }, <snippet>
Sai, did you get a chance to collect above mentioned information from the setup? Meanwhile, can you check with latest puddle. Healthcheck recently moved to using Diagstatus to get the ODL status and this issue should not be seen with the recent changes.
I ran a fresh deployment based on the latest puddle and ODL container is healthy. 28100a419265 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-02.1 "kolla_start" 3 days ago Up 3 days (healthy) opendaylight_api Healthchecks based on diagstatus (merged some time back) are known to be OK and should resolve this issue of unhealthy containers. This commit https://github.com/openstack/tripleo-common/commit/bb98f5aa7db3c4b05e63af48e593841f2a71913a should solve the issue.
According to our records, this should be resolved by openstack-tripleo-common-8.6.3-13.el7ost. This build is available now.
On puddle 2018-10-18.1 (with rpm openstack-tripleo-common-8.6.3-13.el7ost), I'm seeing this unhealthy state of openstack-neutron-server-opendaylight: [root@titan13 ~]# ssh stack@undercloud-0 [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 13 -p 2018-10-18.1 (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-0 "sudo docker ps | grep opendaylight" ebe8c4165108 192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-10-18.1 "kolla_start" 12 days ago Up 11 days (healthy) neutron_api dd7632c20c8c 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-18.1 "kolla_start" 12 days ago Up 11 days (unhealthy) opendaylight_api (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-1 "sudo docker ps | grep opendaylight" 7e6e0a864744 192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-10-18.1 "kolla_start" 12 days ago Up 11 days (healthy) neutron_api 44579ff7cf35 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-18.1 "kolla_start" 12 days ago Up 11 days (unhealthy) opendaylight_api (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-2 "sudo docker ps | grep opendaylight" 41804c0acae4 192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-10-18.1 "kolla_start" 12 days ago Up 11 days (healthy) neutron_api 696921a7060b 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-18.1 "kolla_start" 12 days ago Up 11 days (unhealthy) opendaylight_api
On OSP version: 13 2018-11-05.3 +--------------------------------------+-----------------+--------+-------------------------------------------------------------+--------+-------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------+--------+-------------------------------------------------------------+--------+-------------+ | 9e741ac8-84ce-46b3-8005-3d507dde68c4 | rhel75_vm1_net2 | ACTIVE | net_ipv64_2=2002::f816:3eff:fea6:a0b5, 10.0.2.5, 10.0.0.215 | rhel75 | rhel_flavor | | ac5e5d20-b13a-45f7-bfba-1082be31b47d | rhel75_vm1_net1 | ACTIVE | net_ipv64_1=10.0.1.5, 2001::f816:3eff:fe3e:a2d8, 10.0.0.233 | rhel75 | rhel_flavor | +--------------------------------------+-----------------+--------+-------------------------------------------------------------+--------+-------------+ ---------------------------------------------------------------------------------------------------- Creating and testing multiple VMs and Networks completed. Please verify output contains no failures. To SSH into VM: ssh -i tester_key.pem cloud-user@SERVER_FIP [stack@undercloud-0 ~]$ ssh heat-admin@controller-0 "docker ps | grep opendaylight" Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.26/containers/json: dial unix /var/run/docker.sock: connect: permission denied [stack@undercloud-0 ~]$ ssh heat-admin@controller-0 "sudo docker ps | grep opendaylight" b9c5aa578f6f 192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-11-05.3 "kolla_start" 14 hours ago Up 14 hours (healthy) neutron_api 21b3de2359d9 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-11-05.3 "kolla_start" 14 hours ago Up 14 hours (healthy) opendaylight_api [stack@undercloud-0 ~]$ ssh heat-admin@controller-1 "sudo docker ps | grep opendaylight" f68bed578be5 192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-11-05.3 "kolla_start" 14 hours ago Up 14 hours (healthy) neutron_api 645d872e77dc 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-11-05.3 "kolla_start" 14 hours ago Up 14 hours (healthy) opendaylight_api [stack@undercloud-0 ~]$ ssh heat-admin@controller-2 "sudo docker ps | grep opendaylight" 3129cef6b9dc 192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-11-05.3 "kolla_start" 14 hours ago Up 14 hours (healthy) neutron_api 0556276cb3e8 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-11-05.3 "kolla_start" 14 hours ago Up 14 hours (healthy) opendaylight_api
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3587