Bug 1673412

OC sos reports and stack home are at :
http://rhos-release.virt.bos.redhat.com/log/pkomarov_sosreports/BZ1673412/

Correction:
Version-Release number of selected component (if applicable):
osp14 with ovs setup -> osp14 with OVN setup

Adding additional info : 
Network agent are reported in unhealthy state => docker healthchecks are failing => there is a listener for port 6642  , 
but the healthcheck executable itself is not found...
Adding DFG:DF as main , since this maybe be a kolla configuration issue...


(overcloud) [stack@undercloud-0 ~]$ ansible overcloud -b -mshell -a"docker ps|grep ovn_controller"
 [WARNING]: Found both group and host with same name: undercloud

controller-0 | SUCCESS | rc=0 >>
5a4fef8d0533        192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1            "kolla_start"            16 hours ago        Up 19 minutes (unhealthy)                       ovn_controller

compute-0 | SUCCESS | rc=0 >>
ccda5101a6f6        192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1               "kolla_start"       16 hours ago        Up 19 minutes (unhealthy)                       ovn_controller

compute-1 | SUCCESS | rc=0 >>
5eefa78edb5b        192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1               "kolla_start"       16 hours ago        Up 19 minutes (unhealthy)                       ovn_controller

controller-2 | SUCCESS | rc=0 >>
751944831ac9        192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1            "kolla_start"            16 hours ago        Up 19 minutes (unhealthy)                       ovn_controller

controller-1 | SUCCESS | rc=0 >>
bb7e14729af5        192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1            "kolla_start"            16 hours ago        Up 19 minutes (unhealthy)                       ovn_controller

(overcloud) [stack@undercloud-0 ~]$ ansible overcloud -b -mshell -a"docker inspect ovn_controller|grep healthcheck"
 [WARNING]: Found both group and host with same name: undercloud

compute-0 | SUCCESS | rc=0 >>
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                    "/openstack/healthcheck 6642"
                "config_data": "{\"start_order\": 1, \"healthcheck\": {\"test\": \"/openstack/healthcheck 6642\"}, \"image\": \"192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"user\": \"root\", \"volumes\": [\"/var/lib/kolla/config_files/ovn_controller.json:/var/lib/kolla/config_files/config.json:ro\", \"/lib/modules:/lib/modules:ro\", \"/run:/run\", \"/var/log/containers/openvswitch:/var/log/openvswitch\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}",

controller-0 | SUCCESS | rc=0 >>
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                    "/openstack/healthcheck 6642"
                "config_data": "{\"start_order\": 1, \"healthcheck\": {\"test\": \"/openstack/healthcheck 6642\"}, \"image\": \"192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"user\": \"root\", \"volumes\": [\"/var/lib/kolla/config_files/ovn_controller.json:/var/lib/kolla/config_files/config.json:ro\", \"/lib/modules:/lib/modules:ro\", \"/run:/run\", \"/var/log/containers/openvswitch:/var/log/openvswitch\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}",

compute-1 | SUCCESS | rc=0 >>
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                    "/openstack/healthcheck 6642"
                "config_data": "{\"start_order\": 1, \"healthcheck\": {\"test\": \"/openstack/healthcheck 6642\"}, \"image\": \"192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"user\": \"root\", \"volumes\": [\"/var/lib/kolla/config_files/ovn_controller.json:/var/lib/kolla/config_files/config.json:ro\", \"/lib/modules:/lib/modules:ro\", \"/run:/run\", \"/var/log/containers/openvswitch:/var/log/openvswitch\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}",

controller-1 | SUCCESS | rc=0 >>
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                    "/openstack/healthcheck 6642"
                "config_data": "{\"start_order\": 1, \"healthcheck\": {\"test\": \"/openstack/healthcheck 6642\"}, \"image\": \"192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"user\": \"root\", \"volumes\": [\"/var/lib/kolla/config_files/ovn_controller.json:/var/lib/kolla/config_files/config.json:ro\", \"/lib/modules:/lib/modules:ro\", \"/run:/run\", \"/var/log/containers/openvswitch:/var/log/openvswitch\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}",

controller-2 | SUCCESS | rc=0 >>
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                        "Output": "/bin/sh: /openstack/healthcheck: No such file or directory\n"
                    "/openstack/healthcheck 6642"
                "config_data": "{\"start_order\": 1, \"healthcheck\": {\"test\": \"/openstack/healthcheck 6642\"}, \"image\": \"192.168.24.1:8787/rhosp14/openstack-ovn-controller:2019-02-05.1\", \"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\"], \"user\": \"root\", \"volumes\": [\"/var/lib/kolla/config_files/ovn_controller.json:/var/lib/kolla/config_files/config.json:ro\", \"/lib/modules:/lib/modules:ro\", \"/run:/run\", \"/var/log/containers/openvswitch:/var/log/openvswitch\"], \"net\": \"host\", \"privileged\": true, \"restart\": \"always\"}",

(overcloud) [stack@undercloud-0 ~]$ ansible overcloud -b -mshell -a'docker exec `docker ps -f name=ovn_controller -q`  sh -c "grep -r 6642 /etc"'
 [WARNING]: Found both group and host with same name: undercloud

compute-0 | SUCCESS | rc=0 >>
/etc/selinux/targeted/active/ports.local:portcon tcp 6642 system_u:object_r:ovsdb_port_t:s0

controller-1 | SUCCESS | rc=0 >>
/etc/selinux/targeted/active/ports.local:portcon tcp 6642 system_u:object_r:ovsdb_port_t:s0

controller-2 | SUCCESS | rc=0 >>
/etc/selinux/targeted/active/ports.local:portcon tcp 6642 system_u:object_r:ovsdb_port_t:s0

controller-0 | SUCCESS | rc=0 >>
/etc/selinux/targeted/active/ports.local:portcon tcp 6642 system_u:object_r:ovsdb_port_t:s0

compute-1 | SUCCESS | rc=0 >>
/etc/selinux/targeted/active/ports.local:portcon tcp 6642 system_u:object_r:ovsdb_port_t:s0


(overcloud) [stack@undercloud-0 ~]$ ansible overcloud -b -mshell -a'docker exec `docker ps -f name=ovn_controller -q`  sh -c "ss -atlp|grep 6642"'
 [WARNING]: Found both group and host with same name: undercloud

compute-0 | FAILED | rc=1 >>
non-zero return code

compute-1 | FAILED | rc=1 >>
non-zero return code

controller-1 | SUCCESS | rc=0 >>
LISTEN     0      10     172.17.1.11:6642                     *:*                    

controller-0 | SUCCESS | rc=0 >>
LISTEN     0      10     172.17.1.11:6642                     *:*                    

controller-2 | SUCCESS | rc=0 >>
LISTEN     0      10     172.17.1.11:6642                     *:*

Looks like ovn-controller is not running/restarting on compute nodes as per "ss -atlp|grep 6642" rc=1 there
Need to check in sosreports how ovn-controller logs look like (/var/log/containers/openvswitch/ovn-controller.log*)

(In reply to Daniel Alvarez Sanchez from comment #4)
> Looks like ovn-controller is not running/restarting on compute nodes as per
> "ss -atlp|grep 6642" rc=1 there
> Need to check in sosreports how ovn-controller logs look like
> (/var/log/containers/openvswitch/ovn-controller.log*)

Yeah that would be good. Also, apparently this /openstack/healthcheck script should have been added to the image by TripleO [0][1]. I do not know why it's missing there.

[0] https://github.com/openstack/tripleo-common/blob/fe8dd5c9076ba7ada444da361b4e5533ace90435/container-images/tripleo_kolla_template_overrides.j2#L722-L726
[1] https://github.com/openstack/tripleo-common/blob/fe8dd5c9076ba7ada444da361b4e5533ace90435/healthcheck/ovn-controller

(In reply to Lucas Alvares Gomes from comment #5)
> (In reply to Daniel Alvarez Sanchez from comment #4)
> > Looks like ovn-controller is not running/restarting on compute nodes as per
> > "ss -atlp|grep 6642" rc=1 there
> > Need to check in sosreports how ovn-controller logs look like
> > (/var/log/containers/openvswitch/ovn-controller.log*)
> 
> Yeah that would be good. Also, apparently this /openstack/healthcheck script
> should have been added to the image by TripleO [0][1]. I do not know why
> it's missing there.
> 
> [0]
> https://github.com/openstack/tripleo-common/blob/
> fe8dd5c9076ba7ada444da361b4e5533ace90435/container-images/
> tripleo_kolla_template_overrides.j2#L722-L726
> [1]
> https://github.com/openstack/tripleo-common/blob/
> fe8dd5c9076ba7ada444da361b4e5533ace90435/healthcheck/ovn-controller

also please note that the added healthcheck script is not always missing , first it was there . then we restarted the container and it was gone.
If we do some more restarts the ovn_controller container may load that mount/script like it should , so I'm just reminding that this breaks "sometimes" :)

(In reply to pkomarov from comment #6)
> (In reply to Lucas Alvares Gomes from comment #5)
> > (In reply to Daniel Alvarez Sanchez from comment #4)
> > > Looks like ovn-controller is not running/restarting on compute nodes as per
> > > "ss -atlp|grep 6642" rc=1 there
> > > Need to check in sosreports how ovn-controller logs look like
> > > (/var/log/containers/openvswitch/ovn-controller.log*)
> > 
> > Yeah that would be good. Also, apparently this /openstack/healthcheck script
> > should have been added to the image by TripleO [0][1]. I do not know why
> > it's missing there.
> > 
> > [0]
> > https://github.com/openstack/tripleo-common/blob/
> > fe8dd5c9076ba7ada444da361b4e5533ace90435/container-images/
> > tripleo_kolla_template_overrides.j2#L722-L726
> > [1]
> > https://github.com/openstack/tripleo-common/blob/
> > fe8dd5c9076ba7ada444da361b4e5533ace90435/healthcheck/ovn-controller
> 
> also please note that the added healthcheck script is not always missing ,
> first it was there . then we restarted the container and it was gone.
> If we do some more restarts the ovn_controller container may load that
> mount/script like it should , so I'm just reminding that this breaks
> "sometimes" :)

Interesting, thanks for that pointer.

Btw, I'm changing the component of this bug to python-tripleo-common because that's were the healthcheck script is injected into the image.

This has been fixed by https://review.opendev.org/#/c/568265/5, checked on OSP16 we have the healthchecks in ovn_controller. I'm closing this BZ but feel free to reopen in case there is still an issue.