Bug 1788618
| Summary: | Overcloud Container Health Check Fails | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Rosenfeld <drosenfe> |
| Component: | openstack-tripleo-common | Assignee: | Adriano Petrich <apetrich> |
| Status: | CLOSED ERRATA | QA Contact: | David Rosenfeld <drosenfe> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 16.0 (Train) | CC: | aschultz, kecarter, mburns, michele, slinaber |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-06-03 20:43:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1779249 | ||
| Bug Blocks: | |||
|
Description
David Rosenfeld
2020-01-07 15:54:19 UTC
Logs to a failing test: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/deployment/job/DFG-df-deployment-16-virthost-3cont_1comp_3ceph_3db_2net_3msg-yes_UC_SSL-yes_OC_SSL-ceph-ipv6-geneve-RHELOSP-31893/8/ Look at pytest_runner / pytest-repo.tripleo.container_sanity_v3.Overcloud_Testing.test_check_helthcheck_status_overcloud failure We've addressed the clustercheck, swift and horizon health checks upstream. I'm uncertain about the ceph/rabbit/galera health checks. I think these might need to be excluded from the test itself. (In reply to Alex Schultz from comment #2) > We've addressed the clustercheck, swift and horizon health checks upstream. > I'm uncertain about the ceph/rabbit/galera health checks. I think these > might need to be excluded from the test itself. +1 any container that has '-bundle' in its name needs to be excluded since it is being helthchecked by pacemaker with its own mechanism After podman overcloud healthcheck script was updated to not report status on containers with format: -bundle-, ceph-, and swift_ the overcloud healthcheck still fails. The script reports: Check healtcheck services for podman containers on Overcloud NODE: 192.168.24.32: ● tripleo_horizon_healthcheck.service - horizon healthcheck Loaded: loaded (/etc/systemd/system/tripleo_horizon_healthcheck.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2020-01-31 13:33:01 UTC; 17s ago Process: 560167 ExecStart=/usr/bin/podman exec horizon /openstack/healthcheck (code=exited, status=127) Main PID: 560167 (code=exited, status=127) Jan 31 13:33:00 controller-0 systemd[1]: Starting horizon healthcheck... Jan 31 13:33:01 controller-0 podman[560167]: Error: exec failed: container_linux.go:346: starting container process caused "exec: \"/openstack/healthcheck\": stat /openstack/healthcheck: no such file or directory": OCI runtime command not found error Jan 31 13:33:01 controller-0 systemd[1]: tripleo_horizon_healthcheck.service: Main process exited, code=exited, status=127/n/a Jan 31 13:33:01 controller-0 systemd[1]: tripleo_horizon_healthcheck.service: Failed with result 'exit-code'. Jan 31 13:33:01 controller-0 systemd[1]: Failed to start horizon healthcheck. 192.168.24.32 is controller-0. Other nodes pass(i.e.: compute-0, ceph-0, ceph-1, and ceph-2.) This is why script fails for controller node: [heat-admin@controller-0 ~]$ sudo systemctl list-units --failed --plain --no-legend dhcp-interface loaded failed failed DHCP interface ovs-system tripleo_horizon_healthcheck.service loaded failed failed horizon healthcheck <----- doesn't show failed on other nodes After second podman script update to not perform the health check in Comment 4 still seeing a failed health check for: openstack-cinder-volume-podman-0. This only happens in two tests: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/deployment/job/DFG-df-deployment-16-virthost-3cont_1comp_3ceph-yes_UC_SSL-no_OC_SSL-scalecompute-ceph-ipv4-geneve-RHELOSP-31842/ and https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/df/view/deployment/job/DFG-df-deployment-16-virthost-3cont_2comp_3ceph-yes_UC_SSL-yes_OC_SSL-ceph-ipv4-geneve-conf_changes-RHELOSP-31868/ *** Bug 1793881 has been marked as a duplicate of this bug. *** |