Bug 1717144
| Summary: | Multiple containers reporting "unhealthy" after getting deployed | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | mlammon |
| Component: | openstack-containers | Assignee: | Emilien Macchi <emacchi> |
| Status: | CLOSED ERRATA | QA Contact: | Marius Cornea <mcornea> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 14.0 (Rocky) | CC: | bbonguar, bfournie, emacchi, jjoyce, lnatapov, m.andre, sasha, tvignaud |
| Target Milestone: | z3 | Keywords: | Regression, Triaged, ZStream |
| Target Release: | 14.0 (Rocky) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | python-paunch-3.2.0-0.20180921003261.6d2ec11.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-02 20:09:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1644883, 1651268, 1652523, 1661575, 1672782, 1672785, 1672786, 1672790, 1697273, 1716801 | ||
Looks like mistral errors in container:
ERROR:__main__:Failed to change ownership of /var/lib/mistral/undercloud.conf to 42430:42430
Traceback (most recent call last):
File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
os.chown(path, uid, gid)
OSError: [Errno 30] Read-only file system: '/var/lib/mistral/undercloud.conf'
++ cat /run_command
+ CMD='/usr/bin/mistral-server --config-file=/etc/mistral/mistral.conf --log-file=/var/log/mistral/executor.log --server=executor'
And lots of errors in mistral/mistral-db-manage.log
2019-06-04 13:55:04.041 12 ERROR mistral.actions.openstack.action_generator.base AttributeError: 'Client' object has no attribute 'certs'
2019-06-04 13:55:04.041 12 ERROR mistral.actions.openstack.action_generator.base
2019-06-04 13:55:04.042 12 ERROR mistral.actions.openstack.action_generator.base [-] Failed to create action: nova.certs_create: AttributeError: 'Client' object has no attribute 'certs'
2019-06-04 13:55:04.521 12 ERROR mistral.actions.openstack.action_generator.base NotImplementedError: To use 'consumers' oauthlib must be installed
2019-06-04 13:55:04.521 12 ERROR mistral.actions.openstack.action_generator.base
2019-06-04 13:55:04.523 12 ERROR mistral.actions.openstack.action_generator.base [-] Failed to create action: keystone.oauth1.request_tokens_list: NotImplementedError: To use 'request_tokens' oauthlib must be installed
2019-06-04 13:55:12.379 12 ERROR mistral.actions.openstack.action_generator.base [-] Failed to create action: murano.services_post: AttributeError: 'NoneType' object has no attribute 'Client'
The problem is when a healthcheck is run with an argument (e.g. port):
{
"Start": "2019-06-12T17:29:22.810768802-04:00",
"End": "2019-06-12T17:29:22.922346953-04:00",
"ExitCode": 127,
"Output": "/bin/sh: /openstack/healthcheck 5672: No such file or directory\n"
}
All other healthchecks running without argument work fine.
*** Bug 1721102 has been marked as a duplicate of this bug. *** Environment: python-paunch-3.2.0-0.20180921003261.6d2ec11.el7ost.noarch Installed latest, verifyied FIV, and verified no more unhealthy containers [stack@undercloud-0 ~]$ docker ps | grep unhealthy [stack@undercloud-0 ~]$ CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4c0ca767ea1f 192.168.24.1:8787/rhosp14/openstack-neutron-dhcp-agent:2019-06-17.2 "ip netns exec qdh..." About an hour ago Up About an hour neutron-dnsmasq-qdhcp-74666646-8e2f-4d52-9836-a3b54b4b49cf e272fc588e3f 192.168.24.1:8787/rhosp14/openstack-nova-compute-ironic:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_compute e6dfce1de1af 192.168.24.1:8787/rhosp14/openstack-ironic-inspector:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_inspector_dnsmasq 368b2efadfae 192.168.24.1:8787/rhosp14/openstack-ironic-inspector:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_inspector 23095a9ded55 192.168.24.1:8787/rhosp14/openstack-ironic-pxe:2019-06-17.2 "kolla_start" About an hour ago Up About an hour ironic_pxe_http e8e466e922e3 192.168.24.1:8787/rhosp14/openstack-ironic-pxe:2019-06-17.2 "/bin/bash -c 'BIN..." About an hour ago Up About an hour (healthy) ironic_pxe_tftp e26f2d675e75 192.168.24.1:8787/rhosp14/openstack-ironic-conductor:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_conductor c619ee83d33e 192.168.24.1:8787/rhosp14/openstack-ironic-neutron-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour ironic_neutron_agent 38478bf86ab3 192.168.24.1:8787/rhosp14/openstack-mistral-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_api acb521eeb313 192.168.24.1:8787/rhosp14/openstack-neutron-openvswitch-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_ovs_agent 91e4ade53b70 192.168.24.1:8787/rhosp14/openstack-neutron-l3-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_l3_agent 0961dec2ec07 192.168.24.1:8787/rhosp14/openstack-neutron-dhcp-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_dhcp 8438ea6303dd 192.168.24.1:8787/rhosp14/openstack-ironic-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_api d3fa3f3a3a7b 192.168.24.1:8787/rhosp14/openstack-nova-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_metadata 7b382ceab22d 192.168.24.1:8787/rhosp14/openstack-nova-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_api de1ae57c30c0 192.168.24.1:8787/rhosp14/openstack-glance-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) glance_api 3b22136df711 192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_proxy 5a14b2dc25b7 192.168.24.1:8787/rhosp14/openstack-nova-placement-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_placement 7709d88f47fe 192.168.24.1:8787/rhosp14/openstack-cron:2019-06-17.2 "kolla_start" About an hour ago Up About an hour logrotate_crond 5eafd6368fdd 192.168.24.1:8787/rhosp14/openstack-tripleo-ui:2019-06-17.2 "kolla_start" About an hour ago Up About an hour tripleo_ui 8ef8f412a515 192.168.24.1:8787/rhosp14/openstack-heat-api-cfn:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cfn be220a6b176c 192.168.24.1:8787/rhosp14/openstack-mistral-engine:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_engine 68ce5ea545a1 192.168.24.1:8787/rhosp14/openstack-zaqar:2019-06-17.2 "kolla_start" About an hour ago Up About an hour zaqar d0a369c22e3b 192.168.24.1:8787/rhosp14/openstack-swift-container:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_container_server 7a0f76ee6b60 192.168.24.1:8787/rhosp14/openstack-swift-object:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_object_updater 6ed0a7a249eb 192.168.24.1:8787/rhosp14/openstack-neutron-server:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_api f0282973cab6 192.168.24.1:8787/rhosp14/openstack-swift-account:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_account_server 31adfdc3dd19 192.168.24.1:8787/rhosp14/openstack-nova-conductor:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_conductor 03af7e39709d 192.168.24.1:8787/rhosp14/openstack-heat-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour heat_api_cron b5b0d0e713cf 192.168.24.1:8787/rhosp14/openstack-heat-engine:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) heat_engine 9a311e1443e2 192.168.24.1:8787/rhosp14/openstack-swift-object:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_rsync 5e257f5f36d9 192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_object_expirer 56ae9ea566fd 192.168.24.1:8787/rhosp14/openstack-swift-account:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_account_reaper 2ac5f6085187 192.168.24.1:8787/rhosp14/openstack-mistral-event-engine:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_event_engine 7efcc1a632d1 192.168.24.1:8787/rhosp14/openstack-nova-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour nova_api_cron cccd7b0a7947 192.168.24.1:8787/rhosp14/openstack-zaqar:2019-06-17.2 "kolla_start" About an hour ago Up About an hour zaqar_websocket 02d11f2f72d6 192.168.24.1:8787/rhosp14/openstack-heat-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) heat_api e0b97146e010 192.168.24.1:8787/rhosp14/openstack-swift-object:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_object_server 2709c35c9e4c 192.168.24.1:8787/rhosp14/openstack-nova-scheduler:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_scheduler a9d254865348 192.168.24.1:8787/rhosp14/openstack-mistral-executor:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_executor a70d4aba315a 192.168.24.1:8787/rhosp14/openstack-swift-container:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_container_updater 4c44baa89638 192.168.24.1:8787/rhosp14/openstack-keystone:2019-06-17.2 "/bin/bash -c '/us..." About an hour ago Up About an hour keystone_cron 578ce28f3be3 192.168.24.1:8787/rhosp14/openstack-keystone:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) keystone 1f9a1a4efe02 192.168.24.1:8787/rhosp14/openstack-iscsid:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) iscsid 22ddf08a15aa 192.168.24.1:8787/rhosp14/openstack-mariadb:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mysql 8b8b04c96a47 192.168.24.1:8787/rhosp14/openstack-rabbitmq:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) rabbitmq 42f0450ec405 192.168.24.1:8787/rhosp14/openstack-haproxy:2019-06-17.2 "kolla_start" About an hour ago Up About an hour haproxy 6df187d7ddb1 192.168.24.1:8787/rhosp14/openstack-memcached:2019-06-17.2 "/bin/bash -c 'sou..." About an hour ago Up About an hour (healthy) memcached 2aa0864d7fc7 192.168.24.1:8787/rhosp14/openstack-keepalived:2019-06-17.2 "/usr/local/bin/ko..." About an hour ago Up About an hour keepalived Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1672 |
Description of problem: Multiple containers reporting "unhealthy" after getting deployed Env: instack-undercloud-9.5.1-0.20190507032360.11d9fef.el7ost.noarch instack-9.1.1-0.20190420021823.3a69819.el7ost.noarch openstack-tripleo-common-containers-9.5.0-4.el7ost.noarch container-storage-setup-0.11.0-2.git5eaf76c.el7.noarch ansible-role-container-registry-1.0.1-0.20190424001444.b54b8ac.el7ost.noarch container-selinux-2.95-2.el7_6.noarch containers-common-0.1.35-2.git404c5bd.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Configure, prepare undercloud containers, etc, install undercloud 2. monitor containers (docker ps ) 3. Actual results: I am seeing 11 containers in more then one regression for osp 14 on 2019-05.31 (undercloud) [stack@undercloud-0 ~]$ docker ps | grep unhealthy e24a9927b1e6 192.168.24.1:8787/rhosp14/openstack-nova-compute-ironic:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) nova_compute da57c9698c11 192.168.24.1:8787/rhosp14/openstack-ironic-conductor:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) ironic_conductor a93138891bc7 192.168.24.1:8787/rhosp14/openstack-neutron-openvswitch-agent:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) neutron_ovs_agent f821e266ae8f 192.168.24.1:8787/rhosp14/openstack-neutron-l3-agent:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) neutron_l3_agent 5d78fa0e797e 192.168.24.1:8787/rhosp14/openstack-neutron-dhcp-agent:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) neutron_dhcp 29a411b1b391 192.168.24.1:8787/rhosp14/openstack-mistral-engine:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) mistral_engine a38f1640a64d 192.168.24.1:8787/rhosp14/openstack-nova-conductor:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) nova_conductor a7b820b61fc3 192.168.24.1:8787/rhosp14/openstack-heat-engine:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) heat_engine 9dd2cba003b0 192.168.24.1:8787/rhosp14/openstack-mistral-event-engine:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) mistral_event_engine 5401c0e63997 192.168.24.1:8787/rhosp14/openstack-nova-scheduler:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) nova_scheduler 2097f217e998 192.168.24.1:8787/rhosp14/openstack-mistral-executor:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) mistral_executor Expected results: Healthy containers Additional info: (undercloud) [stack@undercloud-0 ~]$ cat containers-prepare-parameter.yaml # Generated with the following on 2019-06-04T10:57:17.169306 # # openstack tripleo container image prepare default --output-env-file /home/stack/containers-prepare-parameter.yaml --local-push-destination # parameter_defaults: ContainerImagePrepare: - excludes: - openshift push_destination: true set: ceph_image: rhceph ceph_namespace: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 ceph_tag: 3-18 name_prefix: openstack- name_suffix: '' namespace: docker-registry.engineering.redhat.com/rhosp14 neutron_driver: null openshift_asb_namespace: registry.access.redhat.com/openshift3 openshift_cluster_monitoring_namespace: registry.access.redhat.com/openshift3 openshift_cockpit_namespace: registry.access.redhat.com/openshift3 openshift_configmap_reload_namespace: registry.access.redhat.com/openshift3 openshift_etcd_image: etcd openshift_etcd_namespace: registry.access.redhat.com/rhel7 openshift_etcd_tag: latest openshift_gluster_block_image: rhgs-gluster-block-prov-rhel7 openshift_gluster_image: rhgs-server-rhel7 openshift_gluster_namespace: registry.access.redhat.com/rhgs3 openshift_gluster_tag: v3.11 openshift_grafana_namespace: registry.access.redhat.com/openshift3 openshift_heketi_image: rhgs-volmanager-rhel7 openshift_heketi_namespace: registry.access.redhat.com/rhgs3 openshift_heketi_tag: v3.11 openshift_kube_rbac_proxy_namespace: registry.access.redhat.com/openshift3 openshift_kube_state_metrics_namespace: registry.access.redhat.com/openshift3 openshift_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_config_reload_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_operator_namespace: registry.access.redhat.com/openshift3 tag: 2019-05-31.1 - includes: - openshift push_destination: true set: openshift_asb_namespace: registry.access.redhat.com/openshift3 openshift_asb_tag: v3.11 openshift_cluster_monitoring_image: ose-cluster-monitoring-operator openshift_cluster_monitoring_namespace: registry.access.redhat.com/openshift3 openshift_cluster_monitoring_tag: v3.11 openshift_cockpit_image: registry-console openshift_cockpit_namespace: registry.access.redhat.com/openshift3 openshift_cockpit_tag: v3.11 openshift_configmap_reload_image: ose-configmap-reloader openshift_configmap_reload_namespace: registry.access.redhat.com/openshift3 openshift_configmap_reload_tag: v3.11 openshift_grafana_namespace: registry.access.redhat.com/openshift3 openshift_grafana_tag: v3.11 openshift_kube_rbac_proxy_image: ose-kube-rbac-proxy openshift_kube_rbac_proxy_namespace: registry.access.redhat.com/openshift3 openshift_kube_rbac_proxy_tag: v3.11 openshift_kube_state_metrics_image: ose-kube-state-metrics openshift_kube_state_metrics_namespace: registry.access.redhat.com/openshift3 openshift_kube_state_metrics_tag: v3.11 openshift_namespace: registry.access.redhat.com/openshift3 openshift_oauth_proxy_tag: v3.11 openshift_prefix: ose openshift_prometheus_alertmanager_tag: v3.11 openshift_prometheus_config_reload_image: ose-prometheus-config-reloader openshift_prometheus_config_reload_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_config_reload_tag: v3.11 openshift_prometheus_node_exporter_tag: v3.11 openshift_prometheus_operator_image: ose-prometheus-operator openshift_prometheus_operator_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_operator_tag: v3.11 openshift_prometheus_tag: v3.11 openshift_tag: v3.11