Description of problem: Multiple containers reporting "unhealthy" after getting deployed Env: instack-undercloud-9.5.1-0.20190507032360.11d9fef.el7ost.noarch instack-9.1.1-0.20190420021823.3a69819.el7ost.noarch openstack-tripleo-common-containers-9.5.0-4.el7ost.noarch container-storage-setup-0.11.0-2.git5eaf76c.el7.noarch ansible-role-container-registry-1.0.1-0.20190424001444.b54b8ac.el7ost.noarch container-selinux-2.95-2.el7_6.noarch containers-common-0.1.35-2.git404c5bd.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Configure, prepare undercloud containers, etc, install undercloud 2. monitor containers (docker ps ) 3. Actual results: I am seeing 11 containers in more then one regression for osp 14 on 2019-05.31 (undercloud) [stack@undercloud-0 ~]$ docker ps | grep unhealthy e24a9927b1e6 192.168.24.1:8787/rhosp14/openstack-nova-compute-ironic:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) nova_compute da57c9698c11 192.168.24.1:8787/rhosp14/openstack-ironic-conductor:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) ironic_conductor a93138891bc7 192.168.24.1:8787/rhosp14/openstack-neutron-openvswitch-agent:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) neutron_ovs_agent f821e266ae8f 192.168.24.1:8787/rhosp14/openstack-neutron-l3-agent:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) neutron_l3_agent 5d78fa0e797e 192.168.24.1:8787/rhosp14/openstack-neutron-dhcp-agent:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) neutron_dhcp 29a411b1b391 192.168.24.1:8787/rhosp14/openstack-mistral-engine:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) mistral_engine a38f1640a64d 192.168.24.1:8787/rhosp14/openstack-nova-conductor:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) nova_conductor a7b820b61fc3 192.168.24.1:8787/rhosp14/openstack-heat-engine:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) heat_engine 9dd2cba003b0 192.168.24.1:8787/rhosp14/openstack-mistral-event-engine:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) mistral_event_engine 5401c0e63997 192.168.24.1:8787/rhosp14/openstack-nova-scheduler:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) nova_scheduler 2097f217e998 192.168.24.1:8787/rhosp14/openstack-mistral-executor:2019-05-31.1 "kolla_start" 20 hours ago Up 20 hours (unhealthy) mistral_executor Expected results: Healthy containers Additional info: (undercloud) [stack@undercloud-0 ~]$ cat containers-prepare-parameter.yaml # Generated with the following on 2019-06-04T10:57:17.169306 # # openstack tripleo container image prepare default --output-env-file /home/stack/containers-prepare-parameter.yaml --local-push-destination # parameter_defaults: ContainerImagePrepare: - excludes: - openshift push_destination: true set: ceph_image: rhceph ceph_namespace: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 ceph_tag: 3-18 name_prefix: openstack- name_suffix: '' namespace: docker-registry.engineering.redhat.com/rhosp14 neutron_driver: null openshift_asb_namespace: registry.access.redhat.com/openshift3 openshift_cluster_monitoring_namespace: registry.access.redhat.com/openshift3 openshift_cockpit_namespace: registry.access.redhat.com/openshift3 openshift_configmap_reload_namespace: registry.access.redhat.com/openshift3 openshift_etcd_image: etcd openshift_etcd_namespace: registry.access.redhat.com/rhel7 openshift_etcd_tag: latest openshift_gluster_block_image: rhgs-gluster-block-prov-rhel7 openshift_gluster_image: rhgs-server-rhel7 openshift_gluster_namespace: registry.access.redhat.com/rhgs3 openshift_gluster_tag: v3.11 openshift_grafana_namespace: registry.access.redhat.com/openshift3 openshift_heketi_image: rhgs-volmanager-rhel7 openshift_heketi_namespace: registry.access.redhat.com/rhgs3 openshift_heketi_tag: v3.11 openshift_kube_rbac_proxy_namespace: registry.access.redhat.com/openshift3 openshift_kube_state_metrics_namespace: registry.access.redhat.com/openshift3 openshift_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_config_reload_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_operator_namespace: registry.access.redhat.com/openshift3 tag: 2019-05-31.1 - includes: - openshift push_destination: true set: openshift_asb_namespace: registry.access.redhat.com/openshift3 openshift_asb_tag: v3.11 openshift_cluster_monitoring_image: ose-cluster-monitoring-operator openshift_cluster_monitoring_namespace: registry.access.redhat.com/openshift3 openshift_cluster_monitoring_tag: v3.11 openshift_cockpit_image: registry-console openshift_cockpit_namespace: registry.access.redhat.com/openshift3 openshift_cockpit_tag: v3.11 openshift_configmap_reload_image: ose-configmap-reloader openshift_configmap_reload_namespace: registry.access.redhat.com/openshift3 openshift_configmap_reload_tag: v3.11 openshift_grafana_namespace: registry.access.redhat.com/openshift3 openshift_grafana_tag: v3.11 openshift_kube_rbac_proxy_image: ose-kube-rbac-proxy openshift_kube_rbac_proxy_namespace: registry.access.redhat.com/openshift3 openshift_kube_rbac_proxy_tag: v3.11 openshift_kube_state_metrics_image: ose-kube-state-metrics openshift_kube_state_metrics_namespace: registry.access.redhat.com/openshift3 openshift_kube_state_metrics_tag: v3.11 openshift_namespace: registry.access.redhat.com/openshift3 openshift_oauth_proxy_tag: v3.11 openshift_prefix: ose openshift_prometheus_alertmanager_tag: v3.11 openshift_prometheus_config_reload_image: ose-prometheus-config-reloader openshift_prometheus_config_reload_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_config_reload_tag: v3.11 openshift_prometheus_node_exporter_tag: v3.11 openshift_prometheus_operator_image: ose-prometheus-operator openshift_prometheus_operator_namespace: registry.access.redhat.com/openshift3 openshift_prometheus_operator_tag: v3.11 openshift_prometheus_tag: v3.11 openshift_tag: v3.11
Looks like mistral errors in container: ERROR:__main__:Failed to change ownership of /var/lib/mistral/undercloud.conf to 42430:42430 Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 345, in set_perms os.chown(path, uid, gid) OSError: [Errno 30] Read-only file system: '/var/lib/mistral/undercloud.conf' ++ cat /run_command + CMD='/usr/bin/mistral-server --config-file=/etc/mistral/mistral.conf --log-file=/var/log/mistral/executor.log --server=executor' And lots of errors in mistral/mistral-db-manage.log 2019-06-04 13:55:04.041 12 ERROR mistral.actions.openstack.action_generator.base AttributeError: 'Client' object has no attribute 'certs' 2019-06-04 13:55:04.041 12 ERROR mistral.actions.openstack.action_generator.base 2019-06-04 13:55:04.042 12 ERROR mistral.actions.openstack.action_generator.base [-] Failed to create action: nova.certs_create: AttributeError: 'Client' object has no attribute 'certs' 2019-06-04 13:55:04.521 12 ERROR mistral.actions.openstack.action_generator.base NotImplementedError: To use 'consumers' oauthlib must be installed 2019-06-04 13:55:04.521 12 ERROR mistral.actions.openstack.action_generator.base 2019-06-04 13:55:04.523 12 ERROR mistral.actions.openstack.action_generator.base [-] Failed to create action: keystone.oauth1.request_tokens_list: NotImplementedError: To use 'request_tokens' oauthlib must be installed 2019-06-04 13:55:12.379 12 ERROR mistral.actions.openstack.action_generator.base [-] Failed to create action: murano.services_post: AttributeError: 'NoneType' object has no attribute 'Client'
The problem is when a healthcheck is run with an argument (e.g. port): { "Start": "2019-06-12T17:29:22.810768802-04:00", "End": "2019-06-12T17:29:22.922346953-04:00", "ExitCode": 127, "Output": "/bin/sh: /openstack/healthcheck 5672: No such file or directory\n" } All other healthchecks running without argument work fine.
*** Bug 1721102 has been marked as a duplicate of this bug. ***
Environment: python-paunch-3.2.0-0.20180921003261.6d2ec11.el7ost.noarch Installed latest, verifyied FIV, and verified no more unhealthy containers [stack@undercloud-0 ~]$ docker ps | grep unhealthy [stack@undercloud-0 ~]$ CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4c0ca767ea1f 192.168.24.1:8787/rhosp14/openstack-neutron-dhcp-agent:2019-06-17.2 "ip netns exec qdh..." About an hour ago Up About an hour neutron-dnsmasq-qdhcp-74666646-8e2f-4d52-9836-a3b54b4b49cf e272fc588e3f 192.168.24.1:8787/rhosp14/openstack-nova-compute-ironic:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_compute e6dfce1de1af 192.168.24.1:8787/rhosp14/openstack-ironic-inspector:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_inspector_dnsmasq 368b2efadfae 192.168.24.1:8787/rhosp14/openstack-ironic-inspector:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_inspector 23095a9ded55 192.168.24.1:8787/rhosp14/openstack-ironic-pxe:2019-06-17.2 "kolla_start" About an hour ago Up About an hour ironic_pxe_http e8e466e922e3 192.168.24.1:8787/rhosp14/openstack-ironic-pxe:2019-06-17.2 "/bin/bash -c 'BIN..." About an hour ago Up About an hour (healthy) ironic_pxe_tftp e26f2d675e75 192.168.24.1:8787/rhosp14/openstack-ironic-conductor:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_conductor c619ee83d33e 192.168.24.1:8787/rhosp14/openstack-ironic-neutron-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour ironic_neutron_agent 38478bf86ab3 192.168.24.1:8787/rhosp14/openstack-mistral-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_api acb521eeb313 192.168.24.1:8787/rhosp14/openstack-neutron-openvswitch-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_ovs_agent 91e4ade53b70 192.168.24.1:8787/rhosp14/openstack-neutron-l3-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_l3_agent 0961dec2ec07 192.168.24.1:8787/rhosp14/openstack-neutron-dhcp-agent:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_dhcp 8438ea6303dd 192.168.24.1:8787/rhosp14/openstack-ironic-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) ironic_api d3fa3f3a3a7b 192.168.24.1:8787/rhosp14/openstack-nova-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_metadata 7b382ceab22d 192.168.24.1:8787/rhosp14/openstack-nova-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_api de1ae57c30c0 192.168.24.1:8787/rhosp14/openstack-glance-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) glance_api 3b22136df711 192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_proxy 5a14b2dc25b7 192.168.24.1:8787/rhosp14/openstack-nova-placement-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_placement 7709d88f47fe 192.168.24.1:8787/rhosp14/openstack-cron:2019-06-17.2 "kolla_start" About an hour ago Up About an hour logrotate_crond 5eafd6368fdd 192.168.24.1:8787/rhosp14/openstack-tripleo-ui:2019-06-17.2 "kolla_start" About an hour ago Up About an hour tripleo_ui 8ef8f412a515 192.168.24.1:8787/rhosp14/openstack-heat-api-cfn:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) heat_api_cfn be220a6b176c 192.168.24.1:8787/rhosp14/openstack-mistral-engine:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_engine 68ce5ea545a1 192.168.24.1:8787/rhosp14/openstack-zaqar:2019-06-17.2 "kolla_start" About an hour ago Up About an hour zaqar d0a369c22e3b 192.168.24.1:8787/rhosp14/openstack-swift-container:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_container_server 7a0f76ee6b60 192.168.24.1:8787/rhosp14/openstack-swift-object:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_object_updater 6ed0a7a249eb 192.168.24.1:8787/rhosp14/openstack-neutron-server:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) neutron_api f0282973cab6 192.168.24.1:8787/rhosp14/openstack-swift-account:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_account_server 31adfdc3dd19 192.168.24.1:8787/rhosp14/openstack-nova-conductor:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_conductor 03af7e39709d 192.168.24.1:8787/rhosp14/openstack-heat-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour heat_api_cron b5b0d0e713cf 192.168.24.1:8787/rhosp14/openstack-heat-engine:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) heat_engine 9a311e1443e2 192.168.24.1:8787/rhosp14/openstack-swift-object:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_rsync 5e257f5f36d9 192.168.24.1:8787/rhosp14/openstack-swift-proxy-server:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_object_expirer 56ae9ea566fd 192.168.24.1:8787/rhosp14/openstack-swift-account:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_account_reaper 2ac5f6085187 192.168.24.1:8787/rhosp14/openstack-mistral-event-engine:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_event_engine 7efcc1a632d1 192.168.24.1:8787/rhosp14/openstack-nova-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour nova_api_cron cccd7b0a7947 192.168.24.1:8787/rhosp14/openstack-zaqar:2019-06-17.2 "kolla_start" About an hour ago Up About an hour zaqar_websocket 02d11f2f72d6 192.168.24.1:8787/rhosp14/openstack-heat-api:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) heat_api e0b97146e010 192.168.24.1:8787/rhosp14/openstack-swift-object:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) swift_object_server 2709c35c9e4c 192.168.24.1:8787/rhosp14/openstack-nova-scheduler:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) nova_scheduler a9d254865348 192.168.24.1:8787/rhosp14/openstack-mistral-executor:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mistral_executor a70d4aba315a 192.168.24.1:8787/rhosp14/openstack-swift-container:2019-06-17.2 "kolla_start" About an hour ago Up About an hour swift_container_updater 4c44baa89638 192.168.24.1:8787/rhosp14/openstack-keystone:2019-06-17.2 "/bin/bash -c '/us..." About an hour ago Up About an hour keystone_cron 578ce28f3be3 192.168.24.1:8787/rhosp14/openstack-keystone:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) keystone 1f9a1a4efe02 192.168.24.1:8787/rhosp14/openstack-iscsid:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) iscsid 22ddf08a15aa 192.168.24.1:8787/rhosp14/openstack-mariadb:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) mysql 8b8b04c96a47 192.168.24.1:8787/rhosp14/openstack-rabbitmq:2019-06-17.2 "kolla_start" About an hour ago Up About an hour (healthy) rabbitmq 42f0450ec405 192.168.24.1:8787/rhosp14/openstack-haproxy:2019-06-17.2 "kolla_start" About an hour ago Up About an hour haproxy 6df187d7ddb1 192.168.24.1:8787/rhosp14/openstack-memcached:2019-06-17.2 "/bin/bash -c 'sou..." About an hour ago Up About an hour (healthy) memcached 2aa0864d7fc7 192.168.24.1:8787/rhosp14/openstack-keepalived:2019-06-17.2 "/usr/local/bin/ko..." About an hour ago Up About an hour keepalived
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1672