Created attachment 1510273 [details] Full controlle update log Description of problem: When attempting an update from 2018-11-03.1/ to latest, the update fails when attempting to update the controller Reproduction logs, full error will be attached to the bug: (undercloud) [stack@undercloud-0 ~]$ yum list installed | grep nova openstack-nova-api.noarch 1:16.1.4-6.el7ost @rhelosp-12.0-puddle openstack-nova-common.noarch 1:16.1.4-6.el7ost @rhelosp-12.0-puddle openstack-nova-compute.noarch 1:16.1.4-6.el7ost @rhelosp-12.0-puddle openstack-nova-conductor.noarch openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch puppet-nova.noarch 11.5.0-4.el7ost @rhelosp-12.0-puddle python-nova.noarch 1:16.1.4-6.el7ost @rhelosp-12.0-puddle python-novaclient.noarch 1:9.1.2-1.el7ost @rhelosp-12.0-puddle [root@compute-0 ~]# docker exec -u root -it nova_compute /bin/bash ()[root@compute-0 /]# yum list installed | grep nova openstack-nova-common.noarch 1:16.1.4-6.el7ost @rhos-12.0 openstack-nova-compute.noarch 1:16.1.4-6.el7ost @rhos-12.0 openstack-nova-migration.noarch 1:16.1.4-6.el7ost @rhos-12.0 python-nova.noarch 1:16.1.4-6.el7ost @rhos-12.0 python-novaclient.noarch 1:9.1.2-1.el7ost @rhos-12.0 (undercloud) [stack@undercloud-0 ~]$ sudo rhos-release 12 Installed: /etc/yum.repos.d/rhos-release-rhel-7.6.repo Installed: /etc/yum.repos.d/rhos-release-ceph-2.repo Installed: /etc/yum.repos.d/rhos-release-ceph-osd-2.repo Installed: /etc/yum.repos.d/rhos-release-12.repo (undercloud) [stack@undercloud-0 ~]$ sudo yum update -y python-tripleoclient (undercloud) [stack@undercloud-0 ~]$ openstack undercloud upgrade ############################################################################# Undercloud upgrade complete. The file containing this installation's passwords is at /home/stack/undercloud-passwords.conf. There is also a stackrc file at /home/stack/stackrc. These files are needed to interact with the OpenStack services, and should be secured. ############################################################################# (undercloud) [stack@undercloud-0 ~]$ yum list installed | grep nova openstack-nova-api.noarch 1:16.1.5-3.el7ost @rhelosp-12.0-puddle openstack-nova-common.noarch 1:16.1.5-3.el7ost @rhelosp-12.0-puddle openstack-nova-compute.noarch 1:16.1.5-3.el7ost @rhelosp-12.0-puddle openstack-nova-conductor.noarch openstack-nova-placement-api.noarch openstack-nova-scheduler.noarch puppet-nova.noarch 11.5.0-6.el7ost @rhelosp-12.0-puddle python-nova.noarch 1:16.1.5-3.el7ost @rhelosp-12.0-puddle python-novaclient.noarch 1:9.1.2-1.el7ost @rhelosp-12.0-puddle (undercloud) [stack@undercloud-0 ~]$ openstack overcloud container image prepare --namespace=docker-registry.engineering.redhat.com/rhosp12 --prefix=openstack- --tag 2018-11-28.1 --set ceph_namespace=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 --set ceph_image=rhceph --set ceph_tag=2.5-3 --env-file=/home/stack/overcloud_images.yaml container_images: - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-api:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-evaluator:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-listener:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-notifier:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-central:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-compute:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-notification:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-cron:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-metricd:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-statsd:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-heat-api-cfn:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-heat-api:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-horizon:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-iscsid:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-keystone:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-memcached:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-compute:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-conductor:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-consoleauth:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-libvirt:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-novncproxy:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-placement-api:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-scheduler:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-rabbitmq:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-redis:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-account:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-container:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-object:2018-11-28.1 - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-proxy-server:2018-11-28.1 (undercloud) [stack@undercloud-0 ~]$ sudo yum install skopeo (undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1 { "Name": "docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy", "Digest": "sha256:ac1dc586fc7614ff9e861dba76ffcb7b579276818fbc37794e823d3ab6dfdcb5", "RepoTags": [ "12.0-20181029.1", "12.0-20180810.1", "2017-12-02.3", "2017-11-03.4", "2017-11-22.7", "2018-01-13.1", "2018-01-22.1", "20180122.1", "12.0-20180122.1", "2018-03-10.1", "2018-08-15.1", "2018-10-02.1", "12.0-20171122.1", "12.0-20181002.1", "12.0-20180405.1", "2017-11-27.14", "2018-03-07.2", "12.0-20171123.3", "12.0-20171128.1", "12.0-20180519.1", "12.0-20181114.1", "2018-01-10.1", "2018-11-28.1", "12.0-20171201.1", "12.0-20180309.1", "2018-01-24.2", "2018-02-27.1", "12.0-20181102.1", "2018-01-04.2", "2018-01-13.3", "2018-03-21.1", "12.0-20180124.1", "12.0-20180713.1", "2018-01-04.1", "2018-03-19.3", "2018-11-02.1", "2017-07-17.1", "2018-03-09.2", "2018-09-10.2", "2018-01-23.1", "2018-05-19.1", "12.0-20180522.1", "2018-08-10.1", "2018-09-21.1", "12.0-20180813.1", "2017-11-21.5", "2018-11-14.1", "2018-04-05.1", "2018-07-27.1", "2018-07-17.1", "12.0-20180727.1", "12.0-20181128.1", "2017-11-15.2", "2018-10-09.1", "12.0-20171103.1", "12.0-20171129.1", "2017-12-02.4", "2018-02-27.5", "12.0-20180420.1", "12.0-20181103.1", "2017-11-28.3", "2017-12-06.2", "2018-01-16.2", "2018-01-26.2", "2018-06-25.2", "2018-10-25.2", "12.0-20180928.1", "12.0-20181025.1", "2018-10-29.1", "ga", "12.0-20180319.1", "12.0-20180529.1", "12.0-20180113.1", "2017-11-20.1", "2018-04-02.1", "12.0-20170717.1", "12.0-20180719.1", "12.0-20180910.1", "2018-09-28.1", "passed_phase1", "12.0-20180402.1", "12.0-20180717.1", "2017-11-14.4", "2017-11-27.5", "2018-05-29.1", "2018-07-19.1", "passed_phase2", "12.0-20180403.1", "12.0-20180921.1", "12.0-20181009.1", "2018-01-11.1", "2018-02-23.1", "12.0-20171118.1", "12.0-20180104.2", "2018-04-20.1", "2018-05-22.1", "2017-11-16.4", "2017-12-01.4", "2017-11-09.2", "12.0-20171127.1", "12.0-20180110.1", "2017-11-23.3", "latest", "12.0-20180307.1", "12.0-20180222.1", "beta", "12.0-20180227.1", "12.0-20180625.2", "12.0-20180808.1", "12.0-20180815.1", "2017-11-29.2", "2018-02-27.4", "12.0-20171122.7", "12.0-20180112.1", "2018-07-13.1", "2018-04-02.3", "2018-04-03.1", "2018-08-08.1", "2018-08-13.1", "2018-11-03.1" ], "Created": "2018-11-30T04:49:49.924324Z", "DockerVersion": "1.13.1", "Labels": { "Kolla-SHA": "5.0.0-39-g6f1b947b", "architecture": "x86_64", "authoritative-source-url": "registry.access.redhat.com", "build-date": "2018-11-30T04:45:11.992194", "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "openstack-haproxy-container", "description": "Red Hat OpenStack Platform 12.0 haproxy", "distribution-scope": "public", "io.k8s.description": "Red Hat OpenStack Platform 12.0 haproxy", "io.k8s.display-name": "Red Hat OpenStack Platform 12.0 haproxy", "io.openshift.tags": "rhosp osp openstack osp-12.0", "kolla_version": "stable/pike", "name": "rhosp12/openstack-haproxy", "release": "20181128.1", "summary": "Red Hat OpenStack Platform 12.0 haproxy", "tripleo-common_version": "7.6.3-23-g4891cfe", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/openstack-haproxy/images/12.0-20181128.1", "vcs-ref": "5cc8d441405684e3606689ba6a8da838836bc89f", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "12.0", "version-release": "12.0-20181128.1" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:9a1bea865f798d0e4f2359bd39ec69110369e3a1131aba6eb3cbf48707fdf92d", "sha256:602125c154e3e132db63d8e6479c5c93a64cbfd3a5ced509de73891ff7102643", "sha256:eb8caf2926b79a94049da862960e8031cc39870563c44b6f1619248aa8800ffc", "sha256:5258d116dadc93e5057835bc5b6ac7942008a169c2468c8af6bc75bb74916984" ] } (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --init-minor-update --container-registry-file /home/stack/overcloud_images.yaml Stack overcloud UPDATE_COMPLETE Heat stack update init on overcloud complete. Started Mistral Workflow tripleo.package_update.v1.get_config. Execution ID: 03e2b136-e321-456d-b011-238e2572d03d Waiting for messages on queue 'tripleo' with no timeout. Success Init minor update on stack overcloud complete. (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --nodes Controller u'TASK [debug] *******************************************************************', u'ok: [192.168.24.14] => {', u' "failed_when_result": false, ', u' "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [', u' "Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend", ', u' "Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.", ', u' "Notice: Compiled catalog for controller-0.localdomain in environment production in 2.57 seconds", ', u' "Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns: executed successfully", ', u' "Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.lo.disable_ipv6]/Sysctl[net.ipv6.conf.lo.disable_ipv6]/ensure: created", ', u' "Notice: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: executed successfully", ', u' "Notice: Applied catalog in 14.96 seconds", ', u' "Warning: Facter: Could not retrieve fact=\'rabbitmq_nodename\', resolution=\'<anonymous>\': undefined method `[]\' for nil:NilClass", ', u' "Warning: Undefined variable \'deploy_config_name\'; ", ', u' " (file & line not available)", ', u' "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 54]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ', u' " (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation\')", ', u' "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 55]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ', u' "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 56]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ', u' "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 66]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ', u' "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 68]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ', u' "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 76]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ', u' "Warning: ModuleLoader: module \'ssh\' has unresolved dependencies - it will only see those that are resolved. Use \'puppet module list --tree\' to see information about modules", ', u' "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ssh/manifests/server.pp\\", 12]:[\\"/var/lib/tripleo-config/puppet_step_config.pp\\", 28]", ', u' "Warning: ModuleLoader: module \'timezone\' has unresolved dependencies - it will only see those that are resolved. Use \'puppet module list --tree\' to see information about modules"', u' ]', u'}', u'', u'TASK [Run docker-puppet tasks (generate config)] *******************************', u'ok: [192.168.24.14]']
From the logs: "Error: rabbitmqctl status | grep -F \\"{rabbit,\\" returned 1 instead of one of [0]", But there is also a mysql error (?).
I could log into the environment and while I'm not fluent with the minor update procedure, something looks wrong to me: [root@controller-0 ~]# docker ps | head CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3d95989ad915 docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb:2018-11-28.1 "kolla_start" 2 days ago Up 2 days clustercheck ec25bd7ce8ca docker-registry.engineering.redhat.com/rhosp12/openstack-memcached:2018-11-28.1 "/bin/bash -c 'sou..." 2 days ago Up 2 days memcached b38a3fb23270 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd:2018-11-03.1 "kolla_start" 3 days ago Restarting (1) 11 hours ago gnocchi_statsd b5aba06e00c7 192.168.24.1:8787/rhosp12/openstack-gnocchi-api:2018-11-03.1 "kolla_start" 3 days ago Up 2 days gnocchi_api 4761442b1e96 192.168.24.1:8787/rhosp12/openstack-gnocchi-metricd:2018-11-03.1 "kolla_start" 3 days ago Up 2 days gnocchi_metricd [...] In fact, when doing so, the docker base image used for the pacemaker resources changes, and not only the docker tag. This is by design not supposed to work, as only the image tag can be updated in a rolling fashion across many controller node with pacemaker resources. The way we achieve this is by using a floating tag 'pcmklatest' that maps to the new docker image on the node that is being updated. I'm not sure this is a use case that we want to support, but currently at least, this the minor update of pacemaker resources expects that only the tag part of the docker image changes. Andrew, could you re-run your minor update test and make sure to keep the base image unchanged, and report whether this works as expected or not?
Hey Damien, Unfortunately trying with only the tag supplied caused the haproxy image to fail being pulled. Here's the relevant steps after a fresh install of the 11/3 puddle. Please let me know if there's anything else you'd like me to try, this is blocking an urgent bug. I'll keep the system intact in case you'd like to look at it. (undercloud) [stack@undercloud-0 ~]$ openstack overcloud container image prepare --tag 2018-11-28.1 --env-file=/home/stack/overcloud_images.yaml container_images: - imagename: registry.access.redhat.com/rhosp12/openstack-aodh-api:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-aodh-evaluator:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-aodh-listener:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-aodh-notifier:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-ceilometer-central:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-ceilometer-compute:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-ceilometer-notification:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-cron:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-glance-api:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-gnocchi-api:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-gnocchi-metricd:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-gnocchi-statsd:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-heat-api-cfn:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-heat-api:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-heat-engine:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-horizon:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-iscsid:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-keystone:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-mariadb:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-memcached:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-api:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-compute:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-conductor:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-consoleauth:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-libvirt:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-novncproxy:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-placement-api:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-nova-scheduler:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-panko-api:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-rabbitmq:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-redis:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-swift-account:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-swift-container:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-swift-object:2018-11-28.1 - imagename: registry.access.redhat.com/rhosp12/openstack-swift-proxy-server:2018-11-28.1 (undercloud) [stack@undercloud-0 ~]$ sudo yum install skopeo (undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1 { "Name": "docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy", "Digest": "sha256:ac1dc586fc7614ff9e861dba76ffcb7b579276818fbc37794e823d3ab6dfdcb5", "RepoTags": [ "12.0-20180522.1", "12.0-20180727.1", "12.0-20180928.1", "12.0-20181025.1", "12.0-20181103.1", "2017-11-28.3", "2017-12-01.4", "20180122.1", "2018-07-13.1", "2018-02-27.1", "2018-04-02.3", "2018-04-05.1", "2018-01-24.2", "2018-11-02.1", "ga", "2017-11-09.2", "2018-01-11.1", "12.0-20180405.1", "12.0-20171127.1", "12.0-20171201.1", "12.0-20180124.1", "2018-05-19.1", "2018-07-17.1", "2018-08-10.1", "12.0-20171118.1", "12.0-20180815.1", "2017-11-14.4", "2017-11-23.3", "2018-01-22.1", "12.0-20170717.1", "12.0-20180307.1", "12.0-20181029.1", "12.0-20180104.2", "12.0-20180402.1", "2018-10-29.1", "latest", "12.0-20181128.1", "12.0-20181009.1", "12.0-20171123.3", "12.0-20180813.1", "12.0-20181102.1", "2017-11-27.14", "2018-01-04.1", "2018-04-20.1", "12.0-20171122.1", "2018-08-13.1", "beta", "12.0-20180222.1", "2018-07-27.1", "2017-11-03.4", "2018-04-02.1", "2018-04-03.1", "2018-09-10.2", "12.0-20181114.1", "12.0-20180910.1", "12.0-20171128.1", "12.0-20180227.1", "12.0-20180529.1", "12.0-20180717.1", "12.0-20171103.1", "2018-11-28.1", "12.0-20180420.1", "2018-03-09.2", "2018-10-09.1", "2017-11-22.7", "12.0-20180112.1", "2018-01-23.1", "2018-03-21.1", "passed_phase2", "12.0-20171129.1", "2017-11-15.2", "12.0-20180122.1", "12.0-20181002.1", "2018-02-23.1", "2018-10-02.1", "12.0-20180110.1", "2018-01-04.2", "2017-07-17.1", "2017-11-29.2", "2018-01-26.2", "2018-03-07.2", "12.0-20171122.7", "2017-11-16.4", "2018-03-19.3", "2018-05-22.1", "2018-09-21.1", "passed_phase1", "12.0-20180319.1", "12.0-20180403.1", "12.0-20180625.2", "12.0-20180713.1", "12.0-20180810.1", "2017-11-27.5", "2017-12-02.3", "2017-12-02.4", "12.0-20180113.1", "2018-06-25.2", "2018-08-08.1", "2018-09-28.1", "2018-01-16.2", "2018-07-19.1", "12.0-20180719.1", "2018-01-13.1", "2018-02-27.4", "2018-02-27.5", "2018-11-14.1", "12.0-20180808.1", "2018-10-25.2", "2018-03-10.1", "2018-05-29.1", "2017-11-21.5", "2017-11-20.1", "2017-12-06.2", "2018-01-13.3", "2018-08-15.1", "2018-11-03.1", "12.0-20180921.1", "12.0-20180519.1", "2018-01-10.1", "12.0-20180309.1" ], "Created": "2018-11-30T04:49:49.924324Z", "DockerVersion": "1.13.1", "Labels": { "Kolla-SHA": "5.0.0-39-g6f1b947b", "architecture": "x86_64", "authoritative-source-url": "registry.access.redhat.com", "build-date": "2018-11-30T04:45:11.992194", "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "openstack-haproxy-container", "description": "Red Hat OpenStack Platform 12.0 haproxy", "distribution-scope": "public", "io.k8s.description": "Red Hat OpenStack Platform 12.0 haproxy", "io.k8s.display-name": "Red Hat OpenStack Platform 12.0 haproxy", "io.openshift.tags": "rhosp osp openstack osp-12.0", "kolla_version": "stable/pike", "name": "rhosp12/openstack-haproxy", "release": "20181128.1", "summary": "Red Hat OpenStack Platform 12.0 haproxy", "tripleo-common_version": "7.6.3-23-g4891cfe", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/openstack-haproxy/images/12.0-20181128.1", "vcs-ref": "5cc8d441405684e3606689ba6a8da838836bc89f", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "12.0", "version-release": "12.0-20181128.1" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:9a1bea865f798d0e4f2359bd39ec69110369e3a1131aba6eb3cbf48707fdf92d", "sha256:602125c154e3e132db63d8e6479c5c93a64cbfd3a5ced509de73891ff7102643", "sha256:eb8caf2926b79a94049da862960e8031cc39870563c44b6f1619248aa8800ffc", "sha256:5258d116dadc93e5057835bc5b6ac7942008a169c2468c8af6bc75bb74916984" ] } (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --init-minor-update --container-registry-file /home/stack/overcloud_images.yaml Stack overcloud UPDATE_COMPLETE Heat stack update init on overcloud complete. Started Mistral Workflow tripleo.package_update.v1.get_config. Execution ID: 40e09dbc-81d3-4d5e-939e-9717be607439 Waiting for messages on queue 'tripleo' with no timeout. Success Init minor update on stack overcloud complete. (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --nodes Controller u'TASK [Pull latest Haproxy images] **********************************************', u'fatal: [192.168.24.15]: FAILED! => {"changed": true, "cmd": ["docker", "pull", "registry.access.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1"], "delta": "0:00:02.732051", "end": "2018-12-04 21:58:48.093039", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-12-04 21:58:45.360988", "stderr": "error parsing HTTP 404 response body: invalid character \'F\' looking for beginning of value: \\"File not found.\\\\\\"\\"", "stderr_lines": ["error parsing HTTP 404 response body: invalid character \'F\' looking for beginning of value: \\"File not found.\\\\\\"\\""], "stdout": "Trying to pull repository registry.access.redhat.com/rhosp12/openstack-haproxy ... ", "stdout_lines": ["Trying to pull repository registry.access.redhat.com/rhosp12/openstack-haproxy ... "]}', u'', u'PLAY RECAP *********************************************************************', u'192.168.24.15 : ok=39 changed=19 unreachable=0 failed=1 ', u''] Minor update failed with: {u'status': u'FAILED', u'execution': {u'name': u'tripleo.package_update.v1.update_nodes', u'created_at': u'2018-12-04 21:56:32', u'id': u'd9c6479d-2161-4096-9466-a16597a534d5', u'params': {u'namespace': u'', u'env': {}}, u'input': {u'inventory_file': u"[undercloud]\nlocalhost\n\n[undercloud:vars]\nusername = admin\novercloud_keystone_url = http://10.0.0.107:5000/v2.0\nproject_name = admin\nundercloud_service_list = ['openstack-nova-compute', 'openstack-heat-engine', 'openstack-ironic-conductor', 'openstack-swift-container', 'openstack-swift-object', 'openstack-mistral-engine']\novercloud_horizon_url = http://10.0.0.107:80/dashboard\nos_auth_token = gAAAAABcBvgL8psG5rjYTgnMxsXCOF8vSmnhxCIkhVc2Exh4Z3JN61LAHfyvGiQlEEsWiz-JRS0LoOnkIUPaM89rYgUdtUd6XX9eqmx0duDPs54aJV5pReNzjLzRKNwsV89mzpas7kaKOcM7a55pEnfoVHjOOyZZWjIs5aOODjrYi-hb6XXwO-A\novercloud_admin_password = bdFu2BaFfpPZRFPu9N6EfxKs6\nauth_url = http://192.168.24.1:5000/\nansible_connection = local\nundercloud_swift_url = http://192.168.24.1:8080/v1/AUTH_884e302336274f1d80df63acb0e8cd12\nplan = overcloud\n\n[controller-0]\n192.168.24.15\n\n[controller-0:vars]\ndeploy_server_id = 807b1da6-cc24-4f3b-a226-c87377b34847\n\n[Controller:vars]\nrole_name = Controller\nansible_ssh_user = heat-admin\nbootstrap_server_id = 807b1da6-cc24-4f3b-a226-c87377b34847\n\n[Controller:children]\ncontroller-0\n\n[compute-0]\n192.168.24.9\n\n[compute-0:vars]\ndeploy_server_id = 2febf52a-03d7-4525-a00a-f3dcab3d5c3e\n\n[Compute:vars]\nrole_name = Compute\nansible_ssh_user = heat-admin\nbootstrap_server_id = 807b1da6-cc24-4f3b-a226-c87377b34847\n\n[Compute:children]\ncompute-0\n\n[overcloud:children]\nCompute\nController\n\n[aodh_evaluator:vars]\nansible_ssh_user = heat-admin\n\n[aodh_evaluator:children]\nController\n\n[kernel:vars]\nansible_ssh_user = heat-admin\n\n[kernel:children]\nCompute\nController\n\n[neutron_metadata:vars]\nansible_ssh_user = heat-admin\n\n[neutron_metadata:children]\nController\n\n[pacemaker:vars]\nansible_ssh_user = heat-admin\n\n[pacemaker:children]\nController\n\n[nova_placement:vars]\nansible_ssh_user = heat-admin\n\n[nova_placement:children]\nController\n\n[snmp:vars]\nansible_ssh_user = heat-admin\n\n[snmp:children]\nCompute\nController\n\n[heat_api:vars]\nansible_ssh_user = heat-admin\n\n[heat_api:children]\nController\n\n[cinder_api:vars]\nansible_ssh_user = heat-admin\n\n[cinder_api:children]\nController\n\n[neutron_l3:vars]\nansible_ssh_user = heat-admin\n\n[neutron_l3:children]\nController\n\n[swift_proxy:vars]\nansible_ssh_user = heat-admin\n\n[swift_proxy:children]\nController\n\n[aodh_listener:vars]\nansible_ssh_user = heat-admin\n\n[aodh_listener:children]\nController\n\n[swift_ringbuilder:vars]\nansible_ssh_user = heat-admin\n\n[swift_ringbuilder:children]\nController\n\n[neutron_dhcp:vars]\nansible_ssh_user = heat-admin\n\n[neutron_dhcp:children]\nController\n\n[gnocchi_api:vars]\nansible_ssh_user = heat-admin\n\n[gnocchi_api:children]\nController\n\n[timezone:vars]\nansible_ssh_user = heat-admin\n\n[timezone:children]\nCompute\nController\n\n[ceilometer_agent_central:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_agent_central:children]\nController\n\n[heat_api_cloudwatch_disabled:vars]\nansible_ssh_user = heat-admin\n\n[heat_api_cloudwatch_disabled:children]\nController\n\n[aodh_notifier:vars]\nansible_ssh_user = heat-admin\n\n[aodh_notifier:children]\nController\n\n[tripleo_firewall:vars]\nansible_ssh_user = heat-admin\n\n[tripleo_firewall:children]\nCompute\nController\n\n[swift_storage:vars]\nansible_ssh_user = heat-admin\n\n[swift_storage:children]\nController\n\n[redis:vars]\nansible_ssh_user = heat-admin\n\n[redis:children]\nController\n\n[gnocchi_statsd:vars]\nansible_ssh_user = heat-admin\n\n[gnocchi_statsd:children]\nController\n\n[iscsid:vars]\nansible_ssh_user = heat-admin\n\n[iscsid:children]\nCompute\nController\n\n[nova_conductor:vars]\nansible_ssh_user = heat-admin\n\n[nova_conductor:children]\nController\n\n[mysql_client:vars]\nansible_ssh_user = heat-admin\n\n[mysql_client:children]\nCompute\nController\n\n[nova_consoleauth:vars]\nansible_ssh_user = heat-admin\n\n[nova_consoleauth:children]\nController\n\n[glance_api:vars]\nansible_ssh_user = heat-admin\n\n[glance_api:children]\nController\n\n[keystone:vars]\nansible_ssh_user = heat-admin\n\n[keystone:children]\nController\n\n[cinder_volume:vars]\nansible_ssh_user = heat-admin\n\n[cinder_volume:children]\nController\n\n[ceilometer_collector_disabled:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_collector_disabled:children]\nController\n\n[ceilometer_agent_notification:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_agent_notification:children]\nController\n\n[memcached:vars]\nansible_ssh_user = heat-admin\n\n[memcached:children]\nController\n\n[mongodb_disabled:vars]\nansible_ssh_user = heat-admin\n\n[mongodb_disabled:children]\nController\n\n[neutron_plugin_ml2:vars]\nansible_ssh_user = heat-admin\n\n[neutron_plugin_ml2:children]\nCompute\nController\n\n[nova_api:vars]\nansible_ssh_user = heat-admin\n\n[nova_api:children]\nController\n\n[aodh_api:vars]\nansible_ssh_user = heat-admin\n\n[aodh_api:children]\nController\n\n[nova_metadata:vars]\nansible_ssh_user = heat-admin\n\n[nova_metadata:children]\nController\n\n[heat_engine:vars]\nansible_ssh_user = heat-admin\n\n[heat_engine:children]\nController\n\n[ntp:vars]\nansible_ssh_user = heat-admin\n\n[ntp:children]\nCompute\nController\n\n[ceilometer_expirer_disabled:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_expirer_disabled:children]\nController\n\n[ceilometer_api_disabled:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_api_disabled:children]\nController\n\n[nova_migration_target:vars]\nansible_ssh_user = heat-admin\n\n[nova_migration_target:children]\nCompute\n\n[cinder_scheduler:vars]\nansible_ssh_user = heat-admin\n\n[cinder_scheduler:children]\nController\n\n[gnocchi_metricd:vars]\nansible_ssh_user = heat-admin\n\n[gnocchi_metricd:children]\nController\n\n[tripleo_packages:vars]\nansible_ssh_user = heat-admin\n\n[tripleo_packages:children]\nCompute\nController\n\n[nova_scheduler:vars]\nansible_ssh_user = heat-admin\n\n[nova_scheduler:children]\nController\n\n[nova_compute:vars]\nansible_ssh_user = heat-admin\n\n[nova_compute:children]\nCompute\n\n[logrotate_crond:vars]\nansible_ssh_user = heat-admin\n\n[logrotate_crond:children]\nCompute\nController\n\n[neutron_ovs_agent:vars]\nansible_ssh_user = heat-admin\n\n[neutron_ovs_agent:children]\nCompute\nController\n\n[haproxy:vars]\nansible_ssh_user = heat-admin\n\n[haproxy:children]\nController\n\n[sshd:vars]\nansible_ssh_user = heat-admin\n\n[sshd:children]\nCompute\nController\n\n[mysql:vars]\nansible_ssh_user = heat-admin\n\n[mysql:children]\nController\n\n[ceilometer_agent_compute:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_agent_compute:children]\nCompute\n\n[nova_libvirt:vars]\nansible_ssh_user = heat-admin\n\n[nova_libvirt:children]\nCompute\n\n[rabbitmq:vars]\nansible_ssh_user = heat-admin\n\n[rabbitmq:children]\nController\n\n[tuned:vars]\nansible_ssh_user = heat-admin\n\n[tuned:children]\nCompute\nController\n\n[panko_api:vars]\nansible_ssh_user = heat-admin\n\n[panko_api:children]\nController\n\n[horizon:vars]\nansible_ssh_user = heat-admin\n\n[horizon:children]\nController\n\n[neutron_api:vars]\nansible_ssh_user = heat-admin\n\n[neutron_api:children]\nController\n\n[ca_certs:vars]\nansible_ssh_user = heat-admin\n\n[ca_certs:children]\nCompute\nController\n\n[heat_api_cfn:vars]\nansible_ssh_user = heat-admin\n\n[heat_api_cfn:children]\nController\n\n[docker:vars]\nansible_ssh_user = heat-admin\n\n[docker:children]\nCompute\nController\n\n[nova_vnc_proxy:vars]\nansible_ssh_user = heat-admin\n\n[nova_vnc_proxy:children]\nController\n\n[clustercheck:vars]\nansible_ssh_user = heat-admin\n\n[clustercheck:children]\nController\n\n", u'playbook': u'update_steps_playbook.yaml', u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False'}, u'module_path': u'/usr/share/ansible-modules', u'nodes': u'Controller', u'node_user': u'heat-admin', u'ansible_queue_name': u'update'}, u'spec': {u'tasks': {u'node_update': {u'name': u'node_update', u'on-error': u'node_update_failed', u'on-success': [{u'node_update_passed': u'<% task().result.returncode = 0 %>'}, {u'node_update_failed': u'<% task().result.returncode != 0 %>'}], u'publish': {u'output': u'<% task(node_update).result %>'}, u'version': u'2.0', u'action': u'tripleo.ansible-playbook', u'input': {u'remote_user': u'<% $.node_user %>', u'become_user': u'root', u'ssh_private_key': u'<% $.private_key %>', u'verbosity': 0, u'queue_name': u'<% $.ansible_queue_name %>', u'extra_env_variables': u'<% $.ansible_extra_env_variables %>', u'inventory': u'<% $.inventory_file %>', u'execution_id': u'<% execution().id %>', u'module_path': u'<% $.module_path %>', u'become': True, u'trash_output': True, u'limit_hosts': u'<% $.nodes %>', u'playbook': u'<% $.tmp_path %>/<% $.playbook %>'}, u'type': u'direct'}, u'get_private_key': {u'name': u'get_private_key', u'on-success': u'node_update', u'publish': {u'private_key': u'<% task(get_private_key).result %>'}, u'version': u'2.0', u'action': u'tripleo.validations.get_privkey', u'type': u'direct'}, u'node_update_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_failed', u'publish': {u'status': u'FAILED', u'message': u'Failed to update nodes - <% $.nodes %>, please see the logs.'}, u'on-success': u'notify_zaqar'}, u'node_update_passed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_passed', u'publish': {u'status': u'SUCCESS', u'message': u'Updated nodes - <% $.nodes %>'}, u'on-success': u'notify_zaqar'}, u'notify_zaqar': {u'retry': u'count=5 delay=1', u'name': u'notify_zaqar', u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}], u'version': u'2.0', u'action': u'zaqar.queue_post', u'input': {u'queue_name': u'<% $.ansible_queue_name %>', u'messages': {u'body': {u'type': u'tripleo.package_update.v1.update_nodes', u'payload': {u'status': u'<% $.status %>', u'execution': u'<% execution() %>'}}}}, u'type': u'direct'}, u'download_config': {u'name': u'download_config', u'on-error': u'node_update_failed', u'on-success': u'get_private_key', u'publish': {u'tmp_path': u'<% task(download_config).result %>'}, u'version': u'2.0', u'action': u'tripleo.config.download_config', u'type': u'direct'}}, u'name': u'update_nodes', u'tags': [u'tripleo-common-managed'], u'version': u'2.0', u'input': [{u'node_user': u'heat-admin'}, u'nodes', u'playbook', u'inventory_file', {u'ansible_queue_name': u'tripleo'}, {u'module_path': u'/usr/share/ansible-modules'}, {u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False'}}], u'description': u'Take a container and perform an update nodes by nodes'}}}
Hmm it looks like the image you're trying to access is not available, when I try that on your undercloud I get: (undercloud) [stack@undercloud-0 ~]$ docker pull registry.access.redhat.com/rhosp12/openstack-aodh-api:2018-11-28.1 Trying to pull repository registry.access.redhat.com/rhosp12/openstack-aodh-api ... error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\"" This seems to be the error that you're hitting when the update runs.
Hey Raviv, Do you have any input on this bug? RHOS 12 went out last week but we still need to verify this process works. Thanks, Andrew
I was able to get the update working, see linked bug for steps. Having said that, closing this as Notabug
Ack, As osp12 is EOL I will not try to reproduce this BZ. Thanks