Bug 1655181 - Minor Update fails at controller stage
Summary: Minor Update fails at controller stage
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Damien Ciabrini
QA Contact: Raviv Bar-Tal
URL:
Whiteboard:
Depends On:
Blocks: 1627009
TreeView+ depends on / blocked
 
Reported: 2018-11-30 21:28 UTC by awaugama
Modified: 2018-12-25 11:17 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-21 14:38:04 UTC
Target Upstream Version:


Attachments (Terms of Use)
Full controlle update log (44.34 KB, text/plain)
2018-11-30 21:28 UTC, awaugama
no flags Details

Description awaugama 2018-11-30 21:28:02 UTC
Created attachment 1510273 [details]
Full controlle update log

Description of problem: When attempting an update from 2018-11-03.1/ to latest, the update fails when attempting to update the controller

Reproduction logs, full error will be attached to the bug:

    (undercloud) [stack@undercloud-0 ~]$ yum list installed | grep nova
    openstack-nova-api.noarch     1:16.1.4-6.el7ost         @rhelosp-12.0-puddle    
    openstack-nova-common.noarch  1:16.1.4-6.el7ost         @rhelosp-12.0-puddle    
    openstack-nova-compute.noarch 1:16.1.4-6.el7ost         @rhelosp-12.0-puddle    
    openstack-nova-conductor.noarch
    openstack-nova-placement-api.noarch
    openstack-nova-scheduler.noarch
    puppet-nova.noarch            11.5.0-4.el7ost           @rhelosp-12.0-puddle    
    python-nova.noarch            1:16.1.4-6.el7ost         @rhelosp-12.0-puddle    
    python-novaclient.noarch      1:9.1.2-1.el7ost          @rhelosp-12.0-puddle
     
    [root@compute-0 ~]# docker exec -u root -it nova_compute /bin/bash
    ()[root@compute-0 /]# yum list installed | grep nova
    openstack-nova-common.noarch     1:16.1.4-6.el7ost      @rhos-12.0              
    openstack-nova-compute.noarch    1:16.1.4-6.el7ost      @rhos-12.0              
    openstack-nova-migration.noarch  1:16.1.4-6.el7ost      @rhos-12.0              
    python-nova.noarch               1:16.1.4-6.el7ost      @rhos-12.0              
    python-novaclient.noarch         1:9.1.2-1.el7ost       @rhos-12.0      
     
    (undercloud) [stack@undercloud-0 ~]$ sudo rhos-release 12
    Installed: /etc/yum.repos.d/rhos-release-rhel-7.6.repo
    Installed: /etc/yum.repos.d/rhos-release-ceph-2.repo
    Installed: /etc/yum.repos.d/rhos-release-ceph-osd-2.repo
    Installed: /etc/yum.repos.d/rhos-release-12.repo
     
    (undercloud) [stack@undercloud-0 ~]$ sudo yum update -y python-tripleoclient
     
    (undercloud) [stack@undercloud-0 ~]$ openstack undercloud upgrade
     
    #############################################################################
    Undercloud upgrade complete.
     
    The file containing this installation's passwords is at
    /home/stack/undercloud-passwords.conf.
     
    There is also a stackrc file at /home/stack/stackrc.
     
    These files are needed to interact with the OpenStack services, and should be
    secured.
     
    #############################################################################
     
    (undercloud) [stack@undercloud-0 ~]$ yum list installed | grep nova
    openstack-nova-api.noarch     1:16.1.5-3.el7ost         @rhelosp-12.0-puddle    
    openstack-nova-common.noarch  1:16.1.5-3.el7ost         @rhelosp-12.0-puddle    
    openstack-nova-compute.noarch 1:16.1.5-3.el7ost         @rhelosp-12.0-puddle    
    openstack-nova-conductor.noarch
    openstack-nova-placement-api.noarch
    openstack-nova-scheduler.noarch
    puppet-nova.noarch            11.5.0-6.el7ost           @rhelosp-12.0-puddle    
    python-nova.noarch            1:16.1.5-3.el7ost         @rhelosp-12.0-puddle    
    python-novaclient.noarch      1:9.1.2-1.el7ost          @rhelosp-12.0-puddle    
     
    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud container image prepare --namespace=docker-registry.engineering.redhat.com/rhosp12 --prefix=openstack- --tag 2018-11-28.1 --set ceph_namespace=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 --set ceph_image=rhceph --set ceph_tag=2.5-3 --env-file=/home/stack/overcloud_images.yaml
    container_images:
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-api:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-evaluator:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-listener:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-aodh-notifier:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-central:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-compute:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-ceilometer-notification:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-cron:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-glance-api:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-api:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-metricd:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-gnocchi-statsd:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-heat-api-cfn:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-heat-api:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-heat-engine:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-horizon:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-iscsid:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-keystone:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-memcached:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-compute:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-conductor:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-consoleauth:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-libvirt:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-novncproxy:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-placement-api:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-nova-scheduler:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-panko-api:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-rabbitmq:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-redis:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-account:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-container:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-object:2018-11-28.1
    - imagename: docker-registry.engineering.redhat.com/rhosp12/openstack-swift-proxy-server:2018-11-28.1
     
    (undercloud) [stack@undercloud-0 ~]$ sudo yum install skopeo
     
    (undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1
    {
        "Name": "docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy",
        "Digest": "sha256:ac1dc586fc7614ff9e861dba76ffcb7b579276818fbc37794e823d3ab6dfdcb5",
        "RepoTags": [
            "12.0-20181029.1",
            "12.0-20180810.1",
            "2017-12-02.3",
            "2017-11-03.4",
            "2017-11-22.7",
            "2018-01-13.1",
            "2018-01-22.1",
            "20180122.1",
            "12.0-20180122.1",
            "2018-03-10.1",
            "2018-08-15.1",
            "2018-10-02.1",
            "12.0-20171122.1",
            "12.0-20181002.1",
            "12.0-20180405.1",
            "2017-11-27.14",
            "2018-03-07.2",
            "12.0-20171123.3",
            "12.0-20171128.1",
            "12.0-20180519.1",
            "12.0-20181114.1",
            "2018-01-10.1",
            "2018-11-28.1",
            "12.0-20171201.1",
            "12.0-20180309.1",
            "2018-01-24.2",
            "2018-02-27.1",
            "12.0-20181102.1",
            "2018-01-04.2",
            "2018-01-13.3",
            "2018-03-21.1",
            "12.0-20180124.1",
            "12.0-20180713.1",
            "2018-01-04.1",
            "2018-03-19.3",
            "2018-11-02.1",
            "2017-07-17.1",
            "2018-03-09.2",
            "2018-09-10.2",
            "2018-01-23.1",
            "2018-05-19.1",
            "12.0-20180522.1",
            "2018-08-10.1",
            "2018-09-21.1",
            "12.0-20180813.1",
            "2017-11-21.5",
            "2018-11-14.1",
            "2018-04-05.1",
            "2018-07-27.1",
            "2018-07-17.1",
            "12.0-20180727.1",
            "12.0-20181128.1",
            "2017-11-15.2",
            "2018-10-09.1",
            "12.0-20171103.1",
            "12.0-20171129.1",
            "2017-12-02.4",
            "2018-02-27.5",
            "12.0-20180420.1",
            "12.0-20181103.1",
            "2017-11-28.3",
            "2017-12-06.2",
            "2018-01-16.2",
            "2018-01-26.2",
            "2018-06-25.2",
            "2018-10-25.2",
            "12.0-20180928.1",
            "12.0-20181025.1",
            "2018-10-29.1",
            "ga",
            "12.0-20180319.1",
            "12.0-20180529.1",
            "12.0-20180113.1",
            "2017-11-20.1",
            "2018-04-02.1",
            "12.0-20170717.1",
            "12.0-20180719.1",
            "12.0-20180910.1",
            "2018-09-28.1",
            "passed_phase1",
            "12.0-20180402.1",
            "12.0-20180717.1",
            "2017-11-14.4",
            "2017-11-27.5",
            "2018-05-29.1",
            "2018-07-19.1",
            "passed_phase2",
            "12.0-20180403.1",
            "12.0-20180921.1",
            "12.0-20181009.1",
            "2018-01-11.1",
            "2018-02-23.1",
            "12.0-20171118.1",
            "12.0-20180104.2",
            "2018-04-20.1",
            "2018-05-22.1",
            "2017-11-16.4",
            "2017-12-01.4",
            "2017-11-09.2",
            "12.0-20171127.1",
            "12.0-20180110.1",
            "2017-11-23.3",
            "latest",
            "12.0-20180307.1",
            "12.0-20180222.1",
            "beta",
            "12.0-20180227.1",
            "12.0-20180625.2",
            "12.0-20180808.1",
            "12.0-20180815.1",
            "2017-11-29.2",
            "2018-02-27.4",
            "12.0-20171122.7",
            "12.0-20180112.1",
            "2018-07-13.1",
            "2018-04-02.3",
            "2018-04-03.1",
            "2018-08-08.1",
            "2018-08-13.1",
            "2018-11-03.1"
        ],
        "Created": "2018-11-30T04:49:49.924324Z",
        "DockerVersion": "1.13.1",
        "Labels": {
            "Kolla-SHA": "5.0.0-39-g6f1b947b",
            "architecture": "x86_64",
            "authoritative-source-url": "registry.access.redhat.com",
            "build-date": "2018-11-30T04:45:11.992194",
            "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com",
            "com.redhat.component": "openstack-haproxy-container",
            "description": "Red Hat OpenStack Platform 12.0 haproxy",
            "distribution-scope": "public",
            "io.k8s.description": "Red Hat OpenStack Platform 12.0 haproxy",
            "io.k8s.display-name": "Red Hat OpenStack Platform 12.0 haproxy",
            "io.openshift.tags": "rhosp osp openstack osp-12.0",
            "kolla_version": "stable/pike",
            "name": "rhosp12/openstack-haproxy",
            "release": "20181128.1",
            "summary": "Red Hat OpenStack Platform 12.0 haproxy",
            "tripleo-common_version": "7.6.3-23-g4891cfe",
            "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/openstack-haproxy/images/12.0-20181128.1",
            "vcs-ref": "5cc8d441405684e3606689ba6a8da838836bc89f",
            "vcs-type": "git",
            "vendor": "Red Hat, Inc.",
            "version": "12.0",
            "version-release": "12.0-20181128.1"
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Layers": [
            "sha256:9a1bea865f798d0e4f2359bd39ec69110369e3a1131aba6eb3cbf48707fdf92d",
            "sha256:602125c154e3e132db63d8e6479c5c93a64cbfd3a5ced509de73891ff7102643",
            "sha256:eb8caf2926b79a94049da862960e8031cc39870563c44b6f1619248aa8800ffc",
            "sha256:5258d116dadc93e5057835bc5b6ac7942008a169c2468c8af6bc75bb74916984"
        ]
    }
     
    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --init-minor-update --container-registry-file /home/stack/overcloud_images.yaml
     
     Stack overcloud UPDATE_COMPLETE
     
    Heat stack update init on overcloud complete.
    Started Mistral Workflow tripleo.package_update.v1.get_config. Execution ID: 03e2b136-e321-456d-b011-238e2572d03d
    Waiting for messages on queue 'tripleo' with no timeout.
    Success
    Init minor update on stack overcloud complete.
     
    (undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --nodes Controller
     u'TASK [debug] *******************************************************************',
     u'ok: [192.168.24.14] => {',
     u'    "failed_when_result": false, ',
     u'    "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [',
     u'        "Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend", ',
     u'        "Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.", ',
     u'        "Notice: Compiled catalog for controller-0.localdomain in environment production in 2.57 seconds", ',
     u'        "Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack_proto_sctp]/Exec[modprobe nf_conntrack_proto_sctp]/returns: executed successfully", ',
     u'        "Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.lo.disable_ipv6]/Sysctl[net.ipv6.conf.lo.disable_ipv6]/ensure: created", ',
     u'        "Notice: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: executed successfully", ',
     u'        "Notice: Applied catalog in 14.96 seconds", ',
     u'        "Warning: Facter: Could not retrieve fact=\'rabbitmq_nodename\', resolution=\'<anonymous>\': undefined method `[]\' for nil:NilClass", ',
     u'        "Warning: Undefined variable \'deploy_config_name\'; ", ',
     u'        "   (file & line not available)", ',
     u'        "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 54]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ',
     u'        "   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation\')", ',
     u'        "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 55]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ',
     u'        "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 56]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ',
     u'        "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 66]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ',
     u'        "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 68]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ',
     u'        "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ntp/manifests/init.pp\\", 76]:[\\"/etc/puppet/modules/tripleo/manifests/profile/base/time/ntp.pp\\", 29]", ',
     u'        "Warning: ModuleLoader: module \'ssh\' has unresolved dependencies - it will only see those that are resolved. Use \'puppet module list --tree\' to see information about modules", ',
     u'        "Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at [\\"/etc/puppet/modules/ssh/manifests/server.pp\\", 12]:[\\"/var/lib/tripleo-config/puppet_step_config.pp\\", 28]", ',
     u'        "Warning: ModuleLoader: module \'timezone\' has unresolved dependencies - it will only see those that are resolved. Use \'puppet module list --tree\' to see information about modules"',
     u'    ]',
     u'}',
     u'',
     u'TASK [Run docker-puppet tasks (generate config)] *******************************',
     u'ok: [192.168.24.14]']

Comment 1 Thomas Hervé 2018-12-03 08:01:15 UTC
From the logs:

"Error: rabbitmqctl status | grep -F \\"{rabbit,\\" returned 1 instead of one of [0]", 

But there is also a mysql error (?).

Comment 4 Damien Ciabrini 2018-12-03 18:03:58 UTC
I could log into the environment and while I'm not fluent with the minor update procedure, something looks wrong to me:

[root@controller-0 ~]# docker ps | head
CONTAINER ID        IMAGE                                                                             COMMAND                  CREATED             STATUS                          PORTS               NAMES
3d95989ad915        docker-registry.engineering.redhat.com/rhosp12/openstack-mariadb:2018-11-28.1     "kolla_start"            2 days ago          Up 2 days                                           clustercheck
ec25bd7ce8ca        docker-registry.engineering.redhat.com/rhosp12/openstack-memcached:2018-11-28.1   "/bin/bash -c 'sou..."   2 days ago          Up 2 days                                           memcached
b38a3fb23270        192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd:2018-11-03.1                   "kolla_start"            3 days ago          Restarting (1) 11 hours ago                         gnocchi_statsd
b5aba06e00c7        192.168.24.1:8787/rhosp12/openstack-gnocchi-api:2018-11-03.1                      "kolla_start"            3 days ago          Up 2 days                                           gnocchi_api
4761442b1e96        192.168.24.1:8787/rhosp12/openstack-gnocchi-metricd:2018-11-03.1                  "kolla_start"            3 days ago          Up 2 days                                           gnocchi_metricd
[...]

In fact, when doing so, the docker base image used for the pacemaker resources changes, and not only the docker tag.

This is by design not supposed to work, as only the image tag can be updated in a rolling fashion across many controller node with pacemaker resources. The way we achieve this is by using a floating tag 'pcmklatest' that maps to the new docker image on the node that is being updated.

I'm not sure this is a use case that we want to support, but currently at least, this the minor update of pacemaker resources expects that only the tag part of the docker image changes.

Andrew, could you re-run your minor update test and make sure to keep the base image unchanged, and report whether this works as expected or not?

Comment 6 awaugama 2018-12-04 22:02:21 UTC
Hey Damien,

Unfortunately trying with only the tag supplied caused the haproxy image to fail being pulled.  Here's the relevant steps after a fresh install of the 11/3 puddle.  Please let me know if there's anything else you'd like me to try, this is blocking an urgent bug.  I'll keep the system intact in case you'd like to look at it.

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud container image prepare --tag 2018-11-28.1 --env-file=/home/stack/overcloud_images.yaml
container_images:
- imagename: registry.access.redhat.com/rhosp12/openstack-aodh-api:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-aodh-evaluator:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-aodh-listener:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-aodh-notifier:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-ceilometer-central:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-ceilometer-compute:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-ceilometer-notification:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-cron:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-glance-api:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-gnocchi-api:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-gnocchi-metricd:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-gnocchi-statsd:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-heat-api-cfn:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-heat-api:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-heat-engine:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-horizon:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-iscsid:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-keystone:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-mariadb:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-memcached:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-api:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-compute:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-conductor:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-consoleauth:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-libvirt:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-novncproxy:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-placement-api:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-nova-scheduler:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-panko-api:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-rabbitmq:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-redis:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-swift-account:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-swift-container:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-swift-object:2018-11-28.1
- imagename: registry.access.redhat.com/rhosp12/openstack-swift-proxy-server:2018-11-28.1

(undercloud) [stack@undercloud-0 ~]$ sudo yum install skopeo

(undercloud) [stack@undercloud-0 ~]$ skopeo inspect --tls-verify=false docker://docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1
{
    "Name": "docker-registry.engineering.redhat.com/rhosp12/openstack-haproxy",
    "Digest": "sha256:ac1dc586fc7614ff9e861dba76ffcb7b579276818fbc37794e823d3ab6dfdcb5",
    "RepoTags": [
        "12.0-20180522.1",
        "12.0-20180727.1",
        "12.0-20180928.1",
        "12.0-20181025.1",
        "12.0-20181103.1",
        "2017-11-28.3",
        "2017-12-01.4",
        "20180122.1",
        "2018-07-13.1",
        "2018-02-27.1",
        "2018-04-02.3",
        "2018-04-05.1",
        "2018-01-24.2",
        "2018-11-02.1",
        "ga",
        "2017-11-09.2",
        "2018-01-11.1",
        "12.0-20180405.1",
        "12.0-20171127.1",
        "12.0-20171201.1",
        "12.0-20180124.1",
        "2018-05-19.1",
        "2018-07-17.1",
        "2018-08-10.1",
        "12.0-20171118.1",
        "12.0-20180815.1",
        "2017-11-14.4",
        "2017-11-23.3",
        "2018-01-22.1",
        "12.0-20170717.1",
        "12.0-20180307.1",
        "12.0-20181029.1",
        "12.0-20180104.2",
        "12.0-20180402.1",
        "2018-10-29.1",
        "latest",
        "12.0-20181128.1",
        "12.0-20181009.1",
        "12.0-20171123.3",
        "12.0-20180813.1",
        "12.0-20181102.1",
        "2017-11-27.14",
        "2018-01-04.1",
        "2018-04-20.1",
        "12.0-20171122.1",
        "2018-08-13.1",
        "beta",
        "12.0-20180222.1",
        "2018-07-27.1",
        "2017-11-03.4",
        "2018-04-02.1",
        "2018-04-03.1",
        "2018-09-10.2",
        "12.0-20181114.1",
        "12.0-20180910.1",
        "12.0-20171128.1",
        "12.0-20180227.1",
        "12.0-20180529.1",
        "12.0-20180717.1",
        "12.0-20171103.1",
        "2018-11-28.1",
        "12.0-20180420.1",
        "2018-03-09.2",
        "2018-10-09.1",
        "2017-11-22.7",
        "12.0-20180112.1",
        "2018-01-23.1",
        "2018-03-21.1",
        "passed_phase2",
        "12.0-20171129.1",
        "2017-11-15.2",
        "12.0-20180122.1",
        "12.0-20181002.1",
        "2018-02-23.1",
        "2018-10-02.1",
        "12.0-20180110.1",
        "2018-01-04.2",
        "2017-07-17.1",
        "2017-11-29.2",
        "2018-01-26.2",
        "2018-03-07.2",
        "12.0-20171122.7",
        "2017-11-16.4",
        "2018-03-19.3",
        "2018-05-22.1",
        "2018-09-21.1",
        "passed_phase1",
        "12.0-20180319.1",
        "12.0-20180403.1",
        "12.0-20180625.2",
        "12.0-20180713.1",
        "12.0-20180810.1",
        "2017-11-27.5",
        "2017-12-02.3",
        "2017-12-02.4",
        "12.0-20180113.1",
        "2018-06-25.2",
        "2018-08-08.1",
        "2018-09-28.1",
        "2018-01-16.2",
        "2018-07-19.1",
        "12.0-20180719.1",
        "2018-01-13.1",
        "2018-02-27.4",
        "2018-02-27.5",
        "2018-11-14.1",
        "12.0-20180808.1",
        "2018-10-25.2",
        "2018-03-10.1",
        "2018-05-29.1",
        "2017-11-21.5",
        "2017-11-20.1",
        "2017-12-06.2",
        "2018-01-13.3",
        "2018-08-15.1",
        "2018-11-03.1",
        "12.0-20180921.1",
        "12.0-20180519.1",
        "2018-01-10.1",
        "12.0-20180309.1"
    ],
    "Created": "2018-11-30T04:49:49.924324Z",
    "DockerVersion": "1.13.1",
    "Labels": {
        "Kolla-SHA": "5.0.0-39-g6f1b947b",
        "architecture": "x86_64",
        "authoritative-source-url": "registry.access.redhat.com",
        "build-date": "2018-11-30T04:45:11.992194",
        "com.redhat.build-host": "cpt-0001.osbs.prod.upshift.rdu2.redhat.com",
        "com.redhat.component": "openstack-haproxy-container",
        "description": "Red Hat OpenStack Platform 12.0 haproxy",
        "distribution-scope": "public",
        "io.k8s.description": "Red Hat OpenStack Platform 12.0 haproxy",
        "io.k8s.display-name": "Red Hat OpenStack Platform 12.0 haproxy",
        "io.openshift.tags": "rhosp osp openstack osp-12.0",
        "kolla_version": "stable/pike",
        "name": "rhosp12/openstack-haproxy",
        "release": "20181128.1",
        "summary": "Red Hat OpenStack Platform 12.0 haproxy",
        "tripleo-common_version": "7.6.3-23-g4891cfe",
        "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp12/openstack-haproxy/images/12.0-20181128.1",
        "vcs-ref": "5cc8d441405684e3606689ba6a8da838836bc89f",
        "vcs-type": "git",
        "vendor": "Red Hat, Inc.",
        "version": "12.0",
        "version-release": "12.0-20181128.1"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:9a1bea865f798d0e4f2359bd39ec69110369e3a1131aba6eb3cbf48707fdf92d",
        "sha256:602125c154e3e132db63d8e6479c5c93a64cbfd3a5ced509de73891ff7102643",
        "sha256:eb8caf2926b79a94049da862960e8031cc39870563c44b6f1619248aa8800ffc",
        "sha256:5258d116dadc93e5057835bc5b6ac7942008a169c2468c8af6bc75bb74916984"
    ]
}

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --init-minor-update --container-registry-file /home/stack/overcloud_images.yaml 

 Stack overcloud UPDATE_COMPLETE 

Heat stack update init on overcloud complete.
Started Mistral Workflow tripleo.package_update.v1.get_config. Execution ID: 40e09dbc-81d3-4d5e-939e-9717be607439
Waiting for messages on queue 'tripleo' with no timeout.
Success
Init minor update on stack overcloud complete.

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud update stack --nodes Controller

 u'TASK [Pull latest Haproxy images] **********************************************',
 u'fatal: [192.168.24.15]: FAILED! => {"changed": true, "cmd": ["docker", "pull", "registry.access.redhat.com/rhosp12/openstack-haproxy:2018-11-28.1"], "delta": "0:00:02.732051", "end": "2018-12-04 21:58:48.093039", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-12-04 21:58:45.360988", "stderr": "error parsing HTTP 404 response body: invalid character \'F\' looking for beginning of value: \\"File not found.\\\\\\"\\"", "stderr_lines": ["error parsing HTTP 404 response body: invalid character \'F\' looking for beginning of value: \\"File not found.\\\\\\"\\""], "stdout": "Trying to pull repository registry.access.redhat.com/rhosp12/openstack-haproxy ... ", "stdout_lines": ["Trying to pull repository registry.access.redhat.com/rhosp12/openstack-haproxy ... "]}',
 u'',
 u'PLAY RECAP *********************************************************************',
 u'192.168.24.15              : ok=39   changed=19   unreachable=0    failed=1   ',
 u'']
Minor update failed with: {u'status': u'FAILED', u'execution': {u'name': u'tripleo.package_update.v1.update_nodes', u'created_at': u'2018-12-04 21:56:32', u'id': u'd9c6479d-2161-4096-9466-a16597a534d5', u'params': {u'namespace': u'', u'env': {}}, u'input': {u'inventory_file': u"[undercloud]\nlocalhost\n\n[undercloud:vars]\nusername = admin\novercloud_keystone_url = http://10.0.0.107:5000/v2.0\nproject_name = admin\nundercloud_service_list = ['openstack-nova-compute', 'openstack-heat-engine', 'openstack-ironic-conductor', 'openstack-swift-container', 'openstack-swift-object', 'openstack-mistral-engine']\novercloud_horizon_url = http://10.0.0.107:80/dashboard\nos_auth_token = gAAAAABcBvgL8psG5rjYTgnMxsXCOF8vSmnhxCIkhVc2Exh4Z3JN61LAHfyvGiQlEEsWiz-JRS0LoOnkIUPaM89rYgUdtUd6XX9eqmx0duDPs54aJV5pReNzjLzRKNwsV89mzpas7kaKOcM7a55pEnfoVHjOOyZZWjIs5aOODjrYi-hb6XXwO-A\novercloud_admin_password = bdFu2BaFfpPZRFPu9N6EfxKs6\nauth_url = http://192.168.24.1:5000/\nansible_connection = local\nundercloud_swift_url = http://192.168.24.1:8080/v1/AUTH_884e302336274f1d80df63acb0e8cd12\nplan = overcloud\n\n[controller-0]\n192.168.24.15\n\n[controller-0:vars]\ndeploy_server_id = 807b1da6-cc24-4f3b-a226-c87377b34847\n\n[Controller:vars]\nrole_name = Controller\nansible_ssh_user = heat-admin\nbootstrap_server_id = 807b1da6-cc24-4f3b-a226-c87377b34847\n\n[Controller:children]\ncontroller-0\n\n[compute-0]\n192.168.24.9\n\n[compute-0:vars]\ndeploy_server_id = 2febf52a-03d7-4525-a00a-f3dcab3d5c3e\n\n[Compute:vars]\nrole_name = Compute\nansible_ssh_user = heat-admin\nbootstrap_server_id = 807b1da6-cc24-4f3b-a226-c87377b34847\n\n[Compute:children]\ncompute-0\n\n[overcloud:children]\nCompute\nController\n\n[aodh_evaluator:vars]\nansible_ssh_user = heat-admin\n\n[aodh_evaluator:children]\nController\n\n[kernel:vars]\nansible_ssh_user = heat-admin\n\n[kernel:children]\nCompute\nController\n\n[neutron_metadata:vars]\nansible_ssh_user = heat-admin\n\n[neutron_metadata:children]\nController\n\n[pacemaker:vars]\nansible_ssh_user = heat-admin\n\n[pacemaker:children]\nController\n\n[nova_placement:vars]\nansible_ssh_user = heat-admin\n\n[nova_placement:children]\nController\n\n[snmp:vars]\nansible_ssh_user = heat-admin\n\n[snmp:children]\nCompute\nController\n\n[heat_api:vars]\nansible_ssh_user = heat-admin\n\n[heat_api:children]\nController\n\n[cinder_api:vars]\nansible_ssh_user = heat-admin\n\n[cinder_api:children]\nController\n\n[neutron_l3:vars]\nansible_ssh_user = heat-admin\n\n[neutron_l3:children]\nController\n\n[swift_proxy:vars]\nansible_ssh_user = heat-admin\n\n[swift_proxy:children]\nController\n\n[aodh_listener:vars]\nansible_ssh_user = heat-admin\n\n[aodh_listener:children]\nController\n\n[swift_ringbuilder:vars]\nansible_ssh_user = heat-admin\n\n[swift_ringbuilder:children]\nController\n\n[neutron_dhcp:vars]\nansible_ssh_user = heat-admin\n\n[neutron_dhcp:children]\nController\n\n[gnocchi_api:vars]\nansible_ssh_user = heat-admin\n\n[gnocchi_api:children]\nController\n\n[timezone:vars]\nansible_ssh_user = heat-admin\n\n[timezone:children]\nCompute\nController\n\n[ceilometer_agent_central:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_agent_central:children]\nController\n\n[heat_api_cloudwatch_disabled:vars]\nansible_ssh_user = heat-admin\n\n[heat_api_cloudwatch_disabled:children]\nController\n\n[aodh_notifier:vars]\nansible_ssh_user = heat-admin\n\n[aodh_notifier:children]\nController\n\n[tripleo_firewall:vars]\nansible_ssh_user = heat-admin\n\n[tripleo_firewall:children]\nCompute\nController\n\n[swift_storage:vars]\nansible_ssh_user = heat-admin\n\n[swift_storage:children]\nController\n\n[redis:vars]\nansible_ssh_user = heat-admin\n\n[redis:children]\nController\n\n[gnocchi_statsd:vars]\nansible_ssh_user = heat-admin\n\n[gnocchi_statsd:children]\nController\n\n[iscsid:vars]\nansible_ssh_user = heat-admin\n\n[iscsid:children]\nCompute\nController\n\n[nova_conductor:vars]\nansible_ssh_user = heat-admin\n\n[nova_conductor:children]\nController\n\n[mysql_client:vars]\nansible_ssh_user = heat-admin\n\n[mysql_client:children]\nCompute\nController\n\n[nova_consoleauth:vars]\nansible_ssh_user = heat-admin\n\n[nova_consoleauth:children]\nController\n\n[glance_api:vars]\nansible_ssh_user = heat-admin\n\n[glance_api:children]\nController\n\n[keystone:vars]\nansible_ssh_user = heat-admin\n\n[keystone:children]\nController\n\n[cinder_volume:vars]\nansible_ssh_user = heat-admin\n\n[cinder_volume:children]\nController\n\n[ceilometer_collector_disabled:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_collector_disabled:children]\nController\n\n[ceilometer_agent_notification:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_agent_notification:children]\nController\n\n[memcached:vars]\nansible_ssh_user = heat-admin\n\n[memcached:children]\nController\n\n[mongodb_disabled:vars]\nansible_ssh_user = heat-admin\n\n[mongodb_disabled:children]\nController\n\n[neutron_plugin_ml2:vars]\nansible_ssh_user = heat-admin\n\n[neutron_plugin_ml2:children]\nCompute\nController\n\n[nova_api:vars]\nansible_ssh_user = heat-admin\n\n[nova_api:children]\nController\n\n[aodh_api:vars]\nansible_ssh_user = heat-admin\n\n[aodh_api:children]\nController\n\n[nova_metadata:vars]\nansible_ssh_user = heat-admin\n\n[nova_metadata:children]\nController\n\n[heat_engine:vars]\nansible_ssh_user = heat-admin\n\n[heat_engine:children]\nController\n\n[ntp:vars]\nansible_ssh_user = heat-admin\n\n[ntp:children]\nCompute\nController\n\n[ceilometer_expirer_disabled:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_expirer_disabled:children]\nController\n\n[ceilometer_api_disabled:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_api_disabled:children]\nController\n\n[nova_migration_target:vars]\nansible_ssh_user = heat-admin\n\n[nova_migration_target:children]\nCompute\n\n[cinder_scheduler:vars]\nansible_ssh_user = heat-admin\n\n[cinder_scheduler:children]\nController\n\n[gnocchi_metricd:vars]\nansible_ssh_user = heat-admin\n\n[gnocchi_metricd:children]\nController\n\n[tripleo_packages:vars]\nansible_ssh_user = heat-admin\n\n[tripleo_packages:children]\nCompute\nController\n\n[nova_scheduler:vars]\nansible_ssh_user = heat-admin\n\n[nova_scheduler:children]\nController\n\n[nova_compute:vars]\nansible_ssh_user = heat-admin\n\n[nova_compute:children]\nCompute\n\n[logrotate_crond:vars]\nansible_ssh_user = heat-admin\n\n[logrotate_crond:children]\nCompute\nController\n\n[neutron_ovs_agent:vars]\nansible_ssh_user = heat-admin\n\n[neutron_ovs_agent:children]\nCompute\nController\n\n[haproxy:vars]\nansible_ssh_user = heat-admin\n\n[haproxy:children]\nController\n\n[sshd:vars]\nansible_ssh_user = heat-admin\n\n[sshd:children]\nCompute\nController\n\n[mysql:vars]\nansible_ssh_user = heat-admin\n\n[mysql:children]\nController\n\n[ceilometer_agent_compute:vars]\nansible_ssh_user = heat-admin\n\n[ceilometer_agent_compute:children]\nCompute\n\n[nova_libvirt:vars]\nansible_ssh_user = heat-admin\n\n[nova_libvirt:children]\nCompute\n\n[rabbitmq:vars]\nansible_ssh_user = heat-admin\n\n[rabbitmq:children]\nController\n\n[tuned:vars]\nansible_ssh_user = heat-admin\n\n[tuned:children]\nCompute\nController\n\n[panko_api:vars]\nansible_ssh_user = heat-admin\n\n[panko_api:children]\nController\n\n[horizon:vars]\nansible_ssh_user = heat-admin\n\n[horizon:children]\nController\n\n[neutron_api:vars]\nansible_ssh_user = heat-admin\n\n[neutron_api:children]\nController\n\n[ca_certs:vars]\nansible_ssh_user = heat-admin\n\n[ca_certs:children]\nCompute\nController\n\n[heat_api_cfn:vars]\nansible_ssh_user = heat-admin\n\n[heat_api_cfn:children]\nController\n\n[docker:vars]\nansible_ssh_user = heat-admin\n\n[docker:children]\nCompute\nController\n\n[nova_vnc_proxy:vars]\nansible_ssh_user = heat-admin\n\n[nova_vnc_proxy:children]\nController\n\n[clustercheck:vars]\nansible_ssh_user = heat-admin\n\n[clustercheck:children]\nController\n\n", u'playbook': u'update_steps_playbook.yaml', u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False'}, u'module_path': u'/usr/share/ansible-modules', u'nodes': u'Controller', u'node_user': u'heat-admin', u'ansible_queue_name': u'update'}, u'spec': {u'tasks': {u'node_update': {u'name': u'node_update', u'on-error': u'node_update_failed', u'on-success': [{u'node_update_passed': u'<% task().result.returncode = 0 %>'}, {u'node_update_failed': u'<% task().result.returncode != 0 %>'}], u'publish': {u'output': u'<% task(node_update).result %>'}, u'version': u'2.0', u'action': u'tripleo.ansible-playbook', u'input': {u'remote_user': u'<% $.node_user %>', u'become_user': u'root', u'ssh_private_key': u'<% $.private_key %>', u'verbosity': 0, u'queue_name': u'<% $.ansible_queue_name %>', u'extra_env_variables': u'<% $.ansible_extra_env_variables %>', u'inventory': u'<% $.inventory_file %>', u'execution_id': u'<% execution().id %>', u'module_path': u'<% $.module_path %>', u'become': True, u'trash_output': True, u'limit_hosts': u'<% $.nodes %>', u'playbook': u'<% $.tmp_path %>/<% $.playbook %>'}, u'type': u'direct'}, u'get_private_key': {u'name': u'get_private_key', u'on-success': u'node_update', u'publish': {u'private_key': u'<% task(get_private_key).result %>'}, u'version': u'2.0', u'action': u'tripleo.validations.get_privkey', u'type': u'direct'}, u'node_update_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_failed', u'publish': {u'status': u'FAILED', u'message': u'Failed to update nodes - <% $.nodes %>, please see the logs.'}, u'on-success': u'notify_zaqar'}, u'node_update_passed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_passed', u'publish': {u'status': u'SUCCESS', u'message': u'Updated nodes - <% $.nodes %>'}, u'on-success': u'notify_zaqar'}, u'notify_zaqar': {u'retry': u'count=5 delay=1', u'name': u'notify_zaqar', u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}], u'version': u'2.0', u'action': u'zaqar.queue_post', u'input': {u'queue_name': u'<% $.ansible_queue_name %>', u'messages': {u'body': {u'type': u'tripleo.package_update.v1.update_nodes', u'payload': {u'status': u'<% $.status %>', u'execution': u'<% execution() %>'}}}}, u'type': u'direct'}, u'download_config': {u'name': u'download_config', u'on-error': u'node_update_failed', u'on-success': u'get_private_key', u'publish': {u'tmp_path': u'<% task(download_config).result %>'}, u'version': u'2.0', u'action': u'tripleo.config.download_config', u'type': u'direct'}}, u'name': u'update_nodes', u'tags': [u'tripleo-common-managed'], u'version': u'2.0', u'input': [{u'node_user': u'heat-admin'}, u'nodes', u'playbook', u'inventory_file', {u'ansible_queue_name': u'tripleo'}, {u'module_path': u'/usr/share/ansible-modules'}, {u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False'}}], u'description': u'Take a container and perform an update nodes by nodes'}}}

Comment 7 Damien Ciabrini 2018-12-05 16:59:21 UTC
Hmm it looks like the image you're trying to access is not available, when I try that on your undercloud I get:

(undercloud) [stack@undercloud-0 ~]$ docker pull registry.access.redhat.com/rhosp12/openstack-aodh-api:2018-11-28.1
Trying to pull repository registry.access.redhat.com/rhosp12/openstack-aodh-api ... 
error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\""

This seems to be the error that you're hitting when the update runs.

Comment 8 awaugama 2018-12-13 15:26:13 UTC
Hey Raviv,

Do you have any input on this bug?  RHOS 12 went out last week but we still need to verify this process works.

Thanks,
Andrew

Comment 9 awaugama 2018-12-21 14:38:04 UTC
I was able to get the update working, see linked bug for steps.  Having said that, closing this as Notabug

Comment 10 Raviv Bar-Tal 2018-12-25 11:17:05 UTC
Ack, 
As osp12 is EOL I will not try to reproduce this BZ.
Thanks


Note You need to log in before you can comment on or make changes to this bug.