Bug 1667584
| Summary: | Upgrade OSP13->OSP14 fails during converge: "[ERROR]: The python-notario library is missing. Please install it on the node" | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | openstack-tripleo-common | Assignee: | Adriano Petrich <apetrich> |
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 14.0 (Rocky) | CC: | bfournie, ccamacho, dbecker, jfrancoa, mburns, morazi, slinaber, yprokule |
| Target Milestone: | z1 | Keywords: | Triaged, ZStream |
| Target Release: | 14.0 (Rocky) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-common-9.4.1-0.20181012010884.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-03-18 13:03:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The undercloud has: ceph-ansible-3.2.3-1.el7cp.noarch Similar to https://bugzilla.redhat.com/show_bug.cgi?id=1654851, which should have been resolved by https://code.engineering.redhat.com/gerrit/#/c/157654/ in openstack-tripleo-common-9.4.1-0.20181012010884.el7ost Just to confirm, this should be fixed now by https://review.openstack.org/#/c/623962/ If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to -. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0446 |
Upgrade OSP13->OSP14 fails during converge: "[ERROR]: The python-notario library is missing. Please install it on the node" Environment: instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch openstack-tripleo-heat-templates-9.0.1-0.20181013060901.el7ost.noarch Steps to reproduce: 1. Attempt to upgrade a composable role deployment, that was deployed on OSP13 with: openstack overcloud deploy --templates \ --libvirt-type kvm \ -e /home/stack/templates/nodes_data.yaml \ -r /home/stack/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml \ -e /home/stack/templates/ironic.yaml \ -e /home/stack/templates/flat_networks.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \ -e /home/stack/inject-trust-anchor-hiera.yaml \ -e /home/stack/docker-images.yaml (undercloud) [stack@undercloud-0 ~]$ cat /home/stack/templates/nodes_data.yaml parameter_defaults: ControllerCount: '3' OvercloudControlFlavor: 'controller' ComputeCount: '2' OvercloudComputeFlavor: 'compute' NtpServer: ["clock.redhat.com","clock2.redhat.com"] CephStorageCount: 3 OvercloudCephStorageFlavor: ceph IronicConductorCount: 2 OvercloudIronicConductorFlavor: ironrole (undercloud) [stack@undercloud-0 ~]$ cat /home/stack/roles_data.yaml ############################################################################### # File generated by TripleO ############################################################################### ############################################################################### # Role: Controller # ############################################################################### - name: Controller description: | Controller role that has all the controler services loaded and handles Database, Messaging and Network functions. CountDefault: 1 tags: - primary - controller networks: - External - InternalApi - Storage - StorageMgmt - Tenant # For systems with both IPv4 and IPv6, you may specify a gateway network for # each, such as ['ControlPlane', 'External'] default_route_networks: ['External'] HostnameFormatDefault: '%stackname%-controller-%index%' # Deprecated & backward-compatible values (FIXME: Make parameters consistent) # Set uses_deprecated_params to True if any deprecated params are used. uses_deprecated_params: True deprecated_param_extraconfig: 'controllerExtraConfig' deprecated_param_flavor: 'OvercloudControlFlavor' deprecated_param_image: 'controllerImage' deprecated_nic_config_name: 'controller.yaml' ServicesDefault: - OS::TripleO::Services::Aide - OS::TripleO::Services::AodhApi - OS::TripleO::Services::AodhEvaluator - OS::TripleO::Services::AodhListener - OS::TripleO::Services::AodhNotifier - OS::TripleO::Services::AuditD - OS::TripleO::Services::BarbicanApi - OS::TripleO::Services::BarbicanBackendSimpleCrypto - OS::TripleO::Services::BarbicanBackendDogtag - OS::TripleO::Services::BarbicanBackendKmip - OS::TripleO::Services::BarbicanBackendPkcs11Crypto - OS::TripleO::Services::CACerts - OS::TripleO::Services::CeilometerAgentCentral - OS::TripleO::Services::CeilometerAgentNotification - OS::TripleO::Services::CephExternal - OS::TripleO::Services::CephMds - OS::TripleO::Services::CephMgr - OS::TripleO::Services::CephMon - OS::TripleO::Services::CephRbdMirror - OS::TripleO::Services::CephRgw - OS::TripleO::Services::CertmongerUser - OS::TripleO::Services::CinderApi - OS::TripleO::Services::CinderBackendDellPs - OS::TripleO::Services::CinderBackendDellSc - OS::TripleO::Services::CinderBackendDellEMCUnity - OS::TripleO::Services::CinderBackendDellEMCVMAXISCSI - OS::TripleO::Services::CinderBackendDellEMCVNX - OS::TripleO::Services::CinderBackendDellEMCXTREMIOISCSI - OS::TripleO::Services::CinderBackendNetApp - OS::TripleO::Services::CinderBackendScaleIO - OS::TripleO::Services::CinderBackendVRTSHyperScale - OS::TripleO::Services::CinderBackup - OS::TripleO::Services::CinderHPELeftHandISCSI - OS::TripleO::Services::CinderScheduler - OS::TripleO::Services::CinderVolume - OS::TripleO::Services::Clustercheck - OS::TripleO::Services::Collectd - OS::TripleO::Services::Congress - OS::TripleO::Services::Docker - OS::TripleO::Services::Ec2Api - OS::TripleO::Services::Etcd - OS::TripleO::Services::ExternalSwiftProxy - OS::TripleO::Services::Fluentd - OS::TripleO::Services::GlanceApi - OS::TripleO::Services::GlanceRegistry - OS::TripleO::Services::GnocchiApi - OS::TripleO::Services::GnocchiMetricd - OS::TripleO::Services::GnocchiStatsd - OS::TripleO::Services::HAproxy - OS::TripleO::Services::ContainerImagePrepare - OS::TripleO::Services::HeatApi - OS::TripleO::Services::HeatApiCloudwatch - OS::TripleO::Services::HeatApiCfn - OS::TripleO::Services::HeatEngine - OS::TripleO::Services::Horizon - OS::TripleO::Services::Ipsec - OS::TripleO::Services::IronicApi - OS::TripleO::Services::Iscsid - OS::TripleO::Services::Keepalived - OS::TripleO::Services::Kernel - OS::TripleO::Services::Keystone - OS::TripleO::Services::LoginDefs - OS::TripleO::Services::ManilaApi - OS::TripleO::Services::ManilaBackendCephFs - OS::TripleO::Services::ManilaBackendIsilon - OS::TripleO::Services::ManilaBackendNetapp - OS::TripleO::Services::ManilaBackendUnity - OS::TripleO::Services::ManilaBackendVNX - OS::TripleO::Services::ManilaBackendVMAX - OS::TripleO::Services::ManilaScheduler - OS::TripleO::Services::ManilaShare - OS::TripleO::Services::Memcached - OS::TripleO::Services::MistralApi - OS::TripleO::Services::MistralEngine - OS::TripleO::Services::MistralExecutor - OS::TripleO::Services::MistralEventEngine - OS::TripleO::Services::MongoDb - OS::TripleO::Services::MySQL - OS::TripleO::Services::MySQLClient - OS::TripleO::Services::NeutronApi - OS::TripleO::Services::NeutronBgpVpnApi - OS::TripleO::Services::NeutronSfcApi - OS::TripleO::Services::NeutronCorePlugin - OS::TripleO::Services::NeutronDhcpAgent - OS::TripleO::Services::NeutronL2gwAgent - OS::TripleO::Services::NeutronL2gwApi - OS::TripleO::Services::NeutronL3Agent - OS::TripleO::Services::NeutronLbaasv2Agent - OS::TripleO::Services::NeutronLinuxbridgeAgent - OS::TripleO::Services::NeutronMetadataAgent - OS::TripleO::Services::NeutronML2FujitsuCfab - OS::TripleO::Services::NeutronML2FujitsuFossw - OS::TripleO::Services::NeutronOvsAgent - OS::TripleO::Services::NeutronVppAgent - OS::TripleO::Services::NovaApi - OS::TripleO::Services::NovaConductor - OS::TripleO::Services::NovaConsoleauth - OS::TripleO::Services::NovaIronic - OS::TripleO::Services::NovaMetadata - OS::TripleO::Services::NovaPlacement - OS::TripleO::Services::NovaScheduler - OS::TripleO::Services::NovaVncProxy - OS::TripleO::Services::Ntp - OS::TripleO::Services::ContainersLogrotateCrond - OS::TripleO::Services::OctaviaApi - OS::TripleO::Services::OctaviaDeploymentConfig - OS::TripleO::Services::OctaviaHealthManager - OS::TripleO::Services::OctaviaHousekeeping - OS::TripleO::Services::OctaviaWorker - OS::TripleO::Services::OpenDaylightApi - OS::TripleO::Services::OpenDaylightOvs - OS::TripleO::Services::OVNDBs - OS::TripleO::Services::OVNController - OS::TripleO::Services::Pacemaker - OS::TripleO::Services::PankoApi - OS::TripleO::Services::OsloMessagingNotify - OS::TripleO::Services::OsloMessagingRpc - OS::TripleO::Services::Redis - OS::TripleO::Services::Rhsm - OS::TripleO::Services::RsyslogSidecar - OS::TripleO::Services::SaharaApi - OS::TripleO::Services::SaharaEngine - OS::TripleO::Services::Securetty - OS::TripleO::Services::SensuClient - OS::TripleO::Services::SkydiveAgent - OS::TripleO::Services::SkydiveAnalyzer - OS::TripleO::Services::Snmp - OS::TripleO::Services::Sshd - OS::TripleO::Services::SwiftProxy - OS::TripleO::Services::SwiftDispersion - OS::TripleO::Services::SwiftRingBuilder - OS::TripleO::Services::SwiftStorage - OS::TripleO::Services::Tacker - OS::TripleO::Services::Timezone - OS::TripleO::Services::TripleoFirewall - OS::TripleO::Services::TripleoPackages - OS::TripleO::Services::Tuned - OS::TripleO::Services::Vpp - OS::TripleO::Services::Zaqar - OS::TripleO::Services::Ptp ############################################################################### # Role: IronicConductor # ############################################################################### - name: IronicConductor description: | Ironic Conductor node role HostnameFormatDefault: '%stackname%-ironic-%index%' networks: - InternalApi ServicesDefault: - OS::TripleO::Services::Aide - OS::TripleO::Services::AuditD - OS::TripleO::Services::CACerts - OS::TripleO::Services::CertmongerUser - OS::TripleO::Services::Collectd - OS::TripleO::Services::Docker - OS::TripleO::Services::Fluentd - OS::TripleO::Services::Ipsec - OS::TripleO::Services::IronicConductor - OS::TripleO::Services::IronicPxe - OS::TripleO::Services::Kernel - OS::TripleO::Services::LoginDefs - OS::TripleO::Services::MySQLClient - OS::TripleO::Services::Ntp - OS::TripleO::Services::ContainersLogrotateCrond - OS::TripleO::Services::Rhsm - OS::TripleO::Services::SensuClient - OS::TripleO::Services::Snmp - OS::TripleO::Services::Timezone - OS::TripleO::Services::TripleoFirewall - OS::TripleO::Services::TripleoPackages - OS::TripleO::Services::Tuned ############################################################################### # Role: Compute # ############################################################################### - name: Compute description: | Basic Compute Node role CountDefault: 1 networks: - InternalApi - Tenant - Storage HostnameFormatDefault: '%stackname%-novacompute-%index%' # Deprecated & backward-compatible values (FIXME: Make parameters consistent) # Set uses_deprecated_params to True if any deprecated params are used. uses_deprecated_params: True deprecated_param_image: 'NovaImage' deprecated_param_extraconfig: 'NovaComputeExtraConfig' deprecated_param_metadata: 'NovaComputeServerMetadata' deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints' deprecated_param_ips: 'NovaComputeIPs' deprecated_server_resource_name: 'NovaCompute' deprecated_nic_config_name: 'compute.yaml' disable_upgrade_deployment: True ServicesDefault: - OS::TripleO::Services::Aide - OS::TripleO::Services::AuditD - OS::TripleO::Services::CACerts - OS::TripleO::Services::CephClient - OS::TripleO::Services::CephExternal - OS::TripleO::Services::CertmongerUser - OS::TripleO::Services::Collectd - OS::TripleO::Services::ComputeCeilometerAgent - OS::TripleO::Services::ComputeNeutronCorePlugin - OS::TripleO::Services::ComputeNeutronL3Agent - OS::TripleO::Services::ComputeNeutronMetadataAgent - OS::TripleO::Services::ComputeNeutronOvsAgent - OS::TripleO::Services::Docker - OS::TripleO::Services::Fluentd - OS::TripleO::Services::Ipsec - OS::TripleO::Services::Iscsid - OS::TripleO::Services::Kernel - OS::TripleO::Services::LoginDefs - OS::TripleO::Services::MySQLClient - OS::TripleO::Services::NeutronBgpVpnBagpipe - OS::TripleO::Services::NeutronLinuxbridgeAgent - OS::TripleO::Services::NeutronVppAgent - OS::TripleO::Services::NovaCompute - OS::TripleO::Services::NovaLibvirt - OS::TripleO::Services::NovaMigrationTarget - OS::TripleO::Services::Ntp - OS::TripleO::Services::ContainersLogrotateCrond - OS::TripleO::Services::OpenDaylightOvs - OS::TripleO::Services::Rhsm - OS::TripleO::Services::RsyslogSidecar - OS::TripleO::Services::Securetty - OS::TripleO::Services::SensuClient - OS::TripleO::Services::SkydiveAgent - OS::TripleO::Services::Snmp - OS::TripleO::Services::Sshd - OS::TripleO::Services::Timezone - OS::TripleO::Services::TripleoFirewall - OS::TripleO::Services::TripleoPackages - OS::TripleO::Services::Tuned - OS::TripleO::Services::Vpp - OS::TripleO::Services::OVNController - OS::TripleO::Services::OVNMetadataAgent - OS::TripleO::Services::Ptp ############################################################################### # Role: CephStorage # ############################################################################### - name: CephStorage description: | Ceph OSD Storage node role networks: - Storage - StorageMgmt uses_deprecated_params: False deprecated_nic_config_name: 'ceph-storage.yaml' ServicesDefault: - OS::TripleO::Services::Aide - OS::TripleO::Services::AuditD - OS::TripleO::Services::CACerts - OS::TripleO::Services::CephOSD - OS::TripleO::Services::CertmongerUser - OS::TripleO::Services::Collectd - OS::TripleO::Services::Docker - OS::TripleO::Services::Fluentd - OS::TripleO::Services::Ipsec - OS::TripleO::Services::Kernel - OS::TripleO::Services::LoginDefs - OS::TripleO::Services::MySQLClient - OS::TripleO::Services::Ntp - OS::TripleO::Services::ContainersLogrotateCrond - OS::TripleO::Services::Rhsm - OS::TripleO::Services::RsyslogSidecar - OS::TripleO::Services::Securetty - OS::TripleO::Services::SensuClient - OS::TripleO::Services::Snmp - OS::TripleO::Services::Sshd - OS::TripleO::Services::Timezone - OS::TripleO::Services::TripleoFirewall - OS::TripleO::Services::TripleoPackages - OS::TripleO::Services::Tuned - OS::TripleO::Services::Ptp Result: The upgrade fails during converge step in set_fact_is_atomic task. 019-01-18 10:50:02 | "TASK [set_fact is_atomic] ******************************************************", 2019-01-18 10:50:02 | "Friday 18 January 2019 10:49:56 -0500 (0:00:00.726) 0:00:30.877 ******** ", 2019-01-18 10:50:02 | "ok: [overcloud-controller-2] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | "ok: [overcloud-controller-1] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | "ok: [overcloud-controller-0] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | "ok: [overcloud-cephstorage-1] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | "ok: [overcloud-cephstorage-0] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | "ok: [overcloud-cephstorage-2] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | "ok: [overcloud-novacompute-1] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | "ok: [overcloud-novacompute-0] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 2019-01-18 10:50:02 | " [ERROR]: The python-notario library is missing. Please install it on the node", 2019-01-18 10:50:02 | "you are running ceph-ansible to continue.", 2019-01-18 10:50:02 | "The python-notario library is missing. Please install it on the node you are running ceph-ansible to continue." 2019-01-18 10:50:02 | ] 2019-01-18 10:50:02 | } 2019-01-18 10:50:02 | 2019-01-18 10:50:02 | NO MORE HOSTS LEFT ************************************************************* 2019-01-18 10:50:02 | 2019-01-18 10:50:02 | PLAY RECAP ********************************************************************* 2019-01-18 10:50:02 | overcloud-cephstorage-0 : ok=111 changed=26 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-cephstorage-1 : ok=111 changed=26 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-cephstorage-2 : ok=111 changed=26 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-controller-0 : ok=177 changed=28 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-controller-1 : ok=176 changed=28 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-controller-2 : ok=176 changed=28 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-ironic-0 : ok=120 changed=26 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-ironic-1 : ok=120 changed=26 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-novacompute-0 : ok=133 changed=27 unreachable=0 failed=0 2019-01-18 10:50:02 | overcloud-novacompute-1 : ok=133 changed=27 unreachable=0 failed=0 2019-01-18 10:50:02 | undercloud : ok=37 changed=22 unreachable=0 failed=1 2019-01-18 10:50:02 | 2019-01-18 10:50:02 | Friday 18 January 2019 10:49:57 -0500 (0:00:00.067) 0:12:38.708 ******** 2019-01-18 10:50:02 | =============================================================================== 2019-01-18 10:50:02 | 2019-01-18 10:50:02 | Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log. 2019-01-18 10:50:02 | 2019-01-18 10:50:02.707 558373 ERROR openstack [-] Overcloud configuration failed.: DeploymentError: Overcloud configuration failed. 2019-01-18 10:50:02 | 2019-01-18 10:50:02.715 558373 INFO osc_lib.shell [-] END return value: 1