1667584 – Upgrade OSP13->OSP14 fails during converge: "[ERROR]: The python-notario library is missing. Please install it on the node"

Bug 1667584 - Upgrade OSP13->OSP14 fails during converge: "[ERROR]: The python-notario library is missing. Please install it on the node"

Summary: Upgrade OSP13->OSP14 fails during converge: "[ERROR]: The python-notario libr...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-common
Sub Component:
Version:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	z1
Target Release:	14.0 (Rocky)
Assignee:	Adriano Petrich
QA Contact:	Alexander Chuzhoy
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-01-18 21:46 UTC by Alexander Chuzhoy
Modified:	2019-03-18 13:03 UTC (History)
CC List:	8 users (show)
Fixed In Version:	openstack-tripleo-common-9.4.1-0.20181012010884.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-03-18 13:03:28 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:0446	0	None	None	None	2019-03-18 13:03:37 UTC

Description Alexander Chuzhoy 2019-01-18 21:46:46 UTC

Upgrade OSP13->OSP14 fails during converge: "[ERROR]: The python-notario library is missing. Please install it on the node"


Environment:
instack-undercloud-9.4.1-0.20180928005746.15cda5a.el7ost.noarch
openstack-tripleo-heat-templates-9.0.1-0.20181013060901.el7ost.noarch

Steps to reproduce:
1. Attempt to upgrade a composable role deployment, that was deployed on OSP13 with:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /home/stack/templates/nodes_data.yaml \
-r /home/stack/roles_data.yaml \
-e  /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml \
-e /home/stack/templates/ironic.yaml \
-e /home/stack/templates/flat_networks.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml \
-e /home/stack/inject-trust-anchor-hiera.yaml \
-e /home/stack/docker-images.yaml


(undercloud) [stack@undercloud-0 ~]$ cat /home/stack/templates/nodes_data.yaml
parameter_defaults:
    ControllerCount: '3'
    OvercloudControlFlavor: 'controller'
    ComputeCount: '2'
    OvercloudComputeFlavor: 'compute'
    NtpServer: ["clock.redhat.com","clock2.redhat.com"]
    CephStorageCount: 3
    OvercloudCephStorageFlavor: ceph
    IronicConductorCount: 2
    OvercloudIronicConductorFlavor: ironrole


(undercloud) [stack@undercloud-0 ~]$ cat /home/stack/roles_data.yaml
###############################################################################
# File generated by TripleO
###############################################################################
###############################################################################
# Role: Controller                                                            #
###############################################################################
- name: Controller
  description: |
    Controller role that has all the controler services loaded and handles
    Database, Messaging and Network functions.
  CountDefault: 1
  tags:
    - primary
    - controller
  networks:
    - External
    - InternalApi
    - Storage
    - StorageMgmt
    - Tenant
  # For systems with both IPv4 and IPv6, you may specify a gateway network for
  # each, such as ['ControlPlane', 'External']
  default_route_networks: ['External']
  HostnameFormatDefault: '%stackname%-controller-%index%'
  # Deprecated & backward-compatible values (FIXME: Make parameters consistent)
  # Set uses_deprecated_params to True if any deprecated params are used.
  uses_deprecated_params: True
  deprecated_param_extraconfig: 'controllerExtraConfig'
  deprecated_param_flavor: 'OvercloudControlFlavor'
  deprecated_param_image: 'controllerImage'
  deprecated_nic_config_name: 'controller.yaml'
  ServicesDefault:
    - OS::TripleO::Services::Aide
    - OS::TripleO::Services::AodhApi
    - OS::TripleO::Services::AodhEvaluator
    - OS::TripleO::Services::AodhListener
    - OS::TripleO::Services::AodhNotifier
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::BarbicanApi
    - OS::TripleO::Services::BarbicanBackendSimpleCrypto
    - OS::TripleO::Services::BarbicanBackendDogtag
    - OS::TripleO::Services::BarbicanBackendKmip
    - OS::TripleO::Services::BarbicanBackendPkcs11Crypto
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CeilometerAgentCentral
    - OS::TripleO::Services::CeilometerAgentNotification
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CephMds
    - OS::TripleO::Services::CephMgr
    - OS::TripleO::Services::CephMon
    - OS::TripleO::Services::CephRbdMirror
    - OS::TripleO::Services::CephRgw
    - OS::TripleO::Services::CertmongerUser
    - OS::TripleO::Services::CinderApi
    - OS::TripleO::Services::CinderBackendDellPs
    - OS::TripleO::Services::CinderBackendDellSc
    - OS::TripleO::Services::CinderBackendDellEMCUnity
    - OS::TripleO::Services::CinderBackendDellEMCVMAXISCSI
    - OS::TripleO::Services::CinderBackendDellEMCVNX
    - OS::TripleO::Services::CinderBackendDellEMCXTREMIOISCSI
    - OS::TripleO::Services::CinderBackendNetApp
    - OS::TripleO::Services::CinderBackendScaleIO
    - OS::TripleO::Services::CinderBackendVRTSHyperScale
    - OS::TripleO::Services::CinderBackup
    - OS::TripleO::Services::CinderHPELeftHandISCSI
    - OS::TripleO::Services::CinderScheduler
    - OS::TripleO::Services::CinderVolume
    - OS::TripleO::Services::Clustercheck
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::Congress
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::Ec2Api
    - OS::TripleO::Services::Etcd
    - OS::TripleO::Services::ExternalSwiftProxy
    - OS::TripleO::Services::Fluentd
    - OS::TripleO::Services::GlanceApi
    - OS::TripleO::Services::GlanceRegistry
    - OS::TripleO::Services::GnocchiApi
    - OS::TripleO::Services::GnocchiMetricd
    - OS::TripleO::Services::GnocchiStatsd
    - OS::TripleO::Services::HAproxy
    - OS::TripleO::Services::ContainerImagePrepare
    - OS::TripleO::Services::HeatApi
    - OS::TripleO::Services::HeatApiCloudwatch
    - OS::TripleO::Services::HeatApiCfn
    - OS::TripleO::Services::HeatEngine
    - OS::TripleO::Services::Horizon
    - OS::TripleO::Services::Ipsec
    - OS::TripleO::Services::IronicApi
    - OS::TripleO::Services::Iscsid
    - OS::TripleO::Services::Keepalived
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Keystone
    - OS::TripleO::Services::LoginDefs
    - OS::TripleO::Services::ManilaApi
    - OS::TripleO::Services::ManilaBackendCephFs
    - OS::TripleO::Services::ManilaBackendIsilon
    - OS::TripleO::Services::ManilaBackendNetapp
    - OS::TripleO::Services::ManilaBackendUnity
    - OS::TripleO::Services::ManilaBackendVNX
    - OS::TripleO::Services::ManilaBackendVMAX
    - OS::TripleO::Services::ManilaScheduler
    - OS::TripleO::Services::ManilaShare
    - OS::TripleO::Services::Memcached
    - OS::TripleO::Services::MistralApi
    - OS::TripleO::Services::MistralEngine
    - OS::TripleO::Services::MistralExecutor
    - OS::TripleO::Services::MistralEventEngine
    - OS::TripleO::Services::MongoDb
    - OS::TripleO::Services::MySQL
    - OS::TripleO::Services::MySQLClient
    - OS::TripleO::Services::NeutronApi
    - OS::TripleO::Services::NeutronBgpVpnApi
    - OS::TripleO::Services::NeutronSfcApi
    - OS::TripleO::Services::NeutronCorePlugin
    - OS::TripleO::Services::NeutronDhcpAgent
    - OS::TripleO::Services::NeutronL2gwAgent
    - OS::TripleO::Services::NeutronL2gwApi
    - OS::TripleO::Services::NeutronL3Agent
    - OS::TripleO::Services::NeutronLbaasv2Agent
    - OS::TripleO::Services::NeutronLinuxbridgeAgent
    - OS::TripleO::Services::NeutronMetadataAgent
    - OS::TripleO::Services::NeutronML2FujitsuCfab
    - OS::TripleO::Services::NeutronML2FujitsuFossw
    - OS::TripleO::Services::NeutronOvsAgent
    - OS::TripleO::Services::NeutronVppAgent
    - OS::TripleO::Services::NovaApi
    - OS::TripleO::Services::NovaConductor
    - OS::TripleO::Services::NovaConsoleauth
    - OS::TripleO::Services::NovaIronic
    - OS::TripleO::Services::NovaMetadata
    - OS::TripleO::Services::NovaPlacement
    - OS::TripleO::Services::NovaScheduler
    - OS::TripleO::Services::NovaVncProxy
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::ContainersLogrotateCrond
    - OS::TripleO::Services::OctaviaApi
    - OS::TripleO::Services::OctaviaDeploymentConfig
    - OS::TripleO::Services::OctaviaHealthManager
    - OS::TripleO::Services::OctaviaHousekeeping
    - OS::TripleO::Services::OctaviaWorker
    - OS::TripleO::Services::OpenDaylightApi
    - OS::TripleO::Services::OpenDaylightOvs
    - OS::TripleO::Services::OVNDBs
    - OS::TripleO::Services::OVNController
    - OS::TripleO::Services::Pacemaker
    - OS::TripleO::Services::PankoApi
    - OS::TripleO::Services::OsloMessagingNotify
    - OS::TripleO::Services::OsloMessagingRpc
    - OS::TripleO::Services::Redis
    - OS::TripleO::Services::Rhsm
    - OS::TripleO::Services::RsyslogSidecar
    - OS::TripleO::Services::SaharaApi
    - OS::TripleO::Services::SaharaEngine
    - OS::TripleO::Services::Securetty
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::SkydiveAgent
    - OS::TripleO::Services::SkydiveAnalyzer
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Sshd
    - OS::TripleO::Services::SwiftProxy
    - OS::TripleO::Services::SwiftDispersion
    - OS::TripleO::Services::SwiftRingBuilder
    - OS::TripleO::Services::SwiftStorage
    - OS::TripleO::Services::Tacker
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::Tuned
    - OS::TripleO::Services::Vpp
    - OS::TripleO::Services::Zaqar
    - OS::TripleO::Services::Ptp
###############################################################################
# Role: IronicConductor                                                       #
###############################################################################
- name: IronicConductor
  description: |
    Ironic Conductor node role
  HostnameFormatDefault: '%stackname%-ironic-%index%'
  networks:
    - InternalApi
  ServicesDefault:
    - OS::TripleO::Services::Aide
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CertmongerUser
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::Fluentd
    - OS::TripleO::Services::Ipsec
    - OS::TripleO::Services::IronicConductor
    - OS::TripleO::Services::IronicPxe
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::LoginDefs
    - OS::TripleO::Services::MySQLClient
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::ContainersLogrotateCrond
    - OS::TripleO::Services::Rhsm
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::Tuned
###############################################################################
# Role: Compute                                                               #
###############################################################################
- name: Compute
  description: |
    Basic Compute Node role
  CountDefault: 1
  networks:
    - InternalApi
    - Tenant
    - Storage
  HostnameFormatDefault: '%stackname%-novacompute-%index%'
  # Deprecated & backward-compatible values (FIXME: Make parameters consistent)
  # Set uses_deprecated_params to True if any deprecated params are used.
  uses_deprecated_params: True
  deprecated_param_image: 'NovaImage'
  deprecated_param_extraconfig: 'NovaComputeExtraConfig'
  deprecated_param_metadata: 'NovaComputeServerMetadata'
  deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints'
  deprecated_param_ips: 'NovaComputeIPs'
  deprecated_server_resource_name: 'NovaCompute'
  deprecated_nic_config_name: 'compute.yaml'
  disable_upgrade_deployment: True
  ServicesDefault:
    - OS::TripleO::Services::Aide
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephClient
    - OS::TripleO::Services::CephExternal
    - OS::TripleO::Services::CertmongerUser
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::ComputeCeilometerAgent
    - OS::TripleO::Services::ComputeNeutronCorePlugin
    - OS::TripleO::Services::ComputeNeutronL3Agent
    - OS::TripleO::Services::ComputeNeutronMetadataAgent
    - OS::TripleO::Services::ComputeNeutronOvsAgent
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::Fluentd
    - OS::TripleO::Services::Ipsec
    - OS::TripleO::Services::Iscsid
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::LoginDefs
    - OS::TripleO::Services::MySQLClient
    - OS::TripleO::Services::NeutronBgpVpnBagpipe
    - OS::TripleO::Services::NeutronLinuxbridgeAgent
    - OS::TripleO::Services::NeutronVppAgent
    - OS::TripleO::Services::NovaCompute
    - OS::TripleO::Services::NovaLibvirt
    - OS::TripleO::Services::NovaMigrationTarget
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::ContainersLogrotateCrond
    - OS::TripleO::Services::OpenDaylightOvs
    - OS::TripleO::Services::Rhsm
    - OS::TripleO::Services::RsyslogSidecar
    - OS::TripleO::Services::Securetty
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::SkydiveAgent
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Sshd
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::Tuned
    - OS::TripleO::Services::Vpp
    - OS::TripleO::Services::OVNController
    - OS::TripleO::Services::OVNMetadataAgent
    - OS::TripleO::Services::Ptp
###############################################################################
# Role: CephStorage                                                           #
###############################################################################
- name: CephStorage
  description: |
    Ceph OSD Storage node role
  networks:
    - Storage
    - StorageMgmt
  uses_deprecated_params: False
  deprecated_nic_config_name: 'ceph-storage.yaml'
  ServicesDefault:
    - OS::TripleO::Services::Aide
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::CephOSD
    - OS::TripleO::Services::CertmongerUser
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::Fluentd
    - OS::TripleO::Services::Ipsec
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::LoginDefs
    - OS::TripleO::Services::MySQLClient
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::ContainersLogrotateCrond
    - OS::TripleO::Services::Rhsm
    - OS::TripleO::Services::RsyslogSidecar
    - OS::TripleO::Services::Securetty
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Sshd
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::Tuned
    - OS::TripleO::Services::Ptp


Result:
 The upgrade fails during converge step in set_fact_is_atomic task.

019-01-18 10:50:02 |         "TASK [set_fact is_atomic] ******************************************************", 
2019-01-18 10:50:02 |         "Friday 18 January 2019  10:49:56 -0500 (0:00:00.726)       0:00:30.877 ******** ", 
2019-01-18 10:50:02 |         "ok: [overcloud-controller-2] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         "ok: [overcloud-controller-1] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         "ok: [overcloud-controller-0] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         "ok: [overcloud-cephstorage-1] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         "ok: [overcloud-cephstorage-0] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         "ok: [overcloud-cephstorage-2] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         "ok: [overcloud-novacompute-1] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         "ok: [overcloud-novacompute-0] => {\"ansible_facts\": {\"is_atomic\": false}, \"changed\": false}", 
2019-01-18 10:50:02 |         " [ERROR]: The python-notario library is missing. Please install it on the node", 
2019-01-18 10:50:02 |         "you are running ceph-ansible to continue.", 
2019-01-18 10:50:02 |         "The python-notario library is missing. Please install it on the node you are running ceph-ansible to continue."
2019-01-18 10:50:02 |     ]
2019-01-18 10:50:02 | }
2019-01-18 10:50:02 | 
2019-01-18 10:50:02 | NO MORE HOSTS LEFT *************************************************************
2019-01-18 10:50:02 | 
2019-01-18 10:50:02 | PLAY RECAP *********************************************************************
2019-01-18 10:50:02 | overcloud-cephstorage-0    : ok=111  changed=26   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-cephstorage-1    : ok=111  changed=26   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-cephstorage-2    : ok=111  changed=26   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-controller-0     : ok=177  changed=28   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-controller-1     : ok=176  changed=28   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-controller-2     : ok=176  changed=28   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-ironic-0         : ok=120  changed=26   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-ironic-1         : ok=120  changed=26   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-novacompute-0    : ok=133  changed=27   unreachable=0    failed=0   
2019-01-18 10:50:02 | overcloud-novacompute-1    : ok=133  changed=27   unreachable=0    failed=0   
2019-01-18 10:50:02 | undercloud                 : ok=37   changed=22   unreachable=0    failed=1   
2019-01-18 10:50:02 | 
2019-01-18 10:50:02 | Friday 18 January 2019  10:49:57 -0500 (0:00:00.067)       0:12:38.708 ******** 
2019-01-18 10:50:02 | =============================================================================== 
2019-01-18 10:50:02 | 
2019-01-18 10:50:02 | Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log.
2019-01-18 10:50:02 | 2019-01-18 10:50:02.707 558373 ERROR openstack [-] Overcloud configuration failed.: DeploymentError: Overcloud configuration failed.
2019-01-18 10:50:02 | 2019-01-18 10:50:02.715 558373 INFO osc_lib.shell [-] END return value: 1

Comment 2 Alexander Chuzhoy 2019-01-18 22:10:03 UTC

The undercloud has: ceph-ansible-3.2.3-1.el7cp.noarch

Comment 3 Bob Fournier 2019-01-18 22:45:51 UTC

Similar to https://bugzilla.redhat.com/show_bug.cgi?id=1654851, which should have been resolved by https://code.engineering.redhat.com/gerrit/#/c/157654/ in openstack-tripleo-common-9.4.1-0.20181012010884.el7ost

Comment 4 Carlos Camacho 2019-02-04 13:22:46 UTC

Just to confirm, this should be fixed now by https://review.openstack.org/#/c/623962/

Comment 9 Mikey Ariel 2019-02-20 12:44:31 UTC

If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Comment 11 errata-xmlrpc 2019-03-18 13:03:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0446

Note You need to log in before you can comment on or make changes to this bug.