Bug 1777971
| Summary: | [OSP14->15] Upgrade fails during ovn pacemaker bundle configuration | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jose Luis Franco <jfrancoa> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Michele Baldessari <michele> |
| Status: | CLOSED ERRATA | QA Contact: | Sasha Smolyak <ssmolyak> |
| Severity: | high | Docs Contact: | |
| Priority: | low | ||
| Version: | 15.0 (Stein) | CC: | dalvarez, ekuris, jfrancoa, mburns, michele, njohnston, skaplons |
| Target Milestone: | z2 | Keywords: | Triaged, ZStream |
| Target Release: | 15.0 (Stein) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-10.6.2-0.20191205160452.7a6dc87.el8ost, puppet-tripleo-10.5.2-0.20191210093332.b392eb6.el8ost.noarch | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-05 12:00:58 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The failing task belongs to the OVN pacemaker managed service, therefore it should be a matter for the OVN Squad. Assigning it to Networking DFG. Jose, do you have an env with this issue that I can take a look at? Ok so the reason for the failure is that ovn_dbs does not have the hiera node override mechanism implemented in order to support the scaleup during upgrades. In fact in Jose's env we see: Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20191128-9-1ik6l4s create failed: Error: unable to set attribute ovn-dbs-role Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Property[ovn-dbs-role-controller-1]/Pcmk_property[property-controller-1-ovn-dbs-role]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20191128-9-1ik6l4s create failed: Error: unable to set attribute ovn-dbs-role Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Bundle[ovn-dbs-bundle]/Pcmk_bundle[ovn-dbs-bundle]: Dependency Pcmk_property[property-controller-1-ovn-dbs-role] has failures: true Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Bundle[ovn-dbs-bundle]/Pcmk_bundle[ovn-dbs-bundle]: Dependency Pcmk_property[property-controller-2-ovn-dbs-role] has failures: true (Note how controller-0 property worked just fine but controller-1/controller-2 failed) that is because it is trying to set a property of nodes that don't exist just yet. So the fix here is to implement the same stuff we have for the other service in ovn_dbs. See https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/pacemaker/database/mysql_bundle.pp#L33: if $pacemaker_master { if (hiera('mysql_short_node_names_override', undef)) { $mysql_short_node_names = hiera('mysql_short_node_names_override') } else { $mysql_short_node_names = hiera('mysql_short_node_names') } https://review.opendev.org/696708 and https://review.opendev.org/696718 are the initial WIP patches Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0643 |
Description of problem: When executing manually the upgrade from OSP14 to OSP15 with ovn configured. The upgrade run step for the first controller failed in the following task: "Error running ['podman', 'run', '--name', 'ovn_dbs_init_bundle', '--label', 'config_id=tripleo_step3', '--label', 'container_name=ovn_dbs_init_bundle', '--label', ' managed_by=paunch', '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"3\", \"file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::prope rty,pacemaker::resource::ip,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation\", \"include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::ovn_dbs_bundle\", \"\"], \"config_volume\": \"ovn_dbs_init_bundle\", \"detach\": false, \"environment\": [\"TRIPLEO_DEPLOY_IDENTIFIER=15749395 09\"], \"image\": \"192.168.24.1:8787/rh-osbs/rhosp15-openstack-ovn-northd:20191114.1\", \"ipc\": \"host\", \"net\": \"host\", \"start_order\": 1, \"user\": \"root\", \"volu mes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchor s:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/cert s/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/contain er_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\"]}', '--conmon-pidfile=/var/run/ov n_dbs_init_bundle.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/ovn_dbs_init_bundle.log', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1574939509', '- -net=host', '--ipc=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca- trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle .crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/ dev/log:/dev/log', '--volume=/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume =/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--cpuset-cpus=0,1,2,3,4,5,6,7', '192.168.24.1:8787/rh-osbs/rhosp15-openstack-ovn-northd:201911 14.1', '/container_puppet_apply.sh', '3', 'file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ip,pacemaker::resource::ocf,pace maker::constraint::order,pacemaker::constraint::colocation', 'include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::ovn_dbs_bundle', '']. [6]", "stdout: Info: Loading facts", [94/1892] "Info: Loading facts", "Notice: Compiled catalog for controller-0.redhat.local in environment production in 0.46 seconds", "Info: Applying configuration version '1574962169'", "Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Property[ovn-dbs-role-controller-0]/Pcmk_property[property-controller-0-ovn-dbs-role]/en sure: created", "Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Bundle[ovn-dbs-bundle]/Pcmk_bundle[ovn-dbs-bundle]: Dependency Pcmk_property[p roperty-controller-1-ovn-dbs-role] has failures: true", "Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Bundle[ovn-dbs-bundle]/Pcmk_bundle[ovn-dbs-bundle]: Dependency Pcmk_property[p roperty-controller-2-ovn-dbs-role] has failures: true", "Info: Class[Tripleo::Profile::Pacemaker::Ovn_dbs_bundle]: Unscheduling all events on Class[Tripleo::Profile::Pacemaker::Ovn_dbs_bundle]", "Info: Creating state file /var/lib/puppet/state/state.yaml", "Notice: Applied catalog in 419.44 seconds", "Changes:", " Total: 1", "Events:", " Success: 1", " Failure: 2", " Total: 3", "Resources:", " Changed: 1", " Failed: 2", " Skipped: 25", " Out of sync: 3", " Total: 30", "Time:", " File line: 0.00", " Config retrieval: 0.52", " Last run: 1574962589", " Pcmk property: 419.30", " Transaction evaluation: 419.43", " Catalog application: 419.44", " Total: 419.44", "Version:", " Config: 1574962169", " Puppet: 5.5.10", "stderr: + STEP=3", "+ TAGS=file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ip,pacemaker::resource::ocf,pacemaker::constraint::order,pa cemaker::constraint::colocation", "+ CONFIG='include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::ovn_dbs_bundle'", "+ EXTRA_ARGS=", "+ '[' -d /tmp/puppet-etc ']'", "+ cp -a /tmp/puppet-etc/auth.conf /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/hiera.yaml.rpmnew /tmp/puppet-etc/hieradata /tmp/puppet-etc/modules /tmp/puppet-etc/pup pet.conf /tmp/puppet-etc/ssl /etc/puppet", "+ echo '{\"step\": 3}'", "+ export FACTER_deployment_type=containers", "+ FACTER_deployment_type=containers", "+ set +e", "+ puppet apply --verbose --detailed-exitcodes --summarize --color=false --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modul es --tags file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ip,pacemaker::resource::ocf,pacemaker::constraint::order,pacemake r::constraint::colocation -e 'noop_resource('\\''package'\\''); include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::ovn_dbs_bundle'", "Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information a bout modules\\n (file & line not available)", "Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5", " (file: /etc/puppet/hiera.yaml)", "Warning: Undefined variable '::deploy_config_name'; \\n (file & line not available)", "Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_language.html\\n (file & line not availa ble)", "Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see informat ion about modules\\n (file & line not available)", "Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20191128-9-1ik6l4s create failed: Error: unable to set attribute ovn-dbs-role", "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Property[ovn-dbs-role-controller-1]/Pcmk_property[property-controller-1-ovn-dbs-role]/ens ure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20191128-9-1ik6l4s create failed: Error: unable to set attribute ovn-dbs-role", "Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20191128-9-ezp74u create failed: Error: unable to set attribute ovn-dbs-role", "Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Property[ovn-dbs-role-controller-2]/Pcmk_property[property-controller-2-ovn-dbs-role]/ens ure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20191128-9-ezp74u create failed: Error: unable to set attribute ovn-dbs-role", "Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Bundle[ovn-dbs-bundle]/Pcmk_bundle[ovn-dbs-bundle]: Skipping because of faile d dependencies", "Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Resource::Ocf[ovndb_servers]/Pcmk_resource[ovndb_servers]: Skipping because of failed $ ependencies", "Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Constraint::Order[ip-172.17.1.15-with-ovndb_servers]/Pcmk_constraint[order-ovn-dbs-bun$ le-ip-172.17.1.15]: Skipping because of failed dependencies", "Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Constraint::Colocation[ip-172.17.1.15-with-ovndb_servers]/Pcmk_constraint[colo-ip-172.$ 7.1.15-ovn-dbs-bundle]: Skipping because of failed dependencies", "+ rc=6", "+ set -e", "+ set +ux", "stdout: 3f1a6ceb5f69727ebd7e5dd0d3f8afb24fc174bdeb7af0bcbb3584db505ed722", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_iscsid.service → /etc/systemd/system/tripleo_iscsid.service.", "Created symlink /etc/systemd/system/timers.target.wants/tripleo_iscsid_healthcheck.timer → /etc/systemd/system/tripleo_iscsid_healthcheck.timer.", "Created symlink /etc/systemd/system/tripleo_iscsid.service.requires/tripleo_iscsid_healthcheck.timer → /etc/systemd/system/tripleo_iscsid_healthcheck.timer.", "stdout: dd4c92ee3e9dfd8cf4d90ecc0fe686a52e6ab6f7cbaa0b5636f1b132f2e3a899", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_keystone.service → /etc/systemd/system/tripleo_keystone.service.", "Created symlink /etc/systemd/system/timers.target.wants/tripleo_keystone_healthcheck.timer → /etc/systemd/system/tripleo_keystone_healthcheck.timer.", "Created symlink /etc/systemd/system/tripleo_keystone.service.requires/tripleo_keystone_healthcheck.timer → /etc/systemd/system/tripleo_keystone_healthcheck.timer.", "stdout: (cellv2) Updating default cell_v2 cell 1742a0a6-afdc-4991-924c-0d94bfec50cb", "stderr: /usr/lib/python3.6/site-packages/pymysql/cursors.py:165: Warning: (139, 'Row size too large (> 8126). Changing some columns to TEXT or BLOB or using ROW_FOR MAT=DYNAMIC or ROW_FORMAT=COMPRESSED may help. In current row format, BLOB prefix of 768 bytes is stored inline.')", " result = self._query(query)", "stdout: c5cd578dbb29dfeb0373db51259b6ab43cfe9c23db10a3b4c378b2c02296077a", "Created symlink /etc/systemd/system/multi-user.target.wants/tripleo_keystone_cron.service → /etc/systemd/system/tripleo_keystone_cron.service.", "Created symlink /etc/systemd/system/timers.target.wants/tripleo_keystone_cron_healthcheck.timer → /etc/systemd/system/tripleo_keystone_cron_healthcheck.timer.", "Created symlink /etc/systemd/system/tripleo_keystone_cron.service.requires/tripleo_keystone_cron_healthcheck.timer → /etc/systemd/system/tripleo_keystone_cron_healt hcheck.timer." ] } NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* controller-0 : ok=256 changed=134 unreachable=0 failed=1 skipped=407 rescued=0 ignored=2 Thursday 28 November 2019 12:36:50 -0500 (0:00:00.071) 0:25:02.685 ***** =============================================================================== The environment is being kept for further analysis if needed. Version-Release number of selected component (if applicable): (undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa |grep tripleo-heat-templates openstack-tripleo-heat-templates-10.6.2-0.20191113030444.fc5ca30.el8ost.noarch (undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa |grep tripleo-puppet openstack-tripleo-puppet-elements-10.3.2-0.20190820220452.5453b89.el8ost.noarch puppet-tripleo-10.5.2-0.20191009160423.fe160dd.el8ost.noarch How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: