Hey, do we have the logs from that controller? [[1;30m2022-05-23 16:05:54.796051 | 525400f4-e180-641e-7ecb-000000005e12 | WAITING | Wait for puppet host configuration to finish | cell1-controller-0 | 204 retries left Puppet failed on the controller, so we need to see what failed in /var/log/messages there.
Looks like no pacemaker stuff has been installed or configured: [heat-admin@cell1-controller-0 ~]$ sudo pcs status Error: error running crm_mon, is pacemaker running? crm_mon: Error: cluster is not available on this node Which is why that job is failing: May 23 15:27:16 cell1-controller-0 puppet-user[16280]: Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: Exec try 280/360 May 23 15:27:16 cell1-controller-0 puppet-user[16280]: Debug: Exec[wait-for-settle](provider=posix): Executing '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1' May 23 15:27:16 cell1-controller-0 puppet-user[16280]: Debug: Executing: '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1' The roles_data file for the cell0 deployment is missing quite a few of the services we have for standard Controller roles. So I'm not sure if that's causing any issue. But the pacemaker service is failing to start because corosync is failing: [heat-admin@cell1-controller-0 ~]$ sudo systemctl start pacemaker A dependency job for pacemaker.service failed. See 'journalctl -xe' for details. [heat-admin@cell1-controller-0 ~]$ ^start^status sudo systemctl status pacemaker ○ pacemaker.service - Pacemaker High Availability Cluster Manager Loaded: loaded (/usr/lib/systemd/system/pacemaker.service; enabled; vendor preset: disabled) Active: inactive (dead) Docs: man:pacemakerd https://clusterlabs.org/pacemaker/doc/ May 26 11:38:12 cell1-controller-0.redhat.local systemd[1]: Dependency failed for Pacemaker High Availability Cluster Manager. May 26 11:38:12 cell1-controller-0.redhat.local systemd[1]: pacemaker.service: Job pacemaker.service/start failed with result 'dependency'. May 26 11:43:27 cell1-controller-0.redhat.local systemd[1]: Dependency failed for Pacemaker High Availability Cluster Manager. May 26 11:43:27 cell1-controller-0.redhat.local systemd[1]: pacemaker.service: Job pacemaker.service/start failed with result 'dependency'. corosync is failing because there is no corosync.conf file: [heat-admin@cell1-controller-0 ~]$ sudo systemctl start corosync Job for corosync.service failed because the control process exited with error code. See "systemctl status corosync.service" and "journalctl -xeu corosync.service" for details. [heat-admin@cell1-controller-0 ~]$ ^start^status sudo systemctl status corosync × corosync.service - Corosync Cluster Engine Loaded: loaded (/usr/lib/systemd/system/corosync.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2022-05-26 11:44:46 UTC; 7s ago Docs: man:corosync man:corosync.conf man:corosync_overview Process: 49367 ExecStart=/usr/sbin/corosync -f $COROSYNC_OPTIONS (code=exited, status=8) Main PID: 49367 (code=exited, status=8) CPU: 4ms May 26 11:44:46 cell1-controller-0.redhat.local systemd[1]: Starting Corosync Cluster Engine... May 26 11:44:46 cell1-controller-0.redhat.local corosync[49367]: Can't read file /etc/corosync/corosync.conf: No such file or directory May 26 11:44:46 cell1-controller-0.redhat.local systemd[1]: corosync.service: Main process exited, code=exited, status=8/n/a May 26 11:44:46 cell1-controller-0.redhat.local systemd[1]: corosync.service: Failed with result 'exit-code'. May 26 11:44:46 cell1-controller-0.redhat.local systemd[1]: Failed to start Corosync Cluster Engine. Adding dfg:pidone to assist on this one.
It seems that this is rather related to multi-stack export/import than to multi-cells. Adding DFG:DF.
The export files created from "overcloud export" are now created automatically during the deployment. You can find them in the working dir for each stack. E.g., ~/overcloud-deploy/overcloud/overcloud-export.yaml and ~/overcloud-deploy/cell1/cell1-export.yaml. We checked the environment and those files contain the pacemaker data. I'm not sure of the intended difference between "overcloud export" and "cell export" and which one should be used when. That will need to be confirmed with Compute, looks like Martin Schupper wrote most of "cell export" from overcloud_cell.py. overcloud_cell.py is not updated for the ephemeral Heat changes, so it's not going to work, as it's still trying to pass in a Heat client to the export function.
@jparker would you try to use that auto-generated file in ./overcloud_cell_deployment.sh instead of /home/stack/cell1/cell-input.yaml ? I think the latter one is no longer needed for osp17? I'm not sure, maybe cells export CLI is/should still be a thing there
The linked patch may be needed as well: 844636: Update cell export for ephemeral heat | https://review.opendev.org/c/openstack/python-tripleoclient/+/844636
Tentatively assigning Rajesh to this, as we suspect the mechanics of the fix will overlap with https://bugzilla.redhat.com/show_bug.cgi?id=1759007, which he is working on.
Pushing this out of 17.0 for 17.1
Let me clarify, this is a regression that would block multi-cell v2 upgrades.
Know issue for 17.0 GA is tracked at https://bugzilla.redhat.com/show_bug.cgi?id=2120398, this will need a bug fix release note.
*** Bug 2126157 has been marked as a duplicate of this bug. ***
Export was silently failing: $ openstack overcloud cell export -f --output-file /tmp/cell-input.yaml File /home/stack/overcloud-deploy/overcloud/config-download/overcloud/overcloud/group_vars/overcloud.json was not found during export No data returned to export AllNodesExtraMapData from. Cell information exported to /tmp/cell-input.yaml. ... ...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577