Bug 2226812 - Upgrade [OSP16.2 -> OSP17.1] multi-cell SSH Public/Private Key rotation needed to Enable upgrades from 16.x to 17.1
Summary: Upgrade [OSP16.2 -> OSP17.1] multi-cell SSH Public/Private Key rotation neede...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: tripleo-ansible
Version: 17.1 (Wallaby)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z2
: 17.1
Assignee: Andre
QA Contact: Archana Singh
URL:
Whiteboard:
Depends On:
Blocks: 1759007
TreeView+ depends on / blocked
 
Reported: 2023-07-26 16:02 UTC by Juan Badia Payno
Modified: 2024-03-16 04:25 UTC (History)
16 users (show)

Fixed In Version: tripleo-ansible-3.3.1-1.20230518201538.el9ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-16 11:34:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 890200 0 None MERGED Add option to only update a new key on Overcloud 2023-08-30 13:08:19 UTC
OpenStack gerrit 893487 0 None MERGED Fix issue when handling tripleo key 2023-09-11 12:14:00 UTC
Red Hat Issue Tracker OSP-26926 0 None None None 2023-07-26 16:05:39 UTC
Red Hat Issue Tracker OSP-28837 0 None None None 2023-09-19 14:53:44 UTC
Red Hat Issue Tracker OSP-28839 0 None None None 2023-09-19 14:53:46 UTC

Description Juan Badia Payno 2023-07-26 16:02:31 UTC
This bugzilla is a next step of https://bugzilla.redhat.com/show_bug.cgi?id=2151002

The minimum key size for rhel-9 for rsa is 2048, we need to implement a method to enable replace the keys of all the node.

Currently, the method used ssh_key_rotation.yaml [1] it works when there is only one stack.

The current process allows to switch between key with the following command:

ansible-playbook -i tripleo-stack-inventory.yaml ssh_key_rotation.yaml

On the multi-cell environment, there are several inventory files so the playbook is launched for all the inventory files. However the playbook creates a new ssh key pair for every execution, so the connectivity with the first inventory is lost when the second one is executed.

[1] https://github.com/openstack/tripleo-ansible/blob/stable/wallaby/tripleo_ansible/playbooks/ssh_key_rotation.yaml

Comment 1 Juan Badia Payno 2023-07-27 06:59:28 UTC
As a workaround previous to the Undercloud Operative System upgrade:

# This aims to keep the ssh connection from the undercloud to the hypervisor (CI - vbmc issue)
if [ ! -f ~/.ssh/old/id_rsa ]; then
 mkdir ~/.ssh/old/
 cp -a ~/.ssh/id_rsa ~/.ssh/old/
 cp -a ~/.ssh/id_rsa.pub  ~/.ssh/old/
 cat <<'EOF' >> ~/.ssh/config
Host 172.16.0.1 
   StrictHostKeyChecking no
   UserKnownHostsFile=/dev/null
   HostName 172.16.0.1
   User root
   IdentityFile /home/stack/.ssh/old/id_rsa
EOF

fi

# This creates one inventory with all the inventory files and use that inventory to do the ssh key rotation
# this only works for overcloud with less than 10 stacks
if [ ! -f  all_inventory ]; then
 # Create local files for modifiactions
 touch all_inventory
 cp /usr/share/ansible/tripleo-playbooks/ssh_key_rotation.yaml /home/stack/ssh_key_rotation.yaml
           
 # Create one big single inventory file
 STACKS=""
 for i in `ls ~/overcloud-deploy/ | grep -v undercloud`; do
   STACKS="${STACKS},${i}_allovercloud"
   sed -e "s|id00|${i}_id00|g" -e "s|Undercloud|${i}Undercloud|g" -e "s|^allovercloud|${i}allovercloud|g" ~/overcloud-deploy/$i/tripleo-ansible-inventory.yaml >> all_inventory
 done
           
 # Modify the playbook to update all nodes on all overcloud stacks
 REPLACEMENT=`echo $STACKS | cut -c2-`
           
 sed -i -e "s|allovercloud|${REPLACEMENT}|g" /home/stack/ssh_key_rotation.yaml
 ansible-playbook -i all_inventory /home/stack/ssh_key_rotation.yaml
           
fi

Comment 34 Red Hat Bugzilla 2024-03-16 04:25:07 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days


Note You need to log in before you can comment on or make changes to this bug.