Bug 2226812

Summary: Upgrade [OSP16.2 -> OSP17.1] multi-cell SSH Public/Private Key rotation needed to Enable upgrades from 16.x to 17.1
Product: Red Hat OpenStack Reporter: Juan Badia Payno <jbadiapa>
Component: tripleo-ansibleAssignee: Andre <afariasa>
Status: CLOSED CURRENTRELEASE QA Contact: Archana Singh <arcsingh>
Severity: high Docs Contact:
Priority: high    
Version: 17.1 (Wallaby)CC: afariasa, alfrgarc, arcsingh, dwilde, eshames, jagee, jpretori, jschluet, kthakre, mariel, mburns, millevy, pgrist, prgutier, pweeks, yatanaka
Target Milestone: z2Keywords: TestOnly, Triaged
Target Release: 17.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tripleo-ansible-3.3.1-1.20230518201538.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-16 11:34:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1759007    

Description Juan Badia Payno 2023-07-26 16:02:31 UTC 
This bugzilla is a next step of https://bugzilla.redhat.com/show_bug.cgi?id=2151002

The minimum key size for rhel-9 for rsa is 2048, we need to implement a method to enable replace the keys of all the node.

Currently, the method used ssh_key_rotation.yaml [1] it works when there is only one stack.

The current process allows to switch between key with the following command:

ansible-playbook -i tripleo-stack-inventory.yaml ssh_key_rotation.yaml

On the multi-cell environment, there are several inventory files so the playbook is launched for all the inventory files. However the playbook creates a new ssh key pair for every execution, so the connectivity with the first inventory is lost when the second one is executed.

[1] https://github.com/openstack/tripleo-ansible/blob/stable/wallaby/tripleo_ansible/playbooks/ssh_key_rotation.yaml
Comment 1 Juan Badia Payno 2023-07-27 06:59:28 UTC 
As a workaround previous to the Undercloud Operative System upgrade:

# This aims to keep the ssh connection from the undercloud to the hypervisor (CI - vbmc issue)
if [ ! -f ~/.ssh/old/id_rsa ]; then
 mkdir ~/.ssh/old/
 cp -a ~/.ssh/id_rsa ~/.ssh/old/
 cp -a ~/.ssh/id_rsa.pub  ~/.ssh/old/
 cat <<'EOF' >> ~/.ssh/config
Host 172.16.0.1 
   StrictHostKeyChecking no
   UserKnownHostsFile=/dev/null
   HostName 172.16.0.1
   User root
   IdentityFile /home/stack/.ssh/old/id_rsa
EOF

fi

# This creates one inventory with all the inventory files and use that inventory to do the ssh key rotation
# this only works for overcloud with less than 10 stacks
if [ ! -f  all_inventory ]; then
 # Create local files for modifiactions
 touch all_inventory
 cp /usr/share/ansible/tripleo-playbooks/ssh_key_rotation.yaml /home/stack/ssh_key_rotation.yaml
           
 # Create one big single inventory file
 STACKS=""
 for i in `ls ~/overcloud-deploy/ | grep -v undercloud`; do
   STACKS="${STACKS},${i}_allovercloud"
   sed -e "s|id00|${i}_id00|g" -e "s|Undercloud|${i}Undercloud|g" -e "s|^allovercloud|${i}allovercloud|g" ~/overcloud-deploy/$i/tripleo-ansible-inventory.yaml >> all_inventory
 done
           
 # Modify the playbook to update all nodes on all overcloud stacks
 REPLACEMENT=`echo $STACKS | cut -c2-`
           
 sed -i -e "s|allovercloud|${REPLACEMENT}|g" /home/stack/ssh_key_rotation.yaml
 ansible-playbook -i all_inventory /home/stack/ssh_key_rotation.yaml
           
fi
Comment 34 Red Hat Bugzilla 2024-03-16 04:25:07 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days