Description of problem: On a stack update, the only way that customers can change the deployed Octavia public key is by updating OctaviaAmphoraSshKeyFile and also changing OctaviaAmphoraSshKeyName If OctaviaAmphoraSshKeyName does not change, then the public key is not updated. Ideally, the ansible playbooks should detect either a change in filename or a change in the MD5 key and then take action to delete and recreate the keypair with the new public key. Additional info: I'm trying to update the keyfile with an existing key already in place: ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack --os-username octavia --os-password Um3tNxVtrwghuhpuyPFRZWHDR --os-project-name service keypair list +------------------+-------------------------------------------------+ | Name | Fingerprint | +------------------+-------------------------------------------------+ | octavia-test-key | be:09:3f:0e:ca:4f:3d:07:2d:43:3b:b1:45:87:02:15 | +------------------+-------------------------------------------------+ ~~~ ~~~ (overcloud) [stack@undercloud-0 ~]$ ssh-keygen -f ~/.ssh/subsequent-test Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/stack/.ssh/subsequent-test. Your public key has been saved in /home/stack/.ssh/subsequent-test.pub. The key fingerprint is: SHA256:vI+ThW4JiCUrVtKTJ+DNZvFL3Qr87xB+ZR6jbxy52yU stack.local The key's randomart image is: +---[RSA 2048]----+ | | | . . | |. = = . . | | o.@.* o . | | ===.+.S. =. | |..o ..ooo.*oo | |.. =o*..oE . | | O+.+. o | | .oooo.. | +----[SHA256]-----+ (overcloud) [stack@undercloud-0 ~]$ cp /home/stack/.ssh/subsequent-test.pub /tmp (overcloud) [stack@undercloud-0 ~]$ chmod 644 /tmp/subsequent-test.pub (overcloud) [stack@undercloud-0 ~]$ ssh-keygen -l -E md5 -f /tmp/subsequent-test.pub 2048 MD5:16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 stack.local (RSA) ~~~ Updating the configuration: ~~~ (undercloud) [stack@undercloud-0 ~]$ tail -n 2 octavia/network-environment.yaml OctaviaAmphoraSshKeyName: 'octavia-test-key' # change octavia-ssh-key for the actual key name OctaviaAmphoraSshKeyFile: /tmp/subsequent-test.pub ~~~ And kicking off the deployment: ~~~ (undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh ~~~ Now after this test, I can see that the key is *not* updated in the database: ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack keypair list +--------+-------------------------------------------------+ | Name | Fingerprint | +--------+-------------------------------------------------+ | id_rsa | f9:f3:c8:27:3d:0f:ba:b5:0a:13:ef:e4:d6:5d:af:6f | +--------+-------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack --os-username octavia --os-password Um3tNxVtrwghuhpuyPFRZWHDR --os-project-name service keypair list +------------------+-------------------------------------------------+ | Name | Fingerprint | +------------------+-------------------------------------------------+ | octavia-test-key | be:09:3f:0e:ca:4f:3d:07:2d:43:3b:b1:45:87:02:15 | +------------------+-------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ ssh-keygen -l -E md5 -f /tmp/subsequent-test.pub 2048 MD5:16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 stack.local (RSA) (overcloud) [stack@undercloud-0 ~]$ ~~~ However, I now changed this to: ~~~ (undercloud) [stack@undercloud-0 ~]$ tail -n 2 octavia/network-environment.yaml OctaviaAmphoraSshKeyName: 'octavia-test-key-b' # change octavia-ssh-key for the actual key name OctaviaAmphoraSshKeyFile: /tmp/subsequent-test.pub (undercloud) [stack@undercloud-0 ~]$ ~~~ And redeployed: ~~~ (undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh ~~~ And then this shows: ~~~ [root@overcloud-controller-0 ~]# grep amp_ssh /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf -B2 # Upload the ssh key as the service_auth user described elsewhere in this config. # Leaving this variable blank will install no ssh key on the amphora. # amp_ssh_key_name = amp_ssh_key_name=octavia-test-key-b ~~~ ~~~ (overcloud) [stack@undercloud-0 ~]$ openstack --os-username octavia --os-password Um3tNxVtrwghuhpuyPFRZWHDR --os-project-name service keypair list +--------------------+-------------------------------------------------+ | Name | Fingerprint | +--------------------+-------------------------------------------------+ | octavia-test-key | be:09:3f:0e:ca:4f:3d:07:2d:43:3b:b1:45:87:02:15 | | octavia-test-key-b | 16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 | +--------------------+-------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ ~~~ And I can log into a new LB with the new key: (undercloud) [stack@undercloud-0 ~]$ . overcloudrc (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer delete lb1 Unable to locate lb1 in loadbalancers (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --name lb1 --vip-subnet-id private1-subnet sleep 60+---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2020-01-17T17:48:25 | | description | | | flavor | | | id | 7ead6197-cb42-410c-8492-ffed5f74000f | | listeners | | | name | lb1 | | operating_status | OFFLINE | | pools | | | project_id | 34d6b951c06e4d65ba23e669761d0165 | | provider | octavia | | provisioning_status | PENDING_CREATE | | updated_at | None | | vip_address | 192.168.0.110 | | vip_network_id | 7e1c33da-db7d-4003-b962-584e93f7f470 | | vip_port_id | 6b6fe81a-f621-4af7-96de-e7e5a51ef0e8 | | vip_qos_policy_id | None | | vip_subnet_id | 1e44d2df-1acb-4d81-8a5e-3ff22ae7224f | +---------------------+--------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ sleep 60 ^C (overcloud) [stack@undercloud-0 ~]$ LBIP=$(openstack loadbalancer amphora list --loadbalancer lb1 -c lb_network_ip -f value) (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.9 -L 127.0.0.1:2222:$LBIP:22 -N -f Warning: Permanently added '192.168.24.9' (ECDSA) to the list of known hosts. (overcloud) [stack@undercloud-0 ~]$ ssh cloud-user.0.1 -p 2222 Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts. Permission denied (publickey,gssapi-keyex,gssapi-with-mic). (overcloud) [stack@undercloud-0 ~]$ ssh cloud-user.0.1 -p 2222 -i ~/.ssh/ authorized_keys config id_rsa id_rsa.pub subsequent-test subsequent-test.pub test_rsa test_rsa.pub (overcloud) [stack@undercloud-0 ~]$ ssh cloud-user.0.1 -p 2222 -i ~/.ssh/subsequent-test Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts. [cloud-user@amphora-f105c36b-5b99-49d4-8b83-20d0287583ca ~]$ [cloud-user@amphora-f105c36b-5b99-49d4-8b83-20d0287583ca ~]$ exit (overcloud) [stack@undercloud-0 ~]$ ssh-keygen -l -E md5 -f /tmp/subsequent-test.pub 2048 MD5:16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 stack.local (RSA) ~~~
Note that this is problematic as we cannot / should not expect that administrators log into the controllers, retrieve the Octavia credentials and figure out which key names were already used. The operation should be idempotent, meaning that the new key should be pushed when the file name changes (or the MD5 sum of the key, ideally). At the moment, we can only create keys, but not update them with the same operation --> not idempotent
The upstream patch doesn't solve the issue, it fails when the ssh key is updated.
A new patch will fix it: https://review.opendev.org/c/openstack/tripleo-ansible/+/783824 My goal was to backport this patch down to wallaby (OSP17) but I can also try to backport it to train.
https://review.opendev.org/c/openstack/tripleo-ansible/+/783824 cannot be backported easily, but we could use a part of this patch in a downstream-only commit. I need to evaluate it.
We won't fix this issue in OSP16.x because backporting the wallaby/master commit is too risky. But here's a workaround to update the Octavia ssh key: 1. Get octavia user password from the undercloud: [stack@undercloud-0 ~]$ . stackrc (undercloud) [stack@undercloud-0 ~]$ ansible -i /usr/bin/tripleo-ansible-inventory -o -b -m shell -a "crudini --get /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf service_auth password" controller-0 controller-0 | CHANGED | rc=0 | (stdout) zgV0J7CK0S8pqZvp392gLWdiy 2. Update the octavia user's ssh public key on the overcloud using the password: [stack@undercloud-0 ~]$ . overcloudrc (overcloud) [stack@undercloud-0 ~]$ export OCTAVIA_PASSWORD=zgV0J7CK0S8pqZvp392gLWdiy (overcloud) [stack@undercloud-0 ~]$ openstack keypair list --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD +-----------------+-------------------------------------------------+ | Name | Fingerprint | +-----------------+-------------------------------------------------+ | octavia-ssh-key | e7:e2:3d:94:48:cf:f5:8c:5e:54:98:bd:89:a5:29:a9 | +-----------------+-------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack keypair delete octavia-ssh-key --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD (overcloud) [stack@undercloud-0 ~]$ openstack keypair create --public-key ./id_rsa.pub octavia-ssh-key --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD +-------------+-------------------------------------------------+ | Field | Value | +-------------+-------------------------------------------------+ | fingerprint | e2:7b:d5:ae:93:30:f3:69:41:ce:f2:8c:7e:1b:71:a1 | | name | octavia-ssh-key | | type | ssh | | user_id | 18767ff3c12445018c4e108b60f15e1c | +-------------+-------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ openstack keypair list --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD +-----------------+-------------------------------------------------+ | Name | Fingerprint | +-----------------+-------------------------------------------------+ | octavia-ssh-key | e2:7b:d5:ae:93:30:f3:69:41:ce:f2:8c:7e:1b:71:a1 | +-----------------+-------------------------------------------------+