Bug 1792598 - OctaviaAmphoraSshKeyFile does not update public key if the filename / content changes
Summary: OctaviaAmphoraSshKeyFile does not update public key if the filename / content...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z8
: 16.1 (Train on RHEL 8.2)
Assignee: Gregory Thiemonge
QA Contact: Bruna Bonguardo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-18 10:08 UTC by Andreas Karis
Modified: 2023-09-07 21:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-29 07:28:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1861031 0 None None None 2020-01-27 20:31:21 UTC
OpenStack gerrit 783824 0 None NEW Generate Octavia ssh key from the octavia playbook 2021-07-16 08:18:42 UTC
Red Hat Issue Tracker OSP-3626 0 None None None 2021-11-29 07:28:11 UTC
Red Hat Knowledge Base (Solution) 3679791 0 None None None 2020-01-18 10:31:43 UTC

Description Andreas Karis 2020-01-18 10:08:56 UTC
Description of problem:
On a stack update, the only way that customers can change the deployed Octavia public key is by updating OctaviaAmphoraSshKeyFile and also changing OctaviaAmphoraSshKeyName

If OctaviaAmphoraSshKeyName does not change, then the public key is not updated. Ideally, the ansible playbooks should detect either a change in filename or a change in the MD5 key and then take action to delete and recreate the keypair with the new public key.

Additional info:

I'm trying to update the keyfile with an existing key already in place:
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack --os-username octavia --os-password Um3tNxVtrwghuhpuyPFRZWHDR --os-project-name service keypair list
+------------------+-------------------------------------------------+
| Name             | Fingerprint                                     |
+------------------+-------------------------------------------------+
| octavia-test-key | be:09:3f:0e:ca:4f:3d:07:2d:43:3b:b1:45:87:02:15 |
+------------------+-------------------------------------------------+
~~~

~~~
(overcloud) [stack@undercloud-0 ~]$ ssh-keygen -f ~/.ssh/subsequent-test
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/stack/.ssh/subsequent-test.
Your public key has been saved in /home/stack/.ssh/subsequent-test.pub.
The key fingerprint is:
SHA256:vI+ThW4JiCUrVtKTJ+DNZvFL3Qr87xB+ZR6jbxy52yU stack.local
The key's randomart image is:
+---[RSA 2048]----+
|                 |
| . .             |
|. = = . .        |
| o.@.* o .       |
|  ===.+.S. =.    |
|..o ..ooo.*oo    |
|..     =o*..oE . |
|        O+.+. o  |
|       .oooo..   |
+----[SHA256]-----+
(overcloud) [stack@undercloud-0 ~]$ cp /home/stack/.ssh/subsequent-test.pub /tmp
(overcloud) [stack@undercloud-0 ~]$ chmod 644 /tmp/subsequent-test.pub 
(overcloud) [stack@undercloud-0 ~]$ ssh-keygen -l -E md5 -f /tmp/subsequent-test.pub 
2048 MD5:16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 stack.local (RSA) 
~~~

Updating the configuration:
~~~
(undercloud) [stack@undercloud-0 ~]$ tail -n 2 octavia/network-environment.yaml
  OctaviaAmphoraSshKeyName: 'octavia-test-key'  # change octavia-ssh-key for the actual key name
  OctaviaAmphoraSshKeyFile: /tmp/subsequent-test.pub
~~~

And kicking off the deployment:
~~~
(undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh 
~~~

Now after this test, I can see that the key is *not* updated in the database:
~~~
(overcloud) [stack@undercloud-0 ~]$ openstack keypair list
+--------+-------------------------------------------------+
| Name   | Fingerprint                                     |
+--------+-------------------------------------------------+
| id_rsa | f9:f3:c8:27:3d:0f:ba:b5:0a:13:ef:e4:d6:5d:af:6f |
+--------+-------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack --os-username octavia --os-password Um3tNxVtrwghuhpuyPFRZWHDR --os-project-name service keypair list
+------------------+-------------------------------------------------+
| Name             | Fingerprint                                     |
+------------------+-------------------------------------------------+
| octavia-test-key | be:09:3f:0e:ca:4f:3d:07:2d:43:3b:b1:45:87:02:15 |
+------------------+-------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ ssh-keygen -l -E md5 -f /tmp/subsequent-test.pub
2048 MD5:16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 stack.local (RSA)
(overcloud) [stack@undercloud-0 ~]$ 
~~~

However, I now changed this to:
~~~
(undercloud) [stack@undercloud-0 ~]$ tail -n 2 octavia/network-environment.yaml
  OctaviaAmphoraSshKeyName: 'octavia-test-key-b'  # change octavia-ssh-key for the actual key name
  OctaviaAmphoraSshKeyFile: /tmp/subsequent-test.pub
(undercloud) [stack@undercloud-0 ~]$ 
~~~

And redeployed:
~~~
(undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh 
~~~

And then this shows:
~~~
[root@overcloud-controller-0 ~]# grep amp_ssh /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf -B2
# Upload the ssh key as the service_auth user described elsewhere in this config.
# Leaving this variable blank will install no ssh key on the amphora.
# amp_ssh_key_name =
amp_ssh_key_name=octavia-test-key-b
~~~

~~~
(overcloud) [stack@undercloud-0 ~]$ openstack --os-username octavia --os-password Um3tNxVtrwghuhpuyPFRZWHDR --os-project-name service keypair list
+--------------------+-------------------------------------------------+
| Name               | Fingerprint                                     |
+--------------------+-------------------------------------------------+
| octavia-test-key   | be:09:3f:0e:ca:4f:3d:07:2d:43:3b:b1:45:87:02:15 |
| octavia-test-key-b | 16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 |
+--------------------+-------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ 
~~~

And I can log into a new LB with the new key:

(undercloud) [stack@undercloud-0 ~]$ . overcloudrc
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer delete lb1
Unable to locate lb1 in loadbalancers
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --name lb1 --vip-subnet-id private1-subnet
sleep 60+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-01-17T17:48:25                  |
| description         |                                      |
| flavor              |                                      |
| id                  | 7ead6197-cb42-410c-8492-ffed5f74000f |
| listeners           |                                      |
| name                | lb1                                  |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 34d6b951c06e4d65ba23e669761d0165     |
| provider            | octavia                              |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 192.168.0.110                        |
| vip_network_id      | 7e1c33da-db7d-4003-b962-584e93f7f470 |
| vip_port_id         | 6b6fe81a-f621-4af7-96de-e7e5a51ef0e8 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 1e44d2df-1acb-4d81-8a5e-3ff22ae7224f |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ sleep 60
^C
(overcloud) [stack@undercloud-0 ~]$ LBIP=$(openstack loadbalancer amphora list --loadbalancer lb1 -c lb_network_ip -f value)
(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin.24.9 -L 127.0.0.1:2222:$LBIP:22 -N -f
Warning: Permanently added '192.168.24.9' (ECDSA) to the list of known hosts.
(overcloud) [stack@undercloud-0 ~]$ ssh cloud-user.0.1 -p 2222
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
(overcloud) [stack@undercloud-0 ~]$ ssh cloud-user.0.1 -p 2222 -i ~/.ssh/
authorized_keys      config               id_rsa               id_rsa.pub           subsequent-test      subsequent-test.pub  test_rsa             test_rsa.pub
(overcloud) [stack@undercloud-0 ~]$ ssh cloud-user.0.1 -p 2222 -i ~/.ssh/subsequent-test
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
[cloud-user@amphora-f105c36b-5b99-49d4-8b83-20d0287583ca ~]$ 
[cloud-user@amphora-f105c36b-5b99-49d4-8b83-20d0287583ca ~]$ exit
(overcloud) [stack@undercloud-0 ~]$ ssh-keygen -l -E md5 -f /tmp/subsequent-test.pub
2048 MD5:16:bb:9a:f4:d2:05:0f:96:b6:5f:42:8e:13:d6:90:47 stack.local (RSA)
~~~

Comment 1 Andreas Karis 2020-01-18 10:10:36 UTC
Note that this is problematic as we cannot / should not expect that administrators log into the controllers, retrieve the Octavia credentials and figure out which key names were already used. The operation should be idempotent, meaning that the new key should be pushed when the file name changes (or the MD5 sum of the key, ideally). At the moment, we can only create keys, but not update them with the same operation --> not idempotent

Comment 5 Gregory Thiemonge 2021-06-21 14:23:46 UTC
The upstream patch doesn't solve the issue, it fails when the ssh key is updated.

Comment 6 Gregory Thiemonge 2021-07-16 08:18:45 UTC
A new patch will fix it: https://review.opendev.org/c/openstack/tripleo-ansible/+/783824
My goal was to backport this patch down to wallaby (OSP17) but I can also try to backport it to train.

Comment 8 Gregory Thiemonge 2021-10-12 13:03:50 UTC
https://review.opendev.org/c/openstack/tripleo-ansible/+/783824 cannot be backported easily, but we could use a part of this patch in a downstream-only commit.
I need to evaluate it.

Comment 9 Gregory Thiemonge 2021-11-29 07:26:53 UTC
We won't fix this issue in OSP16.x because backporting the wallaby/master commit is too risky.

But here's a workaround to update the Octavia ssh key:


1. Get octavia user password from the undercloud:

[stack@undercloud-0 ~]$ . stackrc
(undercloud) [stack@undercloud-0 ~]$ ansible -i /usr/bin/tripleo-ansible-inventory -o -b -m shell -a "crudini --get /var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf service_auth password" controller-0
controller-0 | CHANGED | rc=0 | (stdout) zgV0J7CK0S8pqZvp392gLWdiy

2. Update the octavia user's ssh public key on the overcloud using the password:

[stack@undercloud-0 ~]$ . overcloudrc
(overcloud) [stack@undercloud-0 ~]$ export OCTAVIA_PASSWORD=zgV0J7CK0S8pqZvp392gLWdiy
(overcloud) [stack@undercloud-0 ~]$ openstack keypair list --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD
+-----------------+-------------------------------------------------+
| Name            | Fingerprint                                     |
+-----------------+-------------------------------------------------+
| octavia-ssh-key | e7:e2:3d:94:48:cf:f5:8c:5e:54:98:bd:89:a5:29:a9 |
+-----------------+-------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack keypair delete octavia-ssh-key --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD
(overcloud) [stack@undercloud-0 ~]$ openstack keypair create --public-key ./id_rsa.pub octavia-ssh-key --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD
+-------------+-------------------------------------------------+
| Field       | Value                                           |
+-------------+-------------------------------------------------+
| fingerprint | e2:7b:d5:ae:93:30:f3:69:41:ce:f2:8c:7e:1b:71:a1 |
| name        | octavia-ssh-key                                 |
| type        | ssh                                             |
| user_id     | 18767ff3c12445018c4e108b60f15e1c                |
+-------------+-------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ openstack keypair list --os-project-name service --os-username octavia --os-password $OCTAVIA_PASSWORD
+-----------------+-------------------------------------------------+
| Name            | Fingerprint                                     |
+-----------------+-------------------------------------------------+
| octavia-ssh-key | e2:7b:d5:ae:93:30:f3:69:41:ce:f2:8c:7e:1b:71:a1 |
+-----------------+-------------------------------------------------+


Note You need to log in before you can comment on or make changes to this bug.