The existing documentation for Barbican explains in [1] and [2] how to create and validate Glance images signature when booting a new guest in Nova or creating a new volume in Cinder but it does not mention that Glance (image) *snapshots* don't have signatures and boostrap of new guests from snapshots or new volumes from snapshots will fail. We need to inform users about that. We should also mention that the same process documented in chapter 8.2 can be used to sign snapshots (manually) and that doing so will make signature verification for snapshots to work. 1. https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/manage_secrets_with_openstack_key_manager/validate_glance_images 2. https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/manage_secrets_with_openstack_key_manager/validate_images_used_for_volume_creation
1. This BZ is flagged for 16.2. The way I understand the issue, I would expect this to affect all versions of OpenStack, is that the case? 2. I'm proposing the admonition below. Let me know if you feel that this satisfies the needs of the bugzilla: "NOTE: When you configure the Compute service (nova) to check for signed images, instances that you create using unsigned images will fail to boot. If you create an instance from a signed image, and then create a snapshot of that instance, the new image that is the snapshot must also be signed using the same process."
I like the #c6 version better - feels easier to understand for a non-native speaker like me. @Mike: Do you think we could have a QE test for this bug?
Hey Mike! I think the test should be something along the lines of 1) Deploy OpenStack in the most Vanilla configuration possible, but make sure Nova is configured to check for signed images (the doc should be at https://docs.openstack.org/nova/train/user/certificate-validation.html) 2) Boot from an unsigned image: $ nova boot --flavor FLAVOR_ID --image <ID of unsigned image> test-failure 3) Check that 2) failed 4) Create a signed image (see https://docs.openstack.org/glance/train/user/signature.html) 5) Boot from this signed image: $ nova boot --flavor FLAVOR_ID --image <ID of signed image> test-success 6) Check that 5 did not fail 7) Make a snapshot of the instance we created in 5): $ nova stop test-success $ nova image-create --poll test-success test-success-snapshot 8) Try to boot a new instance from the snapshot: $ nova boot --flavor FLAVOR_ID --image test-success-snapshot instance-from-snapshot 9) Check that 8 failed 10) Sign the snapshot 11) Try to boot from the signed snapshot 12) Check that 11 worked
@cyril when nova is set to check, and the containers are restarted, it does not fail the unsigned image boot: (overcloud) [stack@undercloud-0 ~]$ for i in 34 53; do echo "=== 192.168.24.$i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root nova_compute sh -c 'grep ^enable_certificate_validation /etc/nova/nova.conf' 2>/dev/null"; done === 192.168.24.34 Warning: Permanently added '192.168.24.34' (ECDSA) to the list of known hosts. enable_certificate_validation=true Connection to 192.168.24.34 closed. === 192.168.24.53 Warning: Permanently added '192.168.24.53' (ECDSA) to the list of known hosts. enable_certificate_validation=true Connection to 192.168.24.53 closed. (overcloud) [stack@undercloud-0 ~]$ nova list +--------------------------------------+------------------+--------+------------+-------------+------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------+--------+------------+-------------+------------------------------------------+ | 4d1bc472-7c0e-44db-9ec3-859afbc101bf | test-failure-001 | ACTIVE | - | Running | nova=10.0.0.163, 2620:52:0:13b8::1000:2f | | 12e0d17c-4268-4bda-a13b-185eb82108f6 | test-failure-002 | ACTIVE | - | Running | nova=10.0.0.229, 2620:52:0:13b8::1000:11 | +--------------------------------------+------------------+--------+------------+-------------+------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ ...is there a THT i need to deploy instead of just changing the value in nova.conf overlay and restarting the nova_compute containers on the compute nodes? please advise. thank you.
OK, so a few pointers: 1) The documentation states: "As of the 18.0.0 Rocky release, only the libvirt compute driver supports trusted image certification validation". Are you using libvirt? 2) Do you have this in the nova config file: [key_manager] backend = barbican 3) Do you have this in the nova config file: [glance] verify_glance_signatures = True I'm not super familiar with this feature, so if what I wrote above is not enough, we might want to find a Nova dev to help us.
hi Cyril, 1) libvirt is installed on the UC, but not on any of the controllers or computes...is that expected? 2) it's like this: (undercloud) [stack@undercloud-0 ~]$ for i in 34 53; do echo "=== 192.168.24.$i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root nova_compute sh -c 'grep ^backend /etc/nova/nova.conf' 2>/dev/null"; done === 192.168.24.34 Warning: Permanently added '192.168.24.34' (ECDSA) to the list of known hosts. backend=castellan.key_manager.barbican_key_manager.BarbicanKeyManager Connection to 192.168.24.34 closed. === 192.168.24.53 Warning: Permanently added '192.168.24.53' (ECDSA) to the list of known hosts. backend=castellan.key_manager.barbican_key_manager.BarbicanKeyManager Connection to 192.168.24.53 closed. (undercloud) [stack@undercloud-0 ~]$ 3) that's not even an option in the controller:/etc/glance/glance-api.conf file...maybe i need to redeploy the OC with a certain THT to enable it? (undercloud) [stack@undercloud-0 ~]$ for i in 8 42 52; do echo "=== 192.168.24.$i"; ssh -t heat-admin.24.$i "sudo podman exec -it -u root glance_api sh -c 'grep verify_glance_signatures /etc/glance/glance-api.conf' 2>/dev/null"; done === 192.168.24.8 Warning: Permanently added '192.168.24.8' (ECDSA) to the list of known hosts. Connection to 192.168.24.8 closed. === 192.168.24.42 Warning: Permanently added '192.168.24.42' (ECDSA) to the list of known hosts. Connection to 192.168.24.42 closed. === 192.168.24.52 Warning: Permanently added '192.168.24.52' (ECDSA) to the list of known hosts. Connection to 192.168.24.52 closed. (undercloud) [stack@undercloud-0 ~]$ please advise. thank you, Mike
We should have a nova_libvirt container on each compute node. Its configuration reside in /var/lib/config-data/puppet-generated/nova_libvirt/etc/. If we want to enable verify_glance_signatures with tht, the parameter is "VerifyGlanceSignatures: True" As for the the KeyManager settings, it's apparently not presented in THT but we do have something in puppet-nova so it would be easy to integrate with THT. I'll clone this BZ and make the required change. Let me know if that answers the question.
Now that I'm looking more closely at THT, apprently, if you have OS::TripleO::Services::BarbicanClient in your role's ServiceDefault, it should configure nova correctly, except for the VerifyGlanceSignatures which I'll submit a fix.
More instructions for step 7 ("Make a snapshot of the instance"): $ openstack server stop myInstance $ openstack server image create -name myInstanceSnapshot myInstance More details in the official doc: https://docs.openstack.org/nova/train/admin/migrate-instance-with-snapshot.html
Oh, thanks Greg for the input. @Mike: if you download the snapshot, sign it and reupload, then it should be just like working with a "regular" image, so we're getting close to verifying this \o/
I'm retargeting this to 16.1, since it affects this version.
Note to reader is merged and published
@rheslop I suggest revising the last sentence in your note in common/key-manager/proc-validating-image-service-images.adoc to: This is true whether the snapshot is from an instance created with signed images, or an instance booted from a volume created from a signed image.
@Mike: I think Brian is right. Can you reverify this but using instructions at https://docs.openstack.org/glance/train/user/signature.html to the image signing?
If you see the log message mentioned in step #8 of section 8.2, that would be the final step. Just to be clear about what you're checking: When nova is configured to verify glance images, it will refuse to boot an image that it can't validate the signature for. In a deployment, the operator will make sure that the public images all have valid signature properties than nova can use, and a user can boot an instance from that image. When that user wants to create a shapshot of the instance, nova creates the image in Glance, but doesn't set any of the 'img_signature*' properties. So, such a snapshot can't be booted until the user adds the required image properties. So you're basically doing the entire section 8.2, the only difference being that the image is already in Glance (nova put it there). In fact, you may be doing section 8.2 twice: the first time, to add an image that nova will boot; the second time, to make sure that if a user creates a snapshot of the booted instance, the documentation is clear about the steps a user needs to take to make the snapshot bootable.
great ok thank you Brian! --- Boot from an unsigned image: $ nova boot --flavor FLAVOR_ID --image <ID of unsigned image> test-failure Check that step 2 failed Create a signed image (see https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/manage_secrets_with_openstack_key_manager/index#validate_an_image) --- (overcloud) [stack@undercloud-0 ~]$ openssl genrsa -out private_key.pem 1024 Generating RSA private key, 1024 bit long modulus (2 primes) .....+++++ ........+++++ e is 65537 (0x010001) (overcloud) [stack@undercloud-0 ~]$ openssl rsa -pubout -in private_key.pem -out public_key.pem writing RSA key (overcloud) [stack@undercloud-0 ~]$ openssl req -new -key private_key.pem -out cert_request.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:US State or Province Name (full name) []:NY Locality Name (eg, city) [Default City]:Manhattan Organization Name (eg, company) [Default Company Ltd]:Red Hat Organizational Unit Name (eg, section) []:QE Common Name (eg, your name or your server's hostname) []:titan50 Email Address []:mabrams Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: (overcloud) [stack@undercloud-0 ~]$ openssl x509 -req -days 14 -in cert_request.csr -signkey private_key.pem -out x509_signing_cert.crt Signature ok subject=C = US, ST = NY, L = Manhattan, O = Red Hat, OU = QE, CN = titan50, emailAddress = mabrams Getting Private key (overcloud) [stack@undercloud-0 ~]$ openstack secret store --name signing-cert --algorithm RSA --secret-type certificate --payload-content-type "application/octet-stream" --payload-content-encoding base64 --payload "$(base64 x509_signing_cert.crt)" -c 'Secret href' -f value http://10.0.0.122:9311/v1/secrets/1b48392a-cdf3-46de-95ba-9242463717a7 (overcloud) [stack@undercloud-0 ~]$ --- (overcloud) [stack@undercloud-0 ~]$ openstack image create --container-format bare --disk-format qcow2 --property img_signature="$cirros_signature_b64" --property img_signature_certificate_uuid="1b48392a-cdf3-46de-95ba-9242463717a7" --property img_signature_hash_method="SHA-256" --property img_signature_key_type="RSA-PSS" cirros_0_3_5_signed --file cirros-0.3.5-x86_64-disk.img +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | checksum | d41d8cd98f00b204e9800998ecf8427e | | container_format | bare | | created_at | 2021-08-24T18:59:45Z | | disk_format | qcow2 | | file | /v2/images/1a029251-4d3f-4023-86ad-341215a1d3eb/file | | id | 1a029251-4d3f-4023-86ad-341215a1d3eb | | min_disk | 0 | | min_ram | 0 | | name | cirros_0_3_5_signed | | owner | c0ea092220b64cfbbf17e61bcb0aa07d | | properties | direct_url='swift+config://ref1/glance/1a029251-4d3f-4023-86ad-341215a1d3eb', img_signature='CW0i0KanMLtSzMaEPR1y87IjDarbXt9Hz2/28dqqfmvgC1RFYlBL6/6L9SDn6VEHF700gSkIJRiG0j8b1JBeyaQ1tJMRvv0jQTMnoOlzpbOId0hyuT9s7eWWt1dNe9F+flEjF3mSPmDK2iLEbpD/LWnnCgru3c8YGOaptR1z8mc=', img_signature_certificate_uuid='1b48392a-cdf3-46de-95ba-9242463717a7', img_signature_hash_method='SHA-256', img_signature_key_type='RSA-PSS', os_hash_algo='sha512', os_hash_value='cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e', os_hidden='False', stores='default_backend' | | protected | False | | schema | /v2/schemas/image | | size | 0 | | status | active | | tags | | | updated_at | 2021-08-24T18:59:47Z | | virtual_size | None | | visibility | shared | +------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-list +--------------------------------------+---------------------+ | ID | Name | +--------------------------------------+---------------------+ | 1a029251-4d3f-4023-86ad-341215a1d3eb | cirros_0_3_5_signed | | 97b8be23-71b9-45cd-99b3-88164e8bb033 | import_scenario | | 1bb89b95-9a10-48f9-a973-ae57c56daa86 | mySignedImage | +--------------------------------------+---------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-show 1a029251-4d3f-4023-86ad-341215a1d3eb | grep status | status | active | (overcloud) [stack@undercloud-0 ~]$ --- Boot from this signed image: $ nova boot --flavor FLAVOR_ID --image <ID of signed image> test-success --- (overcloud) [stack@undercloud-0 ~]$ nova boot --flavor m2.tiny --image 1a029251-4d3f-4023-86ad-341215a1d3eb test-success-002 +--------------------------------------+------------------------------------------------------------+ | Property | Value | +--------------------------------------+------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | - | | OS-EXT-SRV-ATTR:hostname | test-success-002 | | OS-EXT-SRV-ATTR:hypervisor_hostname | - | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-SRV-ATTR:kernel_id | | | OS-EXT-SRV-ATTR:launch_index | 0 | | OS-EXT-SRV-ATTR:ramdisk_id | | | OS-EXT-SRV-ATTR:reservation_id | r-flag0fco | | OS-EXT-SRV-ATTR:root_device_name | - | | OS-EXT-SRV-ATTR:user_data | - | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | adminPass | UsVbys24uddi | | config_drive | | | created | 2021-08-24T19:01:37Z | | description | - | | flavor:disk | 10 | | flavor:ephemeral | 0 | | flavor:extra_specs | {} | | flavor:original_name | m2.tiny | | flavor:ram | 1024 | | flavor:swap | 2048 | | flavor:vcpus | 1 | | hostId | | | host_status | | | id | f083374e-bb12-424c-bb9f-bc7613db562b | | image | cirros_0_3_5_signed (1a029251-4d3f-4023-86ad-341215a1d3eb) | | key_name | - | | locked | False | | locked_reason | - | | metadata | {} | | name | test-success-002 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | server_groups | [] | | status | BUILD | | tags | [] | | tenant_id | c0ea092220b64cfbbf17e61bcb0aa07d | | trusted_image_certificates | - | | updated | 2021-08-24T19:01:37Z | | user_id | 1ad19957f8734e17a57ded4ba12b9a64 | +--------------------------------------+------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ nova list +--------------------------------------+------------------+---------+------------+-------------+-----------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------+---------+------------+-------------+-----------------------------------------+ | 571c26c9-fe4f-4ab0-9d5b-545a978bf779 | test-failure | ERROR | - | NOSTATE | | | 502a5c42-79e2-428c-a6e6-b13a84fae907 | test-success | SHUTOFF | - | Shutdown | nova=2620:52:0:13b8::1000:6, 10.0.0.199 | | f083374e-bb12-424c-bb9f-bc7613db562b | test-success-002 | BUILD | spawning | NOSTATE | | +--------------------------------------+------------------+---------+------------+-------------+-----------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ nova list +--------------------------------------+------------------+---------+------------+-------------+------------------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+------------------+---------+------------+-------------+------------------------------------------+ | 571c26c9-fe4f-4ab0-9d5b-545a978bf779 | test-failure | ERROR | - | NOSTATE | | | 502a5c42-79e2-428c-a6e6-b13a84fae907 | test-success | SHUTOFF | - | Shutdown | nova=2620:52:0:13b8::1000:6, 10.0.0.199 | | f083374e-bb12-424c-bb9f-bc7613db562b | test-success-002 | ACTIVE | - | Running | nova=2620:52:0:13b8::1000:2f, 10.0.0.236 | +--------------------------------------+------------------+---------+------------+-------------+------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ --- Check that step 5 did not fail --- did not fail check the nova log for a successful signature verification --- (undercloud) [stack@undercloud-0 ~]$ . ./overcloudrc (overcloud) [stack@undercloud-0 ~]$ glance image-list +--------------------------------------+---------------------+ | ID | Name | +--------------------------------------+---------------------+ | 1a029251-4d3f-4023-86ad-341215a1d3eb | cirros_0_3_5_signed | | 97b8be23-71b9-45cd-99b3-88164e8bb033 | import_scenario | | 1bb89b95-9a10-48f9-a973-ae57c56daa86 | mySignedImage | +--------------------------------------+---------------------+ (overcloud) [stack@undercloud-0 ~]$ . ./stackrc (undercloud) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ | 22b8beab-5ca0-4e19-9537-33ce9aa34f61 | controller-2 | ACTIVE | ctlplane=192.168.24.30 | overcloud-full | controller | | 6989b006-380c-4b09-841c-c3a423402b3b | controller-0 | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | controller | | ba0fc723-2bc0-4f51-b753-a8d21d3686e9 | controller-1 | ACTIVE | ctlplane=192.168.24.27 | overcloud-full | controller | | 4799cbff-b418-42f0-b3e4-5e039f10086c | compute-0 | ACTIVE | ctlplane=192.168.24.17 | overcloud-full | compute | | 4af3b93c-108c-465f-99f9-7ce466d4aad2 | compute-1 | ACTIVE | ctlplane=192.168.24.47 | overcloud-full | compute | +--------------------------------------+--------------+--------+------------------------+----------------+------------+ (undercloud) [stack@undercloud-0 ~]$ ssh -t heat-admin.24.17 Warning: Permanently added '192.168.24.17' (ECDSA) to the list of known hosts. Last login: Tue Aug 24 19:05:45 2021 from 192.168.24.1 [heat-admin@compute-0 ~]$ sudo grep "Image signature verification succeeded" /var/log/containers/nova/nova-compute.log 2021-08-24 15:23:13.203 7 INFO nova.image.glance [req-6d4998fa-2ec6-4ce1-8c41-d2c1a50d0f58 1ad19957f8734e17a57ded4ba12b9a64 c0ea092220b64cfbbf17e61bcb0aa07d - default default] Image signature verification succeeded for image 1bb89b95-9a10-48f9-a973-ae57c56daa86 [heat-admin@compute-0 ~]$ ---
Content has been updated, merged and published