Bug 2040644

Summary: RFE: libvirt needs to provide a reference implementation tool for validating AMD SEV guest measurements
Product: Red Hat Enterprise Linux 9 Reporter: Daniel Berrangé <berrange>
Component: libvirtAssignee: Daniel Berrangé <berrange>
libvirt sub component: CLI & API QA Contact: Luyao Huang <lhuang>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: coli, crobinso, jdenemar, jsuchane, lcheng, lmen, mrezanin, pkrempa, virt-maint, xuzhang, yicui, ymankad, zixchen
Version: 9.2Keywords: AutomationTriaged, FutureFeature, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-9.0.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-09 07:26:11 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 9.0.0
Embargoed:
Bug Depends On:    
Bug Blocks: 2040646    

Description Daniel Berrangé 2022-01-14 11:01:40 UTC 
Description of problem:
Bug 2040638 describes a need to document how to do validate an AMD SEV guest measurement. Even with such docs, this is still a very difficult problem which will be hard to debug when users mess it up. We need to provide a reference implementation tool for validating AMD SEV guest measurements.

This should be something of a high enough quality that users are able to use it in the real world for validating SEV guests. It does not, however, need to be designed for consumption by layered products or higher level mgmt apps. 

It is something that could be used by a guest owner, to validate a guest provisioned with a simple tool like virt-install / virt-manager / clockpit, *provided* the guest owner has privileges needed to remotely connect to libvirt.

Management apps like oVirt, OpenStack, CNV are expected to need something altogether more clever / advanced, since guest owners won't have any ability to talk to libvirt directly in those cases. 

The tool needs to handle scenarios:

 - Validating a guest booting off hard disk with AMD SEV
 - Validating a guest booting off hard disk with AMD SEV-ES
 - Validating a guest booting off direct supplied kernel with AMD SEV
 - Validating a guest booting off direct supplied kernel with AMD SEV-ES

The tool should also be intended for use by QE to test AMD SEV measured boot. 

Version-Release number of selected component (if applicable):
8.0.0
Comment 2 Luyao Huang 2023-02-16 08:44:29 UTC 
I have found some problems during test this feature:

1. Cannot use virt-qemu-sev-validate after install libvirt-client-qemu

# virt-qemu-sev-validate --help
Traceback (most recent call last):
  File "/usr/bin/virt-qemu-sev-validate", line 50, in <module>
    from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
ModuleNotFoundError: No module named 'cryptography'

# dnf install python-cryptography

# virt-qemu-sev-validate --help
Traceback (most recent call last):
  File "/usr/bin/virt-qemu-sev-validate", line 53, in <module>
    from lxml import etree
ModuleNotFoundError: No module named 'lxml'

# dnf install python-lxml


2. Some small mistakes in the man page of virt-qemu-sev-validate

virt-dom-sev-validate should be virt-qemu-sev-validate

--disk-password passwd.txt should be --inject-secret luks-key:passwd.txt

--loader should be --firmware


3. Incorrect example in https://libvirt.org/kbase/launch_security_sev.html#guest-attestation-for-sev-sev-es-from-a-trusted-host

Example:
$ virt-qemu-sev-validate \
    --measurement LMnv8i8N2QejezMPkscShF0cyPYCslgUoCxGWRqQuyt0Q0aUjVkH/T6NcmkwZkWp
    --api-major 0
    --api-minor 24
    --build-id 15
    --policy 3
    --tik ${myvmname}_tik.bin
    --tek ${myvmname}_tek.bin
OK: Looks good to me

Test Result:
# virt-qemu-sev-validate --tik vm1_tik.bin --tek vm1_tek.bin --measurement Tco/y8PzIlUMuHrJXjxYU5D/43ZAcAu8mrrsnx1CYihG0byA8nQWpTPk8PfnMPz5 --api-major 1 --api-minor 51 --build-id 3 --policy 7
ERROR: Either --firmware or --domain is required


4. amd-sev-es-vmsa.stp cannot work with rhel9.2 kernel

# uname -r
5.14.0-265.el9.x86_64

# stap /usr/share/doc/libvirt-docs/examples/systemtap/amd-sev-es-vmsa.stp 
semantic error: unable to find member 'vmsa' for struct vcpu_svm (alternatives: vmcb, vcpu, asid, msrpm, sev_es, vmcb01, ir_list, svm_data, nested, dfr_reg, ldr_reg, tsc_aux, current_vmcb, next_rip, msr_decfg, spec_ctrl, tsc_ratio_msr, ir_list_lock, lbrv_enabled, nmi_iret_rip, nmi_l1_to_l2, vgif_enabled, nrips_enabled, soft_int_csbase, nmi_singlestep, virt_spec_ctrl, sysenter_eip_hi, sysenter_esp_hi, soft_int_old_rip, avic_backing_page, soft_int_injected, soft_int_next_rip, guest_state_loaded, tsc_scaling_enabled, pause_filter_enabled, shadow_msr_intercept, avic_physical_id_cache, pause_threshold_enabled, v_vmload_vmsave_enabled, x2avic_msrs_intercepted, nmi_singlestep_guest_rflags): operator '->' at /usr/share/doc/libvirt-docs/examples/systemtap/amd-sev-es-vmsa.stp:47:17
        source:   dump_vmsa($svm->vmsa)
                                ^

semantic error: unresolved type : identifier '$svm' at :47:13
        source:   dump_vmsa($svm->vmsa)
                            ^

Pass 2: analysis failed.  [man error::pass2]
Comment 3 Luyao Huang 2023-02-16 09:07:56 UTC 
5. Fail to validate the measurement of a SEV guest with direct kernel boot

# virsh dumpxml vm2
...
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <kernel>/var/lib/libvirt/boot/vmlinuz</kernel>
    <initrd>/var/lib/libvirt/boot/initrd.img</initrd>
    <cmdline>console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0 inst.repo=http://someurl</cmdline>
    <boot dev='hd'/>
  </os>

# virsh start  vm2 --paused
Domain 'vm2' started


Try local insecure validate:
# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --insecure --domain vm2
ERROR: Measurement does not match, VM is not trustworthy

Try offline validate:

# virsh domlaunchsecinfo vm2
sev-measurement: lZbyhHfa6D5gKk0ozHy72RKhXy9cGBPiUBBnEY+11KGBxXl4VlUQ1YhtCbd4bEzF
sev-api-major  : 1
sev-api-minor  : 51
sev-build-id   : 3
sev-policy     : 3

# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement lZbyhHfa6D5gKk0ozHy72RKhXy9cGBPiUBBnEY+11KGBxXl4VlUQ1YhtCbd4bEzF --api-major 1 --api-minor 51 --build-id 3 --policy 3  --kernel /var/lib/libvirt/boot/vmlinuz --initrd /var/lib/libvirt/boot/initrd.img --cmdline "console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0 inst.repo=http://someurl"
ERROR: Measurement does not match, VM is not trustworthy

# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement lZbyhHfa6D5gKk0ozHy72RKhXy9cGBPiUBBnEY+11KGBxXl4VlUQ1YhtCbd4bEzF --api-major 1 --api-minor 51 --build-id 3 --policy 3
OK: Looks good to me
Comment 4 Luyao Huang 2023-02-16 10:04:49 UTC 
Hi Daniel,

I have finished some basic test on latest RHEL9.2 libvirt and found some tiny problems mentioned in comment 2.

And Results of measurement validation:

Validate the measurement of a SEV guest booting from disk: PASS
Validate the measurement of a SEV guest with direct kernel boot: FAIL see comment 3
Validate the measurement of a SEV-ES SMP guest booting from disk: PASS
Validate the measurement of a SEV-ES SMP guest with direct kernel boot: FAIL see comment 3
Validate the measurement of a SEV-ES SMP guest booting from disk, with automatically constructed VMSA: PASS
Validate the measurement of a SEV guest booting from disk and inject a disk password on success: BLOCK (No idea about how to setup the env now)


Could you please check the problems mentioned in comment 2 and comment 3? Thanks in advance for your help!

About inject a disk password testing, I am still researching how to setup a test environment now. I will update this part results later.


Luyao
Comment 5 Daniel Berrangé 2023-02-16 15:46:20 UTC 
(In reply to Luyao Huang from comment #2)
> I have found some problems during test this feature:
> 
> 1. Cannot use virt-qemu-sev-validate after install libvirt-client-qemu

Yes, we've missed two RPM deps - I thought RPM automatically extracted
python deps, it does not seem to be happening.

> 2. Some small mistakes in the man page of virt-qemu-sev-validate
> 
> virt-dom-sev-validate should be virt-qemu-sev-validate
> 
> --disk-password passwd.txt should be --inject-secret luks-key:passwd.txt
> 
> --loader should be --firmware

Yes, we renamed the argument and missed that change in docs

> 3. Incorrect example in
> https://libvirt.org/kbase/launch_security_sev.html#guest-attestation-for-sev-
> sev-es-from-a-trusted-host
> 
> Example:
> $ virt-qemu-sev-validate \
>     --measurement
> LMnv8i8N2QejezMPkscShF0cyPYCslgUoCxGWRqQuyt0Q0aUjVkH/T6NcmkwZkWp
>     --api-major 0
>     --api-minor 24
>     --build-id 15
>     --policy 3
>     --tik ${myvmname}_tik.bin
>     --tek ${myvmname}_tek.bin
> OK: Looks good to me
> 
> Test Result:
> # virt-qemu-sev-validate --tik vm1_tik.bin --tek vm1_tek.bin --measurement
> Tco/y8PzIlUMuHrJXjxYU5D/43ZAcAu8mrrsnx1CYihG0byA8nQWpTPk8PfnMPz5 --api-major
> 1 --api-minor 51 --build-id 3 --policy 7
> ERROR: Either --firmware or --domain is required

This example should mention --domain as its illustrating a fully offline validation

> 4. amd-sev-es-vmsa.stp cannot work with rhel9.2 kernel
> 

Unfortunately it is not practical to write a script that works on any
kernel. The script was known to work with 6.0 kernel, but if it fails
on other kernels that's life. We're not going to attempt to make it
work everywhere, since this is just a demo script illustrating a
general technique, not a fully supported component.
Comment 6 Daniel Berrangé 2023-02-16 15:50:00 UTC 
(In reply to Luyao Huang from comment #3)
> 5. Fail to validate the measurement of a SEV guest with direct kernel boot
> 
> # virsh dumpxml vm2
> ...
>   <os firmware='efi'>
>     <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
>     <kernel>/var/lib/libvirt/boot/vmlinuz</kernel>
>     <initrd>/var/lib/libvirt/boot/initrd.img</initrd>
>     <cmdline>console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0
> inst.repo=http://someurl</cmdline>
>     <boot dev='hd'/>
>   </os>

Could you attach the *full* XML for the guest to this bug.

> 
> # virsh start  vm2 --paused
> Domain 'vm2' started
> 
> 
> Try local insecure validate:
> # virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --insecure
> --domain vm2
> ERROR: Measurement does not match, VM is not trustworthy

Could you re-run this with --debug flags

> # virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware
> /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement
> lZbyhHfa6D5gKk0ozHy72RKhXy9cGBPiUBBnEY+11KGBxXl4VlUQ1YhtCbd4bEzF --api-major
> 1 --api-minor 51 --build-id 3 --policy 3  --kernel
> /var/lib/libvirt/boot/vmlinuz --initrd /var/lib/libvirt/boot/initrd.img
> --cmdline "console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0
> inst.repo=http://someurl"
> ERROR: Measurement does not match, VM is not trustworthy
> 
> # virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware
> /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement
> lZbyhHfa6D5gKk0ozHy72RKhXy9cGBPiUBBnEY+11KGBxXl4VlUQ1YhtCbd4bEzF --api-major
> 1 --api-minor 51 --build-id 3 --policy 3
> OK: Looks good to me

Also re-run both of these with --debug flags

This difference suggests that the firmware measurement is not covering the kernel/initrd/cmdline data. The full guest XML will help us understand things.
Comment 7 Luyao Huang 2023-02-17 02:15:32 UTC 
(In reply to Daniel Berrangé from comment #5)
> (In reply to Luyao Huang from comment #2)
...
> 
> > 4. amd-sev-es-vmsa.stp cannot work with rhel9.2 kernel
> > 
> 
> Unfortunately it is not practical to write a script that works on any
> kernel. The script was known to work with 6.0 kernel, but if it fails
> on other kernels that's life. We're not going to attempt to make it
> work everywhere, since this is just a demo script illustrating a
> general technique, not a fully supported component.


I see, Thanks a lot for your explanation!


(In reply to Daniel Berrangé from comment #6)
> (In reply to Luyao Huang from comment #3)
> > 5. Fail to validate the measurement of a SEV guest with direct kernel boot
> > 
> > # virsh dumpxml vm2
> > ...
> >   <os firmware='efi'>
> >     <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
> >     <kernel>/var/lib/libvirt/boot/vmlinuz</kernel>
> >     <initrd>/var/lib/libvirt/boot/initrd.img</initrd>
> >     <cmdline>console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0
> > inst.repo=http://someurl</cmdline>
> >     <boot dev='hd'/>
> >   </os>
> 
> Could you attach the *full* XML for the guest to this bug.


Sure, I will put full guest xml in attachment.


> 
> > 
> > # virsh start  vm2 --paused
> > Domain 'vm2' started
> > 
> > 
> > Try local insecure validate:
> > # virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --insecure
> > --domain vm2
> > ERROR: Measurement does not match, VM is not trustworthy
> 
> Could you re-run this with --debug flags


# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin --insecure --domain vm2 --debug
[DEBUG]: TIK(hex): dd39132309ccce32a09450ce800ee2c0
[DEBUG]: TEK(hex): 5937b47cebc536542897c217d86e2a5b
[DEBUG]: VM: id=1 name=vm2 uuid=77f3c26b-956f-49e0-b9b3-4c9c81d91a36
[DEBUG]: Firmware(sha256): ba6d627f68a48392ed365f627b41b6eeb279bdf135ddc9a0d2185e8e77b80a69
[DEBUG]: Kernel(sha256): 5490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af963179
[DEBUG]: Initrd(sha256): 56facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a8
[DEBUG]: Cmdline(sha256): 0f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c
[DEBUG]: Table(hex): 06d63894224fc94cb479a793d411fd21a800d82dd09720bd944caa78e7714d36ab2a32000f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c31f7ba442f3ad74b9af141e29169781d320056facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a83794e74dd2ab7f42b835d5b172d2045b32005490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af9631790000000000000000
[DEBUG]: Measured-data(sha256): 10f4b326b2bc7861c88feadaffa7a83efaa0588c67116e8dcb20ef94f41f2372
[DEBUG]: Measured-msg(hex): 040133030300000010f4b326b2bc7861c88feadaffa7a83efaa0588c67116e8dcb20ef94f41f2372e642e35630b4f39443808857e043451b
[DEBUG]: Measurement reported(hex): f1148328fd8fc601095dfc2c73d63dc0934cbff4b75e23325ba4d2851da8aed2
[DEBUG]: Measurement computed(hex): 2d2e611fb8a89fb77002a56c795c9ddae532f8a7ab9a2da23ef3973a9f0f568d
  File "/usr/bin/virt-qemu-sev-validate", line 1300, in main
    attest(args)
  File "/usr/bin/virt-qemu-sev-validate", line 1271, in attest
    cvm.attest()
  File "/usr/bin/virt-qemu-sev-validate", line 800, in attest
    raise AttestationFailedException(
ERROR: Measurement does not match, VM is not trustworthy


> 
> > # virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware
> > /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement
> > lZbyhHfa6D5gKk0ozHy72RKhXy9cGBPiUBBnEY+11KGBxXl4VlUQ1YhtCbd4bEzF --api-major
> > 1 --api-minor 51 --build-id 3 --policy 3  --kernel
> > /var/lib/libvirt/boot/vmlinuz --initrd /var/lib/libvirt/boot/initrd.img
> > --cmdline "console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0
> > inst.repo=http://someurl"
> > ERROR: Measurement does not match, VM is not trustworthy
> > 
> > # virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware
> > /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement
> > lZbyhHfa6D5gKk0ozHy72RKhXy9cGBPiUBBnEY+11KGBxXl4VlUQ1YhtCbd4bEzF --api-major
> > 1 --api-minor 51 --build-id 3 --policy 3
> > OK: Looks good to me
> 
> Also re-run both of these with --debug flags
> 
> This difference suggests that the firmware measurement is not covering the
> kernel/initrd/cmdline data. The full guest XML will help us understand
> things.


# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement 8RSDKP2PxgEJXfwsc9Y9wJNMv/S3XiMyW6TShR2ortLmQuNWMLTzlEOAiFfgQ0Ub --api-major 1 --api-minor 51 --build-id 3 --policy 3  --kernel /var/lib/libvirt/boot/vmlinuz --initrd /var/lib/libvirt/boot/initrd.img --cmdline "console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0 inst.repo=http://someurl" --debug
[DEBUG]: Firmware(sha256): ba6d627f68a48392ed365f627b41b6eeb279bdf135ddc9a0d2185e8e77b80a69
[DEBUG]: TIK(hex): dd39132309ccce32a09450ce800ee2c0
[DEBUG]: TEK(hex): 5937b47cebc536542897c217d86e2a5b
[DEBUG]: Kernel(sha256): 5490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af963179
[DEBUG]: Initrd(sha256): 56facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a8
[DEBUG]: Cmdline(sha256): 0f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c
[DEBUG]: Table(hex): 06d63894224fc94cb479a793d411fd21a800d82dd09720bd944caa78e7714d36ab2a32000f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c31f7ba442f3ad74b9af141e29169781d320056facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a83794e74dd2ab7f42b835d5b172d2045b32005490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af9631790000000000000000
[DEBUG]: Measured-data(sha256): 10f4b326b2bc7861c88feadaffa7a83efaa0588c67116e8dcb20ef94f41f2372
[DEBUG]: Measured-msg(hex): 040133030300000010f4b326b2bc7861c88feadaffa7a83efaa0588c67116e8dcb20ef94f41f2372e642e35630b4f39443808857e043451b
[DEBUG]: Measurement reported(hex): f1148328fd8fc601095dfc2c73d63dc0934cbff4b75e23325ba4d2851da8aed2
[DEBUG]: Measurement computed(hex): 2d2e611fb8a89fb77002a56c795c9ddae532f8a7ab9a2da23ef3973a9f0f568d
  File "/usr/bin/virt-qemu-sev-validate", line 1300, in main
    attest(args)
  File "/usr/bin/virt-qemu-sev-validate", line 1271, in attest
    cvm.attest()
  File "/usr/bin/virt-qemu-sev-validate", line 800, in attest
    raise AttestationFailedException(
ERROR: Measurement does not match, VM is not trustworthy


# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin  --firmware /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd --measurement 8RSDKP2PxgEJXfwsc9Y9wJNMv/S3XiMyW6TShR2ortLmQuNWMLTzlEOAiFfgQ0Ub --api-major 1 --api-minor 51 --build-id 3 --policy 3  --debug
[DEBUG]: Firmware(sha256): ba6d627f68a48392ed365f627b41b6eeb279bdf135ddc9a0d2185e8e77b80a69
[DEBUG]: TIK(hex): dd39132309ccce32a09450ce800ee2c0
[DEBUG]: TEK(hex): 5937b47cebc536542897c217d86e2a5b
[DEBUG]: Measured-data(sha256): ba6d627f68a48392ed365f627b41b6eeb279bdf135ddc9a0d2185e8e77b80a69
[DEBUG]: Measured-msg(hex): 0401330303000000ba6d627f68a48392ed365f627b41b6eeb279bdf135ddc9a0d2185e8e77b80a69e642e35630b4f39443808857e043451b
[DEBUG]: Measurement reported(hex): f1148328fd8fc601095dfc2c73d63dc0934cbff4b75e23325ba4d2851da8aed2
[DEBUG]: Measurement computed(hex): f1148328fd8fc601095dfc2c73d63dc0934cbff4b75e23325ba4d2851da8aed2
OK: Looks good to me


Hi Daniel,

I have updated virt-qemu-sev-validate debug log level output. Could you please check it again? Thanks in advance for your help!

Luyao
Comment 10 Luyao Huang 2023-02-17 11:20:11 UTC 
Hi Daniel,

I have tested use virt-qemu-sev-validate to inject secret to guest without any error, but I didn't find a way to verify the injected secret in the guest. I have tried to follow this blog [1] to verify the injected secret, but later I noticed that grub2 patches [2] didn't get merged in upstream/downstream grub2 and cryptomount not support option -s. Could you please provide a method to verify the injected secret in the guest? Thanks in advance for your help!


[1] https://blog.hansenpartnership.com/deploying-encrypted-images-for-confidential-computing/
[2] https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00257.html

Luyao
Comment 11 Cole Robinson 2023-02-18 19:59:39 UTC 
There's some steps about validating a full setup here: https://gist.github.com/crobinso/830512728bf707a35e73755ed65988c4

But simplest is: `modprobe efi_secret` inside the VM, then you should see the secret in /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b
Comment 12 Luyao Huang 2023-02-20 03:19:27 UTC 
(In reply to Cole Robinson from comment #11)
> There's some steps about validating a full setup here:
> https://gist.github.com/crobinso/830512728bf707a35e73755ed65988c4
> 
> But simplest is: `modprobe efi_secret` inside the VM, then you should see
> the secret in
> /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b

Your suggestion is really helpful! Thanks a lot for your help!

And I found there is no such secret file when test this on RHEL9.2 kernel 5.14.0-267.el9.x86_64:

# modprobe efi_secret 
# lsmod |grep efi_secret
efi_secret             20480  0
# cat /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b
cat: /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b: No such file or directory

I guess maybe this kernel is too old to support this feature, I will retest this on latest fedora.

Luyao
Comment 13 Luyao Huang 2023-02-20 08:26:28 UTC 
Hi Daniel,

Please ignore comment 7 since I noticed that I forgot set the kernelHashes attribute in guest xml. Here is the test results after set kernelHashes=yes:

# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --insecure --debug
[DEBUG]: TIK(hex): dd39132309ccce32a09450ce800ee2c0
[DEBUG]: TEK(hex): 5937b47cebc536542897c217d86e2a5b
[DEBUG]: VM: id=7 name=vm2 uuid=77f3c26b-956f-49e0-b9b3-4c9c81d91a36
[DEBUG]: Firmware(sha256): cb9b26ae60b0f5a9ab0b2a05ff0863cfbb5987a83b644b3c925499e11fdf0093
[DEBUG]: Kernel(sha256): 5490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af963179
[DEBUG]: Initrd(sha256): 56facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a8
[DEBUG]: Cmdline(sha256): 0f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c
[DEBUG]: Table(hex): 06d63894224fc94cb479a793d411fd21a800d82dd09720bd944caa78e7714d36ab2a32000f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c31f7ba442f3ad74b9af141e29169781d320056facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a83794e74dd2ab7f42b835d5b172d2045b32005490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af9631790000000000000000
[DEBUG]: Measured-data(sha256): be177f0053e7097808987c1f0c31f44206ce0dc750a01b910026656ec30585ed
[DEBUG]: Measured-msg(hex): 0401330303000000be177f0053e7097808987c1f0c31f44206ce0dc750a01b910026656ec30585ed3c7dd128e2922fca5ba19b5b7a629514
[DEBUG]: Measurement reported(hex): 27013a5dbc68a37d7e0c1e65581e62df2f5b9672eacdeac0a66afdcf320dc4f1
[DEBUG]: Measurement computed(hex): 229426bba7e45980b977ecbb740f43956e5bbab6502dde9fc2cda7edd843eb10
  File "/usr/bin/virt-qemu-sev-validate", line 1300, in main
    attest(args)
  File "/usr/bin/virt-qemu-sev-validate", line 1271, in attest
    cvm.attest()
  File "/usr/bin/virt-qemu-sev-validate", line 800, in attest
    raise AttestationFailedException(
ERROR: Measurement does not match, VM is not trustworthy

# virt-qemu-sev-validate  --tik vm2_tik.bin --tek vm2_tek.bin --measurement JwE6Xbxoo31+DB5lWB5i3y9blnLqzerApmr9zzINxPE8fdEo4pIvyluhm1t6YpUU --api-major 1 --api-minor 51 --build-id 3 --policy 3 --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --kernel /var/lib/libvirt/boot/vmlinuz --initrd /var/lib/libvirt/boot/initrd.img --cmdline "console=tty0 console=ttyS0,115200 net.ifnames=0 biosdevname=0 inst.repo=http://someurl" --debug
[DEBUG]: Firmware(sha256): cb9b26ae60b0f5a9ab0b2a05ff0863cfbb5987a83b644b3c925499e11fdf0093
[DEBUG]: TIK(hex): dd39132309ccce32a09450ce800ee2c0
[DEBUG]: TEK(hex): 5937b47cebc536542897c217d86e2a5b
[DEBUG]: Kernel(sha256): 5490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af963179
[DEBUG]: Initrd(sha256): 56facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a8
[DEBUG]: Cmdline(sha256): 0f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c
[DEBUG]: Table(hex): 06d63894224fc94cb479a793d411fd21a800d82dd09720bd944caa78e7714d36ab2a32000f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c31f7ba442f3ad74b9af141e29169781d320056facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a83794e74dd2ab7f42b835d5b172d2045b32005490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af9631790000000000000000
[DEBUG]: Measured-data(sha256): be177f0053e7097808987c1f0c31f44206ce0dc750a01b910026656ec30585ed
[DEBUG]: Measured-msg(hex): 0401330303000000be177f0053e7097808987c1f0c31f44206ce0dc750a01b910026656ec30585ed3c7dd128e2922fca5ba19b5b7a629514
[DEBUG]: Measurement reported(hex): 27013a5dbc68a37d7e0c1e65581e62df2f5b9672eacdeac0a66afdcf320dc4f1
[DEBUG]: Measurement computed(hex): 229426bba7e45980b977ecbb740f43956e5bbab6502dde9fc2cda7edd843eb10
  File "/usr/bin/virt-qemu-sev-validate", line 1300, in main
    attest(args)
  File "/usr/bin/virt-qemu-sev-validate", line 1271, in attest
    cvm.attest()
  File "/usr/bin/virt-qemu-sev-validate", line 800, in attest
    raise AttestationFailedException(
ERROR: Measurement does not match, VM is not trustworthy

# virt-qemu-sev-validate  --tik vm2_tik.bin --tek vm2_tek.bin --measurement JwE6Xbxoo31+DB5lWB5i3y9blnLqzerApmr9zzINxPE8fdEo4pIvyluhm1t6YpUU --api-major 1 --api-minor 51 --build-id 3 --policy 3 --debug --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd
[DEBUG]: Firmware(sha256): cb9b26ae60b0f5a9ab0b2a05ff0863cfbb5987a83b644b3c925499e11fdf0093
[DEBUG]: TIK(hex): dd39132309ccce32a09450ce800ee2c0
[DEBUG]: TEK(hex): 5937b47cebc536542897c217d86e2a5b
[DEBUG]: Measured-data(sha256): cb9b26ae60b0f5a9ab0b2a05ff0863cfbb5987a83b644b3c925499e11fdf0093
[DEBUG]: Measured-msg(hex): 0401330303000000cb9b26ae60b0f5a9ab0b2a05ff0863cfbb5987a83b644b3c925499e11fdf00933c7dd128e2922fca5ba19b5b7a629514
[DEBUG]: Measurement reported(hex): 27013a5dbc68a37d7e0c1e65581e62df2f5b9672eacdeac0a66afdcf320dc4f1
[DEBUG]: Measurement computed(hex): a62084b57df331fb3519e4f3f389739f8b581655300d960bbb61894bd03a57fa
  File "/usr/bin/virt-qemu-sev-validate", line 1300, in main
    attest(args)
  File "/usr/bin/virt-qemu-sev-validate", line 1271, in attest
    cvm.attest()
  File "/usr/bin/virt-qemu-sev-validate", line 800, in attest
    raise AttestationFailedException(
ERROR: Measurement does not match, VM is not trustworthy


And I will put new guest xml in attachment.
Comment 15 Daniel Berrangé 2023-02-20 10:41:20 UTC 
(In reply to Luyao Huang from comment #12)
> (In reply to Cole Robinson from comment #11)
> # modprobe efi_secret 
> # lsmod |grep efi_secret
> efi_secret             20480  0
> # cat /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b
> cat: /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b:
> No such file or directory
> 
> I guess maybe this kernel is too old to support this feature, I will retest
> this on latest fedora.

You need to mount the filesystem too

 modprobe efi_secret
 mount -t efi_secret none /sys/kernel/security/secrets
Comment 16 Daniel Berrangé 2023-02-20 10:46:50 UTC 
(In reply to Luyao Huang from comment #13)
> Hi Daniel,
> 
> Please ignore comment 7 since I noticed that I forgot set the kernelHashes
> attribute in guest xml. Here is the test results after set kernelHashes=yes:
> 
> # virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2
> --insecure --debug
> [DEBUG]: TIK(hex): dd39132309ccce32a09450ce800ee2c0
> [DEBUG]: TEK(hex): 5937b47cebc536542897c217d86e2a5b
> [DEBUG]: VM: id=7 name=vm2 uuid=77f3c26b-956f-49e0-b9b3-4c9c81d91a36
> [DEBUG]: Firmware(sha256):
> cb9b26ae60b0f5a9ab0b2a05ff0863cfbb5987a83b644b3c925499e11fdf0093
> [DEBUG]: Kernel(sha256):
> 5490a66856f9a4195a8688cb22ace432d2c6741badcddae0a5068ce8af963179
> [DEBUG]: Initrd(sha256):
> 56facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340756a8
> [DEBUG]: Cmdline(sha256):
> 0f709b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c
> [DEBUG]: Table(hex):
> 06d63894224fc94cb479a793d411fd21a800d82dd09720bd944caa78e7714d36ab2a32000f709
> b04f407634c661c3f11a5f973b57909fca66a3524878193eabf02f38b8c31f7ba442f3ad74b9a
> f141e29169781d320056facd79220ef2cb5466626ea62a5681974d9b8f7e81acacea07139d340
> 756a83794e74dd2ab7f42b835d5b172d2045b32005490a66856f9a4195a8688cb22ace432d2c6
> 741badcddae0a5068ce8af9631790000000000000000
> [DEBUG]: Measured-data(sha256):
> be177f0053e7097808987c1f0c31f44206ce0dc750a01b910026656ec30585ed
> [DEBUG]: Measured-msg(hex):
> 0401330303000000be177f0053e7097808987c1f0c31f44206ce0dc750a01b910026656ec3058
> 5ed3c7dd128e2922fca5ba19b5b7a629514
> [DEBUG]: Measurement reported(hex):
> 27013a5dbc68a37d7e0c1e65581e62df2f5b9672eacdeac0a66afdcf320dc4f1
> [DEBUG]: Measurement computed(hex):
> 229426bba7e45980b977ecbb740f43956e5bbab6502dde9fc2cda7edd843eb10
>   File "/usr/bin/virt-qemu-sev-validate", line 1300, in main
>     attest(args)
>   File "/usr/bin/virt-qemu-sev-validate", line 1271, in attest
>     cvm.attest()
>   File "/usr/bin/virt-qemu-sev-validate", line 800, in attest
>     raise AttestationFailedException(
> ERROR: Measurement does not match, VM is not trustworthy

You are using the pc-q35-rhel9.2.0  which I suspect is suffering from the upstream disaster with the RNG seed patches breaking validation of kernel/initrd files

Please try again with pc-q35-rhel9.1.0 machine type

See https://bugzilla.redhat.com/show_bug.cgi?id=2158430 for more details
Comment 17 Luyao Huang 2023-02-21 07:56:41 UTC 
(In reply to Daniel Berrangé from comment #15)
> (In reply to Luyao Huang from comment #12)
> > (In reply to Cole Robinson from comment #11)
> > # modprobe efi_secret 
> > # lsmod |grep efi_secret
> > efi_secret             20480  0
> > # cat /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b
> > cat: /sys/kernel/security/secrets/coco/736869e5-84f0-4973-92ec-06879ce3da0b:
> > No such file or directory
> > 
> > I guess maybe this kernel is too old to support this feature, I will retest
> > this on latest fedora.
> 
> You need to mount the filesystem too
> 
>  modprobe efi_secret
>  mount -t efi_secret none /sys/kernel/security/secrets


Hi Daniel,

Thanks for your comment! I found the root cause is the incorrect firmware binary in guest xml.
I can get the right efi secrets after use OVMF.amdsev.fd firmware binary.


(In reply to Daniel Berrangé from comment #16)
> (In reply to Luyao Huang from comment #13)
> > Hi Daniel,
> > 
> > Please ignore comment 7 since I noticed that I forgot set the kernelHashes
> > attribute in guest xml. Here is the test results after set kernelHashes=yes:
> > 
...
> 
> You are using the pc-q35-rhel9.2.0  which I suspect is suffering from the
> upstream disaster with the RNG seed patches breaking validation of
> kernel/initrd files
> 
> Please try again with pc-q35-rhel9.1.0 machine type
> 
> See https://bugzilla.redhat.com/show_bug.cgi?id=2158430 for more details


Thanks a lot for pointing out the root cause! After change machine type to pc-q35-rhel9.0.0, The sev guest with direct kernel boot measurement validation related test passed.

Luyao
Comment 18 Luyao Huang 2023-02-21 08:48:37 UTC 
Hi Daniel,

Thanks a lot for your help and I have finished pre-verified testing. The only left problem is how to deal with the document/dependency issues 1,2 and 3 mentioned in comment 2. Do you think is that okay for us to open a new bug to track these tiny issues? or maybe we should move this bug back to ASSIGNED status? Thanks in advance for your reply!

Luyao
Comment 21 Daniel Berrangé 2023-02-21 13:45:05 UTC 
(In reply to Luyao Huang from comment #18)
> Hi Daniel,
> 
> Thanks a lot for your help and I have finished pre-verified testing. The
> only left problem is how to deal with the document/dependency issues 1,2 and
> 3 mentioned in comment 2. Do you think is that okay for us to open a new bug
> to track these tiny issues? or maybe we should move this bug back to
> ASSIGNED status? Thanks in advance for your reply!

Lets open a new bug for those, since they're minor issues. I should still get them fixed in time for 9.2 though as patches are easy
Comment 22 Luyao Huang 2023-02-22 03:40:27 UTC 
(In reply to Daniel Berrangé from comment #21)
> (In reply to Luyao Huang from comment #18)
> > Hi Daniel,
> > 
> > Thanks a lot for your help and I have finished pre-verified testing. The
> > only left problem is how to deal with the document/dependency issues 1,2 and
> > 3 mentioned in comment 2. Do you think is that okay for us to open a new bug
> > to track these tiny issues? or maybe we should move this bug back to
> > ASSIGNED status? Thanks in advance for your reply!
> 
> Lets open a new bug for those, since they're minor issues. I should still
> get them fixed in time for 9.2 though as patches are easy

Thanks a lot for your quick reply! I have opened a new bug 2172347 for these minor issues.
Comment 23 Luyao Huang 2023-02-22 08:47:38 UTC 
Verify this bug on libvirt-9.0.0-6.el9.x86_64:

S1: SEV guest + direct kernel boot

1. prepare guest xml:

    <loader readonly='yes' secure='no' type='pflash' stateless='yes'>/usr/share/edk2/ovmf/OVMF.amdsev.fd</loader>
    <kernel>/var/lib/libvirt/images/vm2-vmlinuz-5.14.0-265.el9.x86_64</kernel>
    <initrd>/var/lib/libvirt/images/vm2-initramfs-5.14.0-265.el9.x86_64.img</initrd>
    <cmdline>BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap</cmdline>

  <launchSecurity type='sev' kernelHashes='yes'>
    <cbitpos>51</cbitpos>
    <reducedPhysBits>1</reducedPhysBits>
    <policy>0x0003</policy>
    <dhCert>...</dhCert>
    <session>...</session>
  </launchSecurity>

2. 
# virsh start vm2 --paused
Domain 'vm2' started

3. use 3 ways to validate guest SEV measurement

Local:

# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --insecure
OK: Looks good to me

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --kernel vm2-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm2-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap"
OK: Looks good to me

offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --kernel vm2-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm2-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" --measurement vHPJlrwIMHV3BRzLaEvC46zx6OQo9s7D9S5aTbrczzOxnGqXPW5T6IBTb70v0ZY3 --api-major 1 --api-minor 51 --build-id 3 --policy 3
OK: Looks good to me

4. Inject secrets test

Local:

# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --insecure --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd.txt 
OK: Looks good to me
OK: Injected 2 secrets

 IN GUEST:
# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --kernel vm2-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm2-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt
OK: Looks good to me
OK: Injected 2 secrets


 IN GUEST:
# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2


offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --kernel vm2-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm2-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" --measurement BQTgAqm5FfpZKJXATypJo2Vl4U4XjtgtRFvdlvM3jNdOrDvmzNx63TIbd/2sGMDz --api-major 1 --api-minor 51 --build-id 3 --policy 3 --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt --secret-header secret-header.b64 --secret-payload secret-payload.b64
OK: Looks good to me
OK: Injected 2 secrets

# virsh domsetlaunchsecstate vm2 --secrethdr secret-header.b64 --secret secret-payload.b64

 IN GUEST:
# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2



S2: SEV-ES guest + direct kernel boot

1. prepare guest xml:
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-rhel9.0.0'>hvm</type>
    <loader stateless='yes'/>
    <kernel>/var/lib/libvirt/images/vm1-vmlinuz-5.14.0-265.el9.x86_64</kernel>
    <initrd>/var/lib/libvirt/images/vm1-initramfs-5.14.0-265.el9.x86_64.img</initrd>
    <cmdline>BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap</cmdline>
    <boot dev='hd'/>
  </os>

  <launchSecurity type='sev' kernelHashes='yes'>
    <cbitpos>51</cbitpos>
    <reducedPhysBits>1</reducedPhysBits>
    <policy>0x0007</policy>
    <dhCert>...</dhCert>
    <session>...</session>
  </launchSecurity>

2.
# virsh start vm1 --paused
Domain 'vm1' started

3. use 3 ways to validate guest SEV measurement

Local:

# virt-qemu-sev-validate --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --insecure
OK: Looks good to me

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --kernel vm1-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm1-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin 
OK: Looks good to me

offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --kernel vm1-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm1-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" --measurement ql3mohMNXtRtHWK4K3tWccxD2B07PkWqC5viZZo3HRHdyTFEQuPIFpZa0DCK93o4 --api-major 1 --api-minor 51 --build-id 3 --policy 7  --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin
OK: Looks good to me

4. Inject secrets test

Local:

# virt-qemu-sev-validate --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --insecure --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt

 IN GUEST:
# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --kernel vm1-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm1-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt
OK: Looks good to me
OK: Injected 2 secrets

 IN GUEST:
# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2

offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --kernel vm1-vmlinuz-5.14.0-265.el9.x86_64 --initrd vm1-initramfs-5.14.0-265.el9.x86_64.img --cmdline "BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-265.el9.x86_64 root=/dev/mapper/rhel-root ro console=tty0 console=ttyS0,115200 reboot=pci biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap" --measurement ql3mohMNXtRtHWK4K3tWccxD2B07PkWqC5viZZo3HRHdyTFEQuPIFpZa0DCK93o4 --api-major 1 --api-minor 51 --build-id 3 --policy 7  --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt --secret-header secret-header.b64 --secret-payload secret-payload.b64
OK: Looks good to me
OK: Injected 2 secrets


# virsh domsetlaunchsecstate vm1 --secrethdr secret-header.b64 --secret secret-payload.b64

# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2
Comment 24 Luyao Huang 2023-02-22 09:24:34 UTC 
S3: SEV guest + booting from disk

1. prepare guest xml:

  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.0.0'>hvm</type>
    <loader readonly='yes' secure='no' type='pflash' stateless='yes'>/usr/share/edk2/ovmf/OVMF.amdsev.fd</loader>
    <boot dev='hd'/>
  </os>

  <launchSecurity type='sev' kernelHashes='no'>
    <cbitpos>51</cbitpos>
    <reducedPhysBits>1</reducedPhysBits>
    <policy>0x0003</policy>
    <dhCert>...</dhCert>
    <session>...</session>
  </launchSecurity>

2. 

# virsh start vm2 --paused
Domain 'vm2' started

3. use 3 ways to validate guest SEV measurement

Local:

# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --insecure
OK: Looks good to me

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2
OK: Looks good to me

offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --measurement h3L4ARtCwOCuTJ3oGq8PMVgq44h0J2ylDfcdRjut5VP0q/yIwEMFUuY6buk9S6Ux --api-major 1 --api-minor 51 --build-id 3 --policy 3


4. Inject secrets test

Local:

# virt-qemu-sev-validate --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --insecure --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt 
OK: Looks good to me
OK: Injected 2 secrets

# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --domain vm2 --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt
OK: Looks good to me
OK: Injected 2 secrets

# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2

offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm2_tik.bin --tek vm2_tek.bin --measurement Z4+hEp3fu1aM5uX1KxiTMSGWxJY2DE6tciqdpv8CFLTAiZHqsVzx5TDMl3Q3XVVk --api-major 1 --api-minor 51 --build-id 3 --policy 3 --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt --secret-header secret-header.b64 --secret-payload secret-payload.b64
OK: Looks good to me
OK: Injected 2 secrets

# virsh domsetlaunchsecstate vm2 --secrethdr secret-header.b64 --secret secret-payload.b64


# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2


S4: SEV-ES guest + booting from disk

1. prepare guest xml:

  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-rhel9.0.0'>hvm</type>
    <loader stateless='yes'/>
    <boot dev='hd'/>
  </os>

  <launchSecurity type='sev' kernelHashes='no'>
    <cbitpos>51</cbitpos>
    <reducedPhysBits>1</reducedPhysBits>
    <policy>0x0007</policy>
    <dhCert>...</dhCert>
    <session>...</session>
  </launchSecurity>


2. 

# virsh start vm2 --paused
Domain 'vm2' started


3. use 3 ways to validate guest SEV measurement

Local:

# virt-qemu-sev-validate --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --insecure
OK: Looks good to me

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin
OK: Looks good to me

offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --measurement WgA/OwvN8EJNvxlRrpRO73r4t5fHW8kRp5MFgPeBvaKIlG1++mFPFt19r+jGA7lu --api-major 1 --api-minor 51 --build-id 3 --policy 7  --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin
OK: Looks good to me


4. Inject secrets test

Local:
# virt-qemu-sev-validate --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --insecure --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt

# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2

Remote:

# virt-qemu-sev-validate --connect qemu+ssh://host2/system --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --domain vm1 --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt
OK: Looks good to me
OK: Injected 2 secrets

# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2

offline:

# virt-qemu-sev-validate --firmware /usr/share/edk2/ovmf/OVMF.amdsev.fd --tik vm1_tik.bin --tek vm1_tek.bin --measurement iSplmo9vTpslq4CIk8sFIgue/2gQv+FdBLyPNaOxY1RWzVOK9gTRWZ0cYeWDhRfv --api-major 1 --api-minor 51 --build-id 3 --policy 7  --num-cpus 2 --vmsa-cpu0 cpu0.bin --vmsa-cpu1 cpu1.bin --inject-secret 736869e5-84f0-4973-92ec-06879ce3da0b:/tmp/passwd.txt --inject-secret 83c83f7f-1356-4975-8b7e-d3a0b54312c6:/tmp/passwd2.txt --secret-header secret-header.b64 --secret-payload secret-payload.b64
OK: Looks good to me
OK: Injected 2 secrets

# virsh domsetlaunchsecstate vm1 --secrethdr secret-header.b64 --secret secret-payload.b64

# cat /sys/kernel/security/secrets/coco/*
redhat
redhat2
Comment 33 errata-xmlrpc 2023-05-09 07:26:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libvirt bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2171