Bug 1934979

Summary: nbd+tls: Convert to remote image failed
Product: Red Hat Enterprise Linux 8 Reporter: zixchen
Component: qemu-kvmAssignee: Eric Blake <eblake>
qemu-kvm sub component: NBD QA Contact: Tingting Mao <timao>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: high CC: coli, eblake, kkiwi, timao, virt-maint, xuwei
Version: 8.4Keywords: Triaged
Target Milestone: rc   
Target Release: 8.6   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1973517 (view as bug list) Environment:
Last Closed: 2021-09-12 02:40:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1973517    

Description zixchen 2021-03-04 06:10:43 UTC
Description of problem:
Convert nbd+tls image to remote nbd+tls failed.

Version-Release number of selected component (if applicable):
qemu-kvm-5.2.0-9.module+el8.4.0+10182+4161bd91.x86_64
kernel-4.18.0-291.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Export nbd image with tls
-Convert source image
# qemu-nbd -t -e 3 --fork -f raw --object tls-creds-x509,id=nbd_system_luks_server,endpoint=server,dir=/etc/pki/qemu-nbd --tls-creds nbd_system_luks_server /home/nbd/nbd_system.luks
-Convert dst image
# qemu-nbd -t -p 10821 -e 3 --fork -f raw --object tls-creds-x509,id=nbd_stg_luks_server,endpoint=server,dir=/etc/pki/qemu-nbd --tls-creds nbd_stg_luks_server /home/nbd/nbd_stg.luks
2. Boot a guest with source image
# -blockdev node-name=nbd_image1,driver=nbd,auto-read-only=on,discard=unmap,server.type=inet,server.host=hp-z238-02.englab.nay.redhat.com,server.port=10809,tls-creds=image1_access,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=luks,read-only=off,key-secret=image1_encrypt0,cache.direct=on,cache.no-flush=off,file=nbd_image1 \

3. Convert image
# qemu-img convert --object secret,id=image1_encrypt0,data=redhat --object secret,id=convert_encrypt0,data=redhat --object tls-creds-x509,id=image1_access,endpoint=client,dir=/etc/pki/qemu-kvm --object tls-creds-x509,id=convert_access,endpoint=client,dir=/etc/pki/qemu-kvm -O luks -o key-secret=convert_encrypt0 'json:{"file": {"driver": "nbd", "server.type": "inet", "server.host": "hp-z238-02.englab.nay.redhat.com", "server.port": "10809", "tls-creds": "image1_access"}, "driver": "luks", "key-secret": "image1_encrypt0"}' nbd://hp-z238-02.englab.nay.redhat.com:10821 


Actual results:
qemu-img: nbd://hp-z238-02.englab.nay.redhat.com:10821: error while converting luks: Protocol driver 'nbd' does not support image creation, and opening the image failed: TLS negotiation required before option 7 (go)
Did you forget a valid tls-creds?
server reported: Option 0x7 not permitted before TLS

Expected results:
Convert should be successful.

Additional info:

Comment 1 Klaus Heinrich Kiwi 2021-06-15 13:40:25 UTC
Eric, were you able to take a look at this one? It's listed as high in priority.

Zix Chen, do you have results for this test in other releases? I.e., can you help us identify if this is a regression, and if so, when it was introduced?

Comment 2 zixchen 2021-06-17 08:17:47 UTC
I checked the earliest version version on rhel8.4, the issue can be reproduced, if it is a regression, it could come from the earlier compose. I will keep looking on the earlier compose, maybe rhel 8.3 compose.

Version:
qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a.x86_64
kernel-4.18.0-259.el8.x86_64

Steps:
1. Export nbd image with tls
qemu-img create -f raw /home/nbd/nbd_stg.raw 21G
/home/nbd/nbd_stg.raw, fmt=raw size=22548578304
54290 ?        Ssl    0:00 qemu-nbd -t -p 10819 -e 3 --fork -f raw --object tls-creds-x509,id=nbd_stg_raw_server,endpoint=server,dir=/etc/pki/qemu-nbd --tls-creds nbd_stg_raw_server /home/nbd/nbd_stg.raw

2. Boot a guest with a local image
# qemu-img info /home/kvm_autotest_root/images/image_convert.raw
image: /home/kvm_autotest_root/images/image_convert.raw
file format: raw
virtual size: 20 GiB (21474836480 bytes)
disk size: 5.8 GiB

3. Convert image
# /usr/bin/qemu-img convert --object tls-creds-x509,id=remote_access,endpoint=client,dir=/etc/pki/qemu-kvm -f raw -O raw /home/kvm_autotest_root/images/image_convert.raw nbd://ibm-x3850x6-04.lab.eng.pek2.redhat.com:10819
qemu-img: nbd://ibm-x3850x6-04.lab.eng.pek2.redhat.com:10819: error while converting raw: Protocol driver 'nbd' does not support image creation, and opening the image failed: TLS negotiation required before option 7 (go)
Did you forget a valid tls-creds?
server reported: Option 0x7 not permitted before TLS


As this issue can be reproduced on rhel9, so I will clone this bug to rhel9.

Comment 3 Klaus Heinrich Kiwi 2021-08-24 17:51:37 UTC
Eric, any updates to this one?

Comment 4 John Ferlan 2021-09-09 11:45:03 UTC
Bulk update: Move RHEL-AV bugs to RHEL8 with existing RHEL9 clone.

Comment 5 Eric Blake 2021-09-10 21:37:35 UTC
(In reply to zixchen from comment #2)
> I checked the earliest version version on rhel8.4, the issue can be
> reproduced, if it is a regression, it could come from the earlier compose. I
> will keep looking on the earlier compose, maybe rhel 8.3 compose.
> 
> Version:
> qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a.x86_64
> kernel-4.18.0-259.el8.x86_64
> 
> Steps:
> 1. Export nbd image with tls
> qemu-img create -f raw /home/nbd/nbd_stg.raw 21G
> /home/nbd/nbd_stg.raw, fmt=raw size=22548578304
> 54290 ?        Ssl    0:00 qemu-nbd -t -p 10819 -e 3 --fork -f raw --object
> tls-creds-x509,id=nbd_stg_raw_server,endpoint=server,dir=/etc/pki/qemu-nbd
> --tls-creds nbd_stg_raw_server /home/nbd/nbd_stg.raw

Note how in this command line, you create a TLS object named nbd_stg_raw_server, AND tell qemu-img to use --tls-creds with that TLS object.

> 
> 2. Boot a guest with a local image
> # qemu-img info /home/kvm_autotest_root/images/image_convert.raw
> image: /home/kvm_autotest_root/images/image_convert.raw
> file format: raw
> virtual size: 20 GiB (21474836480 bytes)
> disk size: 5.8 GiB
> 
> 3. Convert image
> # /usr/bin/qemu-img convert --object
> tls-creds-x509,id=remote_access,endpoint=client,dir=/etc/pki/qemu-kvm -f raw
> -O raw /home/kvm_autotest_root/images/image_convert.raw
> nbd://ibm-x3850x6-04.lab.eng.pek2.redhat.com:10819
> qemu-img: nbd://ibm-x3850x6-04.lab.eng.pek2.redhat.com:10819: error while
> converting raw: Protocol driver 'nbd' does not support image creation, and
> opening the image failed: TLS negotiation required before option 7 (go)
> Did you forget a valid tls-creds?
> server reported: Option 0x7 not permitted before TLS

But in this command line, you are creating the TLS object remote_access, but never using it!  The error message is trying to tell you that the server expects TLS, but that the client isn't requesting to use it.

You need to modify your command line to tell qemu-img to use the just-created TLS object:

myhost=server.type=inet,server.host=ibm-x3850x6-04.lab.eng.pek2.redhat.com,server.port=10819
qemu-img convert --object tls-creds-x509,id=remote_access,endpoint=client,dir=/etc/pki/qemu-kvm \
  -f raw /home/kvm_autotest_root/images/image_convert.raw \
  -n --target-image-opts driver=nbd,$myhost,tls-creds=remote_access

With that in place, you should be able to convert, now.

Comment 6 Eric Blake 2021-09-10 21:39:11 UTC
I'm inclined to close this as not a bug, as the command line usage was wrong; but I'll wait for your confirmation that using the correct command line works for your test.

Comment 7 Tingting Mao 2021-09-12 02:40:46 UTC
Yes, coverting works well when added the TLS objects to the related iamges. Close this bug accordingly. Thanks Eric.


Tested with:
qemu-kvm-6.0.0-28.module+el8.5.0+12271+fffa967b
kernel-4.18.0-339.el8.x86_64


Steps:
Source image:
# qemu-img info rhel850-64-virtio.raw 
image: rhel850-64-virtio.raw
file format: raw
virtual size: 20 GiB (21474836480 bytes)
disk size: 3.91 GiB
# qemu-nbd -t -e 3 --fork -f raw --object tls-creds-x509,id=nbd_system_server,endpoint=server,dir=/etc/pki/qemu-nbd --tls-creds nbd_system_server rhel850-64-virtio.raw &
[1] 118343


Target image:
# qemu-img create -f raw target.img 20G
# qemu-nbd -t -p 10821 -e 3 --fork -f raw --object tls-creds-x509,id=nbd_stg_server,endpoint=server,dir=/etc/pki/qemu-nbd --tls-creds nbd_stg_server target.img &
[1] 118383


Converting:
# qemu-img convert -f raw -O raw --object tls-creds-x509,id=image1_access,endpoint=client,dir=/etc/pki/qemu-kvm 'json:{"file": {"driver": "nbd", "server.type": "inet", "server.host": "ibm-x3650m5-09.lab.eng.pek2.redhat.com", "server.port": "10809", "tls-creds": "image1_access"}}' 'json:{"file": {"driver": "nbd", "server.type": "inet", "server.host": "ibm-x3650m5-09.lab.eng.pek2.redhat.com", "server.port": "10821", "tls-creds": "image1_access"}}' -n -p
    (100.00/100%)

# qemu-img convert -f raw --object tls-creds-x509,id=image1_access,endpoint=client,dir=/etc/pki/qemu-kvm 'json:{"file": {"driver": "nbd", "server.type": "inet", "server.host": "ibm-x3650m5-09.lab.eng.pek2.redhat.com", "server.port": "10809", "tls-creds": "image1_access"}}' --target-image-opts driver=nbd,server.type=inet,server.host=ibm-x3650m5-09.lab.eng.pek2.redhat.com,server.port=10821,tls-creds=image1_access -n -p
    (100.00/100%)