Bug 1822631

Implemented upstream: 

commit 423576679a5235ebb294bb9474f76abb7548e5e5
Author: Peter Krempa <pkrempa>
Date:   Wed Jul 1 12:38:42 2020 +0200

    qemu: backup: Setup TLS environment for pull-mode backup jobs
    
    Use the configured TLS env to setup encryption of the TLS transport.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1822631
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit fc6aaf6a194e6ee01cba3ee59e9fc3603189b77c
Author: Peter Krempa <pkrempa>
Date:   Wed Jul 1 12:25:42 2020 +0200

    conf: backup: Add 'tls' attribute for 'server' element
    
    Allow enabling TLS for the NBD server used to do pull-mode backups. Note
    that documentation already mentions 'tls', so this just implements the
    schema and XML bits.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit d37a2cd5edcdc4aef4df09f5ca4f274a17072468
Author: Peter Krempa <pkrempa>
Date:   Fri Jun 26 16:37:16 2020 +0200

    qemu: conf: Add configuration of TLS environment for NBD transport of pull-backups
    
    TLS is required to transport backed-up data securely when using
    pull-mode backups.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit fffc147ba24aecd661fcb81725aa606386e88b88
Author: Peter Krempa <pkrempa>
Date:   Wed Jul 1 09:48:27 2020 +0200

    conf: backup: Store 'tlsAlias' and 'tlsSecretAlias' as internals of a backup
    
    Add fields for storing the aliases necessary to clean up the TLS env for
    a backup job after it finishes.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

commit bfd972167111c37f53bd8df1aa72474600dbed94
Author: Peter Krempa <pkrempa>
Date:   Wed Jul 1 09:56:46 2020 +0200

    testCompareBackupXML: Add infrastructure for testing internal fields
    
    There are few internal fields of the backup XML. Propagate the
    'internal' flag so that the test can verify the XML infrastructure.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Eric Blake <eblake>

Test with libvirt-6.6.0-2.module+el8.3.0+7567+dc41c0a9.x86_64
Result is: PASS

Test setup
Following actions cwd is /etc/pki/libvirt-backup, which should be same as ‘backup_tls_x509_cert_dir’ in qemu.conf


CA cert setup:

1. prepare a CA private key

[root@dell-per730-67 libvirt-backup]#  certtool --generate-privkey > ca-key.pem


2. prepare a CA info file

[root@dell-per730-67 libvirt-backup]# cat ca.info

cn = ca.redhat.com

ca

cert_signing_key


3. generate a CA cert signed by the private key

[root@dell-per730-67 libvirt-backup]# certtool --generate-self-signed --load-privkey ca-key.pem --template ca.info --outfile ca-cert.pem


Server side cert setup：

1. prepare a server private key

[root@dell-per730-67 libvirt-backup]# certtool --generate-privkey > server-key.pem


if we need to encrypt the private key, use following cmd:
# certtool --password redhat --generate-privkey > server-key.pem



2. prepare a server info file

[root@dell-per730-67 libvirt-backup]# cat server.info 

organization = Red Hat

cn = dell-per730-67.lab.eng.pek2.redhat.com <=== server hostname

tls_www_server

encryption_key

signing_key


3. generate a server cert

If private key is encrypted, we’ll need to setup GUNTLS_PIN before generate certificate

[root@dell-per730-67 libvirt-backup]# export GNUTLS_PIN=redhat


[root@dell-per730-67 libvirt-backup]# certtool --generate-certificate --load-privkey server-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template server.info --outfile server-cert.pem


If private key is encrypted, we’ll need to setup GUNTLS_PIN before generate certificate

[root@dell-per730-67 libvirt-backup]# export GNUTLS_PIN=redhat


Client side cert setup：

1. prepare a client private key

[root@dell-per730-67 libvirt-backup]# certtool --generate-privkey > client-key.pem

Generating a 3072 bit RSA private key…



2. prepare a client info file

[root@dell-per730-67 libvirt-backup]# cat client.info 

country = GB

state = London

locality = London

organization = Red Hat

cn = dell-per730-67.lab.eng.pek2.redhat.com <=== client hostname

tls_www_client

encryption_key

signing_key

 


3. generate a client cert

[root@dell-per730-67 libvirt-backup]# certtool --generate-certificate --load-privkey client-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template client.info --outfile client-cert.pem

Generating a signed certificate…

Test tools
Script to dump data for incremental backup - ‘get_inc_backup.sh’
This script is to get the data from the nbd export, and dump it to a local image. 

[root@dell-per730-67 ~]# cat get_inc_backup.sh 

if test $# -lt 6 || test $# -gt 7; then

  echo 'usage: copyif nbd_server nbd_export nbd_port target_img bitmap tls_dir'

  return 1

fi

NBD_SERVER=$1

NBD_EXPORT=$2

NBD_PORT=$3

TARGET_IMG=$4

BITMAP=$5

TLS_DIR=$6

map_from="--image-opts driver=nbd,export=vdb,server.type=inet"

map_from+=",server.host=$NBD_SERVER,server.port=10809,tls-creds=tls0"

map_from+=",x-dirty-bitmap=qemu:dirty-bitmap:$BITMAP"

state=false

echo $map_from

qemu-img rebase -u -f qcow2 -F raw -b 'json:{"file":{"driver":"nbd", "server":{"host":"'$NBD_SERVER'", "port":'$NBD_PORT', "type":"inet"}, "export":"'$NBD_EXPORT'", "tls-creds":"tls0"}}' $TARGET_IMG

while read line; do

  [[ $line =~ .*start.:.([0-9]*).*length.:.([0-9]*).*data.:.$state.* ]] || continue

  start=${BASH_REMATCH[1]} len=${BASH_REMATCH[2]}

  echo

  echo " $start $len:"

  qemu-io -C -c "r $start $len" -f qcow2 $TARGET_IMG --object tls-creds-x509,id=tls0,endpoint=client,dir=$TLS_DIR

done < <(qemu-img map --object tls-creds-x509,id=tls0,endpoint=client,dir=$TLS_DIR --output=json $map_from)

qemu-img rebase -u -f qcow2 -b '' $TARGET_IMG



Usage:

Vm has a existing checkpoint
[root@dell-per730-67 ~]# virsh checkpoint-list vm1
 Name           Creation Time

-------------------------------------------

 checkpoint_1   2020-08-17 04:16:20 -0400

Prepare a backup xml
[root@dell-per730-67 ~]# cat bkup_inc1.xml 
<domainbackup mode="pull">
  <incremental>checkpoint_1</incremental>
  <server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/>
  <disks>
    <disk backup="no" name="vda" />
    <disk backup="yes" name="vdb" type="file">
      <scratch file="/tmp/scratch_file_0" />
    </disk>
  </disks>
</domainbackup>
Start the backup
[root@dell-per730-67 ~]# virsh backup-begin vm1 bkup_inc1.xml 
Backup started
Prepare the target image
[root@dell-per730-67 ~]# qemu-img create -f qcow2 vdb.inc1.backup 1G
Formatting 'vdb.inc1.backup', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16

Run the script
[root@dell-per730-67 ~]# ./get_inc_backup.sh dell-per730-67.lab.eng.pek2.redhat.com vdb 10809 vdb.inc1.backup backup-vdb /etc/pki/libvirt-backup/
--image-opts driver=nbd,export=vdb,server.type=inet,server.host=dell-per730-67.lab.eng.pek2.redhat.com,server.port=10809,tls-creds=tls0,x-dirty-bitmap=qemu:dirty-bitmap:backup-vdb

20971520 128974848:

read 128974848/128974848 bytes at offset 20971520

123 MiB, 1 ops; 00.53 sec (229.993 MiB/sec and 1.8699 ops/sec)


Test scenarios
backup with default backup_tls_x509_cert_dir
Positive test - Full backup with default backup_tls_x509_cert_dir
Prepare a vdb for vm
[root@dell-per730-67 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/vdb.qcow2 1G
Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16

[root@dell-per730-67 ~]# virsh domblklist vm1

 Target   Source

--------------------------------------------------------
...

 vdb      /var/lib/libvirt/images/vdb.qcow2

Make sure comment out following params in qemu.conf:
# backup_tls_x509_cert_dir = "/etc/pki/libvirt-backup"
# backup_tls_x509_verify = 1

# backup_tls_x509_secret_uuid = "00000000-0000-0000-0000-000000000000"

Restart libvirtd daemon
[root@dell-per730-67 ~]# systemctl restart libvirtd
Generate 123MB data in vm’s vdb
[root@dell-per730-67 ~]# virsh start vm1 --console
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=100 count=123; sync
Start a full backup
[root@dell-per730-67 ~]# cat backup_pull_0.xml 
<domainbackup mode="pull"><server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/><disks><disk backup="no" name="vda" /><disk backup="yes" name="vdb" type="file"><scratch file="/tmp/scratch_file_0" /></disk></disks></domainbackup>

[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_pull_0.xml 

Backup started

Dump the backup data to a local image
[root@dell-per730-67 ~]# qemu-img convert -O qcow2 --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/libvirt-backup/ 'json:{"file":{"driver":"nbd", "server":{"host":"dell-per730-67.lab.eng.pek2.redhat.com", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}' test.qcow2
Check the new image has 123MB data
[root@dell-per730-67 ~]# qemu-img info test.qcow2 
image: test.qcow2

file format: qcow2

virtual size: 1 GiB (1073741824 bytes)

disk size: 123 MiB

cluster_size: 65536

Format specific information:

    compat: 1.1

    compression type: zlib

    lazy refcounts: false

    refcount bits: 16

    corrupt: false

backup with custom backup_tls_x509_cert_dir
Positive test - Incremental backup with custom backup_tls_x509_cert_dir
Mkdir /etc/pki/test and set backup tls dir to it
[root@dell-per730-67 pki]# mkdir /etc/pki/test
[root@dell-per730-67 pki]# cat /etc/libvirt/qemu.conf | grep backup_tls_x509_cert_dir
backup_tls_x509_cert_dir = "/etc/pki/test"
[root@dell-per730-67 ~]# systemctl restart libvirtd

Prepare full backup&checkpoint xml
[root@dell-per730-67 ~]# cat backup_full.xml 
<domainbackup mode="pull">

<server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/>

<disks>

<disk backup="no" name="vda" />

<disk backup="yes" name="vdb" type="file">

<scratch file="/tmp/scratch_file_0" />

</disk>

</disks>

</domainbackup>


[root@dell-per730-67 ~]# cat check_0.xml 

<domaincheckpoint>

<disks>

<disk checkpoint="no" name="vda" />

<disk checkpoint="bitmap" name="vdb" />

</disks>

<name>checkpoint_0</name>

<description>desc of cp_0</description>

</domaincheckpoint>

[root@dell-per730-67 ~]# cat backup_inc.xml 

<domainbackup mode="pull">

<incremental>checkpoint_0</incremental>

<server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/>

<disks>

<disk backup="no" name="vda" />

<disk backup="yes" name="vdb" type="file">

<scratch file="/tmp/scratch_file_0" />

</disk>

</disks>

</domainbackup>

[root@dell-per730-67 ~]# cat check_1.xml 

<domaincheckpoint>

<disks>

<disk checkpoint="no" name="vda" />

<disk checkpoint="bitmap" name="vdb" />

</disks>

<name>checkpoint_1</name>

<description>desc of cp_1</description>

</domaincheckpoint>


Generate 123MB data in vm’s vdb
[root@dell-per730-67 ~]# virsh console vm1
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=20 count=123; sync 
Start full backup without preparing any certs
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml check_0.xml 
error: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/test/ca-cert.pem: No such file or directory

⇐ failed as expected

[root@dell-per730-67 ~]# virsh checkpoint-list vm1

 Name   Creation Time

-----------------------
⇐ no checkpoint created

Prepare ca+server key/certs in /etc/pki/test/
[root@dell-per730-67 pki]# ll /etc/pki/test
total 131056

-rw-r--r--. 1 root root       1448 Aug 18 01:45 ca-cert.pem

-rw-r--r--. 1 root root       8177 Aug 18 01:45 ca-key.pem

-rw-r--r--. 1 root root         39 Aug 18 01:45 ca.info

-rw-r--r--. 1 root root       1639 Aug 18 01:45 client-cert.pem

-rw-r--r--. 1 root root       8177 Aug 18 01:45 client-key.pem

-rw-r--r--. 1 root root        155 Aug 18 01:45 client.info

-rw-r--r--. 1 root root       1578 Aug 18 01:45 server-cert.pem

-rw-r--r--. 1 root root       8180 Aug 18 01:45 server-key.pem

-rw-r--r--. 1 root root        109 Aug 18 01:45 server.info

Start the full backup
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml check_0.xml 
Backup started


[root@dell-per730-67 ~]# virsh checkpoint-list vm1

setlocale: No such file or directory

 Name           Creation Time

-------------------------------------------

 checkpoint_0   2020-08-18 01:46:26 -0400

⇐ checkpoint_0 created


[root@dell-per730-67 ~]# virsh domjobabort vm1

Generate 321MB data in vm1’s vdb
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=500 count=321; sync
Start the incremental backup (from checkpoint_0)
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_inc.xml check_1.xml 
Backup started

Generate incremental backup data to a local image
[root@dell-per730-67 ~]# qemu-img create -f qcow2 vdb.inc.backup 1G
Formatting 'vdb.inc.backup', fmt=qcow2 cluster_size=65536 compression_type=zlib size=1073741824 lazy_refcounts=off refcount_bits=16

[root@dell-per730-67 ~]# ./get_inc_backup.sh dell-per730-67.lab.eng.pek2.redhat.com vdb 10809 vdb.inc.backup backup-vdb /etc/pki/test/

--image-opts driver=nbd,export=vdb,server.type=inet,server.host=dell-per730-67.lab.eng.pek2.redhat.com,server.port=10809,tls-creds=tls0,x-dirty-bitmap=qemu:dirty-bitmap:backup-vdb

 524288000 336592896:

read 336592896/336592896 bytes at offset 524288000

321 MiB, 1 ops; 0:00:01.35 (238.349 MiB/sec and 0.7425 ops/sec)

[root@dell-per730-67 ~]# qmeu-img info vdb.inc.backup 

-bash: qmeu-img: command not found

[root@dell-per730-67 ~]# qemu-img info vdb.inc.backup 

image: vdb.inc.backup

file format: qcow2

virtual size: 1 GiB (1073741824 bytes)

disk size: 321 MiB

⇐ the disk size is 321MB which is same as the incremental data we generated after full backup

cluster_size: 65536

Format specific information:

    compat: 1.1

    compression type: zlib

    lazy refcounts: false

    refcount bits: 16

    corrupt: false

Full backup with backup_tls_x509_secret_uuid
Env preparation
Prepare the server cert:

Prepare CA key+cert as before

Prepare server private key:
[root@dell-per730-67 libvirt-backup]# certtool --password redhat --generate-privkey > server-key.pem
Assuming PKCS #8 format...

Generating a 3072 bit RSA private key...

Set the local variable GNUTLS_PIN
[root@dell-per730-67 libvirt-backup]# export GNUTLS_PIN=redhat
Generate the server’s signed cert
[root@dell-per730-67 libvirt-backup]# certtool --generate-certificate --load-privkey server-key.pem --load-ca-certificate ca-cert.pem --load-ca-privkey ca-key.pem --template server.info --outfile server-cert.pem
Prepare libvirt secret:

Prepare secret xml
[root@dell-per730-67 libvirt-backup]# cat secret.xml
<secret ephemeral='no' private='no'>

  <uuid>e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d</uuid>

  <description>TLS passphrase</description>

  <usage type='tls'>

    <name>backup_tls_secret</name>

  </usage>

</secret>

Define the secret
[root@dell-per730-67 ~]# virsh secret-define secret.xml


Positive test - use a libvirt secret with correct passphrase to do the backup
Set backup_tls_x509_secret_uuid to the libvirt secret uuid we just prepared
[root@dell-per730-67 ~]# cat /etc/libvirt/qemu.conf | grep 'backup_tls_x509_secret_uuid ='
backup_tls_x509_secret_uuid = "e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d"

Set password=redhat to the secret
[root@dell-per730-67 ~]# MYSECRET=`printf %s "redhat" | base64`

[root@dell-per730-67 ~]# virsh secret-set-value e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d $MYSECRET

error: Passing secret value as command-line argument is insecure!

Secret value set

Generate 111MB data in vm’s vdb
[root@dell-per730-67 ~]# virsh start vm1 --console
...

[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M seek=20 count=111; sync

Start the backup
[root@dell-per730-67 ~]# cat backup_full.xml 
<domainbackup mode="pull">

        <server name="dell-per730-67.lab.eng.pek2.redhat.com" port="10809" tls="yes"/>

                    <disks>

<disk backup="no" name="vda" />

<disk backup="yes" name="vdb" type="file">

<scratch file="/tmp/scratch_file_0" />

</disk>

         </disks>

</domainbackup>


[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml  

Backup started

Dump the backup data to a local image
[root@dell-per730-67 ~]# qemu-img convert -O qcow2 --object tls-creds-x509,id=tls0,endpoint=client,dir=/etc/pki/libvirt-backup 'json:{"file":{"driver":"nbd", "server":{"host":"dell-per730-67.lab.eng.pek2.redhat.com", "port":10809, "type":"inet"}, "export":"vdb", "tls-creds":"tls0"}}' test.qcow2

[root@dell-per730-67 ~]# qemu-img info test.qcow2 

image: test.qcow2

file format: qcow2

virtual size: 1 GiB (1073741824 bytes)

disk size: 111 MiB
⇐ the disk size is 111MB, same as the data we generated to vm’s vdb

cluster_size: 65536

Format specific information:

    compat: 1.1

    compression type: zlib

    lazy refcounts: false

    refcount bits: 16

    corrupt: false


Abort the backup job
[root@dell-per730-67 ~]# virsh domjobabort vm1

Negative test - use libvirt secret with incorrect passphrase to do the backup
Set the libvirt secret with a wrong password
[root@dell-per730-67 ~]# MYSECRET=`printf %s "greenhat" | base64`

[root@dell-per730-67 ~]# virsh secret-set-value e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d $MYSECRET

error: Passing secret value as command-line argument is insecure!

Secret value set

Try to start the backup job
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml  
error: internal error: unable to execute QEMU command 'object-add': Cannot load certificate '/etc/pki/libvirt-backup/server-cert.pem' & key '/etc/pki/libvirt-backup/server-key.pem': Decryption has failed.

⇐ failed as expected

Negative test - disable backup_tls_x509_secret_uuid and do the backup
Not provide backup_tls_x509_secret_uuid
[root@dell-per730-67 ~]# cat /etc/libvirt/qemu.conf | grep 'backup_tls_x509_secret_uuid ='
# backup_tls_x509_secret_uuid = "e3ba1f0c-01dc-4edd-97e5-1f5edb93b50d"
⇐ it’s commented out

Restart libvirtd daemon
[root@dell-per730-67 ~]# systemctl restart libvirtd
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml  
error: internal error: unable to execute QEMU command 'object-add': Cannot load certificate '/etc/pki/libvirt-backup/server-cert.pem' & key '/etc/pki/libvirt-backup/server-key.pem': Decryption has failed.

⇐ failed as expected

Test backup_tls_x509_verify param in qemu.conf
Negative test - set ‘backup_tls_x509_verify = 1’ and not providing the ca-cert.pem
Prepare qemu.conf as follow and restart libvirtd
[root@dell-per730-67 pki]# cat /etc/libvirt/qemu.conf | grep "backup_tls_x509_verify ="
backup_tls_x509_verify = 1

[root@dell-per730-67 pki]# systemctl restart libvirtd

Move the ca-cert.pem to tmp dir
[root@dell-per730-67 pki]# mv /etc/pki/libvirt-backup/ca-cert.pem /tmp/
Start the backup, error happens as expected
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml  
error: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/libvirt-backup/ca-cert.pem: No such file or directory

Positive test - set ‘backup_tls_x509_verify = 0’ and not providing the ca-cert.pem
Prepare qemu.conf as follow and restart libvirtd
[root@dell-per730-67 pki]# cat /etc/libvirt/qemu.conf | grep "backup_tls_x509_verify ="
backup_tls_x509_verify = 0


[root@dell-per730-67 pki]# systemctl restart libvirtd

Move the ca-cert.pem to tmp dir
[root@dell-per730-67 pki]# mv /etc/pki/libvirt-backup/ca-cert.pem /tmp/
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_full.xml  
error: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/libvirt-backup/ca-cert.pem: No such file or directory
⇐ not as expected ?

Other corner cases
Set tls=’yes’ for unix socket pull mode backup
Prepare a backup xml as follow:
[root@dell-per730-67 ~]# cat backup_unix.xml 
<domainbackup mode="pull">

<server socket="/tmp/pull_backup.socket" transport="unix" tls="yes"/>

    <disks>

<disk backup="no" name="vda" />

<disk backup="yes" name="vdb" type="file">

                    <scratch file="/tmp/scratch_file_1" />

        </disk>

    </disks>

</domainbackup>

Start the backup
[root@dell-per730-67 ~]# virsh backup-begin vm1 backup_unix.xml 
error: internal error: unable to execute QEMU command 'nbd-server-start': TLS is only supported with IPv4/IPv6
⇐ error msg is clear

well, the format of above comment is a little ugly...
Same test info can be found at:
https://docs.google.com/document/d/1DaFtumX6M9XOl-n1Y3aiICbjbT49QqCAQDKcD2id1F4

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137