Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1939082

Summary: [incremental_backup][--reuse-external] extended attributes will be left on backup target/scratch if we destroy vm during backup process
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: yisun
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: yisun
Severity: low Docs Contact:
Priority: medium    
Version: 8.4CC: jdenemar, jen, nsoffer, pkrempa, virt-maint, ymankad
Target Milestone: rcKeywords: Triaged
Target Release: 8.4Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-7.0.0-11.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-25 06:48:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yisun 2021-03-15 15:08:00 UTC
Description:
[incremental_backup][--reuse-external] extended attributes will be left on backup target/scratch if we destroy vm during backup process

Versions:
libvirt-7.0.0-9.module+el8.4.0+10326+5e50a3b6.x86_64
qemu-kvm-5.2.0-11.module+el8.4.0+10268+62bcbbed.x86_64

How reproducible:
100%

Scene 1. Push mode backup
1. create a qcow2 image as backup target
[root@dell-per740-01 ~]# qemu-img create -f qcow2 /tmp/vda.backup 10G
Formatting '/tmp/vda.backup', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 lazy_refcounts=off refcount_bits=16
[root@dell-per740-01 ~]# cat push.xml 
<domainbackup>
  <disks>
    <disk name='vda' type='file'>
      <target file='/tmp/vda.backup'/>
      <driver type='qcow2'/>
    </disk>
  </disks>
</domainbackup>

2. start push mode backup
[root@dell-per740-01 ~]# virsh backup-begin vm1 push.xml --reuse-external
Backup started

3. before backup ends, destroy vm to cancel the backup job
[root@dell-per740-01 ~]# virsh destroy vm1
Domain 'vm1' destroyed


[root@dell-per740-01 ~]# virsh domjobinfo vm1 --completed
Job type:         Cancelled   
Operation:        Backup      

4. now the extended attributes not cleaned for the target image
[root@dell-per740-01 ~]# getfattr -n trusted.libvirt.security.ref_selinux /tmp/vda.backup 
getfattr: Removing leading '/' from absolute path names
# file: tmp/vda.backup
trusted.libvirt.security.ref_selinux="1"

5. if we do the backup again, it'll be failed
[root@dell-per740-01 ~]# virsh start vm1
Domain 'vm1' started

[root@dell-per740-01 ~]# virsh backup-begin vm1 push.xml --reuse-external
error: Requested operation is not valid: Setting different SELinux label on /tmp/vda.backup which is already in use



Scene 2. Pull mode
similar steps:
[root@dell-per740-01 ~]# qemu-img create -f qcow2 /tmp/scratch.vda 10G
Formatting '/tmp/scratch.vda', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 lazy_refcounts=off refcount_bits=16
[root@dell-per740-01 ~]# cat pull.xml 
<domainbackup mode='pull'>
  <server name="localhost" port="10809"/>
  <disks>
    <disk name='vda' backup='yes' type='file'>
    <scratch file='/tmp/scratch.vda'/>
    </disk>
  </disks>
</domainbackup>
[root@dell-per740-01 ~]# virsh backup-begin vm1 pull.xml --reuse-external
Backup started

[root@dell-per740-01 ~]# virsh destroy vm1
Domain 'vm1' destroyed

[root@dell-per740-01 ~]# getfattr -n trusted.libvirt.security.ref_selinux /tmp/scratch.vda 
getfattr: Removing leading '/' from absolute path names
# file: tmp/scratch.vda
trusted.libvirt.security.ref_selinux="1"

[root@dell-per740-01 ~]# virsh start vm1
Domain 'vm1' started

[root@dell-per740-01 ~]# virsh backup-begin vm1 pull.xml --reuse-external
error: Requested operation is not valid: Setting different SELinux label on /tmp/scratch.vda which is already in use

Expected result:
extended attributes on the backup target/scratch files should be cleaned when vm is down.

Comment 5 Peter Krempa 2021-03-19 16:08:02 UTC
Fixed upstream:

commit 8967ad7be652f9949339748ce0f24e4ce0ade707
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 17 16:30:31 2021 +0100

    qemu: backup: Restore security label on backup disk store image on VM termination
    
    When the backup job is terminated normally the security label is
    restored by the blockjob finishing handler.
    
    If the VM dies or is destroyed that wouldn't happen as the blockjob
    handler wouldn't be called.
    
    Restore the security label on disk store where we remember that the job
    was running at the point when 'qemuBackupJobTerminate' was called.
    
    Not resetting the security label means that we also leak the xattr
    attributes remembering the label which prevents any further use of the
    file, which is a problem for block devices.
    
    This also requires that the call to 'qemuBackupJobTerminate' from
    'qemuProcessStop' happens only after 'vm->pid' was reset as otherwise
    the security subdrivers attempt to enter the process namespace which
    fails if the process isn't running any more.

Comment 8 yisun 2021-03-23 06:55:42 UTC
Tested with scratch build, result is PASS
Set VERIFIED field to TESTED
And a auto job is triggered to do a regression test. I'll give feedback later when it's finished.

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# rpm -qa | egrep '^libvirt-7|^qemu-kvm-5'
libvirt-7.0.0-10.el8_rc.f9bf3fb262.x86_64
qemu-kvm-5.2.0-14.module+el8.4.0+10425+ad586fa5.x86_64

====================
PUSH MODE:
====================
(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# qemu-img create -f qcow2 /tmp/vda.backup 10G
Formatting '/tmp/vda.backup', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 lazy_refcounts=off refcount_bits=16

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# cat push.xml 
<domainbackup>
  <disks>
    <disk name='vda' type='file'>
      <target file='/tmp/vda.backup'/>
      <driver type='qcow2'/>
    </disk>
  </disks>
</domainbackup>


(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# virsh backup-begin vm1 push.xml --reuse-external
Backup started

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# virsh destroy vm1
Domain 'vm1' destroyed

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# virsh domjobinfo vm1 --completed
Job type:         Cancelled   
Operation:        Backup      

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# getfattr -n trusted.libvirt.security.ref_selinux /tmp/vda.backup 
/tmp/vda.backup: trusted.libvirt.security.ref_selinux: No such attribute


====================
PULL MODE:
====================
(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# qemu-img create -f qcow2 /tmp/scratch.vda 10G
Formatting '/tmp/scratch.vda', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=10737418240 lazy_refcounts=off refcount_bits=16

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# cat pull.xml 
<domainbackup mode='pull'>
  <server name="localhost" port="10809"/>
  <disks>
    <disk name='vda' backup='yes' type='file'>
    <scratch file='/tmp/scratch.vda'/>
    </disk>
  </disks>
</domainbackup>


(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]#  virsh backup-begin vm1 pull.xml --reuse-external
Backup started

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# virsh destroy vm1
Domain 'vm1' destroyed

(.libvirt-ci-venv-ci-runtest-XptyQO) [root@dell-per740-08 bz1939082]# getfattr -n trusted.libvirt.security.ref_selinux /tmp/scratch.vda 
/tmp/scratch.vda: trusted.libvirt.security.ref_selinux: No such attribute

Comment 9 yisun 2021-03-23 10:44:05 UTC
no regression failures in the auto test with the scratch build.

Comment 12 yisun 2021-03-29 11:34:52 UTC
test with same steps as https://bugzilla.redhat.com/show_bug.cgi?id=1939082#c8
PASSED with latest build: libvirt-7.0.0-11.module+el8.4.0+10505+3a8d753f.x86_64

Comment 14 errata-xmlrpc 2021-05-25 06:48:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098