Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2190276

Summary: virt-v2v should relabel Linux guest filesystems in parallel
Product: Red Hat Enterprise Linux 9 Reporter: Richard W.M. Jones <rjones>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: mxie <mxie>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.3CC: chhu, hongzliu, juzhou, lersek, mxie, qzhang, rjones, tyan, tzheng, virt-maint, vwu, xiaodwan, ymao, yoguo
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libguestfs-1.50.1-4.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:24:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2177705    
Bug Blocks:    

Description Richard W.M. Jones 2023-04-27 18:10:48 UTC
Description of problem:

virt-v2v relabels (SELinux) the filesystem of Linux guests after
making changes.  Recently a bug was reported by Virtuozzo about
guests taking a very long time to relabel (because they have
apparently enormous filesystems):

https://listman.redhat.com/archives/libguestfs/2023-April/031367.html
https://listman.redhat.com/archives/libguestfs/2023-April/031401.html

Andrey from Virtuozzo also provided a patch so relabelling is
done in parallel, which should significantly speed things up, and
seems low risk.  It is now upstream here:

https://github.com/libguestfs/libguestfs/commit/9ced5fac8c1f0f8ff7ed2b5671c1c7f5f0bfa875
https://github.com/libguestfs/libguestfs/commit/152d6e4bdf2dac88856a4ff83cf73451f897d4d4
https://github.com/libguestfs/libguestfs/commit/d0d8e6738477148a7b752348f9364a3b8faed67f

We should consider including this patch.

The patch is actually for libguestfs, but it affects virt-v2v; and
virt-customize if the --smp option is used.

Version-Release number of selected component (if applicable):

libguestfs-1.50.1-3.el9
but improving the performance of virt-v2v-2.3.4-1.el9

How reproducible:

?

Steps to Reproduce:
1. Take a Fedora or RHEL guest with a large filesystem.
2. Run virt-v2v on it.
3. Conversion should run faster with this patch.

Comment 1 Richard W.M. Jones 2023-05-05 07:28:24 UTC
For testing this, the first, most important thing is that it doesn't break
existing Linux guest conversions.

It should also make them faster.  This may be noticable even with a regular
RHEL or Fedora guest conversion just by timing before and after.  There
may only be a few seconds difference.

The real test would be a RHEL or Fedora guest with a very large filesystem
(ie. lots of small files and directories - many millions of them).  If you
create that kind of guest then the difference in conversion time may be
significant.  You could try something like this inside a guest ...

$ wget https://ftp.gnu.org/gnu/glibc/glibc-2.37.tar.xz
$ xzcat glibc-2.37.tar.xz | tar xf -
$ du -sh glibc-2.37
278M	glibc-2.37
$ find glibc-2.37 | wc -l
20202

(or another large software package would work)

Then create many copies, eg:

$ for f in {01..10}; do cp -a glibc-2.37 glibc-2.37.$f; done

$ du -sh glibc-2.37*
278M	glibc-2.37
278M	glibc-2.37.01
278M	glibc-2.37.02
278M	glibc-2.37.03
278M	glibc-2.37.04
278M	glibc-2.37.05
278M	glibc-2.37.06
278M	glibc-2.37.07
278M	glibc-2.37.08
278M	glibc-2.37.09
278M	glibc-2.37.10
18M	glibc-2.37.tar.xz
$ find glibc-2.37* | wc -l
222223

(increase "10" to a larger number)

Comment 2 mxie@redhat.com 2023-05-06 12:11:24 UTC
Reproduce the bug with below builds:
libguestfs-1.50.1-3.el9.x86_64
guestfs-tools-1.50.1-1.el9.x86_64
virt-v2v-2.3.4-1.el9.x86_64
nbdkit-server-1.34.1-1.el9.x86_64
libnbd-1.16.0-1.el9.x86_64

Steps to reproduce:
Scenario1:
1.1 Prepare a rhel9 guest with large disk and SELINUX=enforcing on VMware environment, then convert the guest from VMware to local libvirt by v2v
# virt-v2v -ic vpx://root.74.72/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=24:40:7E:C8:C8:1F:9C:DF:E4:E0:48:D0:9E:25:64:94:64:AF:C6:8C  -ip /home/passwd  esx6.7-rhel9.3-with-large-disk -v -x |& ts > v2v-convert-guest-with-large-disk-selinux-enforcing-libguestfs-1.50.1-3.log

1.2 Check v2v debug log, find selinux relael takes 121.19 secs
# cat v2v-convert-guest-with-large-disk-selinux-enforcing-libguestfs-1.50.1-3.log |grep 'setfiles -F' -A 4
May 06 16:10:00 commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e /sysroot/selinux -e /sysroot/sys -m -C -r /sysroot -v /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
May 06 16:10:00 Can't stat exclude path "/sysroot/selinux", No such file or directory - ignoring.
May 06 16:12:01 libguestfs: trace: v2v: selinux_relabel = 0
May 06 16:12:01 libguestfs: trace: v2v: rm_f "/.autorelabel"
May 06 16:12:01 guestfsd: => selinux_relabel (0x1d3) took 121.19 secs


Scenario2: 
2.1 Prepare a rhel9 guest with large disk and disabled SELINUX on VMware environment, then convert the guest from VMware to local libvirt by v2v
# virt-v2v -ic vpx://root.74.72/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=24:40:7E:C8:C8:1F:9C:DF:E4:E0:48:D0:9E:25:64:94:64:AF:C6:8C  -ip /home/passwd  esx6.7-rhel9.3-with-large-disk -v -x |& ts > v2v-convert-guest-with-large-disk-selinux-disabled-libguestfs-1.50.1-3.log

2.2 Check v2v debug log, find selinux relael takes 122.63 secs
# cat v2v-convert-guest-with-large-disk-selinux-disabled-libguestfs-1.50.1-3.log |grep 'setfiles -F' -A 4
May 06 18:29:30 commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e /sysroot/selinux -e /sysroot/sys -m -C -r /sysroot -v /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
May 06 18:29:30 Can't stat exclude path "/sysroot/selinux", No such file or directory - ignoring.
May 06 18:31:32 libguestfs: trace: v2v: selinux_relabel = 0
May 06 18:31:32 libguestfs: trace: v2v: rm_f "/.autorelabel"
May 06 18:31:32 guestfsd: => selinux_relabel (0x1d3) took 122.63 secs



Test the bug with below builds:
libguestfs-1.50.1-4.el9.x86_64
guestfs-tools-1.50.1-2.el9.x86_64
virt-v2v-2.3.4-1.el9.x86_64
nbdkit-server-1.34.1-1.el9.x86_64
libnbd-1.16.0-1.el9.x86_64

Steps:
Scenario1:
1.1 Convert the same guest of above reproduction step1.1 by v2v
#virt-v2v -ic vpx://root.74.72/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=24:40:7E:C8:C8:1F:9C:DF:E4:E0:48:D0:9E:25:64:94:64:AF:C6:8C  -ip /home/passwd  esx6.7-rhel9.3-with-large-disk -v -x |& ts > v2v-convert-guest-with-large-disk-selinux-enforcing-libguestfs-1.50.1-4.log

1.2 Check v2v debug log, find selinux relael takes 60.11 secs and '-T 0' parameter is added to setfiles command, besides, warning 'Can't stat exclude path "/sysroot/selinux", No such file or directory - ignoring.' is removed
# cat v2v-convert-guest-with-large-disk-selinux-enforcing-libguestfs-1.50.1-4.log |grep 'setfiles -F' -A 3
May 06 15:44:18 commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e /sysroot/sys -m -C -T 0 -r /sysroot -v /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
May 06 15:45:18 libguestfs: trace: v2v: selinux_relabel = 0
May 06 15:45:18 libguestfs: trace: v2v: rm_f "/.autorelabel"
May 06 15:45:18 guestfsd: => selinux_relabel (0x1d3) took 60.11 secs

Scenario2: 
2.1 Convert the same guest of above reproduction step2.1 by v2v
#virt-v2v -ic vpx://root.74.72/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk8.0.0 -io vddk-thumbprint=24:40:7E:C8:C8:1F:9C:DF:E4:E0:48:D0:9E:25:64:94:64:AF:C6:8C  -ip /home/passwd  esx6.7-rhel9.3-with-large-disk -v -x |& ts > v2v-convert-guest-with-large-disk-selinux-disabled-libguestfs-1.50.1-4.log

2.2 Check v2v debug log,  find selinux relael takes 61.27 secs and '-T 0' parameter is added to setfiles command, besides, warning 'Can't stat exclude path "/sysroot/selinux", No such file or directory - ignoring' is removed
# cat v2v-convert-guest-with-large-disk-selinux-disabled-libguestfs-1.50.1-4.log |grep 'setfiles -F' -A 3
May 06 18:38:59 commandrvf: setfiles -F -e /sysroot/dev -e /sysroot/proc -e /sysroot/sys -m -C -T 0 -r /sysroot -v /sysroot/etc/selinux/targeted/contexts/files/file_contexts /sysroot/
May 06 18:40:00 libguestfs: trace: v2v: selinux_relabel = 0
May 06 18:40:00 libguestfs: trace: v2v: rm_f "/.autorelabel"
May 06 18:40:01 guestfsd: => selinux_relabel (0x1d3) took 61.27 secs

Result:
   The time of selinux relabel for linux guest has reduced with libguestfs-1.50.1-4.el9.x86_64 during v2v conversion

Hi Richard,
   Please help to check scenanio2,  selinux will be relabeling even if selinux is disabled during v2v conversion, is it expected?

Comment 3 Richard W.M. Jones 2023-05-06 12:18:50 UTC
First I think doubling the relabelling speed is pretty good, thanks for verifying that.

Yes, we do relabel the filesystem even if SELinux is disabled.  HOWEVER in future
this might change, because we accepted a patch upstream which entirely skips the
SELinux relabel step if we detect that it is disabled
(https://github.com/libguestfs/libguestfs-common/commit/17ef57a9bc8196d2e521c877a835ea84d02b931d)
It is not in RHEL 9 so far.

Comment 7 mxie@redhat.com 2023-05-08 03:49:42 UTC
Move the bug to verified status according to comment2 and comment3

Comment 9 errata-xmlrpc 2023-11-07 08:24:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libguestfs bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6327