(In reply to Fabien Dupont from comment #0)

> you need to arrange to put RPMs there. However that was apparently difficult.

Can you please provide more background on this? Perhaps a virtio-win RHBZ that was rejected? Thanks.

I'll answer about the background to this request.

We want virt-v2v to install qemu guest agent ("qemu-ga") and in fact it
can do that at the moment.  Currently it will look on the virtio-win
disk in a directory called /linux/el<N> (eg. /linux/el9/...) for some
RPMs -- yes I know it's virtio-win and these are Linux RPMs -- and it
will install them from there.
https://github.com/libguestfs/virt-v2v/blob/68af35d48ca845133ede948d36ee351d171e3de8/convert/windows_virtio.ml#L116

However:

$ rpm -ql virtio-win | grep /linux
<nada>

This is because we've had difficulties completing the implementation of
this with the virtio-win team.

So the request is to rip this out and either install the RPMs from another
known location, or probably easier just add a firstboot command which
does "yum install qemu-guest-agent".

For the second approach we'd need to work out the name of the qemu-ga
package for all distros we care about.  The "package install" command
can be taken from virt-customize:
https://github.com/rwmjones/guestfs-tools/blob/ea83e7693a5ecb0e9d179e77e40f17be2379b182/customize/customize_run.ml#L71

I've implemented comment 2, but now that I'd like to test it (booting a converted Fedora 35 domain), I'm realizing: even though the firstboot script would (hopefully) install QGA, QGA wouldn't be able to "bind" a virtio-serial port -- namely, the converted domain has no virtio-serial device in the first place.

In fact I see no "gcaps_virtio_serial" in virt-v2v at all (@ 426a4e3419ad).

But... this doesn't add up. I gather QGA works in converted Windows guests. Is that right? How is that possible? If we never create a virtio-serial-pci device during conversion, how has QGA ever been successfully tested with Windows domains?

Is it that RHV (which is just one of the possible outputs of virt-v2v), upon importing the converted domain, automatically adds a virtio-serial-pci device, and the agent's channel as a port on that?

... Given that we install qga unconditionally (i.e., it does not depend on the guest), we should probably also install virtio-serial independently of the guest. So that's in favor of *not* introducing gcaps_virtio_serial.

More crap:

(1) The dnf install command in the firstboot script fails because, at
that point, there is no network connectivity.

I've worked this around by prepeding another firstboot script with
"sleep 60" in it. Here's the comment I wrote for it:

            (* Waiting for the guest to go "online" is very complicated. On
             * systemd-based distros, our firstboot service could perhaps depend
             * on "systemd-networkd-wait-online.service" or
             * "NetworkManager-wait-online.service". But not all Linux distros
             * are based on systemd, and even with systemd, a guest could use
             * systemd-networkd vs. NetworkManager... So just do what we do in
             * various Windows firstboot scripts: wait. We could cut the waiting
             * short by pinging a well-known IP address, or by resolving a
             * well-known FQDN, but those might qualify as "phoning home"... So
             * just wait.
             *)

(2) With this, the qemu-guest-agent package gets indeed installed
(Fedora 35 guest), but "/root/virt-sysprep-firstboot.log" contains:

> /usr/lib/virt-sysprep/firstboot.sh start
> Scripts dir: /usr/lib/virt-sysprep/scripts
> === Running /usr/lib/virt-sysprep/scripts/5000-0001-wait-online ===
> === Running /usr/lib/virt-sysprep/scripts/5000-0002-install-qga ===
> Fedora 35 - x86_64                              6.3 MB/s |  79 MB     00:12
> Fedora 35 openh264 (From Cisco) - x86_64        2.4 kB/s | 2.5 kB     00:01
> Fedora Modular 35 - x86_64                      2.9 MB/s | 3.3 MB     00:01
> Fedora 35 - x86_64 - Updates                    8.4 MB/s |  30 MB     00:03
> Fedora Modular 35 - x86_64 - Updates            2.5 MB/s | 3.1 MB     00:01
> Last metadata expiration check: 0:00:01 ago on Fri 03 Jun 2022 07:57:07 AM EDT.
> Dependencies resolved.
> ================================================================================
>  Package                Architecture Version                Repository     Size
> ================================================================================
> Installing:
>  qemu-guest-agent       x86_64       2:6.1.0-14.fc35        updates       197 k
> Installing dependencies:
>  liburing               x86_64       2.0-2.fc35             fedora         26 k
>
> Transaction Summary
> ================================================================================
> Install  2 Packages
>
> Total download size: 223 k
> Installed size: 586 k
> Downloading Packages:
> (1/2): liburing-2.0-2.fc35.x86_64.rpm            81 kB/s |  26 kB     00:00
> (2/2): qemu-guest-agent-6.1.0-14.fc35.x86_64.rp 545 kB/s | 197 kB     00:00
> --------------------------------------------------------------------------------
> Total                                           157 kB/s | 223 kB     00:01
> Running transaction check
> Transaction check succeeded.
> Running transaction test
> Transaction test succeeded.
> Running transaction
>   Preparing        :                                                        1/1
>   Installing       : liburing-2.0-2.fc35.x86_64                             1/2
>   Installing       : qemu-guest-agent-2:6.1.0-14.fc35.x86_64                2/2
>   Running scriptlet: qemu-guest-agent-2:6.1.0-14.fc35.x86_64                2/2
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %post(qemu-guest-agent-2:6.1.0-14.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in POSTIN scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(glibc-common-2.34-29.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(systemd-udev-249.9-1.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(man-db-2.9.4-2.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(systemd-249.9-1.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
>   Verifying        : liburing-2.0-2.fc35.x86_64                             1/2
>   Verifying        : qemu-guest-agent-2:6.1.0-14.fc35.x86_64                2/2
>
> Installed:
>   liburing-2.0-2.fc35.x86_64       qemu-guest-agent-2:6.1.0-14.fc35.x86_64
>
> Complete!

I don't have the slightest idea what those "Permission denied" errors
are!

Searching the web for them does not help. Various web forums finger
SELinux.

(3) So,

sealert -a /var/log/audit/audit.log

> 100% done
> found 1 alerts in /var/log/audit/audit.log
> --------------------------------------------------------------------------------
>
> SELinux is preventing dnf from using the transition access on a process.
>
> *****  Plugin catchall (100. confidence) suggests   **************************
>
> If you believe that dnf should be allowed transition access on processes labeled rpm_script_t by default.
> Then you should report this as a bug.
> You can generate a local policy module to allow this access.
> Do
> allow this access for now by executing:
> # ausearch -c 'dnf' --raw | audit2allow -M my-dnf
> # semodule -X 300 -i my-dnf.pp
>
>
> Additional Information:
> Source Context                system_u:system_r:virtd_t:s0-s0:c0.c1023
> Target Context                system_u:system_r:rpm_script_t:s0-s0:c0.c1023
> Target Objects                /usr/bin/bash [ process ]
> Source                        dnf
> Source Path                   dnf
> Port                          <Unknown>
> Host                          <Unknown>
> Source RPM Packages
> Target RPM Packages           bash-5.1.8-2.fc35.x86_64
> SELinux Policy RPM            selinux-policy-targeted-35.15-1.fc35.noarch
> Local Policy RPM              selinux-policy-targeted-35.15-1.fc35.noarch
> Selinux Enabled               True
> Policy Type                   targeted
> Enforcing Mode                Enforcing
> Host Name                     fedora
> Platform                      Linux fedora 5.16.18-200.fc35.x86_64 #1 SMP
>                               PREEMPT Mon Mar 28 14:10:07 UTC 2022 x86_64 x86_64
> Alert Count                   5
> First Seen                    2022-06-03 07:57:14 EDT
> Last Seen                     2022-06-03 07:57:14 EDT
> Local ID                      4e914889-661c-4d11-8e08-3f907253924e
>
> Raw Audit Messages
> type=AVC msg=audit(1654257434.993:181): avc:  denied  { transition } for  pid=685 comm="dnf" path="/usr/bin/bash" dev="vda3" ino=12634193 scontext=system_u:system_r:virtd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:rpm_script_t:s0-s0:c0.c1023 tclass=process permissive=0
>
>
> Hash: dnf,virtd_t,rpm_script_t,process,transition

This makes absolutely no sense to me. DNF is expected to run RPM shell
scriptlets all the time.

Why does SELinux prevent it only when the package is being installed
from a firstboot script?

Should we apply some SELinux context to the firstboot scripts?

(4) The failed RPM scripts are:

# rpm -q --scripts qemu-guest-agent

> postinstall scriptlet (using /bin/sh):
> if [ $1 -eq 1 ] && [ -x "/usr/lib/systemd/systemd-update-helper" ]; then
>     # Initial installation
>     /usr/lib/systemd/systemd-update-helper install-system-units qemu-guest-agent.service || :
> fi

and

> preuninstall scriptlet (using /bin/sh):
> if [ $1 -eq 0 ] && [ -x "/usr/lib/systemd/systemd-update-helper" ]; then
>     # Package removal, not upgrade
>     /usr/lib/systemd/systemd-update-helper remove-system-units qemu-guest-agent.service || :
> fi

and

> postuninstall scriptlet (using /bin/sh):
> if [ $1 -ge 1 ] && [ -x "/usr/lib/systemd/systemd-update-helper" ]; then
>     # Package upgrade, not uninstall
>     /usr/lib/systemd/systemd-update-helper mark-restart-system-units qemu-guest-agent.service || :
> fi

I don't know what "systemd-update-helper" is supposed to do.
*Obviously*, "systemd-update-helper" has no manual page.

Searching the web for "systemd-update-helper" yields like 5-10 results
only, one of them being
<https://bugzilla.redhat.com/show_bug.cgi?id=2016253#c37>. This suggests
these RPM scripts attempt to enable QGA after installation, but use the
wrong means for that. In other words, packaging bug?

... Maybe relevant: bug 850285; Fedora qemu dist-git commit
6986e10cf3d0e63cd2ac26471489169648974528.

It seems like "%systemd_post" in the spec file is what expands to
"systemd-update-helper".

In the end, the agent is certainly *not started* when the firstboot
scripts finish running as described above, and so I cannot connect to
QGA from the host, without logging in to the guest first and starting
QGA manually.

----*----

Seriously, this is a trash fire. Just because we've been unable to put
the qemu-guest-agent RPM in a well-known, agreed-upon location on the
virtio-win ISO, we are now introducing, a firstboot dependency, a
service management dependency, and a "wait-online" dependency. :/

Why does the script not have permission to execute /bin/sh?

How do I get rid of the RPM script errors above?

How do I ensure that qemu-guest-agent is *immediately started* upon
first boot, in every distro family that we intend to support?

(In reply to Richard W.M. Jones from comment #2)
> So the request is to rip this out and either install the RPMs from
> another known location, or probably easier just add a firstboot
> command which does "yum install qemu-guest-agent".
>
> For the second approach we'd need to work out the name of the qemu-ga
> package for all distros we care about.  The "package install" command
> can be taken from virt-customize:
> https://github.com/rwmjones/guestfs-tools/blob/ea83e7693a5ecb0e9d179e77e40f17be2379b182/customize/customize_run.ml#L71

Yes, I've factored that out to a new interface in
libguestfs-common/mlcustomize, but once we have the package install
command, why do we have to run it as a firstboot script, as opposed to
running it imediately in the appliance? (In other words, why imitate
"--firstboot-install" of virt-customize, when we could just imitate
"--install" too?) I've read
<https://libguestfs.org/virt-builder.1.html#installing-packages>, and I
don't see what it buys us to delay the package installation until first
boot.

BTW I suspect that (a) the network connectivity problem, and (b) the
SELinux problem that I describe in comment 5, break
"--firstboot-install" for the current upstream versions of virt-builder,
virt-customize and virt-sysprep; my factoring out the install script
generation should play no role in that. In other words, I suspect that
the upstream "--firstboot-install" logic has bitrotted (at least for
modern Fedora guests). That would make firstboot effectively unusable
for both (some) Linux guests, and Windows guests (cue the many earlier
unsolvable firstboot issues (BZs) with Windows). Anyway; I'm going to
test this next week.

Yes, the exact same issue reproduces without my work-in-progress
patches.

With:

- libguestfs @ 8fc4d167153a ("appliance, daemon: disable lvm2
  devicesfile", 2022-05-31)
- guestfs-tools @ 40b28512f700 ("Version 1.49.2.", 2022-05-26)

and commands:

virt-builder --root-password password:[redacted] fedora-35
virt-customize -a fedora-35.img --firstboot-command 'sleep 60' \
  --firstboot-install qemu-guest-agent
virt-install --import -n fedora-35 --memory 4096 \
  --disk vol=default/fedora-35.img,format=raw --network bridge=virbr0 \
  --graphics type=vnc --os-variant fedora35

I get, in the guest's "/root/virt-sysprep-firstboot.log":

> /usr/lib/virt-sysprep/firstboot.sh start
> Scripts dir: /usr/lib/virt-sysprep/scripts
> === Running /usr/lib/virt-sysprep/scripts/5000-0001-sleep-60 ===
> === Running /usr/lib/virt-sysprep/scripts/5000-0002-install-qemu-guest-agent ===
> Fedora 35 - x86_64                              5.4 MB/s |  79 MB     00:14
> Fedora 35 openh264 (From Cisco) - x86_64        2.5 kB/s | 2.5 kB     00:00
> Fedora Modular 35 - x86_64                      2.4 MB/s | 3.3 MB     00:01
> Fedora 35 - x86_64 - Updates                    6.9 MB/s |  30 MB     00:04
> Fedora Modular 35 - x86_64 - Updates            4.2 MB/s | 3.2 MB     00:00
> Dependencies resolved.
> ================================================================================
>  Package                Architecture Version                Repository     Size
> ================================================================================
> Installing:
>  qemu-guest-agent       x86_64       2:6.1.0-14.fc35        updates       197 k
> Installing dependencies:
>  liburing               x86_64       2.0-2.fc35             fedora         26 k
>
> Transaction Summary
> ================================================================================
> Install  2 Packages
>
> Total download size: 223 k
> Installed size: 586 k
> Downloading Packages:
> (1/2): qemu-guest-agent-6.1.0-14.fc35.x86_64.rp 386 kB/s | 197 kB     00:00
> (2/2): liburing-2.0-2.fc35.x86_64.rpm            38 kB/s |  26 kB     00:00
> --------------------------------------------------------------------------------
> Total                                           167 kB/s | 223 kB     00:01
> Running transaction check
> Transaction check succeeded.
> Running transaction test
> Transaction test succeeded.
> Running transaction
>   Preparing        :                                                        1/1
>   Installing       : liburing-2.0-2.fc35.x86_64                             1/2
>   Installing       : qemu-guest-agent-2:6.1.0-14.fc35.x86_64                2/2
>   Running scriptlet: qemu-guest-agent-2:6.1.0-14.fc35.x86_64                2/2
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %post(qemu-guest-agent-2:6.1.0-14.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in POSTIN scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(glibc-common-2.34-29.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(systemd-udev-249.9-1.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(man-db-2.9.4-2.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
> error: failed to exec scriptlet interpreter /bin/sh: Permission denied
> warning: %transfiletriggerin(systemd-249.9-1.fc35.x86_64) scriptlet failed, exit status 127
>
> Error in <unknown> scriptlet in rpm package qemu-guest-agent
>   Verifying        : liburing-2.0-2.fc35.x86_64                             1/2
>   Verifying        : qemu-guest-agent-2:6.1.0-14.fc35.x86_64                2/2
>
> Installed:
>   liburing-2.0-2.fc35.x86_64       qemu-guest-agent-2:6.1.0-14.fc35.x86_64
>
> Complete!

SELinux analysis:

sealert -a /var/log/audit/audit.log

> 100% done
> found 1 alerts in /var/log/audit/audit.log
> --------------------------------------------------------------------------------
>
> SELinux is preventing dnf from using the transition access on a process.
>
> *****  Plugin catchall (100. confidence) suggests   **************************
>
> If you believe that dnf should be allowed transition access on processes labeled rpm_script_t by default.
> Then you should report this as a bug.
> You can generate a local policy module to allow this access.
> Do
> allow this access for now by executing:
> # ausearch -c 'dnf' --raw | audit2allow -M my-dnf
> # semodule -X 300 -i my-dnf.pp
>
>
> Additional Information:
> Source Context                system_u:system_r:virtd_t:s0-s0:c0.c1023
> Target Context                system_u:system_r:rpm_script_t:s0-s0:c0.c1023
> Target Objects                /usr/bin/bash [ process ]
> Source                        dnf
> Source Path                   dnf
> Port                          <Unknown>
> Host                          <Unknown>
> Source RPM Packages
> Target RPM Packages           bash-5.1.8-2.fc35.x86_64
> SELinux Policy RPM            selinux-policy-targeted-35.15-1.fc35.noarch
> Local Policy RPM              selinux-policy-targeted-35.15-1.fc35.noarch
> Selinux Enabled               True
> Policy Type                   targeted
> Enforcing Mode                Enforcing
> Host Name                     fedora
> Platform                      Linux fedora 5.16.18-200.fc35.x86_64 #1 SMP
>                               PREEMPT Mon Mar 28 14:10:07 UTC 2022 x86_64 x86_64
> Alert Count                   5
> First Seen                    2022-06-06 05:00:22 EDT
> Last Seen                     2022-06-06 05:00:22 EDT
> Local ID                      2330c9c2-ee8f-4bdf-930c-af51d676aa3c
>
> Raw Audit Messages
> type=AVC msg=audit(1654506022.600:176): avc:  denied  { transition }
> for  pid=750 comm="dnf" path="/usr/bin/bash" dev="vda3" ino=12634193
> scontext=system_u:system_r:virtd_t:s0-s0:c0.c1023
> tcontext=system_u:system_r:rpm_script_t:s0-s0:c0.c1023 tclass=process
> permissive=0
>
>
> Hash: dnf,virtd_t,rpm_script_t,process,transition

"firstboot.sh" itself is "virtd_exec_t":

> system_u:object_r:virtd_exec_t:s0 /usr/lib/virt-sysprep/firstboot.sh

namely from selinux-policy commit 06a183a14d8a ("Label
/usr/lib/virt-sysprep/firstboot.sh as virtd_exec_t", 2017-09-12).

Apparently this context cannot transition to "rpm_script_t", which is
required for running RPM scripts.

... From a bit of browsing / grepping the selinux-policy.git repo, I've
found:

- https://github.com/fedora-selinux/selinux-policy/commit/2014a2e2ae785
- https://bugzilla.redhat.com/show_bug.cgi?id=1872245

That's *exactly* our issue as well!

However, I don't think generally allowing RPM script execution for *all*
"virtd_exec_t" context executables would be a great idea; for example,
libvirt executables have no business duing that. I think we might need a
specific new context for "firstboot.sh", and then we should allow the
transition to "rpm_script_t" from that new context only.

BUT: why am I finding out about this *only now*? The
"--firstboot-install" parameter of virt-customize has existed for ages!

Furthermore: however we fix the selinux policy, it will not help
released OS versions / installed guests.

Should we flip SELinux to "permissive" for the duration of
"firstboot.sh"? That's actually something we could do inside
"firstboot.sh", regardless of Linux distro and/or active SELinux policy.

The exact same issue reproduces with the Fedora 28 image from
virt-builder. The kernel in that particular image was built in April
2018, so this issue must have existed for at least 4 years now -- albeit
dormant, perhaps.

I wonder if the issue is that now we relabel *by default* (from
BZ#2075718). Let me check that, again with Fedora 35 (note the
"--no-selinux-relabel" option on the virt-customize cmdline):

virt-builder --root-password password:[redacted] fedora-35
virt-customize -a fedora-35.img --firstboot-command 'sleep 60' \
  --firstboot-install qemu-guest-agent --no-selinux-relabel
virsh pool-refresh default
virt-install --import -n fedora-35 --memory 4096 \
  --disk vol=default/fedora-35.img,format=raw --network bridge=virbr0 \
  --graphics type=vnc --os-variant fedora35

Well, no; that way, the situation is even worse. With
"--no-selinux-relabel", I get:

> system_u:object_r:unlabeled_t:s0 /usr/lib/virt-sysprep/firstboot.sh

That is, "firstboot.sh" is not even labeled "virtd_exec_t", and then it
won't even start during boot; see the journal:

> systemd[1]: Starting libguestfs firstboot service...
> audit[547]: AVC avc:  denied  { execute } for  pid=547 comm="(tboot.sh)" name="firstboot.sh" dev="vda3" ino=144 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=0
> systemd[547]: guestfs-firstboot.service: Failed to locate executable /usr/lib/virt-sysprep/firstboot.sh: Permission denied
> systemd[547]: guestfs-firstboot.service: Failed at step EXEC spawning /usr/lib/virt-sysprep/firstboot.sh: Permission denied
> systemd[1]: Starting Permit User Sessions...
> systemd[1]: guestfs-firstboot.service: Main process exited, code=exited, status=203/EXEC
> systemd[1]: guestfs-firstboot.service: Failed with result 'exit-code'.
> systemd[1]: Failed to start libguestfs firstboot service.

and "sealert -a /var/log/audit/audit.log" makes three suggestions
(summarized), in decreasing order of "confidence":

- "Plugin file" ->

  /sbin/restorecon -R -v firstboot.sh

- "Plugin catchall_labels" ->

  semanage fcontext -a -t FILE_TYPE 'firstboot.sh'

  where FILE_TYPE is one of the following: [...] virsh_exec_t,
  virt_qemu_ga_exec_t, virtd_exec_t, virtd_initrc_exec_t,
  virtd_lxc_exec_t, virtlogd_exec_t, virtlogd_initrc_exec_t,
  vmtools_exec_t [...]

- "Plugin catchall" ->

  same generic transition problem report / suggestion as the ones quoted
  in the previous comments above

So the issue I'm seeing is certainly not a regression from BZ#2075718 --
without SELinux relabeling, the situation is even worse!

I think it may simply be the case that nobody has used
"--firstboot-install" for years now, and this regression is ancient. I
just have no better explanation.

Good ol' "setenforce 0" does the job of course; see also the
startup-at-once at the end:

virt-customize -a fedora-35.img \
  --firstboot-command 'sleep 60' \
  --firstboot-command 'setenforce 0' \
  --firstboot-install qemu-guest-agent \
  --firstboot-command 'setenforce 1' \
  --firstboot-command 'service qemu-guest-agent start'

virsh pool-refresh default

virt-install --import ...

virsh domifaddr fedora-35 --source agent

> Name       MAC address          Protocol     Address
> -------------------------------------------------------------------------------
> lo         00:00:00:00:00:00    ipv4         127.0.0.1/8
> -          -                    ipv6         ::1/128
> enp1s0     52:54:00:fd:15:0c    ipv4         192.168.122.32/24
> -          -                    ipv6         fe80::1400:8328:d2ee:35af/64

So, next steps for my v2v patches:

- install a firstboot script just before the package installation to:

  - check if the distro uses SELinux ("getenforce" / "setenforce"
    utilities are on the path)

  - save the current value to a file under /root

  - set SELinux to Permissive

- install a firstboot script just after the package installation to:

  - restore the original state of SELinux from the file under /root

- complete the virtio-serial channel addition for the QEMU output as
  well (libvirt is done, RHV probably has its own thing, OpenStack docs
  are silent on it)

- queue "service qemu-guest-agent start" at the end; this should work on
  both systemd-based and sysvinit-based distros (and distro versions)

[v2v PATCH 0/4] convert_linux: install the QEMU guest agent with a firstboot script
Message-Id: <20220606141941.16536-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-June/029082.html

[libguestfs-common PATCH] mlcustomize: factor out pkg install/update/uninstall from guestfs-tools
Message-Id: <20220606142016.16627-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-June/029087.html

[guestfs-tools PATCH] customize: rebase to the common/mlcustomize/Guest_packages interface
Message-Id: <20220606142042.16680-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-June/029088.html

(In reply to Laszlo Ersek from comment #10)
> [v2v PATCH 0/4] convert_linux: install the QEMU guest agent with a firstboot script
> Message-Id: <20220606141941.16536-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-June/029082.html

This will need a v2...

> [libguestfs-common PATCH] mlcustomize: factor out pkg install/update/uninstall from guestfs-tools
> Message-Id: <20220606142016.16627-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-June/029087.html

Upstream commit 9e990f3e4530.

> [guestfs-tools PATCH] customize: rebase to the common/mlcustomize/Guest_packages interface
> Message-Id: <20220606142042.16680-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-June/029088.html

Upstream commit 7eb1ecf467e8.

Sorry, ignore those changes I just made.

[v2v PATCH v2 0/4] convert_linux: install the QEMU guest agent with a firstboot script
Message-Id: <20220613170135.12557-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-June/029217.html

(In reply to Laszlo Ersek from comment #13)
> [v2v PATCH v2 0/4] convert_linux: install the QEMU guest agent with a firstboot script
> Message-Id: <20220613170135.12557-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-June/029217.html

Upstream commit range a91110cc178e..e64356896377.

Tried with the versions:
virt-v2v-2.0.7-2.el9.x86_64
libvirt-8.5.0-2.el9.x86_64
qemu-kvm-7.0.0-9.el9.x86_64
qemu-guest-agent-7.0.0-9.el9.x86_64


1. Check the original VMs (RHEL6.9/RHEL7.10/RHEL8.6/RHEL9.0) in order to ensure NO qemu-guest-agent pkgs on it;

2. Convert them via virt-v2v to rhv; Please check the attachment on the debug log.

# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/root/vddk_libdir/latest -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -ip /v2v-ops/esxpw -o rhv-upload  -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /v2v-ops/rhvpasswd -os nfs_data -b ovirtmgmt esx6.7-rhel9.0-x86_64 
[   0.0] Setting up the source: -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk esx6.7-rhel9.0-x86_64
[   2.0] Opening the source
[   6.8] Inspecting the source
[  14.8] Checking for sufficient free disk space in the guest
[  14.8] Converting Red Hat Enterprise Linux 9.0 Beta (Plow) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[ 124.4] Mapping filesystem data to avoid copying unused and blank areas
[ 125.3] Closing the overlay
[ 125.6] Assigning disks to buses
[ 125.6] Checking if the guest needs BIOS or UEFI to boot
[ 125.6] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 138.3] Copying disk 1/1
█ 100% [****************************************]
[ 361.8] Creating output metadata
[ 385.6] Finishing off


3. Start VMs and check the pkgs, qemu-guest-agent pkgs are NOT installed on any one of VMs.


Mark the Verified: FailedQA and Status to Assigned.

Hello Vera,

the way we attempt installing the guest agent is at the first boot of the converted (output) domain. At that time, we wait for the network to come up, and when it does, we start a dnf / yum install command in the background (as a systemd service) for installing the guest agent. After installation, we also enable & start the guest agent.

If at the first boot of the output domain there is no functional networking, the installation will fail.

Please log in to the converted and first-booted guest, and collect the logfile from under /root. Thanks.

(I tested the patches with the libvirt output module, I have no access to RHV.)

The logfile is "/root/virt-sysprep-firstboot.log", but you can also see the firstboot scripts in action on the normal console (tty1) right after booting the output domain.

I don't see any --bridge, --network or --mac command line options in comment 16, so it's possible that the output domain will not have any networking in RHV.

Hi, Laszlo

Thanks.

That makes sense. It means we need to configure repos for all original VMs that we didn't do it before.
Will add repos to ESX VMs for this new feature.

I re-tried on the RHEL9.0 guest with repo configuration:
virt-v2v-2.0.7-2.el9.x86_64
libvirt-8.5.0-2.el9.x86_64
qemu-kvm-7.0.0-9.el9.x86_64


1. Check the original VMs (RHEL9.0/RHEL8.5) in order to ensure NO qemu-guest-agent rpm/service and configure repos;

2. Convert them via virt-v2v to rhv; 
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -ip /v2v-ops/esxpw -o rhv-upload  -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /v2v-ops/rhvpasswd -os nfs_data -b ovirtmgmt esx6.7-rhel9.0-x86_64
[   0.0] Setting up the source: -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk esx6.7-rhel9.0-x86_64
[   2.0] Opening the source
[   7.3] Inspecting the source
[  14.4] Checking for sufficient free disk space in the guest
[  14.4] Converting Red Hat Enterprise Linux 9.0 Beta (Plow) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[ 122.9] Mapping filesystem data to avoid copying unused and blank areas
[ 123.8] Closing the overlay
[ 124.0] Assigning disks to buses
[ 124.0] Checking if the guest needs BIOS or UEFI to boot
[ 124.0] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 137.4] Copying disk 1/1
█ 100% [****************************************]
[ 325.8] Creating output metadata
[ 343.4] Finishing off

3. Start VM and check the qemu-guest-agent rpm/service.

# rpm -qa qemu-guest-agent
qemu-guest-agent-6.2.0-11.el9_0.2.x86_64
# systemctl status qemu-guest-agent.service 
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/usr/lib/systemd/system/qemu-guest-agent.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-07-25 11:04:30 CST; 14min ago
   Main PID: 6525 (qemu-ga)
      Tasks: 2 (limit: 10680)
     Memory: 608.0K
        CPU: 29ms
     CGroup: /system.slice/qemu-guest-agent.service
             └─6525 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=guest-file-open,guest-fil>

Jul 25 11:04:30 bootp-73-195-94.lab.eng.pek2.redhat.com systemd[1]: Started QEMU Guest Agent.

# cat virt-sysprep-firstboot.log 
/usr/lib/virt-sysprep/firstboot.sh start
Scripts dir: /usr/lib/virt-sysprep/scripts
=== Running /usr/lib/virt-sysprep/scripts/5000-0001-wait-online ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0002-setenforce-0 ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0003-install-qga ===
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:24:48 ago on Mon 25 Jul 2022 10:39:12 AM CST.
Dependencies resolved.
================================================================================
 Package            Arch     Version                  Repository           Size
================================================================================
Installing:
 qemu-guest-agent   x86_64   17:6.2.0-11.el9_0.2      rhel9.0-appstream   298 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 298 k
Installed size: 813 k
Downloading Packages:
qemu-guest-agent-6.2.0-11.el9_0.2.x86_64.rpm    189 kB/s | 298 kB     00:01    
--------------------------------------------------------------------------------
Total                                           188 kB/s | 298 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Installing       : qemu-guest-agent-17:6.2.0-11.el9_0.2.x86_64            1/1 
  Running scriptlet: qemu-guest-agent-17:6.2.0-11.el9_0.2.x86_64            1/1 
  Verifying        : qemu-guest-agent-17:6.2.0-11.el9_0.2.x86_64            1/1 
Installed products updated.

Installed:
  qemu-guest-agent-17:6.2.0-11.el9_0.2.x86_64                                   

Complete!
=== Running /usr/lib/virt-sysprep/scripts/5000-0004-setenforce-restore ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0005-start-qga ===
Redirecting to /bin/systemctl start qemu-guest-agent.service
/usr/lib/virt-sysprep/scripts-done/5000-0005-start-qga: line 3: chkconfig: command not found


Moving to Verified:Tested Stauts.

Hi Laszlo,

Could you help us to confirm these questions on this feature?

1. From the testing, we can get the results: repo should be configured in the original VM, otherwise "yum install" will fail. Is it right?
We tried the one without repo configuration, the error in the virt-sysprep-firstboot.log as below:
# cat virt-sysprep-firstboot.log 
/usr/lib/virt-sysprep/firstboot.sh start
Scripts dir: /usr/lib/virt-sysprep/scripts
=== Running /usr/lib/virt-sysprep/scripts/5000-0001-wait-online ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0002-setenforce-0 ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0003-install-qga ===
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Error: There are no enabled repositories in "/etc/yum.repos.d", "/etc/yum/repos.d", "/etc/distro.repos.d".
=== Running /usr/lib/virt-sysprep/scripts/5000-0004-setenforce-restore ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0005-start-qga ===
Redirecting to /bin/systemctl start qemu-guest-agent.service
Failed to start qemu-guest-agent.service: Unit qemu-guest-agent.service not found.
/usr/lib/virt-sysprep/scripts-done/5000-0005-start-qga: line 3: chkconfig: command not found


If so, we need to add the repos on all VMs on ESX servers. It is a large workload for us.

2. For some guests like Suse/Debian/Ubuntu, we don't know the repos. If definitely yes to Q1, installation of qemu-guest-agent will not work on these kind of VMs.
How about these VMs?


Thanks,
Vera

Hi Vera,

thank you for the repeateed testing; I'm relieved that the way we figured it would work actually does seem to work, at least for RHEL/Fedora guests.

Regarding your other question -- the installation for qga at firstboot uses the generic "firstboot package installation" facility <https://libguestfs.org/virt-builder.1.html#installing-packages>, which is exposed also as the --firstboot-install option of virt-customize and virt-sysprep. Because the installation runs at first boot, the admin has no option to log in to the converted guest and set up the repos; the repos are expected to be in place already when the conversion is started.

I'm unsure if you really need to modify all VMs on the ESX servers -- I think that's only necessary wherever you want to test QGA installation after conversion. If you don't specifically focus on QGA after conversion, I think it's OK to just let this firstboot script to fail. Also I assume in your automated tests, where you might want to check QGA install all the time, you could also add a step to the input domain configuration, for configuring the repos.

Regarding Suse/Debian/Ubuntu, shouldn't those guests come with pre-set repo locations already?

The patch series for this BZ contains a commit (f65e8e68fb4e, "convert_linux: extract qemu-guest-agent package name", 2022-06-14) that centralizes the package *name* handling, so the QGA package name for SUSE and Debian/Ubuntu are already known to virt-v2v. The repo locations are not, but (again) just as I wouldn't expect any Fedora installation to exist without functional DNF repo files, I wouldn't expect any OpenSUSE (at least) or Debian/Ubuntu installs to exist without functional dpkg/apt/whatever configs out of the box.

But... even if that fails, Debian/Ubuntu are still Tech Preview. So the intent is certainly to support QGA installation on those VMs as well, but if they don't work and it's very difficult to set up the pkg repos in the input domains, then I think they can be ignored. Thanks.

> I'm unsure if you really need to modify all VMs on the ESX servers -- I
> think that's only necessary wherever you want to test QGA installation after
> conversion. If you don't specifically focus on QGA after conversion, I think
> it's OK to just let this firstboot script to fail. Also I assume in your
> automated tests, where you might want to check QGA install all the time, you
> could also add a step to the input domain configuration, for configuring the
> repos.
> 
> Regarding Suse/Debian/Ubuntu, shouldn't those guests come with pre-set repo
> locations already?

Hi Laszlo,
   I think the qemu-ga installation method of the bug is more complicated than the previous qemu-ga installation method(pls refer to bug1619665 and bug1791802). For example, I have 100 VMware linux guests to install qemu-ga. If I use the method of the bug, I need to configure repo in the original 100 VMware guest(it's very difficult),  but if I use this previous method, I just need to put the qemu-ga package in /usr/share/virtio-win path and export the path, is it possible that v2v help to configure the repo for guest during conversion? Or v2v helps to download the qemu-ga package from somewhere during conversion?

Hi mxie,

I don't understand why I am getting this requirement only now. The discussion about local or networked (firstboot) installation happened in February 2022 (almost six months ago); see comment#2. Stabilizing the firstboot package installation with selinux etc was a lot of work.

This is a huge waste of resources, if we only figure out that an idea is not good once QE starts testing it. I don't even know what to suggest.

For RHEL guests, generating any repo files by virt-v2v is not good; those come from subscription-manager.

I don't find it realistic to have a 100 Linux VMs without having a functional (networked) package management in them. How do you install normal bugfix and security updates? Even if you use Satellite, that still requires some level of network connectivity.

Sorry, I don't have a constructive idea here. If you cannot accept this approach, feel free to move the BZ back to ASSIGNED status.

(In reply to Laszlo Ersek from comment #25)
> Hi mxie,
> 
> I don't understand why I am getting this requirement only now. The
> discussion about local or networked (firstboot) installation happened in
> February 2022 (almost six months ago); see comment#2. Stabilizing the
> firstboot package installation with selinux etc was a lot of work.
> 
> This is a huge waste of resources, if we only figure out that an idea is not
> good once QE starts testing it. I don't even know what to suggest.
> 
> For RHEL guests, generating any repo files by virt-v2v is not good; those
> come from subscription-manager.
> 
> I don't find it realistic to have a 100 Linux VMs without having a
> functional (networked) package management in them. How do you install normal
> bugfix and security updates? Even if you use Satellite, that still requires
> some level of network connectivity.
> 
> Sorry, I don't have a constructive idea here. If you cannot accept this
> approach, feel free to move the BZ back to ASSIGNED status.

Hi Laszlo,
  
   I tried previous method to install qemu-ga for guest and it still works, besides, v2v will not use the method of this bug to install qemu-ga if use previous method during conversion, I accepted the fix of this bug, I think we can let users choose which way to install qemu-ga, can we propose these two ways in the documentation?


Steps:
1.Make sure there is no repo configured and qemu-guest-agent installed in original guest
# ls /etc/yum.repos.d/
redhat.repo

# rpm -q qemu-guest-agent
package qemu-guest-agent is not installed

2. Download qemu-guest-agent rhel9 package to directory '/usr/share/virtio-win/linux/el9' and set up environment variables
# ls /usr/share/virtio-win/linux/el9
qemu-guest-agent-7.0.0-8.el9.x86_64.rpm

# export VIRTIO_WIN=/usr/share/virtio-win/

3.Convert the guest from VMware to rhv by v2v
# virt-v2v -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io  vddk-thumbprint=76:75:59:0E:32:F5:1E:58:69:93:75:5A:7B:51:32:C5:D1:6D:F1:21 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api  -op /home/rhvpasswd  -os nfs_data -b ovirtmgmt -ip /home/passwd esx7.0-rhel9.1-x86_64 -on esx7.0-rhel9.1-x86_64-old-method-to-install-qemu-ga
[   0.2] Setting up the source: -i libvirt -ic vpx://root.227.27/data/10.73.199.217/?no_verify=1 -it vddk esx7.0-rhel9.1-x86_64
[   2.0] Opening the source
[   8.0] Inspecting the source
[  15.0] Checking for sufficient free disk space in the guest
[  15.0] Converting Red Hat Enterprise Linux 9.1 Beta (Plow) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[ 148.5] Mapping filesystem data to avoid copying unused and blank areas
[ 149.5] Closing the overlay
[ 149.8] Assigning disks to buses
[ 149.8] Checking if the guest needs BIOS or UEFI to boot
[ 149.8] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 182.4] Copying disk 1/1
█ 100% [****************************************]
[ 571.3] Creating output metadata
[ 589.9] Finishing off

4.Check the guest after v2v conversion, found qemu-ga is installed and qemu-ga service is running successfully but there is no virt-sysprep-firstboot.log in root path
# rpm -q qemu-guest-agent
qemu-guest-agent-6.2.0-11.el9.x86_64

# systemctl status qemu-guest-agent
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/usr/lib/systemd/system/qemu-guest-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-07-26 02:08:21 EDT; 19min ago
   Main PID: 744 (qemu-ga)
      Tasks: 2 (limit: 10552)
     Memory: 1.4M
        CPU: 44ms
     CGroup: /system.slice/qemu-guest-agent.service
             └─744 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=gu>

# ls /root/virt-sysprep-firstboot.log
ls: cannot access '/root/virt-sysprep-firstboot.log': No such file or directory

Hi mxie,

(In reply to mxie from comment #26)
>   
> I tried previous method to install qemu-ga for guest and it still
> works, besides, v2v will not use the method of this bug to install
> qemu-ga if use previous method during conversion, I accepted the fix
> of this bug, I think we can let users choose which way to install
> qemu-ga, can we propose these two ways in the documentation?

The two ways do not work simultaneously.

The original way (to install the package from virtio-win) was reported
as "not functioning" in comment#0 and comment#2, and therefore the
firstboot-time installation from the network was implemented as a
*replacement*. Specifically, commit 52e9cd77a8ef ("windows_virtio:
remove "install_linux_tools"", 2022-06-14) removes the original
implementation.

> 2. Download qemu-guest-agent rhel9 package to directory
> '/usr/share/virtio-win/linux/el9' and set up environment variables
> # ls /usr/share/virtio-win/linux/el9
> qemu-guest-agent-7.0.0-8.el9.x86_64.rpm

This step in fact differs from how the original approach is supposed to
work.

The user should not be expected to manually download an RPM file, and
place it under /usr/share. Only installed packages should populate
/usr/share. The expectation of the user is to install the virtio-win
RPM, nothing more.

That is, comment#0 and comment#2 are still correct in stating that the
virtio-win based QGA installation method does not work. (Because the
virtio-win package does not, in fact, provide an "embedded" QGA RPM.)

What we can do here (if you insist on locally providing the qga RPM to
virt-v2v) is to revert the upstream virt-v2v series from both the master
branch and the rhel-9.1 branch, and then reimplement the internals of
the "install_linux_tools" function.

This function was based on the "copy_from_virtio_win" function (which we
still have), but "copy_from_virtio_win" is unsuitable for this job, as
the virtio-win simply *does not contain* the qga RPM file(s). (That's
why this BZ was opened in the first place.)

Again, manually curl'ing or wget'ting an RPM file to a directory under
"/usr/share/virtio-win" is wrong. It corrupts the host OS installation.

As discussed above.

Tested with the versions:
libguestfs-1.48.4-1.el9.x86_64
qemu-kvm-7.0.0-9.el9.x86_64
libnbd-1.12.6-1.el9.x86_64
virt-v2v-2.0.7-4.el9.x86_64
libvirt-8.5.0-4.el9.x86_64

1. Check the original VMs (RHEL9.0/RHEL8.5) in order to ensure NO qemu-guest-agent rpm/service and configure repos;

2. Convert them via virt-v2v to rhv; 
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -ip /v2v-ops/esxpw -o rhv-upload  -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /v2v-ops/rhvpasswd -os nfs_data -b ovirtmgmt esx6.7-rhel9.0-x86_64
[   0.1] Setting up the source: -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk esx6.7-rhel9.0-x86_64
[   2.0] Opening the source
[   7.9] Inspecting the source
[  14.7] Checking for sufficient free disk space in the guest
[  14.7] Converting Red Hat Enterprise Linux 9.0 Beta (Plow) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[ 146.6] Mapping filesystem data to avoid copying unused and blank areas
[ 147.7] Closing the overlay
[ 148.0] Assigning disks to buses
[ 148.0] Checking if the guest needs BIOS or UEFI to boot
[ 148.0] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[ 189.1] Copying disk 1/1
█ 100% [****************************************]
[ 408.3] Creating output metadata
[ 426.0] Finishing off


3. Start VM and check the qemu-guest-agent rpm/service.

# rpm -qa |grep  qemu-guest-agent
qemu-guest-agent-6.2.0-11.el9_0.2.x86_64
# systemctl status qemu-guest-agent.service
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/usr/lib/systemd/system/qemu-guest-agent.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-03 16:15:51 CST; 6min ago
   Main PID: 6571 (qemu-ga)
      Tasks: 2 (limit: 10680)
     Memory: 608.0K
        CPU: 17ms
     CGroup: /system.slice/qemu-guest-agent.service
             └─6571 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=guest-file-open,guest-fil>

Aug 03 16:15:51 bootp-73-195-94.lab.eng.pek2.redhat.com systemd[1]: Started QEMU Guest Agent.
lines 1-11/11 (END)


Moving to Verified.

/etc/rc5.d/S99guestfs-firstboot start
Scripts dir: /usr/lib/virt-sysprep/scripts
=== Running /usr/lib/virt-sysprep/scripts/5000-0001-wait-online ===
Error: Object 'networking' is unknown, try 'nmcli help'.
/usr/lib/virt-sysprep/scripts-done/5000-0001-wait-online: line 12: systemctl: command not found
=== Running /usr/lib/virt-sysprep/scripts/5000-0002-setenforce-0 ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0003-install-qga ===
Loaded plugins: product-id, refresh-packagekit, search-disabled-repos, security,
              : subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package qemu-guest-agent.x86_64 2:0.12.1.2-2.503.el6_9.6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package             Arch      Version                        Repository   Size
================================================================================
Installing:
 qemu-guest-agent    x86_64    2:0.12.1.2-2.503.el6_9.6       rhel6.10    508 k

Transaction Summary
================================================================================
Install       1 Package(s)

Total download size: 508 k
Installed size: 233 k
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 2:qemu-guest-agent-0.12.1.2-2.503.el6_9.6.x86_64             1/1 
  Verifying  : 2:qemu-guest-agent-0.12.1.2-2.503.el6_9.6.x86_64             1/1 

Installed:
  qemu-guest-agent.x86_64 2:0.12.1.2-2.503.el6_9.6                              

Complete!
=== Running /usr/lib/virt-sysprep/scripts/5000-0004-setenforce-restore ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0005-start-qga ===
qemu-guest-agent: unrecognized service
error reading information on service qemu-guest-agent: No such file or directory


The service name is "qemu-ga" instead of "qemu-guest-agent" on RHEL6.

And also the errors in wait-online script on RHEL6.

=== Running /usr/lib/virt-sysprep/scripts/5000-0001-wait-online ===
Error: Object 'networking' is unknown, try 'nmcli help'.
/usr/lib/virt-sysprep/scripts-done/5000-0001-wait-online: line 12: systemctl: command not found

(In reply to Xiaodai Wang from comment #32)

> Error: Object 'networking' is unknown, try 'nmcli help'.
> /usr/lib/virt-sysprep/scripts-done/5000-0001-wait-online: line 12:
> systemctl: command not found

This is OK. Checking for network connectivity is "best effort". If the guest has neither "nmcli networking connectivity" nor "systemctl -q is-active systemd-networkd", then we cannot check for network connectivity. (Note that we've shown that the "nm-online" command, which may be available on RHEL-6 too, is useless; it does not do what we (and other users) expect.) In that case, we just assume "network is up" -- which is not a wrong assumption, as this particular case shows too. RHEL-6 boots "serially", so by the time we reach our firstboot scripts, the network is indeed up, and "yum" installs the package fine.

> === Running /usr/lib/virt-sysprep/scripts/5000-0005-start-qga ===
> qemu-guest-agent: unrecognized service
> error reading information on service qemu-guest-agent: No such file or
> directory
> 
> 
> The service name is "qemu-ga" instead of "qemu-guest-agent" on RHEL6.

Blech, this is a problem however, indeed. We need to distinguish the package name from the service name. How sad.

Please flip this BZ back to ASSIGNED (I'm not doing it myself because the FailedQA keyword is deserved, and if I do it, maybe it won't be set?)

Thanks.

Debian releases providing the qemu-guest-agent package:

https://packages.debian.org/stretch/qemu-guest-agent
https://packages.debian.org/buster/qemu-guest-agent
https://packages.debian.org/buster-backports/qemu-guest-agent
https://packages.debian.org/bullseye/qemu-guest-agent
https://packages.debian.org/bullseye-backports/qemu-guest-agent
https://packages.debian.org/bookworm/qemu-guest-agent
https://packages.debian.org/sid/qemu-guest-agent

Debianization tarballs, respectively:

http://security.debian.org/debian-security/pool/updates/main/q/qemu/qemu_2.8+dfsg-6+deb9u17.debian.tar.xz
http://deb.debian.org/debian/pool/main/q/qemu/qemu_3.1+dfsg-8+deb10u8.debian.tar.xz
http://deb.debian.org/debian/pool/main/q/qemu/qemu_5.2+dfsg-9~bpo10+1.debian.tar.xz
http://deb.debian.org/debian/pool/main/q/qemu/qemu_5.2+dfsg-11+deb11u2.debian.tar.xz
http://deb.debian.org/debian/pool/main/q/qemu/qemu_7.0+dfsg-2~bpo11+2.debian.tar.xz
http://deb.debian.org/debian/pool/main/q/qemu/qemu_7.0+dfsg-7.debian.tar.xz
http://deb.debian.org/debian/pool/main/q/qemu/qemu_7.0+dfsg-7.debian.tar.xz

The service startup file is identical in each (md5sums given on the left):

f61a64ac1e48993023018fd1cff85191  qemu_2.8+dfsg-6+deb9u17/debian/qemu-guest-agent.init
f61a64ac1e48993023018fd1cff85191  qemu_3.1+dfsg-8+deb10u8/debian/qemu-guest-agent.init
f61a64ac1e48993023018fd1cff85191  qemu_5.2+dfsg-11+deb11u2/debian/qemu-guest-agent.init
f61a64ac1e48993023018fd1cff85191  qemu_5.2+dfsg-9~bpo10+1/debian/qemu-guest-agent.init
f61a64ac1e48993023018fd1cff85191  qemu_7.0+dfsg-2~bpo11+2/debian/qemu-guest-agent.init
f61a64ac1e48993023018fd1cff85191  qemu_7.0+dfsg-7/debian/qemu-guest-agent.init

The service is called "qemu-guest-agent" in each:

# Provides:          qemu-guest-agent

The results from comment 39 are not satisfactory. I checked a Debian 11 (bullseye) domain, and it has both an rc.d compatible init file, and a systemd service.

Currently, virt-v2v seeks to handle RHEL, ALT, SUSE and Debian family linux distros. Apparently, I need to install every possible release of RHEL, ALT, OpenSUSE and Debian that provides the qemu-guest-agent package, and manually test out how the service can be activated after installation.

See you next month I guess.

virt-builder provides images for Debian 6 through Debian 11. Once downloaded & decompressed, I imported each image into libvirt, as follows:

$ virt-install --import --disk vol=default/debian-$VER.img --ram 4096 --os-variant debian$VER

After these commands, the Debian 6 and Debian 7 domains would not successfully boot; they hang at some point during the boot process. I'll focus my investigation on Debian 8 and upwards (8 - jessie, 9 - stretch, 10 - buster, 11 - bullseye).

Debian 8: after running "apt-get install qemu-guest-agent", and making sure in advance that the domain config had a QGA channel (this was covered by "virt-install --os-variant debian8"), the package installation enabled the service immediately. No particular command was needed for service enablement. The service was available after reboot too. Verified with "virsh domifaddr --source agent debian8".

Debian 9: I had to set up networking manually in the guest at first, which was extremely unpleasant. "apt-get update" and "apt-get install" did the right thing afterwards -- the service was enabled immediately, and persisted after reboot too.

Debian 10: I couldn't get "apt-get update" to work, even after enabling the network. I got various pointers to "allow-downgrade-to-insecure=yes" and apt-secure(), but I couldn't get it to work. "dist-upgrade" didn't help. Skip.

Debian 11: "apt-get update" refused to work here too, even after enabling the network. "apt-get dist-upgrade" solved it though. The dist-upgrade brought in qemu-guest-agent automatically, which was very surprising (and unwanted). I removed qemu-guest-agent with "apt-get purge qemu-guest-agent", and rebooted. Subsequently, I installed qemu-guest-agent with "apt-get install", and surprisingly, the service was *not* enabled immediately. "service qemu-guest-agent start" started QGA fine. The service persisted after rebooting the domain.

SUMMARY for Debian:
- the package is called qemu-guest-agent
- the service is usually enabled right after package installation, but when it's not, "service qemu-guest-agent start" helps
- persistent enabling requires no additional commands.

ALT Linux:

all releases (branches) that can be searched on the web for packages provide the "qemu-guest-agent" package; from most recent to least recent release:

https://packages.altlinux.org/en/search?branch=sisyphus&name=qemu-guest-agent
https://packages.altlinux.org/en/search?branch=p10&name=qemu-guest-agent
https://packages.altlinux.org/en/search?branch=p9&name=qemu-guest-agent
https://packages.altlinux.org/en/search?branch=p8&name=qemu-guest-agent
https://packages.altlinux.org/en/search?branch=c7&name=qemu-guest-agent

The download locations, per <https://en.altlinux.org/Downloads>, are the following, for all releases (branches) except sisyphus -- sisyphus does not offer ISO images, at <http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/>:

http://ftp.altlinux.org/pub/distributions/ALTLinux/p10/images/server/x86_64/
http://ftp.altlinux.org/pub/distributions/ALTLinux/p9/images/server/x86_64/
http://ftp.altlinux.org/pub/distributions/ALTLinux/p8/images/server/
http://ftp.altlinux.org/pub/distributions/ALTLinux/c7/images/updates/

The install media for x86_64 are:

http://ftp.altlinux.org/pub/distributions/ALTLinux/p10/images/server/x86_64/alt-server-10.0-x86_64.iso
http://ftp.altlinux.org/pub/distributions/ALTLinux/p9/images/server/x86_64/alt-server-9.2-x86_64.iso
http://ftp.altlinux.org/pub/distributions/ALTLinux/p8/images/server/alt-server-8.2.1-20190906-x86_64-ru-install-dvd5.iso
http://ftp.altlinux.org/pub/distributions/ALTLinux/c7/images/updates/cd2.iso

ALT Linux C7 is uninstallable from <http://ftp.altlinux.org/pub/distributions/ALTLinux/c7/images/updates/cd2.iso>, exposed in an IDE CD-ROM to the guest: SeaBIOS reports "Boot failed: Could not read from CDROM (code 0003)". This happens despite <http://ftp.altlinux.org/pub/distributions/ALTLinux/c7/images/updates/README.txt> stating:

-------
i586: CD 1
x86_64: CD 2
SRPMS: CD 3
-------

Therefore I'm skipping ALT Linux C7.

ALT Linux Server 8.2.1 is bootable from <http://ftp.altlinux.org/pub/distributions/ALTLinux/p8/images/server/alt-server-8.2.1-20190906-x86_64-ru-install-dvd5.iso>; however, it is 100% localized to a Russian user base. I can't even read the boot menu due to it being in Cyrillic script.

The same applies to "alt-server-9.2-x86_64.iso" and "alt-server-10.0-x86_64.iso" too.

This means I can't test qemu-guest-agent package installation & service enablement "live".

Then, I've tried looking at the packages themselves, following the links from the first half of comment 43:

c7:       https://git.altlinux.org/tasks/188849/build/100/x86_64/rpms/qemu-guest-agent-2.5.1.1-alt0.M70C.5.x86_64.rpm
p8:       https://git.altlinux.org/tasks/217644/build/100/x86_64/rpms/qemu-guest-agent-2.11.0-alt0.M80P.4.x86_64.rpm
p9:       https://git.altlinux.org/tasks/270488/build/600/x86_64/rpms/qemu-guest-agent-5.2.0-alt6.x86_64.rpm
p10:      https://git.altlinux.org/tasks/301460/build/400/x86_64/rpms/qemu-guest-agent-6.2.0-alt1.p10.x86_64.rpm
sisyphus: https://git.altlinux.org/tasks/301530/build/100/x86_64/rpms/qemu-guest-agent-7.0.0-alt1.x86_64.rpm

Each RPM contains:
- lib/systemd/system/qemu-guest-agent.service
- lib/udev/rules.d/90-qemu-guest-agent.rules

The latter file is identical across all five RPMs:

----------
SUBSYSTEM=="virtio-ports", ATTR{name}=="org.qemu.guest_agent.0", \
  TAG+="systemd" ENV{SYSTEMD_WANTS}="qemu-guest-agent.service"
----------

The former file has two versions, { c7, p8 } and { p9, p10, sisyphus }:

----------
[Unit]
Description=QEMU Guest Agent
BindTo=dev-virtio\x2dports-org.qemu.guest_agent.0.device
After=dev-virtio\x2dports-org.qemu.guest_agent.0.device

[Service]
ExecStart=-/usr/bin/qemu-ga
Restart=always
RestartSec=0

[Install]

----------

and

----------
[Unit]
Description=QEMU Guest Agent
BindsTo=dev-virtio\x2dports-org.qemu.guest_agent.0.device
After=dev-virtio\x2dports-org.qemu.guest_agent.0.device
IgnoreOnIsolate=True

[Service]
UMask=0077
Environment=FSFREEZE_HOOK_PATHNAME=/etc/qemu/fsfreeze-hook DAEMON_ARGS="" BLACKLIST_RPC=""
EnvironmentFile=/etc/sysconfig/qemu-ga
ExecStart=/usr/bin/qemu-ga \
  $DAEMON_ARGS \
  --method=virtio-serial \
  --path=/dev/virtio-ports/org.qemu.guest_agent.0 \
  --blacklist=${BLACKLIST_RPC} \
  -F${FSFREEZE_HOOK_PATHNAME}
Restart=always
RestartSec=0

[Install]
WantedBy=dev-virtio\x2dports-org.qemu.guest_agent.0.device
----------

SUMMARY for ALT Linux:
- the package is called qemu-guest-agent
- the service is called qemu-guest-agent
- the service is systemd and udev based
- the systemd unit is not a template unit but a concrete unit
- "systemctl start" *should* suffice for starting the service right after installation
- after reboot, the udev rule *should* take care of service startup

Virt-builder offers the followin opensuse templates: 13.1, 13.2, 42.1, tumbleweed.

virt-builder --quiet --hostname opensuse-13-1       -o opensuse-13-1.img       --root-password password:redacted opensuse-13.1
virt-builder --quiet --hostname opensuse-13-2       -o opensuse-13-2.img       --root-password password:redacted opensuse-13.2
virt-builder --quiet --hostname opensuse-42-1       -o opensuse-42-1.img       --root-password password:redacted opensuse-42.1
virt-builder --quiet --hostname opensuse-tumbleweed -o opensuse-tumbleweed.img --root-password password:redacted opensuse-tumbleweed

virt-install --import --name opensuse-13-1       --disk vol=default/opensuse-13-1.img       --ram 4096 --os-variant opensuse13.1       --network bridge=virbr0
virt-install --import --name opensuse-13-2       --disk vol=default/opensuse-13-2.img       --ram 4096 --os-variant opensuse13.2       --network bridge=virbr0
virt-install --import --name opensuse-42-1       --disk vol=default/opensuse-42-1.img       --ram 4096 --os-variant opensuse42.1       --network bridge=virbr0
virt-install --import --name opensuse-tumbleweed --disk vol=default/opensuse-tumbleweed.img --ram 4096 --os-variant opensusetumbleweed --network bridge=virbr0

Each domain gets the qga channel set up automatically.

13.1 does not boot successfully; it gets stuck at "Reached target Graphical Interface". (But I can still tell it is a systemd-based distribution.)

The other three domains boot fine (just to the plain text console). I'm going to ignore 13.1 in the rest of this comment.

Testing all three domains (13.2, 42.1, tumbleweed) with "virsh domifaddr --source agent": none of them have QGA installed or running at once.

Now, turns out virt-builder failed to set the root password for the 42.1 and tumbleweed domains (I used the same "--root-password password:..." option all four times), so I'm going to ignore those two domains as well (can't log in -- can't install packages interactively). Therefore I'm left with 13.2 only.

"zypper install qemu-guest-agent" installs the package fine, but does not enable it at once.

Now, consider the following commands:

# rpm -q qemu-guest-agent 
qemu-guest-agent-2.1.3-7.2.x86_64

# rpm -ql qemu-guest-agent 
/usr/bin/qemu-ga

That's it. The qemu guest agent is not available as a service *at all* in opensuse 13.2. It's impossible to enable it.

Now, the opensuse package search reports three hits for qemu-guest-agent though:

https://software.opensuse.org/package/qemu-guest-agent

-> openSUSE Leap 15.2:  4.2.1
-> openSUSE Leap 15.3:  5.2.0
-> openSUSE Tumbleweed: 7.0.0

I intend to download installer ISOs for these distro releases later, and check qemu-guest-agent installation and service enablement live, in newly created domains.

(In reply to Laszlo Ersek from comment #46)

> Now, the opensuse package search reports three hits for qemu-guest-agent
> though:
> 
> https://software.opensuse.org/package/qemu-guest-agent
> 
> -> openSUSE Leap 15.2:  4.2.1
> -> openSUSE Leap 15.3:  5.2.0
> -> openSUSE Tumbleweed: 7.0.0

Correction -- I can't easily interpret the result list on that page, but it seems that Leap 15.3 and Leap 15.4 *both* ship 5.2.0.

ISO images:

0fd2d4e630b6579b933b5cb4930a8100acca6b4e29cd2738c4b7a9b2f76d80e4  openSUSE-Leap-15.2-DVD-x86_64.iso
c1515358daec1ab7c3be2b7c28636eac2e18f0ad550b918ed0550722b87f8b49  openSUSE-Leap-15.3-3-DVD-x86_64-Build38.1-Media.iso
4683345f242397c7fd7d89a50731a120ffd60a24460e21d2634e783b3c169695  openSUSE-Leap-15.4-DVD-x86_64-Build243.2-Media.iso
954efb5c4a5b187e4f993b4c47f15ee98b37e0f1a91eaab97cb82b57ddb6eef8  openSUSE-Tumbleweed-DVD-x86_64-Snapshot20220813-Media.iso

* openSUSE-Leap-15.2:
- the package is called "qemu-guest-agent"
- the service is udev- and systemd-based
- the systemd unit is called qemu-ga@ (note the trailing "at" sign -- this is a template unit that gets customizer with the virtio-serial port pathname upon instantiation)
- the package is automatically installed when selecting the "Server" package set in the openSUSE installer
- when the package is not installed on the system, "zypper install qemu-guest-agent" both installs it and enables the service (checking it right after from the host side, with "virsh domifaddr --source agent", succeeds)
- with the package installed, the next boot auto-starts the service too

* openSUSE-Leap-15.3:
- the package is called "qemu-guest-agent"
- the service is udev- and systemd-based
- the systemd unit is called qemu-guest-agent (!), and it is not a template unit but a concrete unit -- the udev rule differs from the one in 15.2 as well, accordingly
- the package is automatically installed when selecting the "Server" package set in the openSUSE installer
- when the package is not installed on the system, "zypper install qemu-guest-agent" both installs it and enables the service (checking it right after from the host side, with "virsh domifaddr --source agent", succeeds)
- with the package installed, the next boot auto-starts the service too

* openSUSE-Leap-15.4:
- precisely the same behavior as with openSUSE-Leap-15.3

* openSUSE-Tumbleweed (Snapshot20220813):
- precisely the same behavior as with openSUSE-Leap-15.3

SUMMARY for openSUSE:
- refer to openSUSE-Leap-15.3 above.

(The only difference introduced in "openSUSE-Leap-15.3" was that the systemd unit got renamed, and turned into a concrete unit from a template unit, with the accompanying udev rule adapted. However, this does not matter for virt-v2v, as we need to start up the service manually in neither case -- "zypper install qemu-guest-agent" starts/enables the agent immediately and permanently, in all the tested releases above.)

* RHEL-5.11:

- does not provide the qemu guest agent (the "VT" repository provides
  packages built from the "kvm" SRPM, such as kmod-kvm, kmod-kvm-debug,
  kvm, kvm-qemu-img, kvm-tools, kvm-debuginfo; however, in the latest
  build in Brew, 83-277.el5_11 (buildID=540686), those binary packages
  do not provide any guest agent binaries)

* RHEL-6.10 (20180525.0-Server-x86_64):

- the package is called qemu-guest-agent (0.12.1.2-2.503.el6_9.6.x86_64)

- the service is called "qemu-ga"

- the service is *not* systemd-based / udev-based

- installation of the package does not start the service immediately

- installation of the package does enable the service for future boots,
  at runlevels 2 through 5 (checked with "chkconfig" and also with
  actual rebooting)

- the command "service qemu-ga start" is good enough for launching the
  service right after installation

* RHEL-7.9 (20200917.0-Server-x86_64):

- the package is called qemu-guest-agent (2.12.0-3.el7.x86_64)

- the service is systemd- and udev-based

- the systemd unit is called "qemu-guest-agent"; it is a concrete (not
  template) unit

- The package is installed as a part of the default installation. I
  removed it manually and rebooted the guest to find out package
  installation details.

- With the package *just installed*, the systemd service is enabled for
  *future boots only* (per "systemctl is-enabled qemu-guest-agent" and
  "systemctl status" / "vendor preset", also checked via manual
  rebooting); not immediately (per "systemctl status").

- "systemctl start qemu-guest-agent" starts the service right after
  installing the package.

* RHEL-8.6 (20220420.3-x86_64 / AppStream):

- the package is called qemu-guest-agent
  (6.2.0-11.module+el8.6.0+14707+5aa4b42d.x86_64)

- otherwise: same behavior as under RHEL-7.9

* RHEL-9.0 (20220420.0-x86_64 / AppStream):

- the package is called qemu-guest-agent (6.2.0-11.el9_0.2.x86_64)

- otherwise: same behavior as under RHEL-7.9

SUMMARY for RHEL:

- package is available in RHEL-6+

- package is always called "qemu-guest-agent"

- right after package installation, i.e., during firstboot, RHEL-6
  requires "service qemu-ga start", later RHELs require "systemctl start
  qemu-guest-agent", for enabling the service immediately

- for boots after firstboot, the service need not be enabled explicitly

Because the virt-v2v logic may have to work based off the inspection's version
info, I've now checked what virt-inspector reports for the disk images I used
for collecting the above information.

debian-08
    <major_version>8</major_version>
    <minor_version>9</minor_version>

debian-09
    <major_version>9</major_version>
    <minor_version>1</minor_version>

debian-10
    <major_version>10</major_version>
    <minor_version>0</minor_version>

debian-11
    <major_version>11</major_version>
    <minor_version>4</minor_version>

openSUSE-Leap-15.2
    <major_version>15</major_version>
    <minor_version>2</minor_version>

openSUSE-Leap-15.3
    <major_version>15</major_version>
    <minor_version>3</minor_version>

openSUSE-Leap-15.4
    <major_version>15</major_version>
    <minor_version>4</minor_version>

openSUSE-Tumbleweed
    <major_version>20220813</major_version>
    <minor_version>0</minor_version>
    <major_version>20220813</major_version>
    <minor_version>0</minor_version>
    <major_version>20220813</major_version>
    <minor_version>0</minor_version>
    <major_version>20220813</major_version>
    <minor_version>0</minor_version>
    <major_version>20220813</major_version>
    <minor_version>0</minor_version>
    <major_version>20220813</major_version>
    <minor_version>0</minor_version>

rhel-6.10
    <major_version>6</major_version>
    <minor_version>10</minor_version>

rhel-7.9
    <major_version>7</major_version>
    <minor_version>9</minor_version>

rhel-8.6
    <major_version>8</major_version>
    <minor_version>6</minor_version>

rhel-9.0
    <major_version>9</major_version>
    <minor_version>0</minor_version>

So everything works as expected, except openSUSE-Tumbleweed -- the major
version is very confusing there (well, it is a rolling distro), plus it has one
root filesystem, and a bunch of btrfs snapshots of that filesystem. However,
per comment#51, we don't care about the major/minor release numbers for
openSUSE; releases of that distro can be handled uniformly.

Test results, using "-i libvirt -o libvirt -on ...":

- none of the converted debian guests (08 through 11) would boot to grub2 successfully, so no results for Debian (this is independent of the patch)

- openSUSE-Leap-15.2 through openSUSE-Leap-15.4: these three converted domains failed to bring up networking *permanently*. In other words, not only did the firstboot scripts lack network connectivity, but much-much later, after logging in interactively, there was still no networking. (In other words, the problem was not that we attempted the QGA installation before network connectivity was established.) So, no results (independently of the patch).

- openSUSE-Tumbleweed: success (verified with "virsh domifaddr --source agent converted-openSUSE-Tumbleweed" at first and second boots).

- ALT Linux: due to it being entirely localized to a Russian userbase, I couldn't even install such guests interactively (see comment 45). No results.

- rhel-6.10, rhel-7.9: success (verified with "virsh domifaddr --source agent converted-openSUSE-Tumbleweed" at first and second boots).

- rhel-8.6, rhel-9.0: In the original domains, both "ifconfig" and "nmcli connection" reports the ethernet card's name as "ens3", and therefore there is network connectivity. In the converted domains, "nmcli connection" still reports the ethernet card's name as "ens3"; however, according to "ifconfig", it gets renamed to "enp3s0" as a result of the conversion. As a consequence, networking is permanently broken in the converted domains, the NIC is never brought up (impossible even with manual "nmcli connection up"), our wait-online logic in the firstboot script times out, and then package installation fails. No results. (independent of the patch)

[v2v PATCH] convert_linux: start the QEMU guest agent in a distro-specific way
Message-Id: <20220817144736.18850-1-lersek>

(I can't see the message in the archive yet, or in my inbox, so no archive URL for now -- the list server is either being slow, or it's temporarily broken)

Created attachment 1905976 [details]
patch posted in comment 55

> rhel-8.6, rhel-9.0

Are you doing the conversion in a way that preserves the MAC address?
Not all of our output methods can preserve it unfortunately.

But yes, network interface renaming is an absolute nightmare and
something we don't properly address at the moment.

I've converted rhel-7.9, rhel-8.6 and rhel-9.0 again, for an answer.

The MAC address remains unchanged in all three cases.

The machine type gets flipped from pc to q35 in all three cases, thereby causing the virtio-net NIC to move to different spot in the PCI(e) hierarchy. Notably however, this "relocation" is identical for rhel-7.9, rhel-8.6 and rhel-9.0; that is, picking any one of these three domains, the (source PCI address, converted PCI address) pair is identical to that of the other two domains:

     <interface type='bridge'>
       <mac address='UNCHANGED-MAC-ADDRESS'/>
       <source bridge='virbr0'/>
       <model type='virtio'/>
-      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
+      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
     </interface>

To this change, rhel-7.9 does not react (so its networking remains functional), whereas rhel-8.6 and rhel-9.0 rename the interface.

(In reply to Laszlo Ersek from comment #55)
> [v2v PATCH] convert_linux: start the QEMU guest agent in a distro-specific way
> Message-Id: <20220817144736.18850-1-lersek>

https://listman.redhat.com/archives/libguestfs/2022-August/029613.html

upstream virt-v2v commit ad2b4f2e5095

(In reply to Laszlo Ersek from comment #58)

>      <interface type='bridge'>
>        <mac address='UNCHANGED-MAC-ADDRESS'/>
>        <source bridge='virbr0'/>
>        <model type='virtio'/>
> -      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> +      <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> function='0x0'/>
>      </interface>
> 
> To this change, rhel-7.9 does not react (so its networking remains
> functional), whereas rhel-8.6 and rhel-9.0 rename the interface.

(Side comment: I think this is pretty bad for physical machines too: you move a NIC to a different slot for some reason, and now your network does not come up at next boot. I thought that behavior was what we'd been moving away from.)

Verified with the versions:
virt-v2v-2.0.7-6.el9.x86_64
libvirt-8.5.0-5.el9.x86_64
libguestfs-1.48.4-2.el9.x86_64
qemu-img-7.0.0-11.el9.x86_64
qemu-kvm-7.0.0-11.el9.x86_64

The guests success:

RHEL6.10
RHEL7.9
RHEL8.6
RHEL9.0

Ubuntu18.04
Debian11.4.0
SLES15.2
SLES15.4
openSUSE-Leap-15.2 
openSUSE-Leap-15.4

But openSUSE-Tumbleweed failed.

Laszlo, could you check again? Cause I saw Comment54: openSUSE-Tumbleweed: success.

Please check the attachment on debug log.

Just want to say here that if everything is working except OpenSUSE
Tumbleweed, then that's a success!  Tumbleweed is SUSE's rolling release
(like Fedora Rawhide) and it is very troublesome.  It causes constant
problems with our upstream CI too since it's not very stable or reliable.
We shouldn't worry too much if it's sometimes broken here.

(In reply to Richard W.M. Jones from comment #63)
> Just want to say here that if everything is working except OpenSUSE
> Tumbleweed, then that's a success!  Tumbleweed is SUSE's rolling release
> (like Fedora Rawhide) and it is very troublesome.  It causes constant
> problems with our upstream CI too since it's not very stable or reliable.
> We shouldn't worry too much if it's sometimes broken here.

Thanks for the explanation .

Mark this bug to Verified:Tested.

The problem is Packagekit blocks zypper. Packagekit will begin to autoupdate when network
is connected and It has a big chance of conflicting with zypper operation. So this is not
Tumbleweed specific.

/usr/lib/virt-sysprep/firstboot.sh start
Scripts dir: /usr/lib/virt-sysprep/scripts
=== Running /usr/lib/virt-sysprep/scripts/5000-0001-wait-online ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0002-setenforce-0 ===
=== Running /usr/lib/virt-sysprep/scripts/5000-0003-install-qga ===
PackageKit is blocking zypper. This happens if you have an updater applet or other software
management application using PackageKit running.
We can ask PackageKit to interrupt the current action as soon as possible, but it depends on
PackageKit how fast it will respond to this request.
Ask PackageKit to quit? [yes/no] (no): no
System management is locked by the application with pid 1509 (/usr/libexec/packagekitd).
Close this application before trying again.
=== Running /usr/lib/virt-sysprep/scripts/5000-0004-setenforce-restore ===

From the google, there are two main solutions. 
1) wait until packagekit is quit.
2) stop the packagekit service.

And for this bug, I agree to move it verified. and the packagekit issue, it's better to
file a new bug to track it.

Thanks for finding the root of the opensuse-tumbleweed issue. (I'm just back from PTO.) I'd not seen it in my testing.

I'd not think automatic/background package updates (by packagekit or anything else) were really suitable for a server OS. An asynchronously started pkg update can block many other things, not just a separate manual package installation -- such as scheduled reboots, planned service restarts and other planned downtimes (database maintenance etc). At organizations where stability matters, updates are very carefully planned (sometimes weeks in advance). "Update my stuff in the background" is a phone/tablet/consumer market-oriented approach IMO.

Anyway, if this use case (= disabling packagekit) is relevant to us, I agree that we can file a new BZ about it. Same way as we "bracket" the QGA package installation with SELinux disablement, we might handle PackageKit. Too bad that's going to be rather flimsy again, as I'm sure it's either going to have to be distro specific, or it will turn into more "best effort" fluff that does the needful on OpenSUSE-Tumbleweed, but produces useless (albeit harmless) error messages elsewhere.

Laszlo, I have filed a new bug bz2121665 to track this issue on opensuse-tumbleweed.

Thanks.

As comment61, Moving to Verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968