Bug 2083594

+++ This bug was initially created as a clone of Bug #2068905 +++

Description of problem:
virtctl guestfs makes wrong assumption about image name and thus fails to assemble the correct guestfs pod image URL

Version-Release number of selected component (if applicable):
CNV v4.9.4

How reproducible:
100%

Steps to Reproduce:
1. Create DV
2. Attempt to create guestfs pod (`virtctl guestfs simple-dv`)

Actual results:
ErrImagePull

Expected results:
Success

Additional info:
It appears we make a wrong assumption about the image name being "libguestfs-tools" in:
https://github.com/kubevirt/kubevirt/blob/main/pkg/virtctl/guestfs/guestfs.go#L197

This results in an attempt to create a pod from a wrong URL

cnv-qe-jenkins@cnv-qe-infra-01:~/alex$ virtctl guestfs simple-dv
Use image: quay.io/openshift-cnv/libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae

while the correct URL is

cnv-qe-jenkins@cnv-qe-infra-01:~/alex$ oc get csv -n openshift-cnv -o yaml | grep guestfs
    - image: quay.io/openshift-cnv/container-native-virtualization-libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae

Manifests:
cnv-qe-jenkins@cnv-qe-infra-01:~/alex$ cat guestfs_dv.yaml 
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: simple-dv
spec:
  source:
      http:
         url: "http://.../Fedora-Cloud-Base-34-1.2.x86_64.qcow2"
  pvc:
    storageClassName: ocs-storagecluster-ceph-rbd
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 8Gi

cnv-qe-jenkins@cnv-qe-infra-01:~/alex$ virtctl guestfs simple-dv
Use image: quay.io/openshift-cnv/libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae 
The PVC has been mounted at /disk 
Waiting for container libguestfs still in pending, reason: ContainerCreating, message:  
Waiting for container libguestfs still in pending, reason: ContainerCreating, message:  
Waiting for container libguestfs still in pending, reason: ImagePullBackOff, message: Back-off pulling image "quay.io/openshift-cnv/libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae" 
Waiting for container libguestfs still in pending, reason: ImagePullBackOff, message: Back-off pulling image "quay.io/openshift-cnv/libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae" 
Waiting for container libguestfs still in pending, reason: ImagePullBackOff, message: Back-off pulling image "quay.io/openshift-cnv/libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae"

--- Additional comment from Alex Kalenyuk on 2022-03-27 12:21:39 UTC ---

The hotfix efforts (v4.9.4-hotfix.2.1) helped uncover this issue which has been there for a while;
We are releasing our images on quay.io this time around as opposed to registry.redhat.io:

cnv-qe-jenkins@cnv-qe-infra-01:~/alex$ oc get csv -n openshift-cnv kubevirt-hyperconverged-operator.v4.9.4-hotfix.2.1 -o yaml | grep guestfs
  - image: quay.io/openshift-cnv/container-native-virtualization-libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae
    name: registry.redhat.io/container-native-virtualization/libguestfs-tools:v4.9.3-4

For registry.redhat.io notice the image name varies:

[cnv-qe-jenkins@c01-hot-494-01-dgrbl-executor ~]$ oc get csv -n openshift-cnv kubevirt-hyperconverged-operator.v4.9.3 -o yaml | grep guestfs
  - image: registry.redhat.io/container-native-virtualization/libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae
    name: registry.redhat.io/container-native-virtualization/libguestfs-tools:v4.9.3-4

Thanks to Gal Ben Haim's eagle eyes that helped debug this

--- Additional comment from Adam Litke on 2022-03-28 11:13:25 UTC ---

As a workaround I believe you can manually specify the image to use so we could document this in the release notes of the hotfix.

--- Additional comment from Gal Ben Haim on 2022-03-28 12:18:10 UTC ---

Just a note, it may also affect a disconnected install.

--- Additional comment from Vagner Farias on 2022-04-18 17:02:39 UTC ---

I think I'm seeing the same issue, but with a slightly different error message, which didn't tell me much about the real problem. I'm not sure this should be handled in a different bz, but besides fixing the wrong assumption on the image name, a better error message is needed.

vfarias@vfarias-mac ~ % virtctl guestfs -n vfarias-virtualization rhel8-ethical-opossum
unknown

vfarias@vfarias-mac ~ % virtctl guestfs --image quay.io/openshift-cnv/container-native-virtualization-libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae rhel8-ethical-opossum
Use image: quay.io/openshift-cnv/container-native-virtualization-libguestfs-tools@sha256:591e6d1e817bbd42b80f24b1ff9863533acc00fa6032c8abcbc38b9bb9e964ae
The PVC has been mounted at /disk
[.. unrelated error suppressed ..]

vfarias@vfarias-mac ~ % virtctl version
Client Version: version.Info{GitVersion:"v0.49.0-119-gb7f0c6b53", GitCommit:"b7f0c6b535a00e01f4833d8746f2f9be527208f3", GitTreeState:"clean", BuildDate:"2022-04-01T07:23:21Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{GitVersion:"v0.49.0-119-gb7f0c6b53", GitCommit:"b7f0c6b535a00e01f4833d8746f2f9be527208f3", GitTreeState:"clean", BuildDate:"2022-04-01T07:24:56Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

--- Additional comment from Alice Frosi on 2022-04-19 06:50:14 UTC ---

So the libguestfs image name is constructed by getting the registry and sha from the KubeVirt CR (https://github.com/kubevirt/kubevirt/blob/main/pkg/virtctl/guestfs/guestfs.go#L189-L210), but it always uses 'libguestfs-tools' as name.
Here, I see that we use 'container-native-virtualization-libguestfs-tools'. How is this constructed for the other container images like virt-launcher? If required I can extend the the GuestfsVersion command and be able to set also the image name, but this is information needs to be presented somewhere.