Description of problem: When auto-update boot sources feature is enabled, CentOS and Fedora DVs fail to import because: Unable to process data: Docker references with both a tag and digest are currently not supported github.com/containers/image/v5/docker.newReference Version-Release number of selected component (if applicable): HCO v4.10.0-105 How reproducible: 100% Steps to Reproduce: 1. Enable enableCommonBootImageImport in HCO 2. Monitor DVs under openshift-virtualization-os-images ns Actual results: CentOS8 and Fedora DVs are in ImportInProgress state Expected results: All imports should succeed. Additional info: =========================== $ oc get dv -A NAMESPACE NAME PHASE PROGRESS RESTARTS AGE openshift-virtualization-os-images centos8-4a6a0eee7b4a ImportInProgress N/A 51 3h59m openshift-virtualization-os-images fedora-3b3fc310abea ImportInProgress N/A 51 3h59m openshift-virtualization-os-images rhel8-2325b0922721 Succeeded 100.0% 4h openshift-virtualization-os-images rhel9-401cc6250f89 Succeeded 100.0% 4h =========================== $ oc describe dv -n openshift-virtualization-os-images fedora-3b3fc310abea Name: fedora-3b3fc310abea Namespace: openshift-virtualization-os-images Labels: app.kubernetes.io/component=storage app.kubernetes.io/managed-by=cdi-controller app.kubernetes.io/part-of=hyperconverged-cluster app.kubernetes.io/version=4.10.0 cdi.kubevirt.io/dataImportCron=fedora-image-cron Annotations: cdi.kubevirt.io/storage.bind.immediate.requested: true API Version: cdi.kubevirt.io/v1beta1 Kind: DataVolume Metadata: Creation Timestamp: 2022-01-05T06:52:52Z Generation: 129 Managed Fields: API Version: cdi.kubevirt.io/v1beta1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:cdi.kubevirt.io/storage.bind.immediate.requested: f:labels: .: f:app.kubernetes.io/component: f:app.kubernetes.io/managed-by: f:app.kubernetes.io/part-of: f:app.kubernetes.io/version: f:cdi.kubevirt.io/dataImportCron: f:spec: .: f:source: .: f:registry: .: f:url: f:storage: .: f:resources: .: f:requests: .: f:storage: f:status: .: f:conditions: f:phase: f:progress: f:restartCount: Manager: virt-cdi-controller Operation: Update Time: 2022-01-05T06:53:08Z Resource Version: 1009392 UID: d12ae27c-6450-45aa-ab01-df166c67b6e5 Spec: Source: Registry: URL: docker://quay.io/containerdisks/fedora:35@sha256:3b3fc310abeadd38655f2886feffcab5a37c9c3272d95b3147090c3710d8dd72 Storage: Resources: Requests: Storage: 5Gi Status: Conditions: Last Heartbeat Time: 2022-01-05T06:52:57Z Last Transition Time: 2022-01-05T06:52:57Z Message: PVC fedora-3b3fc310abea Bound Reason: Bound Status: True Type: Bound Last Heartbeat Time: 2022-01-05T10:50:32Z Last Transition Time: 2022-01-05T06:52:53Z Status: False Type: Ready Last Heartbeat Time: 2022-01-05T10:50:32Z Last Transition Time: 2022-01-05T10:50:32Z Message: Unable to process data: Docker references with both a tag and digest are currently not supported github.com/containers/image/v5/docker.newReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:80 github.com/containers/image/v5/docker.NewReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:64 github.com/containers/image/v5/docker.ParseReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:59 kubevirt.io/containerized-data-importer/pkg/importer.parseImageName /remote-source/app/pkg/importer/transport.go:91 kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:69 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Could not parse image kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:72 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Failed to read registry image kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:85 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Unable to transfer source data to scratch space kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:202 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Reason: Error Status: False Type: Running Phase: ImportInProgress Progress: N/A Restart Count: 51 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Error 3m25s (x49 over 4h) datavolume-controller Unable to process data: Docker references with both a tag and digest are currently not supported github.com/containers/image/v5/docker.newReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:80 github.com/containers/image/v5/docker.NewReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:64 github.com/containers/image/v5/docker.ParseReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:59 kubevirt.io/containerized-data-importer/pkg/importer.parseImageName /remote-source/app/pkg/importer/transport.go:91 kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:69 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Could not parse image kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:72 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Failed to read registry image kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:85 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Unable to transfer source data to scratch space kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:202 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371
Just for clarification, DataImportCron does not support source URL with tag, and it was never a requirement. We use digests for the imports, and we poll the latest to see if updated. I see no reason why one would use DataImportCron with a tagged url. In the case of a static source why not simply use DataSource? I guess SSP should simply remove the tag.
*** Bug 2036817 has been marked as a duplicate of this bug. ***
I don't see the symptoms of this bug in hco-v4.10.0-600 nor in hco-v4.10.0-601 All DVs imported, and no crashing pods. NAMESPACE NAME PHASE PROGRESS RESTARTS AGE openshift-virtualization-os-images centos-stream8-f09342b27c35 Succeeded 100.0% 28m openshift-virtualization-os-images centos-stream9-530e9c62633a Succeeded 100.0% 28m openshift-virtualization-os-images fedora-3b3fc310abea Succeeded 100.0% 28m openshift-virtualization-os-images rhel8-2325b0922721 Succeeded 100.0% 28m openshift-virtualization-os-images rhel9-401cc6250f89 Succeeded 100.0% 28m
btw, it looks like there is missing field in the defined DataImportCron: spec.template.spec.source.registry.pullMethod: node which is used in the created import DVs: spec.source.registry.pullMethod: node without this one, it can work on external registry but won't work for import from RH registry.
Arnon, Yes, the fedora-image-cron, centos-stream9-image-cron and centos-stream8-image-cron are missing pullMethod:node, but rhel8-image-cron and rhel9-image-cron works well spec: garbageCollect: Outdated managedDataSource: fedora schedule: 19 2/12 * * * template: metadata: {} spec: source: registry: url: docker://quay.io/containerdisks/fedora:35 storage: resources: requests: storage: 5Gi status: {} $ oc get dic rhel8-image-cron -o yaml | grep -i pull pullMethod: node $ oc get dic rhel9-image-cron -o yaml | grep -i pull pullMethod: node
Moving bug to verified. pullMethod: node is currently needed only when the source url/imagestream points RH registry https://github.com/kubevirt/containerized-data-importer/blob/main/doc/image-from-registry.md#import-registry-image-into[…]me-using-node-docker-cache
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947