Bug 2037270
| Summary: | Auto-update boot sources: CentOs and Fedora DVs fail to import due to docker references | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Ruth Netser <rnetser> |
| Component: | Storage | Assignee: | Arnon Gilboa <agilboa> |
| Status: | CLOSED ERRATA | QA Contact: | Yan Du <yadu> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.10.0 | CC: | agilboa, cnv-qe-bugs, dholler, fdeutsch, lbednar, stirabos, yadu, ycui |
| Target Milestone: | --- | Keywords: | AutomationBlocker |
| Target Release: | 4.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | CNV v4.10.0-605 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-16 16:05:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2035008 | ||
Just for clarification, DataImportCron does not support source URL with tag, and it was never a requirement. We use digests for the imports, and we poll the latest to see if updated. I see no reason why one would use DataImportCron with a tagged url. In the case of a static source why not simply use DataSource? I guess SSP should simply remove the tag. *** Bug 2036817 has been marked as a duplicate of this bug. *** I don't see the symptoms of this bug in hco-v4.10.0-600 nor in hco-v4.10.0-601 All DVs imported, and no crashing pods. NAMESPACE NAME PHASE PROGRESS RESTARTS AGE openshift-virtualization-os-images centos-stream8-f09342b27c35 Succeeded 100.0% 28m openshift-virtualization-os-images centos-stream9-530e9c62633a Succeeded 100.0% 28m openshift-virtualization-os-images fedora-3b3fc310abea Succeeded 100.0% 28m openshift-virtualization-os-images rhel8-2325b0922721 Succeeded 100.0% 28m openshift-virtualization-os-images rhel9-401cc6250f89 Succeeded 100.0% 28m btw, it looks like there is missing field in the defined DataImportCron: spec.template.spec.source.registry.pullMethod: node which is used in the created import DVs: spec.source.registry.pullMethod: node without this one, it can work on external registry but won't work for import from RH registry. Arnon, Yes, the fedora-image-cron, centos-stream9-image-cron and centos-stream8-image-cron are missing pullMethod:node,
but rhel8-image-cron and rhel9-image-cron works well
spec:
garbageCollect: Outdated
managedDataSource: fedora
schedule: 19 2/12 * * *
template:
metadata: {}
spec:
source:
registry:
url: docker://quay.io/containerdisks/fedora:35
storage:
resources:
requests:
storage: 5Gi
status: {}
$ oc get dic rhel8-image-cron -o yaml | grep -i pull
pullMethod: node
$ oc get dic rhel9-image-cron -o yaml | grep -i pull
pullMethod: node
Moving bug to verified. pullMethod: node is currently needed only when the source url/imagestream points RH registry https://github.com/kubevirt/containerized-data-importer/blob/main/doc/image-from-registry.md#import-registry-image-into[…]me-using-node-docker-cache Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947 |
Description of problem: When auto-update boot sources feature is enabled, CentOS and Fedora DVs fail to import because: Unable to process data: Docker references with both a tag and digest are currently not supported github.com/containers/image/v5/docker.newReference Version-Release number of selected component (if applicable): HCO v4.10.0-105 How reproducible: 100% Steps to Reproduce: 1. Enable enableCommonBootImageImport in HCO 2. Monitor DVs under openshift-virtualization-os-images ns Actual results: CentOS8 and Fedora DVs are in ImportInProgress state Expected results: All imports should succeed. Additional info: =========================== $ oc get dv -A NAMESPACE NAME PHASE PROGRESS RESTARTS AGE openshift-virtualization-os-images centos8-4a6a0eee7b4a ImportInProgress N/A 51 3h59m openshift-virtualization-os-images fedora-3b3fc310abea ImportInProgress N/A 51 3h59m openshift-virtualization-os-images rhel8-2325b0922721 Succeeded 100.0% 4h openshift-virtualization-os-images rhel9-401cc6250f89 Succeeded 100.0% 4h =========================== $ oc describe dv -n openshift-virtualization-os-images fedora-3b3fc310abea Name: fedora-3b3fc310abea Namespace: openshift-virtualization-os-images Labels: app.kubernetes.io/component=storage app.kubernetes.io/managed-by=cdi-controller app.kubernetes.io/part-of=hyperconverged-cluster app.kubernetes.io/version=4.10.0 cdi.kubevirt.io/dataImportCron=fedora-image-cron Annotations: cdi.kubevirt.io/storage.bind.immediate.requested: true API Version: cdi.kubevirt.io/v1beta1 Kind: DataVolume Metadata: Creation Timestamp: 2022-01-05T06:52:52Z Generation: 129 Managed Fields: API Version: cdi.kubevirt.io/v1beta1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:cdi.kubevirt.io/storage.bind.immediate.requested: f:labels: .: f:app.kubernetes.io/component: f:app.kubernetes.io/managed-by: f:app.kubernetes.io/part-of: f:app.kubernetes.io/version: f:cdi.kubevirt.io/dataImportCron: f:spec: .: f:source: .: f:registry: .: f:url: f:storage: .: f:resources: .: f:requests: .: f:storage: f:status: .: f:conditions: f:phase: f:progress: f:restartCount: Manager: virt-cdi-controller Operation: Update Time: 2022-01-05T06:53:08Z Resource Version: 1009392 UID: d12ae27c-6450-45aa-ab01-df166c67b6e5 Spec: Source: Registry: URL: docker://quay.io/containerdisks/fedora:35@sha256:3b3fc310abeadd38655f2886feffcab5a37c9c3272d95b3147090c3710d8dd72 Storage: Resources: Requests: Storage: 5Gi Status: Conditions: Last Heartbeat Time: 2022-01-05T06:52:57Z Last Transition Time: 2022-01-05T06:52:57Z Message: PVC fedora-3b3fc310abea Bound Reason: Bound Status: True Type: Bound Last Heartbeat Time: 2022-01-05T10:50:32Z Last Transition Time: 2022-01-05T06:52:53Z Status: False Type: Ready Last Heartbeat Time: 2022-01-05T10:50:32Z Last Transition Time: 2022-01-05T10:50:32Z Message: Unable to process data: Docker references with both a tag and digest are currently not supported github.com/containers/image/v5/docker.newReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:80 github.com/containers/image/v5/docker.NewReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:64 github.com/containers/image/v5/docker.ParseReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:59 kubevirt.io/containerized-data-importer/pkg/importer.parseImageName /remote-source/app/pkg/importer/transport.go:91 kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:69 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Could not parse image kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:72 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Failed to read registry image kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:85 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Unable to transfer source data to scratch space kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:202 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Reason: Error Status: False Type: Running Phase: ImportInProgress Progress: N/A Restart Count: 51 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Error 3m25s (x49 over 4h) datavolume-controller Unable to process data: Docker references with both a tag and digest are currently not supported github.com/containers/image/v5/docker.newReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:80 github.com/containers/image/v5/docker.NewReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:64 github.com/containers/image/v5/docker.ParseReference /remote-source/app/vendor/github.com/containers/image/v5/docker/docker_transport.go:59 kubevirt.io/containerized-data-importer/pkg/importer.parseImageName /remote-source/app/pkg/importer/transport.go:91 kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:69 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Could not parse image kubevirt.io/containerized-data-importer/pkg/importer.readImageSource /remote-source/app/pkg/importer/transport.go:72 kubevirt.io/containerized-data-importer/pkg/importer.copyRegistryImage /remote-source/app/pkg/importer/transport.go:181 kubevirt.io/containerized-data-importer/pkg/importer.CopyRegistryImage /remote-source/app/pkg/importer/transport.go:261 kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:83 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Failed to read registry image kubevirt.io/containerized-data-importer/pkg/importer.(*RegistryDataSource).Transfer /remote-source/app/pkg/importer/registry-datasource.go:85 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:197 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371 Unable to transfer source data to scratch space kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause /remote-source/app/pkg/importer/data-processor.go:202 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /remote-source/app/pkg/importer/data-processor.go:172 main.main /remote-source/app/cmd/cdi-importer/importer.go:138 runtime.main /usr/lib/golang/src/runtime/proc.go:225 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1371