Bug 2151817

Summary:	[CNV-4.11.2] Failing to import DataVolumes
Product:	Container Native Virtualization (CNV)	Reporter:	Mor Cohen <mocohen>
Component:	Storage	Assignee:	Maya Rashish <mrashish>
Status:	CLOSED ERRATA	QA Contact:	Natalie Gavrielov <ngavrilo>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	4.11.2	CC:	jpeimer, mrashish, yadu
Target Milestone:	---	Keywords:	AutomationBlocker, TestBlocker
Target Release:	4.11.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	4.11.2-6	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-01-12 14:08:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Mor Cohen 2022-12-08 08:12:56 UTC

Description of problem:
After deploying CNV-4.11.2 DataVolumes are failing to get imported, stuck on the ImportInProgress phase. 



Version-Release number of selected component (if applicable):
CNV-4.11.2-4

How reproducible:
100%

Steps to Reproduce:
1. Deploy CNV from the 388386 IIB.
2.
3.

Actual results:


Expected results:


Additional info:
[cnv-qe-jenkins@c01-vrf-t1-411-xd655-executor ~]$ oc get dv -A
NAMESPACE                            NAME                          PHASE              PROGRESS   RESTARTS   AGE
openshift-virtualization-os-images   centos-stream8-2f16c067b974   ImportInProgress   N/A        159        14h
openshift-virtualization-os-images   centos-stream9-b603efbe0295   ImportInProgress   N/A        160        14h
openshift-virtualization-os-images   centos7-680e9b4e0fba          ImportInProgress   N/A        159        14h
openshift-virtualization-os-images   fedora-56ccabc01cbe           ImportInProgress   N/A        161        14h
openshift-virtualization-os-images   rhel8-13b024fed1a3            ImportInProgress   N/A        161        14h
openshift-virtualization-os-images   rhel9-35d9b2336799            ImportInProgress   N/A        159        14h

[cnv-qe-jenkins@c01-vrf-t1-411-xd655-executor ~]$ oc get pods -A | grep -v Running | grep -v Completed
NAMESPACE                                          NAME                                                              READY   STATUS             RESTARTS          AGE
openshift-virtualization-os-images                 importer-centos-stream8-2f16c067b974                              1/2     CrashLoopBackOff   160 (2m1s ago)    14h
openshift-virtualization-os-images                 importer-centos-stream9-b603efbe0295                              1/2     CrashLoopBackOff   161 (117s ago)    14h
openshift-virtualization-os-images                 importer-fedora-56ccabc01cbe                                      1/2     CrashLoopBackOff   162 (2m52s ago)   14h
openshift-virtualization-os-images                 importer-rhel8-13b024fed1a3                                       1/2     CrashLoopBackOff   162 (3m31s ago)   14h
openshift-virtualization-os-images                 importer-rhel9-35d9b2336799                                       1/2     CrashLoopBackOff   160 (2m45s ago)   14h


$ oc logs -n openshift-virtualization-os-images importer-centos-stream8-2f16c067b974
Defaulted container "importer" out of: importer, server, init (init)
I1208 07:58:43.215458       1 importer.go:83] Starting importer
I1208 07:58:43.215506       1 importer.go:150] begin import process
I1208 07:58:43.219728       1 data-processor.go:379] Calculating available size
I1208 07:58:43.219747       1 data-processor.go:391] Checking out file system volume size.
I1208 07:58:43.219763       1 data-processor.go:399] Request image size not empty.
I1208 07:58:43.219770       1 data-processor.go:404] Target size 34087042032.
I1208 07:58:43.220339       1 nbdkit.go:296] Waiting for nbdkit PID.
I1208 07:58:43.400808       1 nbdkit.go:283] Log line from nbdkit: /var/run/nbdkit.sock: Permission denied
I1208 07:58:43.401560       1 nbdkit.go:289] Stopped watching nbdkit log.
E1208 07:58:58.221381       1 nbdkit.go:264] Failed waiting for nbdkit to start up: timed out waiting for nbdkit to be ready
E1208 07:58:58.221516       1 data-processor.go:278] timed out waiting for nbdkit to be ready
kubevirt.io/containerized-data-importer/pkg/image.waitForNbd
	/remote-source/app/pkg/image/nbdkit.go:321
kubevirt.io/containerized-data-importer/pkg/image.(*Nbdkit).StartNbdkit
	/remote-source/app/pkg/image/nbdkit.go:262
kubevirt.io/containerized-data-importer/pkg/importer.(*HTTPDataSource).Info
	/remote-source/app/pkg/importer/http-datasource.go:147
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).initDefaultPhases.func1
	/remote-source/app/pkg/importer/data-processor.go:201
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessDataWithPause
	/remote-source/app/pkg/importer/data-processor.go:275
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	/remote-source/app/pkg/importer/data-processor.go:184
main.handleImport
	/remote-source/app/cmd/cdi-importer/importer.go:156
main.main
	/remote-source/app/cmd/cdi-importer/importer.go:122
runtime.main
	/usr/lib/golang/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1594
Unable to obtain information about data source

Comment 1 Maya Rashish 2022-12-08 09:07:59 UTC

Sorry, this is likely due to a partial backport (of PSA fixes) that we also decided we don't want in 4.11.
I didn't realize it is also broken, I am reverting it.

Comment 2 Maya Rashish 2022-12-08 11:46:29 UTC

Let me know if it was this change that broke things, and the revert is sufficient.

Comment 3 Mor Cohen 2022-12-08 11:49:18 UTC

Hey Maya, the revert is sufficient, thanks!

[cnv-qe-jenkins@c01-vrf-t1-411-xd655-executor ~]$ oc get dv -A
NAMESPACE                            NAME                          PHASE       PROGRESS   RESTARTS   AGE
openshift-virtualization-os-images   centos-stream8-2f16c067b974   Succeeded   100.0%                6m12s
openshift-virtualization-os-images   centos-stream9-250b87464d0d   Succeeded   100.0%                6m13s
openshift-virtualization-os-images   centos7-680e9b4e0fba          Succeeded   100.0%                6m12s
openshift-virtualization-os-images   fedora-0e5ac38b20ab           Succeeded   100.0%                3m24s
openshift-virtualization-os-images   fedora-56ccabc01cbe           Succeeded   100.0%                6m13s


Thanks!

Comment 11 errata-xmlrpc 2023-01-12 14:08:54 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.11.2 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:0155