Bug 1753654 - clone fails when using filesystem volumeMode on ceph rbd backend
Summary: clone fails when using filesystem volumeMode on ceph rbd backend
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 2.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.1.0
Assignee: Alexander Wels
QA Contact: Kevin Alon Goldblatt
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-19 14:17 UTC by Daniel Erez
Modified: 2019-10-31 14:08 UTC (History)
8 users (show)

Fixed In Version: hco-bundle-registry-container-v2.1.0-69 -- cdi-2.1.0-17
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-31 14:08:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:3282 0 None None None 2019-10-31 14:08:48 UTC

Description Daniel Erez 2019-09-19 14:17:40 UTC
Description of problem:
The cloner tests are failing due to insufficient space in the target, see [logs] of 
"Tests Suite.[rfe_id:1277][crit:high][vendor:cnv-qe][level:component]Cloner Test Suite [test_id:1354]Should clone data within same namespace"

Version-Release number of selected component (if applicable):
2.1.0

How reproducible:
100%

Steps to Reproduce:
1. Create source DataVolume - see [source dv yaml]
2. Create target DataVolume - see [target dv yaml]

Actual results:
Fails to clone.

Expected results:
Should succeed.

Additional info:
[logs]
source:
09:41:38 Logs for container cdi-clone-source
09:41:38 VOLUME_MODE=filesystem
09:41:38 MOUNT_POINT=/var/run/cdi/clone/source
09:41:38 /var/run/cdi/clone/source /
09:41:38 UPLOAD_BYTES=495091734
09:41:38 ./
09:41:38 ./disk.img
09:41:38 I0919 06:41:02.647733      11 clone-source.go:106] content_type is "filesystem-clone"
09:41:38 I0919 06:41:02.648420      11 clone-source.go:107] upload_bytes is 495091734
09:41:38 I0919 06:41:02.648435      11 clone-source.go:117] Starting cloner target
09:41:38 I0919 06:41:03.346326      11 clone-source.go:129] Set header to filesystem-clone
09:41:38 I0919 06:41:03.649783      11 prometheus.go:67] 0.57
09:41:38 I0919 06:41:04.649937      11 prometheus.go:67] 3.40
09:41:38 I0919 06:41:05.650132      11 prometheus.go:67] 30.27
09:41:38 I0919 06:41:06.650325      11 prometheus.go:67] 58.49
09:41:38 I0919 06:41:07.650629      11 prometheus.go:67] 87.04
09:41:38 I0919 06:41:08.083239      11 clone-source.go:96] Wrote 495093760 bytes
09:41:38 I0919 06:41:08.653213      11 prometheus.go:67] 100.00
09:41:38 F0919 06:41:19.711272      11 clone-source.go:138] Unexpected status code 500
09:41:38 goroutine 1 [running]:
09:41:38 kubevirt.io/containerized-data-importer/vendor/k8s.io/klog.stacks(0xc0002bbd00, 0xc0002ec000, 0x4e, 0xa3)
09:41:38 	/go/src/kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/klog.go:828 +0xd4
09:41:38 kubevirt.io/containerized-data-importer/vendor/k8s.io/klog.(*loggingT).output(0x1a999e0, 0xc000000003, 0xc00018c2c0, 0x1a327ef, 0xf, 0x8a, 0x0)
09:41:38 	/go/src/kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/klog.go:779 +0x306
09:41:38 kubevirt.io/containerized-data-importer/vendor/k8s.io/klog.(*loggingT).printf(0x1a999e0, 0xc000000003, 0x1052f41, 0x19, 0xc000521f08, 0x1, 0x1)
09:41:38 	/go/src/kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/klog.go:678 +0x14b
09:41:38 kubevirt.io/containerized-data-importer/vendor/k8s.io/klog.Fatalf(0x1052f41, 0x19, 0xc000521f08, 0x1, 0x1)
09:41:38 	/go/src/kubevirt.io/containerized-data-importer/vendor/k8s.io/klog/klog.go:1207 +0x67
09:41:38 main.main()
09:41:38 	/go/src/kubevirt.io/containerized-data-importer/cmd/cdi-cloner/clone-source.go:138 +0x88b
09:41:38 

target:
09:47:37 Logs for container cdi-upload-server
09:47:37 I0919 06:46:04.256760       1 uploadserver.go:61] Upload destination: /data/disk.img
09:47:37 I0919 06:46:04.257267       1 uploadserver.go:63] Running server on 0.0.0.0:8443
09:47:37 I0919 06:46:45.648267       1 uploadserver.go:244] Content type header is "filesystem-clone"
09:47:37 I0919 06:46:45.648845       1 util.go:156] begin untar to /data...
09:47:37 I0919 06:47:05.118230       1 util.go:175] /usr/bin/tar: ./disk.img: Wrote only 1024 of 10240 bytes
09:47:37 /usr/bin/tar: Exiting with failure status due to previous errors
09:47:37 
09:47:37 E0919 06:47:05.118361       1 util.go:176] exit status 2
09:47:37 E0919 06:47:05.119188       1 uploadserver.go:252] Saving stream failed: error unarchiving to /data: exit status 2
09:47:37 I0919 06:47:07.029362       1 uploadserver.go:244] Content type header is "filesystem-clone"
09:47:37 I0919 06:47:07.029530       1 util.go:37] deleting file: /data/disk.img
09:47:37 I0919 06:47:07.073778       1 util.go:156] begin untar to /data...


[source dv yaml]
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
  name: source-dv
spec:
  pvc:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 500Mi
  source:
    http:
      url: http://cdi-file-host.openshift-cnv/tinyCore.iso


[target dv yaml]
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
  annotations:
  name: target-dv
spec:
  pvc:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 500Mi
  source:
    pvc:
      name: source-dv
      namespace: default

Comment 1 Fred Rolland 2019-09-19 14:46:25 UTC
It looks like a file system overhead . (XFS)

As a workaround, by increasing the size of the target PVC by 1Mi the clone will succeed.

Comment 3 Adam Litke 2019-09-24 11:31:43 UTC
@Alexander, check v1.10.6.  It seems you forgot to backport https://github.com/kubevirt/containerized-data-importer/pull/966 to release-v1.10.  Can you check it please?

Comment 4 Alexander Wels 2019-09-24 11:32:56 UTC
We release v1.10.6 before 996 was merged, next release will contain the PR.

Comment 9 Kevin Alon Goldblatt 2019-10-02 14:00:26 UTC
Verified with the following code:
----------------------------------------
v2.1.0-latest
virt-cdi-apiserver:v2.1.0-20

Verified with the following scenario:
----------------------------------------
1. Create dv on sc rook-ceph-block(file)
2. Create clone of dv >>>>> created successfully


dv yaml;
-----------------
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
  name: source-dv
spec:
  pvc:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 500Mi
  source:
    http:
      url: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img"


apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
  annotations:
  name: target-dv
spec:
  pvc:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 500Mi
  source:
    pvc:
      name: source-dv
      namespace: default


Moving to VERIFIED!

Comment 11 errata-xmlrpc 2019-10-31 14:08:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3282


Note You need to log in before you can comment on or make changes to this bug.