Bug 2226764

Summary: Deletion of DataImportCron PVCs does not result in re-creation of PVCs
Product: Container Native Virtualization (CNV) Reporter: Sarah Bennert <sbennert>
Component: StorageAssignee: Alex Kalenyuk <akalenyu>
Status: ASSIGNED --- QA Contact: Natalie Gavrielov <ngavrilo>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.14.0CC: agilboa, akalenyu, alitke, jpeimer
Target Milestone: ---Keywords: Reopened
Target Release: 4.14.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-27 15:50:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sarah Bennert 2023-07-26 12:44:25 UTC
Description of problem:

Deletion of a Golden DataImportCron PVC does not result in re-creation of PVC


Version-Release number of selected component (if applicable):
4.14.0

How reproducible:
100%

Steps to Reproduce:
1. Delete DataImportCron PVC

Actual results:
DV shows PVC import progressing
After end of import DV and PVC are deleted and never return.

Expected results:
PVC is re-created and remains.

Additional info:
Result is the same if DV is deleted instead of PVC.

Attempts to recreate DV ends with same result.

Reason for deleting the PVC was to have it repopulate on default StorageClass after having been changed.

Comment 1 Arnon Gilboa 2023-07-26 14:24:45 UTC
This is actually a feature and not a bug, but may cause some confusion and failures in tier-2 tests.
The cluster was initially created with default storage class with no snapshot support (hpp), so the DataImportCrons created DVs/PVCs as they used to do in <=4.13.
However, later the default storage class was updated to one with snapshot support (ceph), so when the DV/PVC were manually deleted, they were recreated, but upon import completion they were replaced by a snapshot and deleted.

@akalenyu can we close this issue as notabug? is this new behavior documented U/S & D/S?

Comment 2 Arnon Gilboa 2023-07-26 15:09:12 UTC
@sbennert can you confirm the behavior described in comment 1?

Checking it with @jpeimer we saw this expected behavior:

$ oc get DataImportCron -n openshift-virtualization-os-images rhel9-image-cron -oyaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataImportCron
metadata:
...
  name: rhel9-image-cron
  namespace: openshift-virtualization-os-images
...
status:
  conditions:
  - lastHeartbeatTime: "2023-07-26T12:48:39Z"
    lastTransitionTime: "2023-07-26T12:48:39Z"
    message: No current import
    reason: NoImport
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2023-07-26T12:48:41Z"
    lastTransitionTime: "2023-07-26T12:48:41Z"
    message: Latest import is up to date
    reason: UpToDate
    status: "True"
    type: UpToDate
  currentImports:
  - DataVolumeName: rhel9-b006ef7856b6
    Digest: sha256:b006ef7856b6807bc0448cf5bdf6b803bd7f012a24854d4eb8225bb8463a5493
  lastExecutionTimestamp: "2023-07-26T04:42:19Z"
  lastImportTimestamp: "2023-07-20T13:00:10Z"
  lastImportedPVC:
    name: rhel9-b006ef7856b6
    namespace: openshift-virtualization-os-images

$ oc get dataSource rhel9 -n openshift-virtualization-os-images -oyaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataSource
metadata:
...
  name: rhel9
  namespace: openshift-virtualization-os-images
...
spec:
  source:
    snapshot:
      name: rhel9-b006ef7856b6
      namespace: openshift-virtualization-os-images
status:
  conditions:
  - lastHeartbeatTime: "2023-07-26T12:48:41Z"
    lastTransitionTime: "2023-07-26T12:48:41Z"
    message: DataSource is ready to be consumed
    reason: Ready
    status: "True"
    type: Ready
  source:
    snapshot:
      name: rhel9-b006ef7856b6
      namespace: openshift-virtualization-os-images

Comment 3 Alex Kalenyuk 2023-07-26 17:36:32 UTC
(In reply to Arnon Gilboa from comment #1)
> This is actually a feature and not a bug, but may cause some confusion and
> failures in tier-2 tests.
> The cluster was initially created with default storage class with no
> snapshot support (hpp), so the DataImportCrons created DVs/PVCs as they used
> to do in <=4.13.
> However, later the default storage class was updated to one with snapshot
> support (ceph), so when the DV/PVC were manually deleted, they were
> recreated, but upon import completion they were replaced by a snapshot and
> deleted.
> 
> @akalenyu can we close this issue as notabug? is this new
> behavior documented U/S & D/S?

Yup sounds about right
If this is blocking automated testing,
consider just making sure the DataSource object points at the expected name and has the condition Ready=true
(we have other tests that ensure this is an honest indication).
Otherwise, you can also just observe which type of format (PVC/Snapshot)
is used and GET that
(spec:
  source:
    snapshot/pvc:)

Comment 4 Sarah Bennert 2023-07-27 15:50:46 UTC
Thank you for the clarification!
I confirmed the creation of the snapshot after import, and the datasource pointing to the snapshot.
Reverting the StorageClass change and deleting the snapshot re-created the PVC.

That's what I get for being away for a while. : )

Comment 5 Jenia Peimer 2023-08-02 11:21:16 UTC
Actually, Sarah, you have a good point - it's confusing that DV/PVC disappeared.
We talked with the Storage team, and we should communicate it better - like adding a Condition to the DataImportCron object. 
I will re-open this bug to track the progress

Comment 6 Adam Litke 2023-08-04 18:38:58 UTC
Reassigning to Alex as the feature owner.  Alex, let's see what we can do to make this more visible.