Bug 2226764

Summary: Deletion of DataImportCron PVCs does not result in re-creation of PVCs
Product: Container Native Virtualization (CNV) Reporter: Sarah Bennert <sbennert>
Component: StorageAssignee: Alex Kalenyuk <akalenyu>
Status: CLOSED ERRATA QA Contact: Harel Meir <hmeir>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.14.0CC: agilboa, akalenyu, alitke, dafrank, jpeimer
Target Milestone: ---Keywords: Reopened
Target Release: 4.14.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: CNV v4.14.0.rhel9-1854 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-08 14:06:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sarah Bennert 2023-07-26 12:44:25 UTC
Description of problem:

Deletion of a Golden DataImportCron PVC does not result in re-creation of PVC


Version-Release number of selected component (if applicable):
4.14.0

How reproducible:
100%

Steps to Reproduce:
1. Delete DataImportCron PVC

Actual results:
DV shows PVC import progressing
After end of import DV and PVC are deleted and never return.

Expected results:
PVC is re-created and remains.

Additional info:
Result is the same if DV is deleted instead of PVC.

Attempts to recreate DV ends with same result.

Reason for deleting the PVC was to have it repopulate on default StorageClass after having been changed.

Comment 1 Arnon Gilboa 2023-07-26 14:24:45 UTC
This is actually a feature and not a bug, but may cause some confusion and failures in tier-2 tests.
The cluster was initially created with default storage class with no snapshot support (hpp), so the DataImportCrons created DVs/PVCs as they used to do in <=4.13.
However, later the default storage class was updated to one with snapshot support (ceph), so when the DV/PVC were manually deleted, they were recreated, but upon import completion they were replaced by a snapshot and deleted.

@akalenyu can we close this issue as notabug? is this new behavior documented U/S & D/S?

Comment 2 Arnon Gilboa 2023-07-26 15:09:12 UTC
@sbennert can you confirm the behavior described in comment 1?

Checking it with @jpeimer we saw this expected behavior:

$ oc get DataImportCron -n openshift-virtualization-os-images rhel9-image-cron -oyaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataImportCron
metadata:
...
  name: rhel9-image-cron
  namespace: openshift-virtualization-os-images
...
status:
  conditions:
  - lastHeartbeatTime: "2023-07-26T12:48:39Z"
    lastTransitionTime: "2023-07-26T12:48:39Z"
    message: No current import
    reason: NoImport
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2023-07-26T12:48:41Z"
    lastTransitionTime: "2023-07-26T12:48:41Z"
    message: Latest import is up to date
    reason: UpToDate
    status: "True"
    type: UpToDate
  currentImports:
  - DataVolumeName: rhel9-b006ef7856b6
    Digest: sha256:b006ef7856b6807bc0448cf5bdf6b803bd7f012a24854d4eb8225bb8463a5493
  lastExecutionTimestamp: "2023-07-26T04:42:19Z"
  lastImportTimestamp: "2023-07-20T13:00:10Z"
  lastImportedPVC:
    name: rhel9-b006ef7856b6
    namespace: openshift-virtualization-os-images

$ oc get dataSource rhel9 -n openshift-virtualization-os-images -oyaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataSource
metadata:
...
  name: rhel9
  namespace: openshift-virtualization-os-images
...
spec:
  source:
    snapshot:
      name: rhel9-b006ef7856b6
      namespace: openshift-virtualization-os-images
status:
  conditions:
  - lastHeartbeatTime: "2023-07-26T12:48:41Z"
    lastTransitionTime: "2023-07-26T12:48:41Z"
    message: DataSource is ready to be consumed
    reason: Ready
    status: "True"
    type: Ready
  source:
    snapshot:
      name: rhel9-b006ef7856b6
      namespace: openshift-virtualization-os-images

Comment 3 Alex Kalenyuk 2023-07-26 17:36:32 UTC
(In reply to Arnon Gilboa from comment #1)
> This is actually a feature and not a bug, but may cause some confusion and
> failures in tier-2 tests.
> The cluster was initially created with default storage class with no
> snapshot support (hpp), so the DataImportCrons created DVs/PVCs as they used
> to do in <=4.13.
> However, later the default storage class was updated to one with snapshot
> support (ceph), so when the DV/PVC were manually deleted, they were
> recreated, but upon import completion they were replaced by a snapshot and
> deleted.
> 
> @akalenyu can we close this issue as notabug? is this new
> behavior documented U/S & D/S?

Yup sounds about right
If this is blocking automated testing,
consider just making sure the DataSource object points at the expected name and has the condition Ready=true
(we have other tests that ensure this is an honest indication).
Otherwise, you can also just observe which type of format (PVC/Snapshot)
is used and GET that
(spec:
  source:
    snapshot/pvc:)

Comment 4 Sarah Bennert 2023-07-27 15:50:46 UTC
Thank you for the clarification!
I confirmed the creation of the snapshot after import, and the datasource pointing to the snapshot.
Reverting the StorageClass change and deleting the snapshot re-created the PVC.

That's what I get for being away for a while. : )

Comment 5 Jenia Peimer 2023-08-02 11:21:16 UTC
Actually, Sarah, you have a good point - it's confusing that DV/PVC disappeared.
We talked with the Storage team, and we should communicate it better - like adding a Condition to the DataImportCron object. 
I will re-open this bug to track the progress

Comment 6 Adam Litke 2023-08-04 18:38:58 UTC
Reassigning to Alex as the feature owner.  Alex, let's see what we can do to make this more visible.

Comment 7 dalia 2023-09-13 12:39:03 UTC
Alex, pushing for 4.14.1, do you have plan to address it

Comment 8 Alex Kalenyuk 2023-09-13 12:52:09 UTC
(In reply to dalia from comment #7)
> Alex, pushing for 4.14.1, do you have plan to address it

Handled in https://bugzilla.redhat.com/show_bug.cgi?id=2227066
@jpeimer cc

Comment 9 Alex Kalenyuk 2023-09-13 14:07:53 UTC
$ oc get dataimportcron -A
NAMESPACE                            NAME                        FORMAT
openshift-virtualization-os-images   centos-7-image-cron         pvc

Comment 10 Harel Meir 2023-09-18 10:07:29 UTC
Verified

Steps:

1. Deleted the PVC
[cloud-user@ocp-psi-executor-xl ~]$ oc delete pvc rhel9-a1947a1edca5 $NS
persistentvolumeclaim "rhel9-a1947a1edca5" deleted

2. the DV is get re-imported:
rhel9-a1947a1edca5            ImportInProgress   97.11%                9m27s

3. After importe succeeded, the dv remain and the pvc gets created:
[cloud-user@ocp-psi-executor-xl ~]$ oc get dv $NS  rhel9-a1947a1edca5
NAME                 PHASE       PROGRESS   RESTARTS   AGE
rhel9-a1947a1edca5   Succeeded   100.0%                10m
[cloud-user@ocp-psi-executor-xl ~]$ oc get pvc $NS  rhel9-a1947a1edca5
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
rhel9-a1947a1edca5   Bound    pvc-f9aa6714-03f0-4a2c-a236-0492ed353c61   149Gi      RWO            hostpath-csi-basic   85s

NOTE that the dic has a format condition of pvc:
[cloud-user@ocp-psi-executor-xl ~]$ oc get dic $NS rhel9-image-cron
NAME               FORMAT
rhel9-image-cron   pvc

Comment 12 errata-xmlrpc 2023-11-08 14:06:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Virtualization 4.14.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6817