Bug 1889425

Summary: After a while VirtualMachineSnapshot shows an error message: VolumeSnapshots (snapshot-id) missing
Product: Container Native Virtualization (CNV) Reporter: Natalie Gavrielov <ngavrilo>
Component: StorageAssignee: Michael Henriksen <mhenriks>
Status: CLOSED ERRATA QA Contact: Natalie Gavrielov <ngavrilo>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.5.0CC: cnv-qe-bugs, mhenriks, mrashish, sgott, ycui
Target Milestone: ---   
Target Release: 2.5.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: virt-controller v2.5.0-76 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-18 20:58:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Natalie Gavrielov 2020-10-19 16:05:59 UTC
Description of problem:
A few days after creating a VirtualMachineSnapshot, it shows an error saying VolumeSnapshots missing
(When in fact it exists)

Version-Release number of selected component:
CNV 2.5

How reproducible:
100%

Steps to Reproduce:
1. Create a vm.
2. Create a vm snapshot, example:
apiVersion: snapshot.kubevirt.io/v1alpha1
kind: VirtualMachineSnapshot
metadata:
  name: vm-cirros-snapshot-for-new-vm
spec:
  source:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: vm-cirros-2
3. Wait for 2-3 days

Actual results:
After a few days (2-3), the following is reported for each vm snapshot:
$ oc get vmsnapshots -A
NAMESPACE   NAME                            SOURCEKIND       SOURCENAME    READYTOUSE   CREATIONTIME   ERROR
default     vm-cirros-snapshot              VirtualMachine   vm-cirros     true         3h10m          VolumeSnapshots (vmsnapshot-534c668b-8170-4b5f-88cf-1258628b275a-volume-cirros-dv-new) missing

Expected results:
No errors

Additional info:
Most important - doesn't seem to impact fuctionality.

$ oc get VolumeSnapshots vmsnapshot-534c668b-8170-4b5f-88cf-1258628b275a-volume-cirros-dv-new
NAME                                                                   READYTOUSE   SOURCEPVC       SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
vmsnapshot-534c668b-8170-4b5f-88cf-1258628b275a-volume-cirros-dv-new   true         cirros-dv-new                           2Gi           ocs-storagecluster-rbdplugin-snapclass   snapcontent-03a4ad1c-a339-4b27-9af5-1fb2c896bc6d   13d            13d


$ oc describe vmsnapshots vm-cirros-snapshot
Name:         vm-cirros-snapshot
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.kubevirt.io/v1alpha1
Kind:         VirtualMachineSnapshot
Metadata:
  Creation Timestamp:  2020-10-05T19:52:43Z
  Finalizers:
    snapshot.kubevirt.io/vmsnapshot-protection
  Generation:  21
  Managed Fields:
    API Version:  snapshot.kubevirt.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:source:
          .:
          f:apiGroup:
          f:kind:
          f:name:
    Manager:      kubectl-create
    Operation:    Update
    Time:         2020-10-05T19:52:43Z
    API Version:  snapshot.kubevirt.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:status:
        .:
        f:conditions:
        f:creationTime:
        f:error:
          .:
          f:message:
          f:time:
        f:readyToUse:
        f:sourceUID:
        f:virtualMachineSnapshotContentName:
    Manager:         virt-controller
    Operation:       Update
    Time:            2020-10-19T12:34:46Z
  Resource Version:  39317413
  Self Link:         /apis/snapshot.kubevirt.io/v1alpha1/namespaces/default/virtualmachinesnapshots/vm-cirros-snapshot
  UID:               534c668b-8170-4b5f-88cf-1258628b275a
Spec:
  Source:
    API Group:  kubevirt.io
    Kind:       VirtualMachine
    Name:       vm-cirros
Status:
  Conditions:
    Last Probe Time:       <nil>
    Last Transition Time:  2020-10-05T19:52:56Z
    Reason:                Operation complete
    Status:                False
    Type:                  Progressing
    Last Probe Time:       <nil>
    Last Transition Time:  2020-10-05T23:58:23Z
    Reason:                Error
    Status:                False
    Type:                  Ready
  Creation Time:           2020-10-19T12:34:44Z
  Error:
    Message:                              VolumeSnapshots (vmsnapshot-534c668b-8170-4b5f-88cf-1258628b275a-volume-cirros-dv-new) missing
    Time:                                 2020-10-05T23:58:23Z
  Ready To Use:                           true
  Source UID:                             91817c63-6c60-49b5-a111-d84d585b5b6f
  Virtual Machine Snapshot Content Name:  vmsnapshot-content-534c668b-8170-4b5f-88cf-1258628b275a
Events:                                   <none>

$ oc describe VolumeSnapshots vmsnapshot-534c668b-8170-4b5f-88cf-1258628b275a-volume-cirros-dv-new
Name:         vmsnapshot-534c668b-8170-4b5f-88cf-1258628b275a-volume-cirros-dv-new
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1beta1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2020-10-05T19:52:53Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:  1
  Managed Fields:
    API Version:  snapshot.storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
      f:spec:
        .:
        f:source:
          .:
          f:persistentVolumeClaimName:
        f:volumeSnapshotClassName:
    Manager:      virt-controller
    Operation:    Update
    Time:         2020-10-05T19:52:53Z
    API Version:  snapshot.storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:status:
        .:
        f:boundVolumeSnapshotContentName:
        f:creationTime:
        f:readyToUse:
        f:restoreSize:
    Manager:    snapshot-controller
    Operation:  Update
    Time:       2020-10-05T19:52:56Z
  Owner References:
    API Version:           snapshot.kubevirt.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  VirtualMachineSnapshotContent
    Name:                  vmsnapshot-content-534c668b-8170-4b5f-88cf-1258628b275a
    UID:                   b51060b1-0c97-4bd0-902b-fd086768d9cf
  Resource Version:        344880
  Self Link:               /apis/snapshot.storage.k8s.io/v1beta1/namespaces/default/volumesnapshots/vmsnapshot-534c668b-8170-4b5f-88cf-1258628b275a-volume-cirros-dv-new
  UID:                     03a4ad1c-a339-4b27-9af5-1fb2c896bc6d
Spec:
  Source:
    Persistent Volume Claim Name:  cirros-dv-new
  Volume Snapshot Class Name:      ocs-storagecluster-rbdplugin-snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-03a4ad1c-a339-4b27-9af5-1fb2c896bc6d
  Creation Time:                       2020-10-05T19:52:55Z
  Ready To Use:                        true
  Restore Size:                        2Gi
Events:                                <none>

Comment 1 Maya Rashish 2020-11-18 14:40:55 UTC
michael henriksen points out this bug might have been fixed with the fixes for #1885211 but he would like to double check it first.
This fix might have made it to 2.5.1 already.

Comment 2 Natalie Gavrielov 2020-12-16 13:22:03 UTC
Hi Michael, can you please check if this was resolved already?

Comment 3 Michael Henriksen 2021-01-04 12:56:41 UTC
Yeah this should be addressed by another fix

Comment 4 Maya Rashish 2021-01-04 13:07:38 UTC
Can you mention what the other fix is, so we can set the bug status to MODIFIED/ON_QA depending on whether it's in a downstream release?
thanks

Comment 5 Maya Rashish 2021-01-19 10:40:51 UTC
Hi Stu,
it looks like the fix for this bug isn't available in a downstream build or in an upstream release of release-0.34.
Can we do that when appropriate?

Comment 6 Maya Rashish 2021-01-27 09:59:58 UTC
Sorry for the noise.
I was getting confused by how midstream was not updated in a while, but that's because downstream is updated in another way by automation.

Comment 8 Natalie Gavrielov 2021-02-16 11:06:09 UTC
Verified with CNV 2.5.4.

Comment 14 errata-xmlrpc 2021-02-18 20:58:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.4 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:0613