Bug 1751711

Summary: External CSI provisioner creates volumes for in-tree storage class
Product: OpenShift Container Platform Reporter: Jan Safranek <jsafrane>
Component: StorageAssignee: Jan Safranek <jsafrane>
Status: CLOSED ERRATA QA Contact: Chao Yang <chaoyang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: aos-bugs, aos-storage-staff, chaoyang
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1752929 (view as bug list) Environment:
Last Closed: 2019-10-16 06:41:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1752929    

Description Jan Safranek 2019-09-12 12:52:31 UTC 
This is clone of bug #1737897
 
Description of problem:
With AWS EBS CSI driver installed, both in-tree and CSI provision will provision volumes for stroage class with provisioner "kubernetes.io/aws-ebs".

It's caused by CSI migration turned on by default in the external CSI provisioner. It affects only CSI drivers that have migration implemented, i.e. AWS EBS, CSI PD and OpenStack, other CSI drivers are OK.

Since we don't ship nor support CSI drivers for these clouds, it's not super serious error, but it should be fixed.

Version-Release number of selected component (if applicable):
(reported by the provisioner): v4.2.0-201909081401-0-g75088ae-dirty

How reproducible:
always

Steps to Reproduce:
1. install a cluster on AWS
2. check the default storage class uses provisioner: kubernetes.io/aws-ebs 
3. install AWS CSI driver
4. create a PVC (using the default storage class)

Actual results:
AWS EBS CSI driver provisions the volume:
$ oc logs -n kube-system <provisioner pod> <provisioner container>

I0911 08:00:12.787394       1 controller.go:354] translating storage class parameters for in-tree plugin kubernetes.io/aws-ebs to CSI
I0911 08:00:12.802651       1 controller.go:451] CreateVolumeRequest {Name:pvc-1cea5684-d46a-11e9-904d-020ff73b0330 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[encrypted:true type:gp2] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:requisite:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-east-2a" > > requisite:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-east-2b" > > preferred:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-east-2a" > > preferred:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-east-2b" > >  XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0911 08:00:12.802831       1 connection.go:180] GRPC call: /csi.v1.Controller/CreateVolume
I0911 08:00:12.802843       1 connection.go:181] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}],"requisite":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2b"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-1cea5684-d46a-11e9-904d-020ff73b0330","parameters":{"encrypted":"true","type":"gp2"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0911 08:00:19.369130       1 connection.go:183] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-east-2a"}}],"capacity_bytes":1073741824,"volume_id":"vol-025d1c353748c897c"}}
I0911 08:00:19.372560       1 connection.go:184] GRPC error: <nil>
I0911 08:00:19.372578       1 controller.go:498] create volume rep: {CapacityBytes:1073741824 VolumeId:vol-025d1c353748c897c VolumeContext:map[] ContentSource:<nil> AccessibleTopology:[segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-east-2a" > ] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0911 08:00:19.372709       1 controller.go:573] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:&AWSElasticBlockStoreVolumeSource{VolumeID:vol-025d1c353748c897c,FSType:ext4,Partition:0,ReadOnly:false,} HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:nil}
I0911 08:00:19.372862       1 controller.go:1306] provision "kube-system/pvc5" class "gp2": volume "pvc-1cea5684-d46a-11e9-904d-020ff73b0330" provisioned
I0911 08:00:19.372891       1 controller.go:1323] provision "kube-system/pvc5" class "gp2": succeeded
I0911 08:00:19.372900       1 volume_store.go:154] Saving volume pvc-1cea5684-d46a-11e9-904d-020ff73b0330
I0911 08:00:19.418227       1 volume_store.go:157] Volume pvc-1cea5684-d46a-11e9-904d-020ff73b0330 saved


Expected results:
Only in-tree volume plugin provisions the volume.
Comment 4 Chao Yang 2019-09-17 06:24:54 UTC 
It is failed on 4.2.0-0.nightly-2019-09-15-052022
There is some error message need fix here.


1.Create pvc using default sc gp2
2.Create a pod

I0917 05:55:47.851186       1 csi-provisioner.go:177] Supports migration from in-tree plugin: kubernetes.io/aws-ebs
I0917 05:55:47.852005       1 controller.go:680] Using saving PVs to API server in background
I0917 05:55:47.852035       1 controller.go:770] Starting provisioner controller ebs.csi.aws.com_ip-10-0-152-107_cbcde704-d90f-11e9-87b2-068e32ee4a96!
I0917 05:55:47.852125       1 volume_store.go:97] Starting save volume queue
I0917 05:55:47.852361       1 reflector.go:123] Starting reflector *v1.PersistentVolumeClaim (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:801
I0917 05:55:47.852383       1 reflector.go:161] Listing and watching *v1.PersistentVolumeClaim from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:801
I0917 05:55:47.852766       1 reflector.go:123] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:804
I0917 05:55:47.852781       1 reflector.go:161] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:804
I0917 05:55:47.853364       1 reflector.go:123] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:807
I0917 05:55:47.853388       1 reflector.go:161] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/controller/controller.go:807
I0917 05:55:47.952945       1 shared_informer.go:123] caches populated
I0917 05:55:47.953146       1 controller.go:819] Started provisioner controller ebs.csi.aws.com_ip-10-0-152-107_cbcde704-d90f-11e9-87b2-068e32ee4a96!
I0917 05:59:57.213902       1 controller.go:1199] provision "kube-system/pvc5" class "gp2": started
I0917 05:59:57.222165       1 controller.go:1298] provision "kube-system/pvc5" class "gp2": volume provision ignored: ignored because PVC annotated with external-provisioner name kubernetes.io/aws-ebs does not match provisioner driver name ebs.csi.aws.com. This could mean the PVC is not migrated
I0917 05:59:57.222208       1 controller.go:1014] Claim processing succeeded, removing PVC 57c27a28-d910-11e9-87f7-06c7d3dd1cd8 from claims in progress
I0917 05:59:57.222263       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"kube-system", Name:"pvc5", UID:"57c27a28-d910-11e9-87f7-06c7d3dd1cd8", APIVersion:"v1", ResourceVersion:"58584", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "kube-system/pvc5"


oc describe pvc pvc5
Name:          pvc5
Namespace:     kube-system
StorageClass:  gp2
Status:        Bound
Volume:        pvc-57c27a28-d910-11e9-87f7-06c7d3dd1cd8
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
               volume.kubernetes.io/selected-node: ip-10-0-138-245.us-east-2.compute.internal
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                 Age                    From                                                                  Message
  ----       ------                 ----                   ----                                                                  -------
  Normal     WaitForFirstConsumer   3m33s (x2 over 3m36s)  persistentvolume-controller                                           waiting for first consumer to be created before binding
  Normal     Provisioning           3m21s                  ebs.csi.aws.com_ip-10-0-152-107_cbcde704-d90f-11e9-87b2-068e32ee4a96  External provisioner is provisioning volume for claim "kube-system/pvc5"
  Normal     ProvisioningSucceeded  3m21s                  persistentvolume-controller                                           Successfully provisioned volume pvc-57c27a28-d910-11e9-87f7-06c7d3dd1cd8 using kubernetes.io/aws-ebs
Comment 7 Chao Yang 2019-09-18 00:22:00 UTC 
Update the bug status to "Verified".
Comment 8 errata-xmlrpc 2019-10-16 06:41:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922