Bug 1502945

Summary: End user can not create/list/describe/delete volumesnapshot.
Product: OpenShift Container Platform Reporter: Liang Xia <lxia>
Component: StorageAssignee: Tomas Smetana <tsmetana>
Status: CLOSED NEXTRELEASE QA Contact: Liang Xia <lxia>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.0CC: aos-bugs, aos-storage-staff, bchilds, chaoyang, lxia, tmorriso, vigoyal, xtian
Target Milestone: ---   
Target Release: 3.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-19 16:21:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Liang Xia 2017-10-17 06:26:32 UTC
Description of problem:
End user can not create/list/describe/delete volumesnapshot.

Version-Release number of selected component (if applicable):
openshift v3.7.0-0.153.0
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

How reproducible:
Always

Steps to Reproduce:
1. Set up OCP cluster.
2. Start snapshot controller and provisioner.
# snapshot-controller -cloudprovider=gce -kubeconfig=$HOME/.kube/config &> controller.log &
# snapshot-pv-provisioner -cloudprovider=gce -kubeconfig=$HOME/.kube/config &> provisioner.log &
3. Login in as end user, and create project and pvc.
$ cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gce-pvc
spec:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 3Gi
4. Check pvc is bound with dynamic provisioned pv/volume.
5. Create a snapshot.
$ cat snapshot.yaml 
apiVersion: volume-snapshot-data.external-storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: snapshot-1
  namespace: myns
spec:
  persistentVolumeClaimName: gce-pvc
6. Try to create/list/describe/delete volumesnapshot.

Actual results:
$ oc create -f snapshot.yaml 
Error from server (Forbidden): error when creating "snapshot.yaml": User "lxia" cannot create volumesnapshots.volume-snapshot-data.external-storage.k8s.io in the namespace "myns": User "lxia" cannot create volumesnapshots.volume-snapshot-data.external-storage.k8s.io in project "myns" (post volumesnapshots.volume-snapshot-data.external-storage.k8s.io)

Expected results:
End user should be able to create/list/describe/delete volumesnapshot in owned project.

Comment 7 Liang Xia 2017-10-24 10:31:36 UTC
Still got the same errors.

[lxia@lxia-laptop-nay workdir]$ oc create -f snapshot.yaml 
Error from server (Forbidden): error when creating "snapshot.yaml": User "lxia" cannot create volumesnapshots.volume-snapshot-data.external-storage.k8s.io in the namespace "myns": User "lxia" cannot create volumesnapshots.volume-snapshot-data.external-storage.k8s.io in project "myns" (post volumesnapshots.volume-snapshot-data.external-storage.k8s.io)


[lxia@lxia-laptop-nay workdir]$ oc get volumesnapshot
Error from server (Forbidden): User "lxia" cannot list volumesnapshots.volume-snapshot-data.external-storage.k8s.io in the namespace "myns": User "lxia" cannot list volumesnapshots.volume-snapshot-data.external-storage.k8s.io in project "myns" (get volumesnapshots.volume-snapshot-data.external-storage.k8s.io)


[lxia@lxia-laptop-nay workdir]$ oc version
oc v3.7.0-0.176.0
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://master:8443
openshift v3.7.0-0.176.0
kubernetes v1.7.6+a08f5eeb62

[root@qe-lxia-master-etcd-1 ~]# rpm -qa | grep openshift-external-storage
openshift-external-storage-snapshot-controller-0.0.1-3.git78d6339.el7.x86_64
openshift-external-storage-snapshot-provisioner-0.0.1-3.git78d6339.el7.x86_64

Comment 8 Tomas Smetana 2017-10-24 12:36:27 UTC
Could you please also provide 'oc get pvc gce-pvc -o json'? Looks like I have not been able to reproduce the bug yet...

Comment 9 Tomas Smetana 2017-10-24 13:07:00 UTC
OK. My apologies. I get it now. The policy needs to be modified to allow the users to create VolumeSnapshots (oc policy can-i create VolumeSnapshot must answer yes...)

Comment 10 Tomas Smetana 2017-10-24 14:04:39 UTC
Just to clarify: this is basically a configuration problem. We need a role that would allow for the VolumeSnapshot object creation and bind the users to that role. So there are additional steps required for the admin to make snapshots work.

Comment 11 Tomas Smetana 2017-10-25 14:49:28 UTC
To allow a particular user to create volume snapshots the user needs to have permissions. So let's create a ClusterRole, i.e. create a yaml file (snapshot-role.yaml):

apiVersion: v1
kind: ClusterRole
metadata:
  annotations:
    openshift.io/description: A user that can work with VolumeSnapshots
    openshift.io/reconcile-protect: "false"
  name: volumesnapshot-admin
rules:
- apiGroups:
  - "volume-snapshot-data.external-storage.k8s.io"
  attributeRestrictions: null
  resources:
  - volumesnapshots
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch

Then as the system:admin user run:
oc create -f snapshot-role.yaml

Then add the desired user this role:
oadm policy add-cluster-role-to-user volumesnapshot-admin  <username>

(It is possible to add the whole group to the role too.)
Now the user should be able to create the VolumeSnapshot objects.

Comment 12 Liang Xia 2017-10-26 10:23:22 UTC
Tested on below version,
# openshift version
openshift v3.7.0-0.178.0
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

openshift-external-storage-snapshot-provisioner-0.0.1-3.git78d6339.el7.x86_64
openshift-external-storage-snapshot-controller-0.0.1-3.git78d6339.el7.x86_64

With steps in #comment 11, end users can create/list/describe/delete volumesnapshot now.

Bug is fixed.

Comment 13 Liang Xia 2017-10-27 03:28:28 UTC
Sorry, this should be an temporary workaround, since we will not add cluster role volumesnapshot-admin to all users.

So this still need a better fix.

Comment 14 Liang Xia 2017-10-27 08:06:35 UTC
Assign back to get it really fixed.

Comment 15 Tomas Smetana 2017-10-30 11:21:56 UTC
The role has to be created and assigned to the users. To avoid adding each user separately the admin may decide to assign the role to the whole group. I will update the documentation but I'm afraid we will not avoid some admin action to enable this feature to "ordinary" users. I'm not sure I would be able to come up with some more convenient solution.

Comment 16 Tomas Smetana 2017-11-06 08:59:47 UTC
This will have to be documented. The API RBAC simply must be configured to allow the users to modify the new API objects. I can't fix this in the controller itself.

Comment 22 Liang Xia 2018-01-16 02:14:53 UTC
# oc version
oc v3.9.0-0.19.0
kubernetes v1.9.0-beta1
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://xxxx:8443
openshift v3.9.0-0.19.0
kubernetes v1.9.0-beta1

# oc get volumesnapshot
NAME         AGE
snapshot-1   3m

# oc describe volumesnapshot 
Name:         snapshot-1
Namespace:    lxiap
Labels:       Timestamp=1516068790881181828
Annotations:  <none>
API Version:  volumesnapshot.external-storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-01-16T02:09:00Z
  Generation:          0
  Resource Version:    6942
  Self Link:           /apis/volumesnapshot.external-storage.k8s.io/v1/namespaces/lxiap/volumesnapshots/snapshot-1
  UID:                 3763842c-fa62-11e7-92bf-0e997db73baa
Spec:
  Persistent Volume Claim Name:  gce-pvc
  Snapshot Data Name:            
Status:
  Conditions:          <nil>
  Creation Timestamp:  <nil>
Events:                <none>

Comment 24 Liang Xia 2018-02-06 01:55:02 UTC
There are known issue which need to be fixed in OCP 3.9

https://docs.openshift.com/container-platform/3.7/release_notes/ocp_3_7_release_notes.html#ocp-37-known-issues

Comment 25 Traci Morrison 2018-02-06 13:56:23 UTC
Hi Vikram, please take a look at this bug. This information is documented in the 3.7 release notes, but the issue still persists for 3.9. I'm not sure if there are any docs updates required here. Please advise. Thanks.

Comment 26 Traci Morrison 2018-02-06 14:01:05 UTC
(In reply to Traci Morrison from comment #25)
> Hi Vikram, please take a look at this bug. This information is documented in
> the 3.7 release notes, but the issue still persists for 3.9. I'm not sure if
> there are any docs updates required here. Please advise. Thanks.

Vikram, you can ignore this comment. The doc updates are linked in the Trello card: https://trello.com/c/x3mXFMeX/717-document-volume-snapshot

Comment 27 Traci Morrison 2018-02-16 15:02:25 UTC
Work in progress: https://github.com/openshift/openshift-docs/pull/7634

The pull request has been merged to master and needs to be pushed to the 3.9 branch.

Comment 28 Traci Morrison 2018-02-19 16:21:42 UTC
This update was published on 02/16/2018. This will be available in the 3.9 release at Installation and Configuration > Configuring Persistent Storage > Persistent Volume Snapshots.

Comment 30 Red Hat Bugzilla 2023-09-14 04:10:12 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days