Bug 1246649
| Summary: | EBS Persistent volumes aren't writable | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Kenny Woodson <kwoodson> |
| Component: | Storage | Assignee: | Sami Wagiaalla <swagiaal> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Chao Yang <chaoyang> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.0.0 | CC: | dmcphers, jkrieger, libra-bugs, lxia, mmcgrath, twiest |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-23 14:44:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Here's a workaround, but it opens the security of the volumes a bit. 1. Create an EBS volume 2. Mount that EBS volume on the master (can be any instance in the same region, really) 3. Create an FS on that volume 4. Chmod that volume to g+rwXs 5. Change the selinux type of the volume to svirt_sandbox_file_t 6. Unmount and detach the volume 7. Register the volume with the master as a PV Here is a playbook that does these steps: https://github.com/openshift/openshift-ansible/blob/master/playbooks/adhoc/create_pv/create_pv.yaml Hi Thomas, The SELinux label issue will be solved by this PR: https://github.com/kubernetes/kubernetes/pull/15323 The ownership issue should be solved by this PR: https://github.com/kubernetes/kubernetes/pull/15352 You no longer need to do your own mkfs on the volume that functionality has been merged into master for a while Thanks Sami, I'm now watching those PRs. Also, thanks for the heads up on the mkfs stuff. We'll remove that once we no longer have to muck with the permissions, as we can't muck with the permissions unless we already have an FS. :) https://github.com/openshift/origin/pull/5169 is now merged. Pushing this to QA. using this case test https://tcms.engineering.redhat.com/case/510564/ it is correct on the load oc v3.1.0.0 kubernetes v1.2.0-alpha.1-1107-g4c8e6f4 [root@ip-172-18-9-252 ec2-user]# oc exec awspd -it -- bash bash-4.2$ ls -lZd /mnt/aws drwxrwsr-x. root 123456 system_u:object_r:svirt_sandbox_file_t:s0:c2,c13 /mnt/aws bash-4.2$ touch /mnt/aws/file bash-4.2$ ls -lZ /mnt/aws/file -rw-rwSr--. 1000000000 123456 system_u:object_r:svirt_sandbox_file_t:s0:c2,c13 /mnt/aws/file bash-4.2$ id uid=1000000000 gid=0(root) groups=123456,1000000000 |
Description of problem: When setting up ebs persistent volumes on openshift v3 the volume comes up as not writable due to 2 issues. 1. Selinux issue ll /var/lib/openshift/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1b/vol-33b97ad2/ -dZ drwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /var/lib/openshift/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1b/vol-33b97ad2/ unlabled_t is incorrect 2. The owner should not be root but should be the user the container is running as. Permission on the directory should be 700. Version-Release number of selected component (if applicable): openshift-master-3.0.1.0-0.git.205.2c9a9b0.el7ose.x86_64 openshift-3.0.1.0-0.git.205.2c9a9b0.el7ose.x86_64 openshift-sdn-ovs-3.0.1.0-0.git.205.2c9a9b0.el7ose.x86_64 openshift-node-3.0.1.0-0.git.205.2c9a9b0.el7ose.x86_64 How reproducible: Very, happens every time. Steps to Reproduce: 1. Create ebs persistentvolume inside of openshift-master as kube admin apiVersion: v1 kind: PersistentVolume metadata: name: pv-ebs-kwoodson10g0001 labels: type: ebs spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle awsElasticBlockStore: volumeID: aws://us-east-1b/vol-33b97ad2 fsType: ext4 2. Create persistentvolumeclaim for your application. apiVersion: v1 kind: PersistentVolumeClaim metadata: name: mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi volumeName: pv-ebs-kwoodson10g0001 3. Deploy the pod that contains the pvc. 4. Verify that pv is bound by running this command. $] KUBECONFIG=/etc/openshift/master/admin.kubeconfig oc get pv NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON pv-ebs-kwoodson10g0001 type=ebs 10737418240 RWO Bound monitoring/mysql 5. Go to the node in which the pod is running and run 'lsblk'. Verify the drive is mounted.: $] lsblk xvdg 202:96 0 10G 0 disk /var/lib/openshift/openshift.local.volumes/pods/86364b56-3233 6. ll -Zd /var/lib/openshift/path/to/container/ drwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /var/lib/openshift/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1b/vol-33b97ad2/ 7. Verify permissions and selinux labels are incorrect. 'unlabeled_t' and the permissions and ownership: 'drwxr-xr-x. root root' Actual results: The drive does mount but labels, permissions, and ownership are incorrect. These prevent the container from writing to the drive. Expected results: The drive would be mounted with the correct labels, permissions, and ownership and allow the container's user to properly write to the drive. Permissions should be (700) 'drwx------.' Owner should be '<container_user> root' Label should be a proper selinux label that would allow the container_user to write. Additional info: We were able to work around this issue by disabling selinux and setting permissions so that the container user can write to the corresponding drive.