Issue : Moving Registry to Container Native Storage does not work as documented Issue description: Customer is not able to copy the registry over to the gluster volume, failing on the rsync step. Need to have a workaround for them. I am able to reproduce the issue. The rsync command as written in the documentation will not work. The customer and I tried to rsync to a local temporary directory and then rsync from it to gluster but received a permission denied error. The steps taken: =-=-=-=-= [olim@olim ~]$ ssh -i ~/Downloads/qwikLABS-L74-13841.pem -l cloud-user master-0.sbrglustercns35.quicklab.rdu2.cee.redhat.com Warning: Permanently added 'master-0.sbrglustercns35.quicklab.rdu2.cee.redhat.com,10.10.94.147' (ECDSA) to the list of known hosts. Last login: Thu Jan 18 14:15:38 2018 from ovpn-117-124.phx2.redhat.com [cloud-user@master-0 ~]$ sudo -s [root@master-0 cloud-user]# oc projects You have access to the following projects and can switch between them with 'oc project <projectname>': default kube-system logging management-infra openshift openshift-infra * storage-project Using project "storage-project" on server "https://openshift.internal.sbrglustercns35.quicklab.rdu2.cee.redhat.com:443". [root@master-0 cloud-user]# oc get pods NAME READY STATUS RESTARTS AGE glusterfs-0kzj7 1/1 Running 0 15d glusterfs-krq0q 1/1 Running 0 15d glusterfs-lcgxg 1/1 Running 0 15d heketi-1-zfw5t 1/1 Running 0 15d storage-project-router-1-fcknm 1/1 Running 0 15d [root@master-0 cloud-user]# oc project default Now using project "default" on server "https://openshift.internal.sbrglustercns35.quicklab.rdu2.cee.redhat.com:443". [root@master-0 cloud-user]# oc projects You have access to the following projects and can switch between them with 'oc project <projectname>': * default kube-system logging management-infra openshift openshift-infra storage-project Using project "default" on server "https://openshift.internal.sbrglustercns35.quicklab.rdu2.cee.redhat.com:443". [root@master-0 cloud-user]# oc get endpoints heketi-storage-endpoints -o yaml --namespace=storage-project > gluster-registry-endpoints.yaml [root@master-0 cloud-user]# vi gluster-registry-endpoints.yaml [root@master-0 cloud-user]# oc create -f gluster-registry-endpoints.yaml endpoints "heketi-storage-endpoints" created [root@master-0 cloud-user]# oc get endpoints NAME ENDPOINTS AGE docker-registry 10.128.2.2:5000 15d heketi-storage-endpoints 10.10.94.141:1,10.10.94.146:1,10.10.94.15:1 15s kubernetes 10.10.94.147:443,10.10.94.147:8053,10.10.94.147:8053 15d registry-console 10.131.0.3:9090 15d router 10.10.94.141:443,10.10.94.146:443,10.10.94.15:443 + 6 more... 15d [root@master-0 cloud-user]# oc get services heketi-storage-endpoints -o yaml --namespace=storage-project > gluster-registry-service.yaml [root@master-0 cloud-user]# vi gluster-registry-service.yaml [root@master-0 cloud-user]# oc create -f gluster-registry-service.yaml service "heketi-storage-endpoints" created [root@master-0 cloud-user]# vi gluster-registry-service.yaml [root@master-0 cloud-user]# oc get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE docker-registry 172.30.154.223 <none> 5000/TCP 15d heketi-storage-endpoints 172.30.213.41 <none> 1/TCP 5m kubernetes 172.30.0.1 <none> 443/TCP,53/UDP,53/TCP 15d registry-console 172.30.35.84 <none> 9000/TCP 15d router 172.30.8.27 <none> 80/TCP,443/TCP,1936/TCP 15d [root@master-0 cloud-user]# export GID=$(oc get po --selector="docker-registry=default" -o go-template --template='{{printf "%.0f" ((index .items 0).spec.securityContext.fsGroup)}}') [root@master-0 cloud-user]# oc get routes NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD docker-registry docker-registry-default.apps.sbrglustercns35.quicklab.rdu2.cee.redhat.com ... 1 more docker-registry <all> passthrough None registry-console registry-console-default.apps.sbrglustercns35.quicklab.rdu2.cee.redhat.com ... 1 more registry-console <all> passthrough None [root@master-0 cloud-user]# export HEKETI_CLI_SERVER=http://heketi-storage-project.apps.sbrglustercns35.quicklab.rdu2.cee.redhat.com [root@master-0 cloud-user]# heketi-cli volume create --size=5 --name=gluster-registry-volume --gid=${GID} Name: gluster-registry-volume Size: 5 Volume Id: 4512d998a316a6d05109e9ecea89da55 Cluster Id: d844fd37336673c202a4870b770e7ecc Mount: 10.10.94.141:gluster-registry-volume Mount Options: backup-volfile-servers=10.10.94.146,10.10.94.15 Block: false Free Size: 0 Block Volumes: [] Durability Type: replicate Distributed+Replica: 3 [root@master-0 cloud-user]# cat > gluster-registry-volume.yaml kind: PersistentVolume apiVersion: v1 metadata: name: gluster-registry-volume labels: glusterfs: registry-volume spec: capacity: storage: 5Gi glusterfs: endpoints: gluster-registry-endpoints path: gluster-registry-volume accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain [root@master-0 cloud-user]# oc create -f gluster-registry-volume.yaml persistentvolume "gluster-registry-volume" created [root@master-0 cloud-user]# oc get pv/gluster-registry-volume NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE gluster-registry-volume 5Gi RWX Retain Available 27s [root@master-0 cloud-user]# cat > gluster-registry-claim.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: gluster-registry-claim spec: accessModes: - ReadWriteMany resources: requests: storage: 5Gi selector: matchLabels: glusterfs: registry-volume [root@master-0 cloud-user]# oc create -f gluster-registry-claim.yaml persistentvolumeclaim "gluster-registry-claim" created [root@master-0 cloud-user]# oc set env dc/docker-registry REGISTRY_STORAGE_MAINTENANCE_READONLY_ENABLED=true deploymentconfig "docker-registry" updated [root@master-0 cloud-user]# [root@master-0 cloud-user]# [root@master-0 cloud-user]# export REGISTRY_POD=$(oc get po --selector="docker-registry=default" -o go-template --template='{{printf "%s" ((index .items 0).metadata.name)}}') [root@master-0 cloud-user]# [root@master-0 cloud-user]# oc rsync $REGISTRY_POD:/registry/ $REGISTRY_POD:/gluster-registry/ error: rsync is only valid between a local directory and a pod directory; specify a pod directory as [PODNAME]:[DIR] [root@master-0 cloud-user]# echo $REGISTRY_POD docker-registry-1-hnr1x [root@master-0 cloud-user]# mkdir /tmp/registry [root@master-0 cloud-user]# oc rsync $REGISTRY_POD:/registry/ /tmp/registry receiving incremental file list ./ sent 14 bytes received 38 bytes 104.00 bytes/sec total size is 0 speedup is 0.00 [root@master-0 cloud-user]# oc rsync /tmp/registry/ $REGISTRY_POD:/gluster-registry/ sending incremental file list rsync: mkdir "/gluster-registry" failed: Permission denied (13) rsync error: error in file IO (code 11) at main.c(587) [Receiver=3.0.9] rsync: connection unexpectedly closed (9 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] error: exit status 12 =-=-=-=-= Is reproducible ? Yes Criticality of Issue - Business impact - Cannot move the registry to gluster Environment details : GlusterFS version (from all nodes) Specific component version (if required) ----> OCP v3.6 CNS v3.6 RHGS v3.3 Number of nodes in cluster - 3 Number of nodes participating in volume with issue - 3 How is Gluster being used : containerized gluster.
Looking at the attached customer case and the output of rsync, I am guessing that the new gluster volume was never properly mounted by the registry pods. The only relevant documentation thing I saw was a note on the case that indicated the following syntax would fix the READONLY registry problem: oc env -n default dc/docker-registry 'REGISTRY_STORAGE_MAINTENANCE_READONLY={"enabled":true}'
Updated the CNS 3.10 Operations guide with the changes mentioned above. The changes can be seen in step 18 substep 1 of section 8.2. The link to updated doc: https://access.qa.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.10/html-single/operations_guide/#idm140391437582784