Issue : Moving Registry to Container Native Storage does not work as documented
Issue description:
Customer is not able to copy the registry over to the gluster volume, failing on the rsync step. Need to have a workaround for them.
I am able to reproduce the issue. The rsync command as written in the documentation will not work. The customer and I tried to rsync to a local temporary directory and then rsync from it to gluster but received a permission denied error.
The steps taken:
=-=-=-=-=
[olim@olim ~]$ ssh -i ~/Downloads/qwikLABS-L74-13841.pem -l cloud-user master-0.sbrglustercns35.quicklab.rdu2.cee.redhat.com
Warning: Permanently added 'master-0.sbrglustercns35.quicklab.rdu2.cee.redhat.com,10.10.94.147' (ECDSA) to the list of known hosts.
Last login: Thu Jan 18 14:15:38 2018 from ovpn-117-124.phx2.redhat.com
[cloud-user@master-0 ~]$ sudo -s
[root@master-0 cloud-user]# oc projects
You have access to the following projects and can switch between them with 'oc project <projectname>':
default
kube-system
logging
management-infra
openshift
openshift-infra
* storage-project
Using project "storage-project" on server "https://openshift.internal.sbrglustercns35.quicklab.rdu2.cee.redhat.com:443".
[root@master-0 cloud-user]# oc get pods
NAME READY STATUS RESTARTS AGE
glusterfs-0kzj7 1/1 Running 0 15d
glusterfs-krq0q 1/1 Running 0 15d
glusterfs-lcgxg 1/1 Running 0 15d
heketi-1-zfw5t 1/1 Running 0 15d
storage-project-router-1-fcknm 1/1 Running 0 15d
[root@master-0 cloud-user]# oc project default
Now using project "default" on server "https://openshift.internal.sbrglustercns35.quicklab.rdu2.cee.redhat.com:443".
[root@master-0 cloud-user]# oc projects
You have access to the following projects and can switch between them with 'oc project <projectname>':
* default
kube-system
logging
management-infra
openshift
openshift-infra
storage-project
Using project "default" on server "https://openshift.internal.sbrglustercns35.quicklab.rdu2.cee.redhat.com:443".
[root@master-0 cloud-user]# oc get endpoints heketi-storage-endpoints -o yaml --namespace=storage-project > gluster-registry-endpoints.yaml
[root@master-0 cloud-user]# vi gluster-registry-endpoints.yaml
[root@master-0 cloud-user]# oc create -f gluster-registry-endpoints.yaml
endpoints "heketi-storage-endpoints" created
[root@master-0 cloud-user]# oc get endpoints
NAME ENDPOINTS AGE
docker-registry 10.128.2.2:5000 15d
heketi-storage-endpoints 10.10.94.141:1,10.10.94.146:1,10.10.94.15:1 15s
kubernetes 10.10.94.147:443,10.10.94.147:8053,10.10.94.147:8053 15d
registry-console 10.131.0.3:9090 15d
router 10.10.94.141:443,10.10.94.146:443,10.10.94.15:443 + 6 more... 15d
[root@master-0 cloud-user]# oc get services heketi-storage-endpoints -o yaml --namespace=storage-project > gluster-registry-service.yaml
[root@master-0 cloud-user]# vi gluster-registry-service.yaml
[root@master-0 cloud-user]# oc create -f gluster-registry-service.yaml
service "heketi-storage-endpoints" created
[root@master-0 cloud-user]# vi gluster-registry-service.yaml
[root@master-0 cloud-user]# oc get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
docker-registry 172.30.154.223 <none> 5000/TCP 15d
heketi-storage-endpoints 172.30.213.41 <none> 1/TCP 5m
kubernetes 172.30.0.1 <none> 443/TCP,53/UDP,53/TCP 15d
registry-console 172.30.35.84 <none> 9000/TCP 15d
router 172.30.8.27 <none> 80/TCP,443/TCP,1936/TCP 15d
[root@master-0 cloud-user]# export GID=$(oc get po --selector="docker-registry=default" -o go-template --template='{{printf "%.0f" ((index .items 0).spec.securityContext.fsGroup)}}')
[root@master-0 cloud-user]# oc get routes
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
docker-registry docker-registry-default.apps.sbrglustercns35.quicklab.rdu2.cee.redhat.com ... 1 more docker-registry <all> passthrough None
registry-console registry-console-default.apps.sbrglustercns35.quicklab.rdu2.cee.redhat.com ... 1 more registry-console <all> passthrough None
[root@master-0 cloud-user]# export HEKETI_CLI_SERVER=http://heketi-storage-project.apps.sbrglustercns35.quicklab.rdu2.cee.redhat.com
[root@master-0 cloud-user]# heketi-cli volume create --size=5 --name=gluster-registry-volume --gid=${GID}
Name: gluster-registry-volume
Size: 5
Volume Id: 4512d998a316a6d05109e9ecea89da55
Cluster Id: d844fd37336673c202a4870b770e7ecc
Mount: 10.10.94.141:gluster-registry-volume
Mount Options: backup-volfile-servers=10.10.94.146,10.10.94.15
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3
[root@master-0 cloud-user]# cat > gluster-registry-volume.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: gluster-registry-volume
labels:
glusterfs: registry-volume
spec:
capacity:
storage: 5Gi
glusterfs:
endpoints: gluster-registry-endpoints
path: gluster-registry-volume
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
[root@master-0 cloud-user]# oc create -f gluster-registry-volume.yaml
persistentvolume "gluster-registry-volume" created
[root@master-0 cloud-user]# oc get pv/gluster-registry-volume
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
gluster-registry-volume 5Gi RWX Retain Available 27s
[root@master-0 cloud-user]# cat > gluster-registry-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster-registry-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
selector:
matchLabels:
glusterfs: registry-volume
[root@master-0 cloud-user]# oc create -f gluster-registry-claim.yaml
persistentvolumeclaim "gluster-registry-claim" created
[root@master-0 cloud-user]# oc set env dc/docker-registry REGISTRY_STORAGE_MAINTENANCE_READONLY_ENABLED=true
deploymentconfig "docker-registry" updated
[root@master-0 cloud-user]#
[root@master-0 cloud-user]#
[root@master-0 cloud-user]# export REGISTRY_POD=$(oc get po --selector="docker-registry=default" -o go-template --template='{{printf "%s" ((index .items 0).metadata.name)}}')
[root@master-0 cloud-user]#
[root@master-0 cloud-user]# oc rsync $REGISTRY_POD:/registry/ $REGISTRY_POD:/gluster-registry/
error: rsync is only valid between a local directory and a pod directory; specify a pod directory as [PODNAME]:[DIR]
[root@master-0 cloud-user]# echo $REGISTRY_POD
docker-registry-1-hnr1x
[root@master-0 cloud-user]# mkdir /tmp/registry
[root@master-0 cloud-user]# oc rsync $REGISTRY_POD:/registry/ /tmp/registry
receiving incremental file list
./
sent 14 bytes received 38 bytes 104.00 bytes/sec
total size is 0 speedup is 0.00
[root@master-0 cloud-user]# oc rsync /tmp/registry/ $REGISTRY_POD:/gluster-registry/
sending incremental file list
rsync: mkdir "/gluster-registry" failed: Permission denied (13)
rsync error: error in file IO (code 11) at main.c(587) [Receiver=3.0.9]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
error: exit status 12
=-=-=-=-=
Is reproducible ? Yes
Criticality of Issue -
Business impact -
Cannot move the registry to gluster
Environment details :
GlusterFS version (from all nodes)
Specific component version (if required)
---->
OCP v3.6
CNS v3.6
RHGS v3.3
Number of nodes in cluster - 3
Number of nodes participating in volume with issue - 3
How is Gluster being used : containerized gluster.
Looking at the attached customer case and the output of rsync, I am guessing that the new gluster volume was never properly mounted by the registry pods. The only relevant documentation thing I saw was a note on the case that indicated the following syntax would fix the READONLY registry problem:
oc env -n default dc/docker-registry 'REGISTRY_STORAGE_MAINTENANCE_READONLY={"enabled":true}'