Bug 1384738

Summary: error while mounting a dynamically provided pv in google.
Product: OpenShift Container Platform Reporter: raffaele spazzoli <rspazzol>
Component: StorageAssignee: Hemant Kumar <hekumar>
Status: CLOSED NOTABUG QA Contact: Jianwei Hou <jhou>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.3.0CC: aos-bugs, rspazzol
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-27 20:03:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description raffaele spazzoli 2016-10-14 02:58:46 UTC
Description of problem:
I'm trying to start the metrics component. This component was not correctly deployed by the ansible script for the error below. I see the same error when I try manual deployment. The error is that the cassandra-data volume cannot be mounted.
here is the log of the node that is trying to mount the hawkular-cassandra pod:

Oct 14 02:33:25 node3 atomic-openshift-node[18488]: E1014 02:33:25.160070   18488 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/kubernetes-dynamic-pvc-9e490c27-91ab-11e6-b63e-42010a800009\"" failed. No retries permitted until 2016-10-14 02:35:25.16003303 +0000 UTC (durationBeforeRetry 2m0s). Error: AttachVolume.Attach failed for volume "kubernetes.io/gce-pd/kubernetes-dynamic-pvc-9e490c27-91ab-11e6-b63e-42010a800009" (spec.Name: "pvc-9e490c27-91ab-11e6-b63e-42010a800009") from node "node3.c.openshift-enablement-exam2.internal" with: googleapi: Error 400: Invalid value 'node3.c.openshift-enablement-exam2.internal'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', invalidParameter
Oct 14 02:35:25 node3 atomic-openshift-node[18488]: I1014 02:35:25.199485   18488 reconciler.go:215] AttachVolume operation started for volume "kubernetes.io/gce-pd/kubernetes-dynamic-pvc-9e490c27-91ab-11e6-b63e-42010a800009" (spec.Name: "pvc-9e490c27-91ab-11e6-b63e-42010a800009") pod "c3f9bee8-91ac-11e6-b5b6-42010a800008" (UID: "c3f9bee8-91ac-11e6-b5b6-42010a800008")
Oct 14 02:35:25 node3 atomic-openshift-node[18488]: E1014 02:35:25.445834   18488 attacher.go:84] Error attaching PD "kubernetes-dynamic-pvc-9e490c27-91ab-11e6-b63e-42010a800009" to node "node3.c.openshift-enablement-exam2.internal": googleapi: Error 400: Invalid value 'node3.c.openshift-enablement-exam2.internal'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', invalidParameter
Oct 14 02:35:25 node3 atomic-openshift-node[18488]: E1014 02:35:25.445928   18488 nestedpendingoperations.go:233] Operation for "\"kubernetes.io/gce-pd/kubernetes-dynamic-pvc-9e490c27-91ab-11e6-b63e-42010a800009\"" failed. No retries permitted until 2016-10-14 02:37:25.445907014 +0000 UTC (durationBeforeRetry 2m0s). Error: AttachVolume.Attach failed for volume "kubernetes.io/gce-pd/kubernetes-dynamic-pvc-9e490c27-91ab-11e6-b63e-42010a800009" (spec.Name: "pvc-9e490c27-91ab-11e6-b63e-42010a800009") from node "node3.c.openshift-enablement-exam2.internal" with: googleapi: Error 400: Invalid value 'node3.c.openshift-enablement-exam2.internal'. Values must match the following regular expression: '[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?', invalidParameter



Version-Release number of selected component (if applicable):
OpenShift Master:
v3.3.0.34
Kubernetes Master:
v1.3.0+52492b4


How reproducible:
consistently for me.

Steps to Reproduce:
see these instructions:
https://github.com/raffaelespazzoli/openshift-enablement-exam

Actual results:
dynamic pv is created, disk exists in google cloud, but node cannot mount it.

Expected results:
pv is correctly mounted.

Additional info:

Comment 1 Hemant Kumar 2016-10-17 16:52:12 UTC
Can we confirm if - kubernetes.io/gce-pd/kubernetes-dynamic-pvc-9e490c27-91ab-11e6-b63e-42010a800009 is a pvc name or volume name or something entirely different?

If it is a PVC name - a pvc name can't have '.' or '/' in their names.

Comment 2 Hemant Kumar 2016-10-17 21:16:18 UTC
In general can you confirm that if pv or pvc names are not using any of those those characters? It may be too involved to reproduce this at my end.

Comment 3 Hemant Kumar 2016-10-27 20:03:42 UTC
This looks like a case of your code generating node hostnames/instance names which are not accepted by Google Cloud.

I could easily reproduce this via:

╰─~/Downloads> gcloud compute instances create hekumar-node-test2.internal
ERROR: (gcloud.compute.instances.create) Some requests did not succeed:
 - Invalid value for field 'resource.name': 'hekumar-node-test2.internal'. Must be a match of regex '(?:[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?)'

I am going to close this ticket. If you think the resolution here does not addresses the bug - please reopen it.

Comment 4 raffaele spazzoli 2016-10-27 20:26:51 UTC
sorry I sent this via email, but it was not added as a comment in bugzilla:

oc get pvc --all-namespaces
NAMESPACE         NAME                  STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE
default           registry-claim        Bound     registry-volume                            10Gi       RWX           1h
logging           logging-es1           Bound     pvc-798494ab-94a1-11e6-b470-42010a800007   10Gi       RWO           1h
openshift-infra   metrics-cassandra-1   Bound     pvc-1e156b7a-94a0-11e6-87df-42010a800002   10Gi       RWO           1h

NAME                                       CAPACITY   ACCESSMODES   STATUS    CLAIM                                 REASON    AGE
pvc-1e156b7a-94a0-11e6-87df-42010a800002   10Gi       RWO           Bound     openshift-infra/metrics-cassandra-1             1h
pvc-798494ab-94a1-11e6-b470-42010a800007   10Gi       RWO           Bound     logging/logging-es1                             1h
registry-volume                            10Gi       RWX           Bound     default/registry-claim                          1h