Bug 1747474
| Summary: | Dynamically provisioned storage doesn't work on OpenStack | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Pavol Pitonak <ppitonak> |
| Component: | Installer | Assignee: | Mike Fedosin <mfedosin> |
| Installer sub component: | openshift-installer | QA Contact: | David Sanz <dsanzmor> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | eduen, mfedosin |
| Version: | 4.2.0 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-10-16 06:39:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi! I can't reproduce the bug on MOC: http://paste.openstack.org/show/l3tvMtlEZjlVtitUjnVL/ Can you provide any additional information about that can help me to investigate the issue? Tested the issue on Upshift cloud and couldn't reproduce it as well: http://paste.openstack.org/show/ITkURf5eqPmjMRXHCizB/ @ppitonak Can you tell me exactly what kind of configuration you're having trouble with? I'm installing cluster on internal Red Hat infra - UpShift/PSI. I usually try latest green 4.2.0-0.nightly build. Firstly I allocate a floating IP and set lbFloatingIP in install-config, I'm using this install-config:
apiVersion: v1
baseDomain: crt.codereadyqe.com
compute:
- hyperthreading: Enabled
name: worker
platform:
openstack:
type: ci.m1.xlarge
replicas: 3
controlPlane:
hyperthreading: Enabled
name: master
platform: {}
replicas: 3
metadata:
creationTimestamp: null
name: tmp
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineCIDR: 192.168.0.0/16
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
openstack:
cloud: openstack
computeFlavor: ci.m1.xlarge
externalNetwork: provider_net_cci_10
lbFloatingIP: "xxx"
octaviaSupport: "0"
region: regionOne
trunkSupport: "1"
pullSecret: xxx
sshKey: |
ssh-rsa xxx
Then I run following commands so that I can use nightly builds
$ export OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE=rhcos-42.80.20190827.1
$ openshift-install --dir=$CLUSTER_NAME --log-level debug create manifests
$ sed -i "s/api.openshift.com\/api\/upgrades_info\/v1\/graph/openshift-release.svc.ci.openshift.org\/graph/" $CLUSTER_NAME/manifests/cvo-overrides.yaml
Then I configure our domain in Route53, I just add an A record for cluster's API
$ aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch '{ "Comment": "Update A record for cluster API", "Changes": [ { "Action": "CREATE", "ResourceRecordSet": { "Name": "api.'$CLUSTER_NAME'.'$DOMAIN'", "Type": "A", "TTL": 60, "ResourceRecords": [ { "Value": "'$FIP'" } ] } } ] }'
Then install cluster by running
$ openshift-install --log-level debug create cluster --dir=$CLUSTER_NAME
After cluster is installed, I allocate a new floating IP
$ openstack floating ip create --description "$CLUSTER_NAME-ingress" -f value -c floating_ip_address --port $INGRESS_PORT $NETWORK
setup DNS records in Route53
$ aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch '{ "Comment": "Update A record for cluster API", "Changes": [ { "Action": "CREATE", "ResourceRecordSet": { "Name": "*.apps.'$CLUSTER_NAME'.'$DOMAIN'", "Type": "A", "TTL": 60, "ResourceRecords": [ { "Value": "'$FIP'" } ] } } ] }'
and follow instructions in bug description
I did try the installation
* with a different account
* with a different flavor, external network, with private domain registered not on AWS Route53
* with latest build of installer (from source)
* with 4.2.0-ci build of installer
* with password in clouds.yaml or ~/.config/openstack/secure.yaml
I simply can't reproduce it... The only one thing I noticed is that your pod.yaml is incorrect and requires "apiVersion: v1" This are my results on MOC and Upshift: $ oc new-project testpvc Now using project "testpvc" on server "https://api.mfedosin.shiftstack.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git to build a new example application in Ruby. $ oc apply -f pvc-pp.yaml persistentvolumeclaim/cinder-claim created $ oc apply -f pod-pp.yaml pod/example created $ oc get pod NAME READY STATUS RESTARTS AGE example 1/1 Running 0 93s My install-config.yaml for Upshift: apiVersion: v1 baseDomain: shiftstack.com clusterID: 3f3030b8-7f91-4ff3-9015-7e10147bd6e2 compute: - hyperthreading: Enabled name: worker platform: openstack: type: ocp-master replicas: 3 controlPlane: hyperthreading: Enabled name: master platform: {} replicas: 3 metadata: name: mfedosin networking: clusterNetworks: - cidr: 10.128.0.0/14 hostSubnetLength: 9 serviceCIDR: 172.30.0.0/16 machineCIDR: 10.0.128.0/17 type: OpenshiftSDN platform: openstack: cloud: upshift externalNetwork: provider_net_shared_2 region: regionOne computeFlavor: ocp-master lbFloatingIP: 10.0.79.139 pullSecret: | {"auths": { "quay.io": { "auth": "xxx", "email": "" }}} sshKey: | ssh-rsa xxx The only thing I can suggest is that you use an outdated RHCOS image, because we had a similar bug, but it was fixed about a month ago. Verified on 4.2.0-0.nightly-2019-09-11-032939 # oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cinder-claim Bound pvc-13f25765-d46f-11e9-8730-fa163e0445b6 1Gi RWO standard 16m # oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-13f25765-d46f-11e9-8730-fa163e0445b6 1Gi RWO Delete Bound testpvc/cinder-claim standard 115s # oc get pods NAME READY STATUS RESTARTS AGE example 1/1 Running 0 119s Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922 |
Description of problem: Dynamically provisioned storage does not work. In freshly installed cluster, when I create a PVC of storage class "standard", and use it in a pod, I expect a volume to be provisioned in OpenStack but it stays in unbound state forver. Version-Release number of the following components: $ openshift-install version openshift-install v4.2.0-201908282219-dirty built from commit 4f3e73a0143ba36229f42e8b65b6e65342bb826b release image registry.svc.ci.openshift.org/ocp/release@sha256:7319aa4ba33821a78710b27f9f4bbef68daee41ab3e9a7286976f0d2e050a85b How reproducible: Steps to Reproduce: 1. Install new OCP cluster on OpenStack 13 2. oc new-project testpvc 3. oc apply -f pvc.yaml 4. oc apply -f pod.yaml content of pvc.yaml: kind: PersistentVolumeClaim apiVersion: v1 metadata: name: cinder-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: standard content of pod.yaml: kind: Pod metadata: name: example labels: app: hello-openshift spec: volumes: - name: cinder-claim persistentVolumeClaim: claimName: cinder-claim containers: - name: hello-openshift image: openshift/hello-openshift ports: - containerPort: 8080 volumeMounts: - mountPath: "/usr/share/nginx/html" name: cinder-claim Actual results: $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cinder-claim Pending standard 2m25s $ oc get pod NAME READY STATUS RESTARTS AGE example 0/1 Pending 0 2m20s $ oc get ev -n testpvc LAST SEEN TYPE REASON OBJECT MESSAGE 2m31s Normal WaitForFirstConsumer persistentvolumeclaim/cinder-claim waiting for first consumer to be created before binding Expected results: A new volume is provisioned in OpenStack, a PV is created in OpenShift and PVC bounds to this PV. Additional info: