Bug 1747474 - Dynamically provisioned storage doesn't work on OpenStack
Summary: Dynamically provisioned storage doesn't work on OpenStack
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.2.0
Assignee: Mike Fedosin
QA Contact: David Sanz
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-30 14:10 UTC by Pavol Pitonak
Modified: 2019-10-16 06:39 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:39:13 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift installer pull 2337 'None' closed Bug 1747474: OpenStack: Escape special characters in cloud provider password 2020-08-04 15:28:17 UTC
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:39:23 UTC

Description Pavol Pitonak 2019-08-30 14:10:51 UTC
Description of problem:

Dynamically provisioned storage does not work. In freshly installed cluster, when I create a PVC of storage class "standard", and use it in a pod, I expect a volume to be provisioned in OpenStack but it stays in unbound state forver.

Version-Release number of the following components:
$ openshift-install version
openshift-install v4.2.0-201908282219-dirty
built from commit 4f3e73a0143ba36229f42e8b65b6e65342bb826b
release image registry.svc.ci.openshift.org/ocp/release@sha256:7319aa4ba33821a78710b27f9f4bbef68daee41ab3e9a7286976f0d2e050a85b


How reproducible:

Steps to Reproduce:
1. Install new OCP cluster on OpenStack 13
2. oc new-project testpvc
3. oc apply -f pvc.yaml
4. oc apply -f pod.yaml

content of pvc.yaml:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cinder-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard

content of pod.yaml:
kind: Pod
metadata:
  name: example
  labels:
    app: hello-openshift
spec:
  volumes:
    - name: cinder-claim
      persistentVolumeClaim:
        claimName: cinder-claim
  containers:
    - name: hello-openshift
      image: openshift/hello-openshift
      ports:
        - containerPort: 8080
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: cinder-claim


Actual results:

$ oc get pvc
NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cinder-claim   Pending                                      standard       2m25s

$ oc get pod
NAME      READY   STATUS    RESTARTS   AGE
example   0/1     Pending   0          2m20s

$ oc get ev -n testpvc
LAST SEEN   TYPE     REASON                 OBJECT                               MESSAGE
2m31s       Normal   WaitForFirstConsumer   persistentvolumeclaim/cinder-claim   waiting for first consumer to be created before binding

Expected results:

A new volume is provisioned in OpenStack, a PV is created in OpenShift and PVC bounds to this PV.

Additional info:

Comment 1 Mike Fedosin 2019-09-03 18:11:13 UTC
Hi! I can't reproduce the bug on MOC: http://paste.openstack.org/show/l3tvMtlEZjlVtitUjnVL/

Can you provide any additional information about that can help me to investigate the issue?

Comment 2 Mike Fedosin 2019-09-03 19:07:48 UTC
Tested the issue on Upshift cloud and couldn't reproduce it as well: http://paste.openstack.org/show/ITkURf5eqPmjMRXHCizB/

Comment 3 Mike Fedosin 2019-09-03 19:09:53 UTC
@ppitonak@redhat.com Can you tell me exactly what kind of configuration you're having trouble with?

Comment 5 Pavol Pitonak 2019-09-04 06:31:47 UTC
I'm installing cluster on internal Red Hat infra - UpShift/PSI. I usually try latest green 4.2.0-0.nightly build. Firstly I allocate a floating IP and set lbFloatingIP in install-config, I'm using this install-config:

apiVersion: v1
baseDomain: crt.codereadyqe.com
compute:
- hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      type: ci.m1.xlarge
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  creationTimestamp: null
  name: tmp
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineCIDR: 192.168.0.0/16
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  openstack:
    cloud: openstack
    computeFlavor: ci.m1.xlarge
    externalNetwork: provider_net_cci_10
    lbFloatingIP: "xxx"
    octaviaSupport: "0"
    region: regionOne
    trunkSupport: "1"
pullSecret: xxx 
sshKey: |
  ssh-rsa xxx 

Then I run following commands so that I can use nightly builds

$ export OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE=rhcos-42.80.20190827.1
$ openshift-install --dir=$CLUSTER_NAME --log-level debug create manifests
$ sed -i "s/api.openshift.com\/api\/upgrades_info\/v1\/graph/openshift-release.svc.ci.openshift.org\/graph/" $CLUSTER_NAME/manifests/cvo-overrides.yaml

Then I configure our domain in Route53, I just add an A record for cluster's API
$ aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch '{ "Comment": "Update A record for cluster API", "Changes": [ { "Action": "CREATE", "ResourceRecordSet": { "Name": "api.'$CLUSTER_NAME'.'$DOMAIN'", "Type": "A", "TTL":  60, "ResourceRecords": [ { "Value": "'$FIP'" } ] } } ] }'

Then install cluster by running
$ openshift-install --log-level debug create cluster --dir=$CLUSTER_NAME

After cluster is installed, I allocate a new floating IP 
$ openstack floating ip create --description "$CLUSTER_NAME-ingress" -f value -c floating_ip_address --port $INGRESS_PORT $NETWORK

setup DNS records in Route53
$ aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch '{ "Comment": "Update A record for cluster API", "Changes": [ { "Action": "CREATE", "ResourceRecordSet": { "Name": "*.apps.'$CLUSTER_NAME'.'$DOMAIN'", "Type": "A", "TTL":  60, "ResourceRecords": [ { "Value": "'$FIP'" } ] } } ] }'

and follow instructions in bug description

I did try the installation
* with a different account
* with a different flavor, external network, with private domain registered not on AWS Route53
* with latest build of installer (from source)
* with 4.2.0-ci build of installer
* with password in clouds.yaml or ~/.config/openstack/secure.yaml

Comment 8 Mike Fedosin 2019-09-05 13:29:37 UTC
I simply can't reproduce it... The only one thing I noticed is that your pod.yaml is incorrect and requires "apiVersion: v1"

This are my results on MOC and Upshift:

$ oc new-project testpvc
Now using project "testpvc" on server "https://api.mfedosin.shiftstack.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git
to build a new example application in Ruby.
$ oc apply -f pvc-pp.yaml 
persistentvolumeclaim/cinder-claim created
$ oc apply -f pod-pp.yaml
pod/example created
$ oc get pod
NAME      READY   STATUS    RESTARTS   AGE
example   1/1     Running   0          93s

My install-config.yaml for Upshift:

apiVersion: v1
baseDomain: shiftstack.com
clusterID:  3f3030b8-7f91-4ff3-9015-7e10147bd6e2
compute:
- hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      type: ocp-master
  replicas: 3
controlPlane:
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3
metadata:
  name: mfedosin
networking:
  clusterNetworks:
  - cidr:             10.128.0.0/14
    hostSubnetLength: 9
  serviceCIDR: 172.30.0.0/16
  machineCIDR: 10.0.128.0/17
  type:        OpenshiftSDN
platform:
  openstack:
    cloud:            upshift
    externalNetwork:  provider_net_shared_2
    region:           regionOne
    computeFlavor:    ocp-master
    lbFloatingIP:     10.0.79.139
pullSecret: |
  {"auths": { "quay.io": { "auth": "xxx", "email": "" }}}
sshKey: |
  ssh-rsa xxx

Comment 9 Mike Fedosin 2019-09-05 13:31:12 UTC
The only thing I can suggest is that you use an outdated RHCOS image, because we had a similar bug, but it was fixed about a month ago.

Comment 12 David Sanz 2019-09-11 08:52:35 UTC
Verified on 4.2.0-0.nightly-2019-09-11-032939

# oc get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
cinder-claim   Bound    pvc-13f25765-d46f-11e9-8730-fa163e0445b6   1Gi        RWO            standard       16m
# oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS   REASON   AGE
pvc-13f25765-d46f-11e9-8730-fa163e0445b6   1Gi        RWO            Delete           Bound    testpvc/cinder-claim   standard                115s
# oc get pods
NAME      READY   STATUS    RESTARTS   AGE
example   1/1     Running   0          119s

Comment 13 errata-xmlrpc 2019-10-16 06:39:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.