Bug 1657003

Hi! I think there may be some confusion; the listed parameters are for a Cinder-backed openshift registry, not persistent volume claims. Separate volumes are created for those: https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/configuration.md#cinder-backed-persistent-volumes-configuration

Let me know if there's something else that's not quite working!

Hi sorry, let me explain more clearly. When we define these variables to attach cinder storage to an internal registry, it does not use the storage we specified. Instead it uses dynamically provisioned storage.

Thanks!

Odd, we've tested this successfully in the past, but maybe there's some combination of parameters we didn't account for. Could you
attach your inventory files, as well as the output of 'openstack volume list' and 'openstack server list'?

So this is what we have in our inventory:

openshift_hosted_registry_storage_kind: openstack
openshift_hosted_registry_storage_access_modes:
- ReadWriteOnce
openshift_hosted_registry_storage_openstack_filesystem: "ext4"
openshift_hosted_registry_storage_openstack_volumeID: "e4ae25b4-930a-4ad4-b674-2e360746f647"
openshift_hosted_registry_storage_volume_size: "20Gi"


And the commands give the following:

> openstack volume list
| e4ae25b4-930a-4ad4-b674-2e360746f647 | pvbugfx.mydomain.internal-registry | available |   20 ||
| 4f1f0ad8-5835-4c78-b1b9-b7a44b28b5c4 | kubernetes-dynamic-pvc-da077dbd-a050-11e8-a561-fa163e2847e3 | in-use  |   20 | Attached to infra-node-0.pvbugfx.mydomain.internal on /dev/vdc |


> oc get pv | grep registry
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                 STORAGECLASS   REASON    AGE 
registry-volume                            20Gi       RWO           Retain          Available                                               118d
pvc-da077dbd-a050-11e8-a561-fa163e2847e3   20Gi       RWO            Delete           Bound   default/registry-claim                     standard                    118d


> openstack server list
| 4b4e7e2d-5a90-4d1e-9256-4714f68adb03 | infra-node-0.pvbugfx.mydomain.internal | ACTIVE | openshift-ansible-pvbugfx.mydomain.internal-net=192.168.40.7, 10.9.53.44  | rhel-server-7.6-x86_64 | m1.xlarge   |
| bc88f735-6292-4221-8845-a9d3cb9dc547 | master-0.pvbugfx.mydomain.internal     | ACTIVE | openshift-ansible-pvbugfx.mydomain.internal-net=192.168.40.5, 10.9.53.123 | rhel-server-7.6-x86_64 | m1.xlarge   |
| 1c889094-43fc-4f6d-adf2-d4f3afd736ea | app-node-0.pvbugfx.mydomain.internal   | ACTIVE | openshift-ansible-pvbugfx.mydomain.internal-net=192.168.40.8, 10.9.53.140 | rhel-server-7.6-x86_64 | m1.xlarge   |

Interesting - could you attach the full all.yml and OSEv3.yml (with any passwords removed)? I want to make sure I get the exact right parameters before I try and reproduce.

Here's an inventory that produces this issue:

all.yml is empty


OSEv3.yml
---------

ansible_user: openshift
ansible_become: True

dns_domain: "mydomain.com"
dns_nameservers:
- 10.8.15.4

deployment_type: openshift-enterprise
openshift_deployment_type: openshift-enterprise
openshift_release: v3.11

# Enable the Multi-Tenant plugin
os_sdn_network_plugin_name: 'redhat/openshift-ovs-multitenant'

# Change default ports to use standard 443
openshift_master_api_port: 443
openshift_master_console_port: 443


# OpenShift FQDNs, DNS, App domain specific configurations
openshift_master_default_subdomain: "apps.pvbugfx.mydomain.com"
openshift_master_cluster_hostname: "master-0.pvbugfx.mydomain.com"
openshift_master_cluster_public_hostname: "console.pvbugfx.mydomain.com"

oreg_auth_user: "{{ lookup('env', 'OREG_AUTH_USER' )}}"
oreg_auth_password: "{{ lookup('env', 'OREG_AUTH_PASSWORD' )}}"

openshift_cloudprovider_kind: openstack
openshift_cloudprovider_openstack_auth_url: "{{ lookup('env','OCP_OSP_AUTH_URL') }}"
openshift_cloudprovider_openstack_username: "{{ lookup('env','OCP_OSP_USERNAME') }}"
openshift_cloudprovider_openstack_password: "{{ lookup('env','OCP_OSP_PASSWORD') }}"
openshift_cloudprovider_openstack_domain_name: Default
openshift_cloudprovider_openstack_tenant_id: "{{ lookup('env','OS_PROJECT_ID') }}"
openshift_cloudprovider_openstack_tenant_name: "{{ lookup('env','OS_PROJECT_NAME') }}"
openshift_cloudprovider_openstack_region: "{{ lookup('env','OS_REGION_NAME') }}"
openshift_cloudprovider_openstack_lb_subnet_id: subnet_id
openshift_cloudprovider_openstack_blockstorage_version: v2


openshift_hosted_registry_storage_kind: openstack
openshift_hosted_registry_storage_access_modes:
- 'ReadWriteOnce'
openshift_hosted_registry_storage_openstack_filesystem: ext4
openshift_hosted_registry_storage_openstack_volumeID: 64ff44bf-ecd1-4ca4-8fbc-f57a365a6467
openshift_hosted_registry_storage_volume_size: 20Gi


openshift_master_identity_providers:
- 'name': 'htpasswd_auth'
  'login': 'true'
  'challenge': 'true'
  'kind': 'HTPasswdPasswordIdentityProvider'

# OpenShift Node specific parameters
openshift_node_groups:
- name: node-config-master
  labels:
  - 'node-role.kubernetes.io/master=true'
  edits:
  - key: kubeletArguments.kube-reserved
    value:
    - 'cpu={{ ansible_processor_vcpus * 50 }}m'
    - 'memory={{ ansible_processor_vcpus * 50 }}M'
  - key: kubeletArguments.system-reserved
    value:
    - 'cpu={{ ansible_processor_vcpus * 50 }}m'
    - 'memory={{ ansible_processor_vcpus * 100 }}M'
- name: node-config-infra
  labels:
  - 'node-role.kubernetes.io/infra=true'
  edits:
  - key: kubeletArguments.kube-reserved
    value:
    - 'cpu={{ ansible_processor_vcpus * 50 }}m'
    - 'memory={{ ansible_processor_vcpus * 50 }}M'
  - key: kubeletArguments.system-reserved
    value:
    - 'cpu={{ ansible_processor_vcpus * 50 }}m'
    - 'memory={{ ansible_processor_vcpus * 100 }}M'
- name: node-config-compute
  labels:
  - 'node-role.kubernetes.io/compute=true'
  edits:
  - key: kubeletArguments.kube-reserved
    value:
    - 'cpu={{ ansible_processor_vcpus * 50 }}m'
    - 'memory={{ ansible_processor_vcpus * 50 }}M'
  - key: kubeletArguments.system-reserved
    value:
    - 'cpu={{ ansible_processor_vcpus * 50 }}m'
    - 'memory={{ ansible_processor_vcpus * 100 }}M'

Ah, I've reproduced the issue; strange! I'll dig in and let you know what I find.

Okay, I figured out the source of the issue. I need to dig in deeper to understand what the right fix may be, but one thing that works is setting this in your OSEv3.yml inventory file:

openshift_storageclass_default: "false"

It looks like default storage classes were introduced relatively recently. This feature is active by default, and if on then PVCs will have their storage class set to the default; however this doesn't happen for PVs, resulting in the behavior you reported.

Fixed by https://github.com/openshift/openshift-ansible/pull/10948

Merge fixed. Note that part of the fix includes a doc update that explains an added parameter that needs to be present in order for this to work.

Hi, 

I still meet this issue with openshift-ansible-3.11.95

openshift_hosted_registry_storage_kind=openstack
openshift_hosted_registry_storage_provider=
openshift_hosted_registry_storage_access_modes=["ReadWriteOnce"]
openshift_hosted_registry_storage_openstack_filesystem=xfs
openshift_hosted_registry_storage_volume_size=20Gi
openshift_hosted_registry_storage_openstack_volumeID=d063fd5a-fc1a-4e6e-9524-fb6641aff7eb


# oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                    STORAGECLASS   REASON    AGE
pvc-60d1d64d-46f3-11e9-9624-fa163e2e3062   20Gi       RWO            Delete           Bound       default/registry-claim   standard                 15m
registry-volume                            20Gi       RWO            Retain           Available                                                     15m
# oc get pvc
NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
registry-claim   Bound     pvc-60d1d64d-46f3-11e9-9624-fa163e2e3062   20Gi       RWO            standard       15m

Thanks.

Fixed.

openshift-ansible-3.11.96-1.git.0.c2a3f89.el7

[root@preserve-wmengl6a311d2rcin-me-1 ~]# oc get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                    STORAGECLASS   REASON    AGE
registry-volume   20Gi       RWO            Retain           Bound     default/registry-claim   standard                 11m
[root@preserve-wmengl6a311d2rcin-me-1 ~]# oc get pvc
NAME             STATUS    VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
registry-claim   Bound     registry-volume   20Gi       RWO            standard       11m


Need to update on-line doc for this.

https://docs.openshift.com/container-platform/3.11/install/configuring_inventory_file.html
not only v3.11 but all relevant versions.

Chen, is there doc bug to track on-line doc update? 
If yes, could you put bug link here?
or need I file doc bug to track this?

one more thing, 
we also need bugs to ensure all related versions get this fix 
Need I clone bug to track?

Ah, I don't believe there's a doc bug, nor has the bug been cloned to previous version. Note that this fix only applies to shiftstack install, so we can only clone as far back as 3.10. Thanks!

Fixed.

openshift-ansible-3.11.98-1.git.0.3cfa7c3.el7

# oc get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                    STORAGECLASS   REASON    AGE
registry-volume   20Gi       RWO            Retain           Bound     default/registry-claim   standard                 10m
[root@qe-wmengl6a31198d2rcin-master-etcd-1 ~]# oc get pvc
NAME             STATUS    VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
registry-claim   Bound     registry-volume   20Gi       RWO            standard       10m

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0636