1856475 – Authentication Operator Degraded complaining unable to attach or mount volumes (unmounted volumes=[v4-0-config-system-cliconfig])

Bug 1856475 - Authentication Operator Degraded complaining unable to attach or mount volumes (unmounted volumes=[v4-0-config-system-cliconfig])

Summary: Authentication Operator Degraded complaining unable to attach or mount volume...

Keywords:
Status:	CLOSED DUPLICATE of bug 1856316
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	apiserver-auth
Sub Component:
Version:	4.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Standa Laznicka
QA Contact:	pmali
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-07-13 18:07 UTC by Anurag saxena
Modified:	2020-07-14 07:47 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-07-14 07:46:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Anurag saxena 2020-07-13 18:07:04 UTC

Description of problem:Found apiserver operator degraded post installing IPI AWS cluster with OpenShiftSdn. Apparently oauth-openshift pods stuck at ContainerCreating with following events:

Events:
  Type     Reason       Age                   From                                                      Message
  ----     ------       ----                  ----                                                      -------
  Normal   Scheduled    <unknown>             default-scheduler                                         Successfully assigned openshift-authentication/oauth-openshift-5644dbc46b-bhv5t to ip-10-0-151-234.ap-northeast-1.compute.internal
  Warning  FailedMount  57m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-system-cliconfig v4-0-config-user-template-error v4-0-config-system-trusted-ca-bundle v4-0-config-system-ocp-branding-template v4-0-config-user-template-provider-selection oauth-openshift-token-lxk8h v4-0-config-system-serving-cert v4-0-config-system-session v4-0-config-system-service-ca v4-0-config-system-router-certs v4-0-config-user-template-login]: timed out waiting for the condition


Check additional info for CLI captures. Must gather here http://file.bos.redhat.com/~anusaxen/must_gather_apiserver.tar.gz

Version-Release number of selected component (if applicable):4.6.0-0.nightly-2020-07-13-051854


How reproducible:Intermittent


Steps to Reproduce:
1.Install IPI OCP cluster on 4.6 
2.
3.

Actual results:Cluster failed with apiserver degraded


Expected results:Cluster should be installed fine without any errors


Additional info:

$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                                                                 False       True          True       58m
cloud-credential                           4.6.0-0.nightly-2020-07-13-051854   True        False         False      80m
cluster-autoscaler                         4.6.0-0.nightly-2020-07-13-051854   True        False         False      67m
config-operator                            4.6.0-0.nightly-2020-07-13-051854   True        False         False      71m
console                                    4.6.0-0.nightly-2020-07-13-051854   False       True          True       58m
csi-snapshot-controller                    4.6.0-0.nightly-2020-07-13-051854   True        False         False      58m
dns                                        4.6.0-0.nightly-2020-07-13-051854   True        False         False      69m
etcd                                       4.6.0-0.nightly-2020-07-13-051854   True        False         False      70m
image-registry                             4.6.0-0.nightly-2020-07-13-051854   True        False         False      59m
ingress                                    4.6.0-0.nightly-2020-07-13-051854   True        False         False      58m
insights                                   4.6.0-0.nightly-2020-07-13-051854   True        False         False      67m
kube-apiserver                             4.6.0-0.nightly-2020-07-13-051854   True        False         False      69m
kube-controller-manager                    4.6.0-0.nightly-2020-07-13-051854   True        False         False      68m
kube-scheduler                             4.6.0-0.nightly-2020-07-13-051854   True        False         False      66m
kube-storage-version-migrator              4.6.0-0.nightly-2020-07-13-051854   True        False         False      58m
machine-api                                4.6.0-0.nightly-2020-07-13-051854   True        False         False      62m
machine-approver                           4.6.0-0.nightly-2020-07-13-051854   True        False         False      69m
machine-config                             4.6.0-0.nightly-2020-07-13-051854   True        False         False      69m
marketplace                                4.6.0-0.nightly-2020-07-13-051854   True        False         False      66m
monitoring                                 4.6.0-0.nightly-2020-07-13-051854   True        False         False      57m
network                                    4.6.0-0.nightly-2020-07-13-051854   True        False         False      71m
node-tuning                                4.6.0-0.nightly-2020-07-13-051854   True        False         False      71m
openshift-apiserver                        4.6.0-0.nightly-2020-07-13-051854   True        False         False      67m
openshift-controller-manager               4.6.0-0.nightly-2020-07-13-051854   True        False         False      64m
openshift-samples                          4.6.0-0.nightly-2020-07-13-051854   True        False         False      66m
operator-lifecycle-manager                 4.6.0-0.nightly-2020-07-13-051854   True        False         False      70m
operator-lifecycle-manager-catalog         4.6.0-0.nightly-2020-07-13-051854   True        False         False      70m
operator-lifecycle-manager-packageserver   4.6.0-0.nightly-2020-07-13-051854   True        False         False      67m
service-ca                                 4.6.0-0.nightly-2020-07-13-051854   True        False         False      71m
storage                                    4.6.0-0.nightly-2020-07-13-051854   True        False         False      67m

$ oc get co console -oyaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: "2020-07-13T14:42:03Z"
  generation: 1
  managedFields:
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec: {}
      f:status:
        .: {}
        f:extension: {}
    manager: cluster-version-operator
    operation: Update
    time: "2020-07-13T14:42:03Z"
  - apiVersion: config.openshift.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions: {}
        f:relatedObjects: {}
        f:versions: {}
    manager: console
    operation: Update
    time: "2020-07-13T15:06:36Z"
  name: console
  resourceVersion: "22888"
  selfLink: /apis/config.openshift.io/v1/clusteroperators/console
  uid: 009ebade-6e20-4e91-9ff0-e5b2bd94fc79
spec: {}
status:
  conditions:
  - lastTransitionTime: "2020-07-13T15:06:36Z"
    message: 'RouteHealthDegraded: route not yet available, https://console-openshift-console.apps.qe-anusaxen50.qe.devcluster.openshift.com/health
      returns ''503 Service Unavailable'''
    reason: RouteHealth_StatusError
    status: "True"
    type: Degraded
  - lastTransitionTime: "2020-07-13T14:56:41Z"
    message: 'SyncLoopRefreshProgressing: Working toward version 4.6.0-0.nightly-2020-07-13-051854'
    reason: SyncLoopRefresh_InProgress
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-07-13T15:04:39Z"
    message: 'DeploymentAvailable: 0 pods available for console deployment'
    reason: Deployment_InsufficientReplicas
    status: "False"
    type: Available
  - lastTransitionTime: "2020-07-13T14:56:39Z"
    reason: AsExpected
    status: "True"
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: operator.openshift.io
    name: cluster
    resource: consoles
  - group: config.openshift.io
    name: cluster
    resource: consoles
  - group: config.openshift.io
    name: cluster
    resource: infrastructures
  - group: config.openshift.io
    name: cluster
    resource: proxies
  - group: oauth.openshift.io
    name: console
    resource: oauthclients
  - group: ""
    name: openshift-console-operator
    resource: namespaces
  - group: ""
    name: openshift-console
    resource: namespaces
  - group: ""
    name: console-public
    namespace: openshift-config-managed
    resource: configmaps
  versions:
  - name: operator
    version: 4.6.0-0.nightly-2020-07-13-051854

$ oc get pods -n openshift-authentication
NAME                               READY   STATUS              RESTARTS   AGE
oauth-openshift-5644dbc46b-bhv5t   0/1     ContainerCreating   0          59m
oauth-openshift-5644dbc46b-mfdf2   0/1     ContainerCreating   0          59m
oauth-openshift-7868fcbb97-2rgg7   0/1     ContainerCreating   0          59m

$ oc describe pod oauth-openshift-5644dbc46b-bhv5t -n openshift-authentication
Name:                 oauth-openshift-5644dbc46b-bhv5t
Namespace:            openshift-authentication
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ip-10-0-151-234.ap-northeast-1.compute.internal/10.0.151.234
Start Time:           Mon, 13 Jul 2020 11:04:36 -0400
Labels:               app=oauth-openshift
                      pod-template-hash=5644dbc46b
Annotations:          openshift.io/scc: anyuid
                      operator.openshift.io/bootstrap-user-exists: true
                      operator.openshift.io/rvs-hash: v_j8mpu4AwmjIv76BQ8qgeXg2SGW6t1stuPbTCp8isQ5h9_NU9BsNkC_UsqB7uppyaN1EUegzFN1HRJQUhnz0A
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/oauth-openshift-5644dbc46b
Containers:
  oauth-openshift:
    Container ID:  
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c26fb8d9759f6997dfdfe04b46f2576540873f76afd4dc05dea656708e7b87a0
    Image ID:      
    Port:          6443/TCP
    Host Port:     0/TCP
    Command:
      /bin/bash
      -ec
    Args:
      if [ -s /var/config/system/configmaps/v4-0-config-system-trusted-ca-bundle/ca-bundle.crt ]; then
          echo "Copying system trust bundle"
          cp -f /var/config/system/configmaps/v4-0-config-system-trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      fi
      exec oauth-server osinserver \
      --config=/var/config/system/configmaps/v4-0-config-system-cliconfig/v4-0-config-system-cliconfig \
      --v=2
      
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     50Mi
    Liveness:     http-get https://:6443/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:6443/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/config/system/configmaps/v4-0-config-system-cliconfig from v4-0-config-system-cliconfig (ro)
      /var/config/system/configmaps/v4-0-config-system-service-ca from v4-0-config-system-service-ca (ro)
      /var/config/system/configmaps/v4-0-config-system-trusted-ca-bundle from v4-0-config-system-trusted-ca-bundle (ro)
      /var/config/system/secrets/v4-0-config-system-ocp-branding-template from v4-0-config-system-ocp-branding-template (ro)
      /var/config/system/secrets/v4-0-config-system-router-certs from v4-0-config-system-router-certs (ro)
      /var/config/system/secrets/v4-0-config-system-serving-cert from v4-0-config-system-serving-cert (ro)
      /var/config/system/secrets/v4-0-config-system-session from v4-0-config-system-session (ro)
      /var/config/user/template/secret/v4-0-config-user-template-error from v4-0-config-user-template-error (ro)
      /var/config/user/template/secret/v4-0-config-user-template-login from v4-0-config-user-template-login (ro)
      /var/config/user/template/secret/v4-0-config-user-template-provider-selection from v4-0-config-user-template-provider-selection (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from oauth-openshift-token-lxk8h (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  v4-0-config-system-session:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-session
    Optional:    false
  v4-0-config-system-cliconfig:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      v4-0-config-system-cliconfig
    Optional:  false
  v4-0-config-system-serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-serving-cert
    Optional:    false
  v4-0-config-system-service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      v4-0-config-system-service-ca
    Optional:  false
  v4-0-config-system-router-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-router-certs
    Optional:    false
  v4-0-config-system-ocp-branding-template:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-ocp-branding-template
    Optional:    false
  v4-0-config-user-template-login:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-user-template-login
    Optional:    true
  v4-0-config-user-template-provider-selection:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-user-template-provider-selection
    Optional:    true
  v4-0-config-user-template-error:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-user-template-error
    Optional:    true
  v4-0-config-system-trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      v4-0-config-system-trusted-ca-bundle
    Optional:  true
  oauth-openshift-token-lxk8h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  oauth-openshift-token-lxk8h
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 120s
                 node.kubernetes.io/unreachable:NoExecute for 120s
Events:
  Type     Reason       Age                   From                                                      Message
  ----     ------       ----                  ----                                                      -------
  Normal   Scheduled    <unknown>             default-scheduler                                         Successfully assigned openshift-authentication/oauth-openshift-5644dbc46b-bhv5t to ip-10-0-151-234.ap-northeast-1.compute.internal
  Warning  FailedMount  57m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-system-cliconfig v4-0-config-user-template-error v4-0-config-system-trusted-ca-bundle v4-0-config-system-ocp-branding-template v4-0-config-user-template-provider-selection oauth-openshift-token-lxk8h v4-0-config-system-serving-cert v4-0-config-system-session v4-0-config-system-service-ca v4-0-config-system-router-certs v4-0-config-user-template-login]: timed out waiting for the condition
  Warning  FailedMount  55m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-user-template-login v4-0-config-user-template-provider-selection v4-0-config-user-template-error v4-0-config-system-session v4-0-config-system-serving-cert v4-0-config-system-ocp-branding-template v4-0-config-system-cliconfig v4-0-config-system-router-certs v4-0-config-system-trusted-ca-bundle oauth-openshift-token-lxk8h v4-0-config-system-service-ca]: timed out waiting for the condition
  Warning  FailedMount  53m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-system-cliconfig v4-0-config-system-serving-cert v4-0-config-system-ocp-branding-template v4-0-config-user-template-provider-selection v4-0-config-system-router-certs v4-0-config-system-service-ca v4-0-config-system-trusted-ca-bundle oauth-openshift-token-lxk8h v4-0-config-user-template-error v4-0-config-user-template-login v4-0-config-system-session]: timed out waiting for the condition
  Warning  FailedMount  50m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-system-serving-cert v4-0-config-system-ocp-branding-template v4-0-config-user-template-provider-selection v4-0-config-system-cliconfig v4-0-config-system-trusted-ca-bundle oauth-openshift-token-lxk8h v4-0-config-system-session v4-0-config-system-router-certs v4-0-config-user-template-error v4-0-config-system-service-ca v4-0-config-user-template-login]: timed out waiting for the condition
  Warning  FailedMount  48m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-system-serving-cert v4-0-config-system-ocp-branding-template v4-0-config-user-template-login v4-0-config-system-trusted-ca-bundle v4-0-config-system-router-certs v4-0-config-user-template-provider-selection v4-0-config-system-session v4-0-config-system-service-ca v4-0-config-user-template-error v4-0-config-system-cliconfig oauth-openshift-token-lxk8h]: timed out waiting for the condition
  Warning  FailedMount  46m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-system-session v4-0-config-system-cliconfig v4-0-config-system-ocp-branding-template v4-0-config-user-template-login v4-0-config-user-template-error v4-0-config-system-trusted-ca-bundle v4-0-config-system-service-ca v4-0-config-user-template-provider-selection oauth-openshift-token-lxk8h v4-0-config-system-serving-cert v4-0-config-system-router-certs]: timed out waiting for the condition
  Warning  FailedMount  44m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-user-template-login v4-0-config-user-template-provider-selection v4-0-config-system-cliconfig v4-0-config-system-ocp-branding-template v4-0-config-system-trusted-ca-bundle v4-0-config-user-template-error v4-0-config-system-session v4-0-config-system-serving-cert v4-0-config-system-service-ca v4-0-config-system-router-certs oauth-openshift-token-lxk8h]: timed out waiting for the condition
  Warning  FailedMount  41m                   kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[oauth-openshift-token-lxk8h v4-0-config-system-service-ca v4-0-config-system-router-certs v4-0-config-user-template-error v4-0-config-system-trusted-ca-bundle v4-0-config-user-template-login v4-0-config-user-template-provider-selection v4-0-config-system-session v4-0-config-system-cliconfig v4-0-config-system-serving-cert v4-0-config-system-ocp-branding-template]: timed out waiting for the condition
  Warning  FailedMount  14m (x16 over 39m)    kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[v4-0-config-system-cliconfig], unattached volumes=[v4-0-config-user-template-login v4-0-config-user-template-error oauth-openshift-token-lxk8h v4-0-config-system-service-ca v4-0-config-system-ocp-branding-template v4-0-config-system-trusted-ca-bundle v4-0-config-system-session v4-0-config-system-cliconfig v4-0-config-system-router-certs v4-0-config-user-template-provider-selection v4-0-config-system-serving-cert]: timed out waiting for the condition
  Warning  FailedMount  4m46s (x30 over 59m)  kubelet, ip-10-0-151-234.ap-northeast-1.compute.internal  MountVolume.SetUp failed for volume "v4-0-config-system-cliconfig" : configmap "v4-0-config-system-cliconfig" not found

Comment 1 Maru Newby 2020-07-13 21:22:39 UTC

Is this a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1852312 ?

Comment 2 Standa Laznicka 2020-07-14 07:46:38 UTC

no, it's a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1856316

*** This bug has been marked as a duplicate of bug 1856316 ***

Note You need to log in before you can comment on or make changes to this bug.