Bug 1508345 - Unmountable PVC storage (mongodb)
Summary: Unmountable PVC storage (mongodb)
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Storage
Version: 3.x
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Pavel Pospisil
QA Contact: Jianwei Hou
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-01 09:33 UTC by eds
Modified: 2017-11-13 09:40 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-13 09:40:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description eds 2017-11-01 09:33:23 UTC
Description of problem:

Actual results:
My MongoDB application fails to deploy every single time, and raising errors while mounting the external PVC storage.

Cluster:
starter-us-east-1

Expected results:
Deployable mongodb application with PVC storage

Master Log:

LASTSEEN   FIRSTSEEN   COUNT     NAME               KIND      SUBOBJECT   TYPE      REASON        SOURCE                                   MESSAGE
1h         1h          1         mongodb-12-t5qnk   Pod                   Normal    Scheduled     default-scheduler                        Successfully assigned mongodb-12-t5qnk to ip-172-31-48-167.ec2.internal
17m        1h          29        mongodb-12-t5qnk   Pod                   Warning   FailedMount   kubelet, ip-172-31-48-167.ec2.internal   Unable to mount volumes for pod "mongodb-12-t5qnk_lc-lunch(1d685852-bedb-11e7-9157-1250f17a13c8)": timeout expired waiting for volumes to attach/mount for pod "lc-lunch"/"mongodb-12-t5qnk". list of unattached/unmounted volumes=[mongodb-data]
1m         1h          36        mongodb-12-t5qnk   Pod                   Warning   FailedSync    kubelet, ip-172-31-48-167.ec2.internal   Error syncing pod
1h         1h          1         mongodb-12-t5qnk   Pod                   Warning   FailedMount   attachdetach                             Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": RequestLimitExceeded: Request limit exceeded.
           status code: 503, request id: 6b03de7b-5922-469c-8e7a-6b5d03ddbb23
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: c4a018b1-c645-4129-af42-be96f1153447
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: 99712d06-536c-48ab-9fca-f0197a9452c9
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: 56c469c8-c36d-4c82-8768-851d723785f1
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: e2dcbb31-0e09-4d64-83c0-e8562e6f77b8
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: 247a425c-5a88-4279-85ad-63a2fbd01b85
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: 760cb844-a8ee-45fd-b7a4-daf97f698007
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: 250d39ac-d4a7-4fd8-9c33-f47510011bd2
1h         1h        1         mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: 7cf209ab-7ec0-4164-9b10-6539a8687acc
2m         1h        20        mongodb-12-t5qnk   Pod                 Warning   FailedMount   attachdetach   (combined from similar events): Failed to attach volume "pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610" on node "ip-172-31-48-167.ec2.internal" with: Error attaching EBS volume "vol-0454f9b2fb14bf0ef" to instance "i-082a1bdddb2dd2128": VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance
           status code: 400, request id: ed8e9a51-a524-4bb9-b00a-d2b9ffdca303
1h         1h        1         mongodb-12   ReplicationController             Normal    SuccessfulCreate              replication-controller        Created pod: mongodb-12-t5qnk
1h         1h        1         mongodb      DeploymentConfig                  Normal    ReplicationControllerScaled   deploymentconfig-controller   Scaled replication controller "mongodb-12" from 843294412116 to 1

PVC YAML:
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
    creationTimestamp: 2017-09-28T12:55:47Z
    labels:
      app: nodejs-mongo-persistent
      template: nodejs-mongo-persistent
    name: mongodb
    namespace: lc-lunch
    resourceVersion: "1491828556"
    selfLink: /api/v1/namespaces/lc-lunch/persistentvolumeclaims/mongodb
    uid: 58ea3ea9-a44c-11e7-9603-12d641ec7610
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 1Gi
    storageClassName: ebs
    volumeName: pvc-58ea3ea9-a44c-11e7-9603-12d641ec7610
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 1Gi
    phase: Bound
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""

Comment 1 eds 2017-11-01 09:35:25 UTC
It says 'VolumeInUse: vol-0454f9b2fb14bf0ef is already attached to an instance', but I do not have any other pods running.

Comment 2 eds 2017-11-07 07:41:00 UTC
What's the status of this ticket, because I cannot reach my data now. Please fix ASAP...

Comment 3 Pavel Pospisil 2017-11-07 15:14:25 UTC
As I was informed by my colleague the volume should be now detached.

Comment 4 eds 2017-11-08 10:56:19 UTC
Please respond. I am held hostage, because I cannot reach my storage.

Comment 5 eds 2017-11-13 09:40:05 UTC
Thanks. I managed to rsync the data through another NodeJS pod. Deployment of MongoDB is still not working. I am ditching OpenShift as a serious hosting partner though. Bad support response, unstable deployments, hanging processes and a lot of errors. Would not advice anyone to use your platform. Bye.


Note You need to log in before you can comment on or make changes to this bug.