Bug 1547718
| Summary: | [free-int] Unable to scale up webconsole | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Stefanie Forrester <dakini> |
| Component: | Website | Assignee: | Jessica Forrester <jforrest> |
| Status: | CLOSED DUPLICATE | QA Contact: | Bing Li <bingli> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, jokerman, mmccomas, spadgett |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-02-21 19:55:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I also opened https://bugzilla.redhat.com/show_bug.cgi?id=1547604 for this problem. *** This bug has been marked as a duplicate of bug 1547604 *** |
Description of problem: When attempting to scale up the web console, it immediately scales back down. Version-Release number of selected component (if applicable): registry.reg-aws.openshift.com:443/openshift3/ose-web-console:v3.9.0-0.46.0 How reproducible: Every time Steps to Reproduce: 1. oc scale rs webconsole-586497459 -n openshift-web-console --replicas=3 2. 3. Actual results: All requested pods terminate immediately. Expected results: One or more pods should start. Additional info: [root@free-int-master-3c664 ~]# oc get pods -n openshift-web-console NAME READY STATUS RESTARTS AGE webconsole-586497459-j98xc 0/1 MatchNodeSelector 0 1d [root@free-int-master-3c664 ~]# oc scale rs webconsole-586497459 -n openshift-web-console --replicas=3 replicaset "webconsole-586497459" scaled [root@free-int-master-3c664 ~]# oc get pods -n openshift-web-console NAME READY STATUS RESTARTS AGE webconsole-586497459-4zl6d 0/1 Terminating 0 2s webconsole-586497459-j98xc 0/1 MatchNodeSelector 0 1d webconsole-586497459-lc4nr 0/1 Terminating 0 2s webconsole-586497459-vgq4l 0/1 Terminating 0 2s [root@free-int-master-3c664 ~]# [root@free-int-master-3c664 ~]# oc get pods -n openshift-web-console NAME READY STATUS RESTARTS AGE webconsole-586497459-j98xc 0/1 MatchNodeSelector 0 1d [root@free-int-master-3c664 ~]# A couple minutes later, there are also errors of "failed to mount volume" associated with the scale-up attempts. ------------------------------------ 2018-02-21 19:39:25 +0000 UTC 2018-02-21 18:54:07 +0000 UTC 4 webconsole.15156c0bdfc7963a Deployment Normal ScalingReplicaSet deployment-controller Scaled down replica set webconsole-586497459 to 0 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459.15156e84adb726c3 ReplicaSet Normal SuccessfulCreate replicaset-controller Created pod: webconsole-586497459-vgq4l 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459.15156e84af0c6123 ReplicaSet Normal SuccessfulCreate replicaset-controller Created pod: webconsole-586497459-4zl6d 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459-vgq4l.15156e84adf19fd6 Pod Normal Scheduled default-scheduler Successfully assigned webconsole-586497459-vgq4l to ip-172-31-60-182.ec2.internal 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459.15156e84af0e1a71 ReplicaSet Normal SuccessfulCreate replicaset-controller Created pod: webconsole-586497459-lc4nr 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459-lc4nr.15156e84af112a0e Pod Normal Scheduled default-scheduler Successfully assigned webconsole-586497459-lc4nr to ip-172-31-56-130.ec2.internal 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459-4zl6d.15156e84af643e7f Pod Normal Scheduled default-scheduler Successfully assigned webconsole-586497459-4zl6d to ip-172-31-50-177.ec2.internal 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459.15156e84b206549a ReplicaSet Normal SuccessfulDelete replicaset-controller Deleted pod: webconsole-586497459-lc4nr 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459.15156e84b261fbe4 ReplicaSet Normal SuccessfulDelete replicaset-controller Deleted pod: webconsole-586497459-4zl6d 2018-02-21 19:39:25 +0000 UTC 2018-02-21 19:39:25 +0000 UTC 1 webconsole-586497459.15156e84b262f54f ReplicaSet Normal SuccessfulDelete replicaset-controller Deleted pod: webconsole-586497459-vgq4l 2018-02-21 19:40:34 +0000 UTC 2018-02-21 19:40:34 +0000 UTC 1 webconsole-586497459-bb27x.15156e94a6da9780 Pod Warning FailedMount kubelet, ip-172-31-56-130.ec2.internal Unable to mount volumes for pod "webconsole-586497459-bb27x_openshift-web-console(cbd68756-173e-11e8-9e8f-0a46c474dfe0)": timeout expired waiting for volumes to attach/mount for pod "openshift-web-console"/"webconsole-586497459-bb27x". list of unattached/unmounted volumes=[serving-cert webconsole-config webconsole-token-rfngz] 2018-02-21 19:40:34 +0000 UTC 2018-02-21 19:40:34 +0000 UTC 1 webconsole-586497459-6r9gv.15156e94a82d4f10 Pod Warning FailedMount kubelet, ip-172-31-50-177.ec2.internal Unable to mount volumes for pod "webconsole-586497459-6r9gv_openshift-web-console(cbd99fc1-173e-11e8-9e8f-0a46c474dfe0)": timeout expired waiting for volumes to attach/mount for pod "openshift-web-console"/"webconsole-586497459-6r9gv". list of unattached/unmounted volumes=[serving-cert webconsole-config webconsole-token-rfngz] 2018-02-21 19:40:34 +0000 UTC 2018-02-21 19:40:34 +0000 UTC 1 webconsole-586497459-746m7.15156e94a8602ac7 Pod Warning FailedMount kubelet, ip-172-31-60-182.ec2.internal Unable to mount volumes for pod "webconsole-586497459-746m7_openshift-web-console(cbd97af2-173e-11e8-9e8f-0a46c474dfe0)": timeout expired waiting for volumes to attach/mount for pod "openshift-web-console"/"webconsole-586497459-746m7". list of unattached/unmounted volumes=[serving-cert webconsole-config webconsole-token-rfngz] ------------------------------------ Systems attempting to run the webconsole pods are close to OOM. [sedgar@use-tower2 ~]$ opssh -i -c free-int -t master "free -m" [1] 19:43:45 [SUCCESS] free-int-master-5470f total used free shared buff/cache available Mem: 15885 2881 1183 1 11821 12531 Swap: 0 0 0 [2] 19:43:45 [SUCCESS] free-int-master-3c664 total used free shared buff/cache available Mem: 15885 3781 324 1 11778 11582 Swap: 0 0 0 [3] 19:43:45 [SUCCESS] free-int-master-de987 total used free shared buff/cache available Mem: 15885 3062 584 1 12238 12348 Swap: 0 0 0