Bug 1477440 - Only heapster pod was generated after metrics deployed in a fresh OCP environment
Summary: Only heapster pod was generated after metrics deployed in a fresh OCP environ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: ewolinet
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-02 05:50 UTC by Junqi Zhao
Modified: 2017-08-16 19:51 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The service accounts that belonged in the 'openshift-infra' namespace were being created in 'default' after a different fix to create them before role bindings. Consequence: The pods weren't able to find their SA for running Fix: Updated to create the SAs in the correct namespace Result: Pods are able to start as before
Clone Of:
Environment:
Last Closed: 2017-08-10 05:32:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
heapster pod log (14.67 KB, text/plain)
2017-08-02 05:50 UTC, Junqi Zhao
no flags Details
ansible inventory file (441 bytes, text/plain)
2017-08-02 05:51 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1716 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 09:02:50 UTC

Description Junqi Zhao 2017-08-02 05:50:09 UTC
Created attachment 1307917 [details]
heapster pod log

Description of problem:
In one new built OCP environment where metrics was not deployed, deploy metrics 3.6 via ansible, only heapster pod was generated, hawkular-metrics and hawkular-cassandra was not generated.

Note: Deploy metrics along with OCP would encounter the same issue.

# oc get po
NAME             READY     STATUS    RESTARTS   AGE
heapster-0tz1v   0/1       Running   7          1h

And only heapster image was pulled, maybe it is because hawkular-metrics and hawkular-cassandra is too large to pull within the required time caused this issue.

# docker images | grep metrics
registry.ops.openshift.com/openshift3/metrics-heapster                 v3.6                d7f375c5d7c3        6 hours ago         274.4 MB

All pods would be generated if we re-deploy metrics again.

Version-Release number of the following components:
# rpm -qa | grep openshift-ansible
openshift-ansible-lookup-plugins-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-filter-plugins-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-callback-plugins-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-playbooks-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-docs-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-roles-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch

# openshift version
openshift v3.6.173.0.2
kubernetes v1.6.1+5115d708d7
etcd 3.2.1


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.6 in one new built OCP environment
2.
3.

Actual results:
Only heapster pod was generated

Expected results:
All pods should be ready

Additional info:
Attached inventory file.

Comment 1 Junqi Zhao 2017-08-02 05:51:33 UTC
Created attachment 1307918 [details]
ansible inventory file

Comment 4 Junqi Zhao 2017-08-03 01:21:59 UTC
PR was merged, tested with the following playbooks, all pods could be created successfully. Please change the status to ON_QA.
# rpm -qa | grep openshift-ansible
openshift-ansible-filter-plugins-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-docs-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-lookup-plugins-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-callback-plugins-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-playbooks-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-roles-3.6.173.0.3-1.git.0.7cd103b.el7.noarch

# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-33cbm   1/1       Running   0          17m
hawkular-metrics-s9xn4       1/1       Running   0          17m
heapster-1zv7n               1/1       Running   0          17m

Comment 6 Junqi Zhao 2017-08-03 01:28:34 UTC
Close it based on Comment 4

Comment 8 errata-xmlrpc 2017-08-10 05:32:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716


Note You need to log in before you can comment on or make changes to this bug.