This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1477440 - Only heapster pod was generated after metrics deployed in a fresh OCP environment
Only heapster pod was generated after metrics deployed in a fresh OCP environ...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
3.6.0
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: ewolinet
Junqi Zhao
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-02 01:50 EDT by Junqi Zhao
Modified: 2017-08-16 15 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The service accounts that belonged in the 'openshift-infra' namespace were being created in 'default' after a different fix to create them before role bindings. Consequence: The pods weren't able to find their SA for running Fix: Updated to create the SAs in the correct namespace Result: Pods are able to start as before
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-10 01:32:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
heapster pod log (14.67 KB, text/plain)
2017-08-02 01:50 EDT, Junqi Zhao
no flags Details
ansible inventory file (441 bytes, text/plain)
2017-08-02 01:51 EDT, Junqi Zhao
no flags Details

  None (edit)
Description Junqi Zhao 2017-08-02 01:50:09 EDT
Created attachment 1307917 [details]
heapster pod log

Description of problem:
In one new built OCP environment where metrics was not deployed, deploy metrics 3.6 via ansible, only heapster pod was generated, hawkular-metrics and hawkular-cassandra was not generated.

Note: Deploy metrics along with OCP would encounter the same issue.

# oc get po
NAME             READY     STATUS    RESTARTS   AGE
heapster-0tz1v   0/1       Running   7          1h

And only heapster image was pulled, maybe it is because hawkular-metrics and hawkular-cassandra is too large to pull within the required time caused this issue.

# docker images | grep metrics
registry.ops.openshift.com/openshift3/metrics-heapster                 v3.6                d7f375c5d7c3        6 hours ago         274.4 MB

All pods would be generated if we re-deploy metrics again.

Version-Release number of the following components:
# rpm -qa | grep openshift-ansible
openshift-ansible-lookup-plugins-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-filter-plugins-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-callback-plugins-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-playbooks-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-docs-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch
openshift-ansible-roles-3.6.173.0.2-1.git.0.f6cbecf.el7.noarch

# openshift version
openshift v3.6.173.0.2
kubernetes v1.6.1+5115d708d7
etcd 3.2.1


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.6 in one new built OCP environment
2.
3.

Actual results:
Only heapster pod was generated

Expected results:
All pods should be ready

Additional info:
Attached inventory file.
Comment 1 Junqi Zhao 2017-08-02 01:51 EDT
Created attachment 1307918 [details]
ansible inventory file
Comment 4 Junqi Zhao 2017-08-02 21:21:59 EDT
PR was merged, tested with the following playbooks, all pods could be created successfully. Please change the status to ON_QA.
# rpm -qa | grep openshift-ansible
openshift-ansible-filter-plugins-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-docs-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-lookup-plugins-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-callback-plugins-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-playbooks-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-3.6.173.0.3-1.git.0.7cd103b.el7.noarch
openshift-ansible-roles-3.6.173.0.3-1.git.0.7cd103b.el7.noarch

# oc get po
NAME                         READY     STATUS    RESTARTS   AGE
hawkular-cassandra-1-33cbm   1/1       Running   0          17m
hawkular-metrics-s9xn4       1/1       Running   0          17m
heapster-1zv7n               1/1       Running   0          17m
Comment 6 Junqi Zhao 2017-08-02 21:28:34 EDT
Close it based on Comment 4
Comment 8 errata-xmlrpc 2017-08-10 01:32:16 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.