Bug 1414966

Summary: Metrics Does not install using ansible
Product: OpenShift Container Platform Reporter: Christian Hernandez <chernand>
Component: InstallerAssignee: Troy Dawson <tdawson>
Status: CLOSED CURRENTRELEASE QA Contact: Gaoyun Pei <gpei>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: aos-bugs, erich, jokerman, mmccomas, nschuetz, veer
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-13 23:19:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christian Hernandez 2017-01-19 20:24:28 UTC
Description of problem:

When trying to deploy metrics from the ansible installer I get the following error in the `oc get events`

```
1h        1h        6         metrics-deployer-xmxav   Pod                                   Warning   FailedSync    {kubelet ip-172-31-22-210.us-west-1.compute.internal}   Error syncing pod, skipping: failed to "StartContainer" for "deployer" with ErrImagePull: "image pull failed for registry.access.redhat.com/openshift3/metrics-deployer:3.4.0, this may be because there are no credentials on this request.  details: (error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: \"File not found.\\\"\")"
```

I'm able to pull the image from docker

```
[root@ip-172-31-22-210 ~]# docker pull  registry.access.redhat.com/openshift3/metrics-deployer:v3.4
Trying to pull repository registry.access.redhat.com/openshift3/metrics-deployer ... 
sha256:86bfb0af56b50863a4dbd77a1b93a7f6375111a5eb389e2c88398001fdd3e003: Pulling from registry.access.redhat.com/openshift3/metrics-deployer
7bd78273b666: Already exists 
c196631bd9ac: Already exists 
2a244ccc5a0c: Already exists 
95e389f1da09: Already exists 
13c112094d90: Already exists 
Digest: sha256:86bfb0af56b50863a4dbd77a1b93a7f6375111a5eb389e2c88398001fdd3e003
Status: Image is up to date for registry.access.redhat.com/openshift3/metrics-deployer:v3.4

```


Version-Release number of selected component (if applicable):

[root@ip-172-31-22-210 ~]# oc version
oc v3.4.0.39
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-31-22-210.us-west-1.compute.internal:8443
openshift v3.4.0.39
kubernetes v1.4.0+776c994


How reproducible:

All the time


Steps to Reproduce:
1. Set up ansible to deploy metrics (example with aws cloud provider preset):

openshift_hosted_metrics_public_url=https://hawkular.{{openshift_master_default_subdomain}}/hawkular/metrics 
openshift_hosted_metrics_deploy=true
openshift_hosted_metrics_storage_kind=dynamic

2. Run the installer

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml 


Actual results:

"ImagePullBackoff" for the metrics-deployer pod


Expected results:

Metrics gets deployed

Additional info:

Running with aws cloud provider

Comment 1 Veer Muchandi 2017-01-20 04:41:28 UTC
I am guessing it is because it is trying to pull registry.access.redhat.com/openshift3/metrics-deployer:3.4.0

instead of 

registry.access.redhat.com/openshift3/metrics-deployer:v3.4

Comment 2 Scott Dodson 2017-01-20 17:00:49 UTC
We're working to ensure that the 3.4.0 tag is applied to all of the metrics and logging images as soon as possible.

Comment 3 Troy Dawson 2017-01-20 22:09:01 UTC
3.4.0 tag has successfully been applied to registry.access.redhat.com/openshift3/metrics-deployer and registry.access.redhat.com/openshift3/logging-deployer

Please try again and let us know how it goes.

Comment 4 Gaoyun Pei 2017-01-22 08:32:31 UTC
Metrics and logging images with 3.4.0 tag are available on registry.access.redhat.com

1. Enable metrics and logging deployment 
openshift_hosted_metrics_deploy=true
openshift_hosted_metrics_storage_kind=dynamic

openshift_hosted_logging_deploy: true
openshift_hosted_logging_storage_kind: dynamic


2. Run the installer

3. Check metrics and logging pods
[root@ip-172-18-3-11 ~]# oc get pod -n openshift-infra
NAME                         READY     STATUS      RESTARTS   AGE
hawkular-cassandra-1-ez3sx   1/1       Running     0          56m
hawkular-metrics-0ob04       1/1       Running     0          56m
heapster-py6pw               1/1       Running     0          56m
metrics-deployer-zxqe7       0/1       Completed   0          56m

[root@ip-172-18-3-11 ~]# oc get pod -n logging
NAME                          READY     STATUS      RESTARTS   AGE
logging-curator-1-n8g8h       1/1       Running     0          51m
logging-deployer-wn2vs        0/1       Completed   0          52m
logging-es-8carpsgr-1-f6gn7   1/1       Running     0          51m
logging-fluentd-c9qvf         1/1       Running     0          51m
logging-fluentd-fpv6e         1/1       Running     0          51m
logging-kibana-1-soams        2/2       Running     0          51m

4. Check the images pulled on node

registry.access.redhat.com/openshift3/logging-elasticsearch      3.4.0               3faf59ca2a97        8 days ago          399 MB
registry.access.redhat.com/openshift3/logging-deployer           3.4.0               910537ac6658        8 days ago          763 MB
registry.access.redhat.com/openshift3/metrics-deployer           3.4.0               2f1483e3e389        8 days ago          769.7 MB
registry.access.redhat.com/openshift3/logging-curator            3.4.0               e8e3b80ea540        8 days ago          244.3 MB
registry.access.redhat.com/openshift3/logging-auth-proxy         3.4.0               d22da7f89efe        8 days ago          219.8 MB
registry.access.redhat.com/openshift3/logging-kibana             3.4.0               fa619423c707        8 days ago          338.6 MB
registry.access.redhat.com/openshift3/metrics-heapster           3.4.0               04c115b270b4        8 days ago          317.8 MB
registry.access.redhat.com/openshift3/logging-fluentd            3.4.0               ceadd7faf90e        8 days ago          232.5 MB
registry.access.redhat.com/openshift3/metrics-cassandra          3.4.0               b5d700281ef7        4 weeks ago         649.2 MB
registry.access.redhat.com/openshift3/metrics-hawkular-metrics   3.4.0               ef113cd9dc4a        4 weeks ago         1.508 GB