Bug 1393775
| Summary: | Logging upgrade to 3.4.0 failed by "Unable to find log message from cluster.service from pod logging-es-3bjvollr-4-mhyt5 within 300 seconds" | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Xia Zhao <xiazhao> | ||||
| Component: | Logging | Assignee: | ewolinet | ||||
| Status: | CLOSED ERRATA | QA Contact: | Xia Zhao <xiazhao> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 3.4.0 | CC: | aos-bugs, tdawson, xiazhao | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: |
undefined
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-01-18 12:54:35 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Xia Zhao
2016-11-10 10:24:08 UTC
This looks to be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1393769 The deployer failed while waiting for the EFK components to scale up. The difference in the error message is that there was a window of time that the Deployer was able to see the ES pod had started, but it couldn't find a message in the logs to confirm that the service was available. It's fixed. Tested with latest deployer image of 3.4.0, upgraded successfully, and kibana & kibana ops UI accesible with log entries:
$ oc get po
NAME READY STATUS RESTARTS AGE
logging-curator-1-n27sm 1/1 Running 0 4m
logging-curator-ops-1-izno3 1/1 Running 0 4m
logging-deployer-o8b77 0/1 Completed 0 10m
logging-deployer-r8kpd 0/1 Completed 0 6m
logging-es-flruj8ta-4-4gnaz 1/1 Running 0 4m
logging-es-ops-rpbmoj63-4-mgqhe 1/1 Running 0 4m
logging-fluentd-qxxjh 1/1 Running 0 4m
logging-kibana-2-j5ohm 2/2 Running 0 3m
logging-kibana-ops-3-9pr69 2/2 Running 0 3m
I'm not sure why the upgrade pod refused to show all logs by a short write issue:
$ oc logs -f logging-deployer-r8kpd
++ oc get dc -l logging-infra=elasticsearch -o 'jsonpath={.items[*].metadata.name}'
+ for dc in '$(oc get dc -l $label -o jsonpath='\''{.items[*].metadata.name}'\'')'
+ patchDCImage logging-es-flruj8ta logging-elasticsearch false
+ local dc=logging-es-flruj8ta
+ local image=logging-elasticsearch
+ local kibana=false
++ oc get dc/logging-es-flruj8ta -o 'jsonpath={.status.latestVersion}'
+ local version=1
+ local authProxy_patch
+ '[' false = true ']'
+ patchIfValid dc/logging-es-flruj8ta
'{.spec.template.spec.containers[0].image}=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:3.4.0
'
error: short write
# openshift version
openshift v3.4.0.25+1f36858
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0
Images tested with:
brew....:xxxx/openshift3/logging-deployer 3.4.0 08eaf2753130 2 days ago 764.3 MB
Prerelease issue, no docs needed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066 |