Bug 1728101
| Summary: | oc adm node-logs cannot get full logs due to 30 sec timeout | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Takayoshi Kimura <tkimura> | |
| Component: | oc | Assignee: | Sally <somalley> | |
| Status: | CLOSED DUPLICATE | QA Contact: | zhou ying <yinzhou> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.1.0 | CC: | aos-bugs, jokerman, mfojtik, rkrawitz, tnozicka | |
| Target Milestone: | --- | |||
| Target Release: | 4.1.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1744840 (view as bug list) | Environment: | ||
| Last Closed: | 2019-08-26 15:33:54 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1744840 | |||
| Bug Blocks: | ||||
Note that the 30 sec timeout is not oc command request timeout, so setting oc --request-timeout won't help. It may be load balancer or kubelet request processing timeout. This also affects to oc adm must-gather command. In must-gather, logs are all old and truncated. Can you please try something like this? oc adm node-logs ip-10-0-137-206.us-west-2.compute.internal --since -1d Will that serve as a workaround to get the latest logs? Meanwhile, I'm investigating the timeout issue Yes the --since option works. However it's not ideal that every time we get partial old logs and need to ask users to get the logs once again with that option. see here: https://bugzilla.redhat.com/show_bug.cgi?id=1744840#c1 , The limit is to protect the API server from being DDoS-ed. with available workarounds, closing the current bug as won't fix. *** This bug has been marked as a duplicate of bug 1744840 *** |
Description of problem: oc adm node-logs cannot get full logs due to 30 sec timeout. > $ time oc adm node-logs ip-10-0-134-117.ap-northeast-1.compute.internal | gzip > log.gz > > real 0m30.821s > user 0m19.329s > sys 0m1.746s > > $ zcat log.gz | head > -- Logs begin at Fri 2019-07-05 10:15:42 UTC, end at Tue 2019-07-09 03:33:53 UTC. -- > Jul 05 10:15:42 ip-10-0-134-117 hyperkube[2264]: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request > > $ zcat log.gz | tail > Jul 06 19:04:11 ip-10-0-134-117 hyperkube[2264]: I0706 19:04:11.145001 2264 configmap.go:207] Received configMap openshift-dns/dns-default containing (1) pieces of data, 247 total bytes > Jul 06 19:04:11 ip-10-0-134-117 hyperkube[2264]: I0706 19:04:11.145583 2264 secret.go:206] Received secret openshift-dns/dns-token-p2cr6 containing (4) pieces of data, 11239 total bytes > > $ time oc adm node-logs ip-10-0-134-117.ap-northeast-1.compute.internal --tail=1000000 | gzip > log-tail.gz > > real 0m3.833s > user 0m2.217s > sys 0m0.216s > > $ $ zcat log-tail.gz | wc -l > 200002 > > $ zcat log-tail.gz | tail > Jul 09 03:37:32 ip-10-0-134-117 hyperkube[2073]: I0709 03:37:32.070347 2073 helpers.go:846] eviction manager: observations: signal=nodefs.inodesFree, available: 62299002, capacity: 62389184, time: 2019-07-09 03:37:32.062073369 +0000 UTC m=+87787.582782338 > Jul 09 03:37:32 ip-10-0-134-117 hyperkube[2073]: I0709 03:37:32.070368 2073 helpers.go:846] eviction manager: observations: signal=imagefs.available, available: 107220488Ki, capacity: 124768236Ki, time: 2019-07-09 03:37:32.062073369 +0000 UTC m=+87787.582782338 > > $ TZ=UTC date > Tue 09 Jul 2019 03:37:43 AM UTC Version-Release number of selected component (if applicable): OpenShift 4.1.4 AWS IPI How reproducible: Always Steps to Reproduce: 1. Perform oc adm node-logs with long-running node 2. 3. Actual results: oc adm node-logs command can get very first of the logs and finished, no error report Expected results: oc adm node-logs command can get full logs. Or at least it needs to report timeout error. Additional info: Workarounds are: - to use --tail option, but it can only get 200k lines max. The --since option doesn't work atm. - use ssh access