Bug 1829113

Summary: Baremetal infra container logs missing from must-gather
Product: OpenShift Container Platform Reporter: Ben Nemec <bnemec>
Component: Machine Config OperatorAssignee: Ben Nemec <bnemec>
Status: CLOSED ERRATA QA Contact: Eldar Weiss <eweiss>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.5CC: amurdaca, augol, vvoronko
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.5.0-0.nightly-2020-05-03-051551 Doc Type: Bug Fix
Doc Text:
Cause: must-gather was not configured to collect details of some baremetal services Consequence: It was difficult to debug problems in baremetal deployments. Fix: Added the openshift-kni-infra namespace to the list of namespaces to be collected. Result: Baremetal service logs are now included in must-gather output.
Story Points: ---
Clone Of: Environment:
titan37
Last Closed: 2020-07-13 17:32:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ben Nemec 2020-04-28 22:13:14 UTC
Description of problem: Running must-gather against a baremetal deployment does not collect the logs from the KNI infra static pods. This makes debugging problematic.


Version-Release number of selected component (if applicable): 4.5


How reproducible: Always


Steps to Reproduce:
1. Run oc adm must-gather against baremetal cluster

Actual results: No logs for the containers in openshift-kni-infra


Expected results: Logs for containers in openshift-kni-infra (coredns, mdns-publisher, keepalived, haproxy)

Comment 1 Eldar Weiss 2020-04-30 07:16:43 UTC
]# oc adm must-gather
[must-gather      ] OUT unable to resolve the imagestream tag openshift/must-gather:latest
[must-gather      ] OUT 
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest
[must-gather      ] OUT namespace/openshift-must-gather-f8tdw created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-nzrfb created
[must-gather      ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created
[must-gather-wxqwm] OUT gather did not start: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp 52.1.239.254:443: connect: network is unreachable
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-nzrfb deleted
[must-gather      ] OUT namespace/openshift-must-gather-f8tdw deleted
error: gather did not start for pod must-gather-wxqwm: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp 52.1.239.254:443: connect: network is unreachable

Comment 2 Ben Nemec 2020-04-30 14:48:37 UTC
I'm guessing you ran that on an ipv6 deployment? By default, must-gather pulls an image from quay.io, which doesn't work with ipv6. I believe this is a known issue. To make it work, you either need to deploy with ipv4 or pass a must-gather image to the command that exists in an ipv6-accessible registry.

Comment 6 Eldar Weiss 2020-05-03 17:57:54 UTC
Hi,

I've ran it on titan37
Ran this version:
version   4.5.0-0.nightly-2020-05-03-051551

all logs created as expected.

Comment 7 Eldar Weiss 2020-05-04 06:50:59 UTC
Expected results: Logs for containers in openshift-kni-infra (coredns, mdns-publisher, keepalived, haproxy)
actual results : all logs created, including the above, etc, too many to list here.

OUT namespaces/openshift-kni-infra/pods/coredns-worker-0-0/coredns-monitor/coredns-monitor/logs/current.log

OUT namespaces/openshift-kni-infra/pods/coredns-worker-0-0/coredns-monitor/coredns-monitor/logs/current.log
OUT namespaces/openshift-kni-infra/pods/keepalived-worker-0-0/keepalived/keepalived/logs/current.log
OUT namespaces/openshift-kni-infra/pods/haproxy-master-0-0/haproxy/haproxy/logs/current.log

Comment 8 errata-xmlrpc 2020-07-13 17:32:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409