Bug 1976387 - sdn containers - readiness probe failing - nonexisting /etc/cni/net.d/80-openshift-network.conf file
Summary: sdn containers - readiness probe failing - nonexisting /etc/cni/net.d/80-open...
Keywords:
Status: CLOSED DUPLICATE of bug 1978268
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Mohamed Mahmoud
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-25 22:16 UTC by Jan Chaloupka
Modified: 2021-07-15 14:09 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-15 14:09:37 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Jan Chaloupka 2021-06-25 22:16:53 UTC 
From https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt/1407422770530226176:

Checking node journal logs of worker nodes (https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt/1408366552603955200/artifacts/e2e-gcp-rt/gather-extra/artifacts/nodes/), readiness probe fails quite often:

From https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt/1408366552603955200/artifacts/e2e-gcp-rt/gather-extra/artifacts/nodes/ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv/journal:

```
Jun 25 11:01:20.677955 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:01:20.677656    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:02:00.807446 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:02:00.807407    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:03:10.710387 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:03:10.710339    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:06:11.350940 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:06:11.350896    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:06:15.777798 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:06:15.777713    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:06:21.088379 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:06:21.088009    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:06:26.264294 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:06:25.684829    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:06:35.794481 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:06:35.794437    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:06:41.342158 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:06:41.342091    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:06:45.705525 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:06:45.705496    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:00.717824 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:00.717776    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:05.732753 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:05.732569    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:20.769501 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:20.769331    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:25.850425 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:25.850366    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:35.719186 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:35.719017    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:40.855662 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:40.855617    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:45.829372 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:45.828909    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:07:50.699617 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:07:50.698869    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:08:05.671451 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:08:05.671400    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:08:11.143178 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:08:11.027482    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:08:15.692882 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:08:15.692727    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:08:20.933775 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:08:20.933425    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:08:40.932511 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:08:40.932464    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:08:50.669462 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:08:50.669359    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:08:55.807184 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:08:55.807143    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:09:10.816198 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:09:10.816158    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:09:15.656379 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:09:15.656335    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:09:20.667670 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:09:20.667387    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:09:26.030911 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:09:26.030225    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:09:30.652761 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:09:30.652636    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:09:56.012834 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:09:56.012788    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:10:00.765203 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:10:00.765149    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:10:16.347773 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:10:15.892404    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:10:22.001232 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:10:21.119987    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:10:50.721765 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:10:50.717945    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:11:00.714396 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:11:00.714353    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:11:20.746326 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:11:20.745658    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:11:30.800656 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:11:30.800583    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:12:00.808023 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:12:00.807982    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:12:05.711429 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:12:05.710550    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:12:10.678721 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:12:10.678647    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:12:25.687209 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:12:25.686534    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:12:30.657986 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:12:30.657944    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:12:35.739135 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:12:35.739094    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:12:50.637890 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:12:50.637837    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:13:00.637404 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:13:00.637342    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:13:12.541394 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:13:10.716726    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:13:15.683426 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:13:15.683387    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:13:26.518393 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:13:26.277138    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:13:30.688180 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:13:30.688135    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:13:35.763572 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:13:35.763531    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:13:50.681099 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:13:50.681057    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:14:10.824726 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:14:10.824653    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:14:15.764808 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:14:15.683360    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:14:20.753331 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:14:20.753289    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:14:25.676977 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:14:25.676938    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:14:40.679082 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:14:40.679026    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:00.776228 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:00.776187    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:05.715041 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:05.714995    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:10.745446 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:10.744838    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:26.090753 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:26.090489    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:30.660011 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:30.659787    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:40.651797 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:40.651611    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:45.692040 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:45.687068    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:50.646763 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:50.646642    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:15:55.781989 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:15:55.773118    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:16:21.491592 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:16:21.470460    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:16:25.651399 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:16:25.651199    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:16:40.750208 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:16:40.750123    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:16:45.715188 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:16:45.712923    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:16:50.652234 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:16:50.651988    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:16:55.803216 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:16:55.803152    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:17:05.751725 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:17:05.751688    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:17:10.753584 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:17:10.753521    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:17:30.697642 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:17:30.697601    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:17:40.742368 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:17:40.742186    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:17:45.678554 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:17:45.678507    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:18:01.103781 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:18:01.103456    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:18:06.415553 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:18:06.415279    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:18:10.786647 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:18:10.786595    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:18:40.663703 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:18:40.662792    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:18:45.674045 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:18:45.673726    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:18:50.753279 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:18:50.741063    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:18:55.642951 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:18:55.642893    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:19:00.681472 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:19:00.681323    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:19:20.747072 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:19:20.747031    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:19:45.913477 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:19:45.913212    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:20:00.923970 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:20:00.923871    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:20:10.760757 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:20:10.760653    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:21:10.918160 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:21:10.918062    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:21:46.297528 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:21:46.297493    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:21:50.851773 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:21:50.851396    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:21:56.062892 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:21:56.062848    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:22:01.215618 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:22:01.215568    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:22:30.741595 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:22:30.741553    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:22:40.690810 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:22:40.690766    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:22:45.956699 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:22:45.912808    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:23:01.829301 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:23:00.693669    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:23:10.726053 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:23:10.726001    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:23:15.684095 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:23:15.671917    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:23:20.679193 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:23:20.679144    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:23:30.840559 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:23:30.840423    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:23:45.676156 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:23:45.675861    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:24:10.733719 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:24:10.733475    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:24:15.686919 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:24:15.686860    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:24:20.695718 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:24:20.695561    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:24:25.888723 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:24:25.888651    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:24:40.681031 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:24:40.680998    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:24:45.893437 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:24:45.785320    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:24:55.867940 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:24:55.867896    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:25:00.956916 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:25:00.956879    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:25:05.787653 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:25:05.787617    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:25:10.955689 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:25:10.955625    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:25:30.693859 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:25:30.693789    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:25:55.688184 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:25:55.687533    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:26:05.796646 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:26:05.796543    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:26:11.040721 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:26:11.040407    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:26:15.650419 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:26:15.650371    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:27:10.923262 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:27:10.922951    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:27:55.669811 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:27:55.669771    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:29:20.935041 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:29:20.934969    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:31:20.696708 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:31:20.696633    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:31:30.690032 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:31:30.689969    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:31:40.828906 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:31:40.828864    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:32:05.859770 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:32:05.859738    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:32:10.867098 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:32:10.863480    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:32:20.655128 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:32:20.655031    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:32:40.801847 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:32:40.801626    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:32:50.727074 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:32:50.727028    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:00.646333 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:00.646088    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:05.745701 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:05.745302    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:10.643138 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:10.643085    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:20.847050 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:20.846677    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:30.668098 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:30.668039    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:35.731980 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:35.731940    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:40.717320 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:40.717098    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:45.947746 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:45.724822    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:33:50.713110 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:33:50.712839    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:34:00.649186 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:34:00.649136    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:34:15.725438 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:34:15.725387    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:34:20.763182 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:34:20.763131    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:34:50.694517 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:34:50.694462    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:35:20.659232 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:35:20.636698    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:35:25.767521 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:35:25.767228    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:35:30.637387 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:35:30.637342    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:35:45.844475 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:35:45.844437    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:35:50.762636 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:35:50.762527    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:36:00.804902 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:36:00.802250    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:36:06.098217 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:36:06.098180    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:36:10.755262 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:36:10.755185    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:36:15.733106 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:36:15.733050    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:36:20.853874 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:36:20.853492    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:36:51.224576 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:36:51.224525    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:36:56.116256 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:36:56.116203    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:37:00.656431 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:37:00.656375    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:37:10.993263 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:37:10.930934    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:37:41.019973 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:37:41.019940    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:37:55.835663 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:37:55.660882    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:38:11.023691 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:38:11.011355    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:38:25.807575 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:38:25.807523    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:38:30.656431 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:38:30.656388    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:38:35.861861 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:38:35.861701    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:38:40.920576 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:38:40.920293    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:38:51.125480 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:38:51.125431    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:38:55.656935 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:38:55.656333    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:39:10.796272 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:39:10.796007    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:39:25.744383 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:39:25.732106    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:39:30.875786 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:39:30.875560    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:39:35.797486 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:39:35.797168    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:39:45.791842 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:39:45.791778    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:39:50.875926 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:39:50.875881    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:39:55.687181 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:39:55.687137    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:40:00.663665 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:40:00.663623    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:40:05.643962 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:40:05.643564    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:40:10.726203 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:40:10.725974    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:40:25.731632 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:40:25.731377    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:40:30.656974 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:40:30.656932    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:40:50.873793 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:40:50.873595    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:41:05.744212 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:41:05.744099    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:41:15.699756 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:41:15.699097    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:41:35.662934 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:41:35.662885    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:41:40.728134 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:41:40.728036    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:41:45.636979 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:41:45.636925    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:41:50.733482 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:41:50.732839    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:42:00.828701 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:42:00.828629    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:42:10.655882 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:42:10.655817    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:42:15.820502 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:42:15.820439    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:42:35.702527 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:42:35.702462    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:42:40.654477 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:42:40.654020    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:42:45.679121 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:42:45.679069    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:43:01.140562 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:43:01.140538    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:43:11.126791 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:43:11.126731    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
Jun 25 11:43:20.666649 ci-op-k72hr7js-f0f18-wpzls-worker-a-qtzrv hyperkube[1572]: I0625 11:43:20.666598    1572 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-sdn/sdn-xpfgc" podUID=f443a18c-2d9a-4165-a700-7a4636f19897 containerName="sdn" probeResult=failure output="command timed out"
```

The readiness probe for the sdn container is configured as:
```
                        "readinessProbe": {
                            "exec": {
                                "command": [
                                    "test",
                                    "-f",
                                    "/etc/cni/net.d/80-openshift-network.conf"
                                ]
                            },
                            "failureThreshold": 3,
                            "initialDelaySeconds": 5,
                            "periodSeconds": 5,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
```

The probe is expected to be ran every 5 seconds. Correlating with the kubelet logs above the probe does not fail every 5s. Histogram of the differences between probe failures:
     75 5
     28 10
     18 15
     14 4
     11 20
      8 6
      6 16
      4 30
      4 25
      3 9
      2 31
      2 26
      2 14
      2 11
      1 85
      1 70
      1 60
      1 55
      1 45
      1 40
      1 36
      1 3
      1 29
      1 28
      1 181
      1 120
      1 12

I'd expect the file to exists in the sdn container all the time (after it's created). Given the probe command only checks if a file exists, it's strange the command can time out. Is it because it takes longer for a kubelet to run a command inside a container?

Checking other jobs where the readiness probe is timing out:
- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt/1408215657794768896
- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt/1408071254895759360
Comment 1 Jan Chaloupka 2021-06-25 22:17:17 UTC 
Also, not localized to a single job. There are others, # of "Probe failed" for "openshift-sdn" lines summed through all nodes.

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-ci-4.9-e2e-aws
  1408499853863948288 -> 55
  1408137236276318208 -> 54
  1407774700578279424 -> 49
  1407412255246520320 -> 48
  1407049775383056384 -> 46
  1406687261579284480 -> 0
  1406324869435494400 -> 0
  1405962259166924800 -> 47
  1405599673976098817 -> 0
  1405237081269080064 -> 0
  1404874471273140224 -> 44 (6-15 11:52)
  1404511939131871232 -> 0
  1404149543762661376 -> 0
  1403787152164130816 -> 2
  1403424759693185024 -> 0
  1403062368040128512 -> 0
  1402699862897594368 -> 0
  1402337406559981568 -> 0
  1401974886473142272 -> 0
  1401612310719500288 -> 0
  1401249710009749504 -> 0
  1400887319149416448 -> 0
  1400524803466596352 -> 3
  1400162364472430592 -> 4
  1399799932109459456 -> 5

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-ci-4.9-e2e-azure:
  1408499853989777408 -> 49
  1408137236427313152 -> 53
  1407774700871880704 -> 47
  1407412256152489984 -> 90
  1407049775517274112 -> 0
  1406687261705113600 -> 57
  1406324869557129216 -> 58
  1405962259393417216 -> 48
  1405599674072567808 -> 7
  1405237081441046528 -> 77
  1404874471415746560 -> 57 (6-15 11:52)
  1404511939651964928 -> 0
  1404149543888490496 -> 0
  1403787152361263104 -> 0
  1403424759898705920 -> 0
  1403062368266620928 -> 0
  1402699863031812096 -> 0
  1402337406731948032 -> 0
  1401974886645108736 -> 0
  1401612311164096512 -> 0
  1401249710789890048 -> 0
  1400887319367520256 -> 1
  1400524803760197632 -> 0
  1400162365361623040 -> 0
  1399799940405792768 -> 0

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-ci-4.9-e2e-gcp-serial:
  1408499854119800832 -> 8
  1408137236557336576 -> 5
  1407774701412945920 -> 8
  1407412256626446336 -> 6
  1407049775655686144 -> 6
  1406687262351036416 -> 5
  1406324870190469120 -> 8
  1405962260718817280 -> 8
  1405599674206785536 -> 7
  1405237081633984512 -> 8
  1404874471545769984 -> 12 (6-15 11:53)
  1404511939865874432 -> 0
  1404149544530219008 -> 0
  1403787152927494144 -> 2
  1403424760028729344 -> 0
  1403062368509890560 -> 0
  1402699863178612736 -> 0
  1402337407499505664 -> 0
  1401974887265865728 -> 0
  1401612312141369344 -> 0
  1401249718952005632 -> 0
  1400887323310166016 -> 0
  1400524811716792320 -> 0
  1400162365575532544 -> 0
  1399799949662621696 -> 0

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp:
  1408523026344972288 -> 0
  1408426965920124928 -> 57
  1408399956833734656 -> 50
  1408366545993732096 -> 100
  1408333565816475648 -> 37
  1408290809438015488 -> 0
  1408215655278186496 -> 43
  1408188377177526272 -> 119
  1408112656614690816 -> 47
  1408071249019539456 -> 26
  1408042914583416832 -> 35
  1407979468877729792 -> 30
  1407965466604867584 -> 35
  1407909975249915904 -> 71
  1407826111701716992 -> 61
  1407779446575861760 -> 42
  1407730976733270016 -> 40
  1407679009713557504 -> 49
  1407617307236110336 -> 36
  1407422783708729344 -> 27
  1407368834712604672 -> 23
  1407286713490870272 -> 26
  1407256069159260160 -> 26
  1407158247877513216 -> 16
  1407141805060788224 -> 20
  1407056138754592768 -> 25
  1407022796508237824 -> 24
  1406990828429119488 -> 16
  1406965032025067520 -> 0
  1406897173617971200 -> 0
  1406877277626568704 -> 0
  1406534725148872704 -> 18
  1406509018096078848 -> 2
  1406431749080092672 -> 23
  1406400607555686400 -> 32
  1406344090815041536 -> 23
  1406301013836566528 -> 17
  1406260944895479808 -> 17
  1406238368362139648 -> 25
  1406175443307991040 -> 15
  1406153596495466496 -> 20
  1406097444055289856 -> 36
  1406035929683988480 -> 37
  1405981743948763136 -> 26
  1405780442413535232 -> 25
  1405770476600430592 -> 23
  1405684672725258240 -> 19
  1405661817270702080 -> 31
  1405616686576439296 -> 26
  1405527699652349952 -> 24
  1405508999712870400 -> 25
  1405453021546024960 -> 37
  1405424083033657344 -> 23
  1405397031186337792 -> 29
  1405349724776566784 -> 36
  1405264231183421440 -> 32
  1405248914688315392 -> 24
  1405156016810627072 -> 16
  1405110770324213760 -> 24
  1405091023717142528 -> 0
  1405047589841145856 -> 54
  1405033782586642432 -> 21
  1404992772485681152 -> 21
  1404974946408468480 -> 30
  1404872603113361408 -> 34
  1404526770002071552 -> 18
  1404508939034300416 -> 29 (6-14 11:40)
  1404195348502548480 -> 0
  1403470568166002688 -> 0
  1402745667595538432 -> 0
  1402020678927912960 -> 0

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt
  1408523029662666752 -> 0
  1408426946819264512 -> 548
  1408399958633091072 -> 455
  1408366552603955200 -> 461
  1408333569146753024 -> 488
  1408290814869639168 -> 0
  1408215657794768896 -> 323
  1408188380587495424 -> 509
  1408112660024659968 -> 485
  1408071254895759360 -> 425
  1408042924704272384 -> 382
  1407979473063645184 -> 421
  1407965469939339264 -> 295
  1407909976097165312 -> 441
  1407826114226688000 -> 449
  1407779446768799744 -> 469
  1407730981636411392 -> 465
  1407679017875673088 -> 445
  1407617318988550144 -> 372
  1407422770530226176 -> 96
  1407368845160615936 -> 79
  1407256072518897664 -> 147
  1407158251291676672 -> 105
  1407141808449785856 -> 119
  1407056139593453568 -> 102
  1407022799905624064 -> 114
  1406990832505982976 -> 162
  1406965032100564992 -> 0
  1406897176881139712 -> 0
  1406877281799901184 -> 0
  1406534730026848256 -> 89
  1406509021455716352 -> 139
  1406431752439730176 -> 129
  1406400610902740992 -> 114
  1406344094812213248 -> 122
  1406301017192009728 -> 128
  1406260945075834880 -> 112
  1406238348372086784 -> 132
  1406175448244686848 -> 129
  1406153600513609728 -> 89
  1406097444885762048 -> 155
  1406035908980903936 -> 125
  1405780442572918784 -> 117
  1405770480123645952 -> 165
  1405684675984232448 -> 113
  1405661820621950976 -> 137
  1405616691475386368 -> 120
  1405527699706875904 -> 83
  1405509000614645760 -> 108
  1405453021898346496 -> 96
  1405424078902267904 -> 180
  1405397031219892224 -> 116
  1405349725611233280 -> 131
  1405264227836366848 -> 134
  1405248915506204672 -> 125
  1405156033394905088 -> 106
  1405110754855620608 -> 120
  1405090997704069120 -> 72
  1405047570882891776 -> 112
  1405033772675502080 -> 95
  1404992764315176960 -> 98
  1404974920785465344 -> 137
  1404872622969196544 -> 96

https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#periodic-ci-openshift-release-master-nightly-4.9-e2e-vsphere
  1408533142368686080 -> 0
  1408523026382721024 -> 0
  1408455104478056448 -> 34
  1408426967753035776 -> 6
  1408399952639430656 -> 13
  1408366547570790400 -> 15
  1408364467191812096 -> 10
  1408333567477420032 -> 10
  1408290810675335168 -> 0
  1408273825409273856 -> 15
  1408215657039794176 -> 15
  1408188379769606144 -> 18
  1408183217189556224 -> 23
  1408112658313383936 -> 8
  1408071251561287680 -> 13
  1408042917049667584 -> 10
  1407979470547062784 -> 15
  1407965468270006272 -> 18
  1407909971059806208 -> 0
  1407826113379438592 -> 28
  1407779446651359232 -> 8
  1407730979124023296 -> 0
  1407730464503894016 -> 7
  1407679012892839936 -> 34
  1407617315037515776 -> 52
  1407549227453648896 -> 3
  1407458618147606528 -> 11
  1407422791266865152 -> 9
  1407368842723725312 -> 9
  1407368005796499456 -> 7
  1407286722638647296 -> 13
  1407186691839496192 -> 12
  1407158255431454720 -> 8
  1407141813474562048 -> 3
  1407096090112561152 -> 12
  1407056143800340480 -> 3
  1407022804217368576 -> 10
  1407005659186073600 -> 0
  1406990846770810880 -> 0
  1406965035137241088 -> 13
  1406897181897527296 -> 5
  1406877286845648896 -> 5
  1406824431300382720 -> 7
  1406733823135191040 -> 0
  1406643266799013888 -> 4
  1406534735080984576 -> 4
  1406509001989951488 -> 3
  1406462038309343232 -> 16
  1406431756634034176 -> 3
  1406400615940100096 -> 2
  1406371430127374336 -> 28
  1406344099010711552 -> 5
  1406301022229368832 -> 5
  1406260952050962432 -> 5
  1406238349148033024 -> 12
  1406175453277851648 -> 5
  1406153605597106176 -> 3
  1406097449088454656 -> 2
  1406035909350002688 -> 4
  1406009075359027200 -> 11
  1405981753193009152 -> 5
  1405918513444425728 -> 7
  1405780443298533376 -> 10
  1405770485215531008 -> 0
  1405737286787665920 -> 10
  1405684680161759232 -> 5
  1405661800703201280 -> 8
  1405646487823585280 -> 6
  1405616696584048640 -> 11
  1405527699794956288 -> 10
  1405508997385031680 -> 3
  1405453023580262400 -> 3
  1405424080579989504 -> 6
  1405397031106646016 -> 12
  1405349727280566272 -> 2
  1405264229514088448 -> 0
  1405248917204897792 -> 12
  1405193584629518336 -> 3
  1405156015153876992 -> 5
  1405110762778660864 -> 12
  1405091002804342784 -> 4
  1405047577249845248 -> 6
  1405033775854784512 -> 2
  1404992765158232064 -> 8
  1404974929681584128 -> 5
  1404921787745046528 -> 5
  1404872632188276736 -> 5
  1404831229940862976 -> 4
  1404740595171201024 -> 7
  1404650009047076864 -> 9 (6-14 21:00)
  1404559350072086528 -> 0
  1404526756559327232 -> 4
  1404508953294934016 -> 7 (6-14 11:40)
  1404378314050637824 -> 0
  1404287719776980992 -> 0
  1404197112677142528 -> 0
  1404106554839404544 -> 0
  1404015920824717312 -> 0
  1403925325989023744 -> 0
  1403834720696930304 -> 0
  1403744159000432640 -> 4
  1403653527003205632 -> 7
  1403562933157367808 -> 0
  1403472327181602816 -> 0
  1403381772640587776 -> 0
  1403291136822349824 -> 0
  1403200542074736640 -> 0
  1403109934266060800 -> 0

Based on the increased occurrence of the "Probe failed" logs there's a chance something changed/got merged around (6-15 11:52)/(6-14 11:40). Could be worth investigation what it was.
Comment 2 Jan Chaloupka 2021-06-25 22:19:17 UTC 
The following script was used to collect the occurrences:

```
JOB=periodic-ci-openshift-release-master-ci-4.9-e2e-azure
JOB=periodic-ci-openshift-release-master-ci-4.9-e2e-gcp-serial
JOB=periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp
JOB=periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt
JOB=periodic-ci-openshift-release-master-nightly-4.9-e2e-vsphere
IDS=$(curl https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing/table?tab=${JOB} 2>/dev/null | jq --raw-output ".changelists[]")

pullData() {
        ID=$1
        total=0
        for dir in $(gsutil ls gs://origin-ci-test/logs/${JOB}/${ID}/artifacts/e2e-vsphere/gather-extra/artifacts/nodes 2>/dev/null); do
                count=$(gsutil cp ${dir}journal - 2>/dev/null | gunzip -c | grep "Probe failed" | grep "openshift-sdn" | wc -l)
                total=$(( $total + $count ))
        done
        echo "  ${ID} -> $total"
}

i=0
for ID in ${IDS}; do
        #echo -e "Checking https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/${JOB}/${ID}"
        if [ $i -le 30 ]; then
                i=$(($i + 1))
                pullData $ID &
        else
                i=0
                wait
        fi
done
```

The scripts were run for each job separately. Results then sorted by job id.
Comment 4 Jan Chaloupka 2021-06-28 22:00:42 UTC 
All the mentioned jobs have must-gather in the bucket. E.g. https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.9-e2e-gcp-rt/1407422770530226176/artifacts/e2e-gcp-rt/gather-must-gather/artifacts/
Comment 10 Mohamed Mahmoud 2021-07-15 11:13:46 UTC 
it seems that recent crio version 1.20+ doesn't respect timeout seconds settings that is why from the logs the readiness probe check took ~3sec however it still timed out even though the limit was set to 5secs
crio team has PR to fix this issue https://bugzilla.redhat.com/show_bug.cgi?id=1978268
once they confirmed this bz will get duplicated to 1978268
Comment 11 Peter Hunt 2021-07-15 14:09:37 UTC 
yeah this does loook like a dup

*** This bug has been marked as a duplicate of bug 1978268 ***
Note You need to log in before you can comment on or make changes to this bug.