Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1827022

Summary:	Failed to destroy network for pod when using crictl stop
Product:	OpenShift Container Platform	Reporter:	Praveen Kumar <prkumar>
Component:	Networking	Assignee:	Juan Luis de Sousa-Valadas <jdesousa>
Networking sub component:	openshift-sdn	QA Contact:	zhaozhanqi <zzhao>
Status:	CLOSED NOTABUG	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	aos-bugs, cfergeau, jdesousa, jokerman, pehunt
Version:	4.4
Target Milestone:	---
Target Release:	4.5.0
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-04-24 11:03:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Praveen Kumar 2020-04-23 04:52:07 UTC

Description of problem: As part of CRC we need to stop and delete all the pods before creating the cluster and do the following which was working till 4.4.0.rc-9 but it start failing for 4.4.0-rc.10

```
# systemctl stop kubelet
# pods=$(crictl pods -q)
# crictl stopp $(pods)
Stopped sandbox 121a861d0b3b3c08deb3f96bd49712acd5544ebd8caf15968b971622104c3bd2
Stopped sandbox 8459987a6becf751402e2add4ae67cb1d3325585ee9cefaf751b50304db65195
Stopped sandbox 076bb586028877289bab881e4779507e6829e209c342b84200f013b21e6ab0df
Stopped sandbox b1a71fa54dabbb4a193b8aedd1ae09858d1aa7dd293217aa829d5215b7d7b898
Stopped sandbox 7ee1e34e707e0eed021c860e89f375bdb6c27bc707897c42b354adc61c35bfde
Stopped sandbox 600dc52e727df6fba4c50a6998548919803b49fa0f8e362e592928705cb6fcdd
Stopped sandbox 4ee5b3119c90c54f71bc6d1dec0b26f68424e88bf3ed2d9645abc55f45cfd03b
Stopped sandbox 40ff0c862fb80c017c17c8d088996fb24e53499e7b86abdab6af7337b6aa8bb0
Stopped sandbox 0101a7600325fec73f15bb7d77b28c29aa8e90a463d70b664eccad66c9660b71
Stopped sandbox d8302314220dfd10d1bd9f8bc8fc0420f638d47c775276224288b368efbc036a
Stopped sandbox 2195f6ae111ccfd99cbceaedd8dedd72610d615cd5cd49417dc8ad13fbf842ee
FATA[0000] stopping the pod sandbox "45fdd46acdd061100ef807b58f182b87ab90ff5cf3c702880d4c703a12d88e53" failed: rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_cluster-image-registry-operator-7b7f4f8fff-ctfxn_openshift-image-registry_9bc334c8-2bf9-4ebd-84d8-afbbb7f5e336_0(45fdd46acdd061100ef807b58f182b87ab90ff5cf3c702880d4c703a12d88e53): delegateDel: error invoking DelegateDel - "openshift-sdn": error in getting result from DelNetwork: failed to send CNI request: Post http://dummy/: dial unix /var/run/openshift-sdn/cniserver/socket: connect: connection refused
```

```
$ rpm -aq | grep cri
subscription-manager-rhsm-certificates-1.25.17-1.el8.x86_64
criu-3.12-9.el8.x86_64
initscripts-10.00.4-1.el8.x86_64
cri-o-1.17.4-2.dev.rhaos4.4.gitfe61deb.el8.x86_64
cri-tools-1.17.0-2.el8.x86_64
```

```
$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="44.81.202004211631-0"
VERSION_ID="4.4"
OPENSHIFT_VERSION="4.4"
RHEL_VERSION="8.1"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 44.81.202004211631-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.4"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.4"
OSTREE_VERSION='44.81.202004211631-0'
```

Comment 1 Praveen Kumar 2020-04-23 05:03:48 UTC

On 4.4.0-rc.9 where it all worked without any issue have following packages.

```
$ rpm -qa | grep cri
subscription-manager-rhsm-certificates-1.25.17-1.el8.x86_64
criu-3.12-9.el8.x86_64
initscripts-10.00.4-1.el8.x86_64
cri-o-1.17.3-1.rhaos4.4.el8.x86_64
cri-tools-1.17.0-2.el8.x86_64
```

Comment 2 Peter Hunt 2020-04-23 14:58:10 UTC

This error comes from openshift-sdn pod being stopped before the final container. If you want to do this, you'll need to order the shutdown of the pods (all but sdn first, then sdn). I do not know why it worked with rc9, but it may have been luck (the order crictl shut them down happened to work). Moving over to Networking to see if they have insight, but I'm inclined to think this is not a bug

Comment 3 Juan Luis de Sousa-Valadas 2020-04-24 11:03:08 UTC

This isn't a case we can solve. By design the SDN pod to be running for CNI to work. Like Peter said, you can't delete the SDN pod before deleting the pods that are on the SDN. Pods with hostNetwork: true can be deleted without the SDN running.