Bug 1898672

+++ This bug was initially created as a clone of Bug #1898670 +++

Description of problem: When using a daemonset and a Whereabouts range gets exhausted, a pod can get stuck in a containercreating state indefinitely.

See: https://github.com/intel/multus-cni/issues/578


How reproducible: Always.


Steps to Reproduce:

Create a net-attach-def:

```
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
 name: whereabouts-nad2
 namespace: whereabouts
spec:
 config: '{
  "cniVersion": "0.3.1",
  "type": "macvlan",
  "master": "eth0",
  "ipam": {
   "type": "whereabouts",
   "range": "10.10.10.0/24",
   "range_start": "10.10.10.100",
   "range_end": "10.10.10.128",
   "exclude": ["10.10.10.0/25"]
  }
 }'
```

Create a deployment:

```
apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
  run: multustest2
 name: multustest2
 namespace: whereabouts
spec:
 replicas: 1
 selector:
  matchLabels:
   run: multustest2
 template:
  metadata:
   namespace: whereabouts
   labels:
    run: multustest2
   annotations:
    k8s.v1.cni.cncf.io/networks: whereabouts-nad2
  spec:
   containers:
   - image: busybox:latest
     name: multustest2
     command: [ "/bin/sh", "-c", "while true; do date; sleep 10; done"]
     securityContext:
       runAsNonRoot: true
       runAsUser: 1000
```

List the pods (oc get pods -w), you'll show one pod running.

Delete that pod. (oc delete pod multustest2-594975985d-clp6z) for example.

Actual results:

That pod is stuck in a ContainerCreating status, indefinitely.


Expected results:

Pod is recreated normally after it's deleted.