Bug 1642530

Description of problem:

If we delete our VirtualMachineInstance CRs with the "Foreground" PropagationPolicy once everything works fine. If we send the delete request which looks like this:


```
{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}
```

multiple times, it happens that the DELETE request adds the Foreground finalizer again to the CR after the Garbage Collector already removed it. The Garbabe Collector does not remove it afterwards again and the CR is stuck.


Here the VMI directly after it got deleted:

```
apiVersion: kubevirt.io/v1alpha2
kind: VirtualMachineInstance
metadata:
  creationTimestamp: 2018-10-24T14:34:28Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2018-10-24T14:48:18Z
  finalizers:
  - foregroundDeleteVirtualMachine
  uid: e99fc418-d799-11e8-a4cd-fa163e0953ea
```

Here the GC logs:

```
I1024 14:48:18.907722       1 graph_builder.go:553] add [kubevirt.io/v1alpha2/VirtualMachineInstance, namespace: default, name: fedora, uid: e99fc418-d799-11e8-a4cd-fa163e0953ea] to the attemptToDelete, because it's waiting for its dependents to be deleted
I1024 14:48:18.908062       1 garbagecollector.go:408] processing item [kubevirt.io/v1alpha2/VirtualMachineInstance, namespace: default, name: fedora, uid: e99fc418-d799-11e8-a4cd-fa163e0953ea]
I1024 14:48:18.913571       1 garbagecollector.go:530] remove DeleteDependents finalizer for item [kubevirt.io/v1alpha2/VirtualMachineInstance, namespace: default, name: fedora, uid: e99fc418-d799-11e8-a4cd-fa163e0953ea]
```

As you see the "Foreground" finalizer got immediately deleted and only our "foregroundDeleteVirtualMachine" finalizer is still left there. After our controller is done and removes our finalizer the object disappears as expected.

Now with multiple times sending the delete request:

```
apiVersion: kubevirt.io/v1alpha2
kind: VirtualMachineInstance
metadata:
  creationTimestamp: 2018-10-24T14:54:41Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2018-10-24T14:54:58Z
  finalizers:
  - foregroundDeleteVirtualMachine
  - foregroundDeletion
  uid: bca97fd7-d79c-11e8-a4cd-fa163e0953ea
```

here the logs again:

```
I1024 14:54:57.368392       1 graph_builder.go:553] add [kubevirt.io/v1alpha2/VirtualMachineInstance, namespace: default, name: fedora, uid: bca97fd7-d79c-11e8-a4cd-fa163e0953ea] to the attemptToDelete, because it's waiting for its dependents to be deleted
I1024 14:54:57.368910       1 garbagecollector.go:408] processing item [kubevirt.io/v1alpha2/VirtualMachineInstance, namespace: default, name: fedora, uid: bca97fd7-d79c-11e8-a4cd-fa163e0953ea]
I1024 14:54:57.388385       1 garbagecollector.go:530] remove DeleteDependents finalizer for item [kubevirt.io/v1alpha2/VirtualMachineInstance, namespace: default, name: fedora, uid: bca97fd7-d79c-11e8-a4cd-fa163e0953ea]
```

The gc reports that it removed the finalizer but we added it with a follow-up DELETE request again and it stays tere. After our controller is done and removes its own finalizer, we still see the Foreground finalizer on the object:

```
apiVersion: kubevirt.io/v1alpha2
kind: VirtualMachineInstance
metadata:
  creationTimestamp: 2018-10-24T14:54:41Z
  deletionGracePeriodSeconds: 0
  deletionTimestamp: 2018-10-24T14:54:58Z
  finalizers:
  - foregroundDeletion
  uid: bca97fd7-d79c-11e8-a4cd-fa163e0953ea

```

The conclusion here is, that rightfully the gc immediately removes the finalizer, because we don't set an owner reference on the pod. If we do the delete request again, the finalizer gets re-added, but the gc does not process the item again.

Version-Release number of selected component (if applicable):

oc version
oc v3.10.0+dd10d17
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://cnv-executor-gouyang-master1.example.com:8443
openshift v3.11.16
kubernetes v1.11.0+d4cacc0


How reproducible:


Steps to Reproduce:
1. Create an arbitrary CRD
2. Create an arbitrary CR of that CRD
3. Set a custom finalizer
4. Do a foregreound delete
5. Wait until the foregreoundDelete finalizer disappears
6. Do another foreground delete
7. The finalizer will be back
8. Remove the custom finalizer


Actual results:

The gc will not remove the finalizer again and the CR is stuck

Expected results:

The garbage collector should remove the foreground finalizer as often as necessary.

Additional info: