Description of problem: Using LSO we have multiple available PVs, when we bound a PVC to a PV for the first time everything works fine. When we delete the workload (deployments + pvcs) we see that the PVs are recreated multiple times by the diskmaker. This is causing two issues: 1. The PVs will be created and deleted multiple times, at some point the PV will be available before being deleted and if there are PVCs requesting storage they will bound to that PV that will be deleted by diskmaker causing the PV to move into "Terminating" status. 2. If you delete the workload after issue nº1 happened, the PVs will likely get stuck in "Released" state. We are seeing PVs being recreated >5 times. Version-Release number of selected component (if applicable): 4.8.13 How reproducible: Always Steps to Reproduce: Using reproducer here https://gist.github.com/mvazquezc/0dd0dabb673a6d822cd8fa8ccdefe9e0 Issue 1: 1. Run deploy.sh and wait until all pods are running 2. Run clean.sh and wait until all pods are gone 3. Check PVs and you will see that diskmaker is continuously recreating them Issue 2: 1. Run deploy.sh and wait until all pods are running 2. Run clean.sh and wait until all pods are gone 3. Check PVs and you will see that diskmaker is continuously recreating them 4. Run deploy.sh and wait until all pods are running 5. Check PVs, you will see some in "Terminating" state 6. Run clean.sh and wait until all pods are gone 7. You will see PVs stuck in "Released" state Actual results: PVs are continuously re-created PVs are stuck in Released state Expected results: PVs are re-created once (Delete reclaim policy) PVs are not stuck in Released state Additional info: I have an environment where I can reproduce the issue 100% of the time, ping me if you want to access it / have a session.
The symptoms look a bit different, but should be fixed by #2008088, which should land in 4.8 soon-ish. *** This bug has been marked as a duplicate of bug 2008088 ***
Hey @Jan, it's not fixed by #2008088. This was actually reproduced using the code that fixes #2008088. Could you re-open?
I was able to reproduce it with today's 4.8 LSO, I saw PVs being created and immediately deleted. With LSO from current 4.8.z errata I was not able to reproduce the issue. I am very interested in diskmaker logs then.
Friendly reminder: we're still waiting for diskmaker logs (from a build with #2008088 fixed).
Hey @jsafrane apologies for the delay in the response. I just attached a tar.gz file with three log files: - diskmakerlogs-before-workload.log -> I deleted the diskmaker when workload was not deployed and PVs where available, this are the logs at that point - diskmakerlogs-after-workload-before-cleanup.log -> Diskmaker logs after all workload pods were running and PVs were bound to their respective pvcs. - diskmakerlogs-after-workload-after-cleanup.log -> Diskmaker logs after workload was removed from the cluster and after PVs stopped being re-created (It took around 4m30s for diskmaker to stop re-creating PVs). One of the PVs that got re-created several times was local-pv-a154f0f1. I used the CatalogSource "quay.io/gnufied/gnufied-index:oct4-1120" which iiuc was the dev build that QE used to verify the fix on 4.8 (diskmaker image: quay.io/gnufied/local-diskmaker:oct4-1120) I'm going to test now with LSO from 4.8.14 and will update the BZ.
Tested again with the latest LSO from 4.8.14: local-storage-operator.4.8.0-202110011559 Local Storage 4.8.0-202110011559 Succeeded I was able to reproduce the issue. Attached is a new tar.gz with the same logs.
Tested with the latest LSO from 4.8.15: local-storage-operator.4.8.0-202110121407 Local Storage 4.8.0-202110121407 local-storage-operator.4.8.0-202110011559 Succeeded The issue cannot be reproduced. Closing now.