Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2065713

Summary: ZTP sync failure leaves lock in place, subsequent syncs hang waiting to obtain lock
Product: OpenShift Container Platform Reporter: Ian Miller <imiller>
Component: Telco EdgeAssignee: Vitaly Grinberg <vgrinber>
Telco Edge sub component: ZTP QA Contact: yliu1
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: unspecified CC: jgato, mcornea, shreepat, vgrinber
Version: 4.9   
Target Milestone: ---   
Target Release: 4.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2066102 (view as bug list) Environment:
Last Closed: 2022-04-11 17:20:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2066102    
Bug Blocks:    

Description Ian Miller 2022-03-18 14:54:39 UTC 
Description of problem:
If the argocd sync fails the mutex lock put in place by the pre-sync-entrypoint remains in place even though the argocd sync is completed. This causes all subsequent sync processes to hang waiting for the lock

Version-Release number of selected component (if applicable): 4.9.0-42


How reproducible: 100%


Steps to Reproduce:
1. Ensure hub cluster does not have namespace "test123"
2. Create SiteConfig CR with namepaces "test123"
3. Push new SiteConfig CR to git
4. Pre-sync hook runs OK, sync fails with "cannot apply one or more resources"
5. All subsequent syncs will cause pre-sync hook to hang "waiting for lock"

Actual results: 
ConfigMap openshift-ztp-lock in clusters-sub namespace remains defined when argocd sync fails.


Expected results:
openshift-ztp-lock configmap gets deleted when sync phase is complete (even in failure)

Additional info:
Workaround: oc delete -n clusters-sub openshift-ztp-lock
Comment 1 Marius Cornea 2022-03-28 08:42:52 UTC 
*** Bug 2069066 has been marked as a duplicate of this bug. ***
Comment 2 Ian Miller 2022-03-28 16:32:59 UTC 
Issue can be resolved by updating the post-sync.yaml hooks within the monitored git repository (one each in the directories containing SiteConfig and PolicyGenTemplate files). The argocd.argoproj.io/hook annotation needs to be updated to include SyncFail:

    argocd.argoproj.io/hook: PostSync,SyncFail
Comment 5 yliu1 2022-04-04 19:55:46 UTC 
Verified in release-4.9 branch. And added additional configs in PGT to verify it after a failed sync.
Comment 7 errata-xmlrpc 2022-04-11 17:20:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (4.9 27 Stream bugfixes for Telco RAN), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1219
Comment 8 Jose Gato 2022-04-20 15:08:01 UTC 
(In reply to Ian Miller from comment #2)
> Issue can be resolved by updating the post-sync.yaml hooks within the
> monitored git repository (one each in the directories containing SiteConfig
> and PolicyGenTemplate files). The argocd.argoproj.io/hook annotation needs
> to be updated to include SyncFail:
> 
>     argocd.argoproj.io/hook: PostSync,SyncFail

Once the bug has been fixed, is still need it to add the SyncFail?
Comment 9 Ian Miller 2022-04-20 16:45:32 UTC 
Yes. The fix for this issue is to trigger the post-sync logic when the argocd sync fails. To do this the SyncFail must be added.
Comment 10 shreepat 2022-05-02 20:59:18 UTC 
Hit This issue in Test harness:

OC 4.10

SNO 4.9.29 and 4.9.25

ZTP 4.9.42 (registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.9.0-42)


ztp-hooks.presync Mon, 02 May 2022 20:56:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
1
ztp-hooks.presync Mon, 02 May 2022 20:57:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
Comment 11 Ian Miller 2022-05-02 21:04:01 UTC 
Can you please provide the "post-sync.yaml" file from the git repository which is being synchronized?
Comment 12 shreepat 2022-05-02 22:25:18 UTC 
hreepat@infra1 site-policies]$ cat post-sync.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: policygentemplates-post
  namespace: policies-sub
  generateName: post-sync-
  annotations:
    argocd.argoproj.io/hook: PostSync,SyncFail
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
        - name: ztp-site-generator
          image: registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.9.0-42
          command: ["/bin/bash", "post-sync-entrypoint.sh", "policygentemplates"]
      restartPolicy: Never
  backoffLimit: 0
  # ttlSecondsAfterFinished: 60
Comment 13 shreepat 2022-05-02 22:26:53 UTC 
the workaround not working 

a) oc delete -n clusters-sub openshift-ztp-lock

or 

post-sync
  annotations:
    argocd.argoproj.io/hook: PostSync,SyncFail

ztp-hooks.presync Mon, 02 May 2022 21:30:55 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:31:55 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:32:55 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:33:55 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:34:55 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:35:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:36:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:37:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:38:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:39:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:40:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:41:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:42:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:43:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:44:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:45:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:46:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:47:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:48:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:49:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:50:56 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:51:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:52:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:53:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:54:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:55:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:56:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:57:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:58:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 21:59:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:00:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:01:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:02:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:03:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:04:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:05:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:06:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:07:57 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:08:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:09:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:10:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:11:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:12:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:13:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:14:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:15:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:16:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:17:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:18:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:19:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:20:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:21:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:22:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:23:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:24:58 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
ztp-hooks.presync Mon, 02 May 2022 22:25:59 +0000 INFO [pre-sync-entrypoint] Waiting to acquire sync lock
 ^C
[kni@infra1 ~]$ oc get pods
NAME                           READY   STATUS    RESTARTS   AGE
policygentemplates-pre-tjh4c   1/1     Running   0          70m
Comment 14 Vitaly Grinberg 2022-05-03 07:36:59 UTC 
(In reply to shreepat from comment #12, #13)

> hreepat@infra1 site-policies]$ cat post-sync.yaml
> apiVersion: batch/v1
> kind: Job
> metadata:
>  name: policygentemplates-post
>  namespace: policies-sub
>  generateName: post-sync-
>  annotations:
>    argocd.argoproj.io/hook: PostSync,SyncFail
------------------------------------
> the workaround not working 
> 
> a) oc delete -n clusters-sub openshift-ztp-lock
> 
> or 
> 
> post-sync
>   annotations:
>     argocd.argoproj.io/hook: PostSync,SyncFail

Hi, it seems the lock was deleted in the "clusters-sub" namespace, but the pod waiting on the lock is in the policies-sub namespace.