Description of problem: Having ACM, GitOps Operator and the CNF-Deploy-Features deployed sucessfully and having deployed an SNO (using the hooks). Once you push the definition to the repo to delete that SNO, the hook start sync the code and ends with no actions telling you that needs prune to do deletion actions. Version-Release number of selected component (if applicable): ACM 2.3.3 Hub 4.8.5 SNO 4.8.11 How reproducible: Always Steps to Reproduce: 1. Update the SiteConfig files (deleting them) on the repo with git push 2. The hooks comes in action and the pod tells you that needs prune Actual results: No actions Expected results: Delete the cluster Additional info: clusters-app and policies-app ``` syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true ```
These are the logs when you updates the Git repo and force a SNO deletion with the --prune flag on the argocd client: [root@bastion1 gitops-verizon]# argocd app sync clusters --prune TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE 2021-09-30T08:15:44-04:00 Namespace vz-wc-lab Synced 2021-09-30T08:15:44-04:00 batch Job clusters-sub siteconfig-post Healthy 2021-09-30T08:15:44-04:00 ran.openshift.io SiteConfig vz-wc-lab vz-wc-lab Synced 2021-09-30T08:15:56-04:00 batch Job clusters-sub siteconfig-pre Progressing 2021-09-30T08:15:58-04:00 batch Job clusters-sub siteconfig-pre Running Synced PreSync job.batch/siteconfig-pre created 2021-09-30T08:16:01-04:00 Namespace vz-wc-lab Synced Progressing 2021-09-30T08:16:01-04:00 ran.openshift.io SiteConfig vz-wc-lab vz-wc-lab Synced Progressing 2021-09-30T08:16:01-04:00 ran.openshift.io SiteConfig vz-wc-lab vz-wc-lab OutOfSync Missing 2021-09-30T08:16:03-04:00 batch Job clusters-sub siteconfig-pre Succeeded Synced PreSync job.batch/siteconfig-pre created 2021-09-30T08:16:03-04:00 ran.openshift.io SiteConfig vz-wc-lab vz-wc-lab OutOfSync Missing pruned 2021-09-30T08:16:03-04:00 project.openshift.io Project vz-wc-lab Succeeded Pruned pruned 2021-09-30T08:16:03-04:00 batch Job clusters-sub siteconfig-post Progressing 2021-09-30T08:16:07-04:00 project.openshift.io Project vz-wc-lab OutOfSync Progressing pruned 2021-09-30T08:16:07-04:00 Namespace vz-wc-lab OutOfSync Missing 2021-09-30T08:16:07-04:00 project.openshift.io Project vz-wc-lab Succeeded Pruned pruned 2021-09-30T08:16:09-04:00 batch Job clusters-sub siteconfig-post Running Synced PostSync job.batch/siteconfig-post created 2021-09-30T08:16:17-04:00 ran.openshift.io SiteConfig vz-wc-lab vz-wc-lab Succeeded Pruned pruned 2021-09-30T08:16:20-04:00 batch Job clusters-sub siteconfig-post Failed Synced PostSync Job has reached the specified backoff limit Name: clusters Project: default Server: https://kubernetes.default.svc Namespace: clusters-sub URL: https://openshift-gitops-server-openshift-gitops.apps.mgmt-hub-2.hqplan.lan/applications/clusters Repo: http://dummy:dummy@bastion1.mgmt-hub-2.hqplan.lan:3000/wc-vz-faredge/vz-faregde-wcl-ztp.git Target: rfe/argo_pipeline Path: gitops-verizon/site-configs SyncWindow: Sync Allowed Sync Policy: Automated (Prune) Sync Status: Synced to rfe/argo_pipeline (2768f45) Health Status: Healthy Operation: Sync Sync Revision: 2768f45acca32f7570709faf5935f6d942781ac0 Phase: Failed Start: 2021-09-30 08:16:15 -0400 EDT Finished: 2021-09-30 08:16:51 -0400 EDT Duration: 36s Message: one or more synchronization tasks completed unsuccessfully GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE batch Job clusters-sub siteconfig-pre Succeeded PreSync job.batch/siteconfig-pre created ran.openshift.io SiteConfig vz-wc-lab vz-wc-lab Succeeded Pruned pruned project.openshift.io Project vz-wc-lab Succeeded Pruned pruned batch Job clusters-sub siteconfig-post Failed PostSync Job has reached the specified backoff limit ============================= Hook Logs ztp-hooks.postsync Thu, 30 Sep 2021 12:15:25 +0000 INFO [post-sync-entrypoint] Retrieved RAN sites resourceVersion 24987272 ztp-hooks.watcher 2021-09-30 12:15:25 UTC [DEBUG] [watcher:217]: 24987272, siteconfigs ztp-hooks.watcher 2021-09-30 12:15:30 UTC [DEBUG] [watcher:177]: {"type":"MODIFIED","object":{"apiVersion":"ran.openshift.io/v1","kind":"SiteConfig","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"ran.openshift.io/v1\",\"kind\":\"SiteConfig\",\"met$ data\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"clusters\"},\"name\":\"vz-wc-lab\",\"namespace\":\"vz-wc-lab\"},\"spec\":{\"baseDomain\":\"mgmt-hub-2.hqplan.lan\",\"clusterImageSetNameRef\":\"openshift-v4.8.11\",\"clusters\":[{\"additionalNTPSources\":[\"fd00:4888:2000:1099::11\"],\"c$ usterLabels\":{\"common\":true,\"group-du-sno\":\"\",\"infra\":\"faregde\",\"mcp\":\"master\",\"pao\":\"true\",\"server\":\"HP\",\"siteName\":\"vz-wc-lab\",\"sno\":\"true\",\"sriov\":\"true\",\"test\":\"true\"},\"clusterName\":\"sno-1\",\"clusterNetwork\":[{\"cidr\":\"fd01::/48\",\"hostPrefix\":64}],\"cluste$ Profile\":\"du\",\"clusterType\":\"sno\",\"machineNetwork\":[{\"cidr\":\"fd00:4888:2000:119c::/64\"}],\"nodes\":[{\"bmcAddress\":\"redfish-virtualmedia://[fd00:4888:2000:1181::16]/redfish/v1/Systems/1\",\"bmcCredentialsName\":{\"name\":\"sno-1-master0\"},\"bootMACAddress\":\"b4:96:91:a3:ab:94\",\"bootMode\":$ "UEFI\",\"cpuset\":\"0-2,32-34\",\"hostName\":\"master-0.sno.hqlan.lan\",\"ignitionConfigOverride\":\"{\\\"ignition\\\":{\\\"version\\\":\\\"3.2.0\\\"},\\\"storage\\\":{\\\"disks\\\":[{\\\"device\\\":\\\"/dev/nvme1n1\\\",\\\"wipeTable\\\":true,\\\"partitions\\\":[{\\\"label\\\":\\\"part1\\\",\\\"number\\\":1$ \\\"sizeMiB\\\":409600,\\\"startMiB\\\":0},{\\\"label\\\":\\\"part2\\\",\\\"number\\\":2,\\\"sizeMiB\\\":409600,\\\"startMiB\\\":409601},{\\\"label\\\":\\\"part3\\\",\\\"number\\\":3,\\\"sizeMiB\\\":409600,\\\"startMiB\\\":819202},{\\\"label\\\":\\\"part4\\\",\\\"number\\\":4,\\\"sizeMiB\\\":409600,\\\"start$ iB\\\":1228803}]}]}}\",\"nodeNetwork\":{\"config\":{\"dns-resolver\":{\"config\":{\"server\":[\"fd00:4888:2000:1099::11\"]}},\"interfaces\":[{\"ipv6\":{\"address\":[{\"ip\":\"fd00:4888:2000:119c::100\",\"prefix-length\":64}],\"dhcp\":false,\"enabled\":true},\"name\":\"ens1f0.304\",\"state\":\"up\",\"type\":\$ vlan\",\"vlan\":{\"base-iface\":\"ens1f0\",\"id\":304}}],\"routes\":{\"config\":[{\"destination\":\"::/0\",\"next-hop-address\":\"fd00:4888:2000:119c::\",\"next-hop-interface\":\"ens1f0.304\"}]}},\"interfaces\":[{\"macAddress\":\"b4:96:91:a3:ab:94\",\"name\":\"ens1f0\"}]},\"rootDeviceHints\":{\"deviceName\":$ "/dev/nvme0n1\"}}],\"serviceNetwork\":[\"fd02::/112\"]}],\"pullSecretRef\":{\"name\":\"assisted-deployment-pull-secret-sno-1\"},\"sshPublicKey\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC26dCKccwEuxyYmStKzvRrWOjrBvpRbseiixKWAUJ/ZPulXjIR6oGZZNd7h/w6Rg0ETakgvSQRJjoMfsf49EXUawxAwKcnDmMXBytkLnsawAk+1raaKjrM27a46f$ jFlcNDjuNMwgvE9PHTX3yUfEXqenJIhwHdod1w2GNE2jsKyZmqOF8Fey1Y/YIl/zK7STCSZn0+nc2iQx08NYsx/s/aaMgzr9Zggoq/9dD1BMljswFcu8YHhzVyLWu5TY0m4PbzMJ8QjPxkfWXz3hYQMbCS20gdymskgoMGluDBYsqiZDBd933ff9K+yPwNdov7ZSoXuGKwWfSHPXfIm+L3dN2lNblz3gr2ncDcflNxpGiGbUwHEai/k5fxf8SiL6HydCSfsdtHB9hyxu14CNI5f+3QyRcO4q50251r2bPWzKiO7tpqXXS$ GSINc5SWZcp/ohb4eVvpRUwjWc2Uhezo/igkoc+rNmxpJG6VEo0BlOBE3p5ByN/II32M/VG/ReNBXE= root.lan\"}}\n"},"creationTimestamp":"2021-09-30T08:49:03Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2021-09-30T12:15:12Z","finalizers":["foregroundDeletion"],"generation":2,"labels":{"app.kubernetes.io$ instance":"clusters"},"managedFields":[{"apiVersion":"ran.openshift.io/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{}}},"f:spec":{".":{},"f:baseDomain":{},"f:clusterIma$ eSetNameRef":{},"f:clusters":{},"f:pullSecretRef":{".":{},"f:name":{}},"f:sshPublicKey":{}}},"manager":"argocd-application-controller","operation":"Update","time":"2021-09-30T08:49:03Z"}],"name":"vz-wc-lab","namespace":"vz-wc-lab","resourceVersion":"24987304","uid":"b9f23f9b-135b-4db4-b7cd-09a4be899192"},"sp$ c":{"baseDomain":"mgmt-hub-2.hqplan.lan","clusterImageSetNameRef":"openshift-v4.8.11","clusters":[{"additionalNTPSources":["fd00:4888:2000:1099::11"],"clusterLabels":{"common":true,"group-du-sno":"","infra":"faregde","mcp":"master","pao":"true","server":"HP","siteName":"vz-wc-lab","sno":"true","sriov":"true"$ "test":"true"},"clusterName":"sno-1","clusterNetwork":[{"cidr":"fd01::/48","hostPrefix":64}],"clusterProfile":"du","clusterType":"sno","machineNetwork":[{"cidr":"fd00:4888:2000:119c::/64"}],"nodes":[{"bmcAddress":"redfish-virtualmedia://[fd00:4888:2000:1181::16]/redfish/v1/Systems/1","bmcCredentialsName":{"n$ me":"sno-1-master0"},"bootMACAddress":"b4:96:91:a3:ab:94","bootMode":"UEFI","cpuset":"0-2,32-34","hostName":"master-0.sno.hqlan.lan","ignitionConfigOverride":"{\"ignition\":{\"version\":\"3.2.0\"},\"storage\":{\"disks\":[{\"device\":\"/dev/nvme1n1\",\"wipeTable\":true,\"partitions\":[{\"label\":\"part1\",\"n$ mber\":1,\"sizeMiB\":409600,\"startMiB\":0},{\"label\":\"part2\",\"number\":2,\"sizeMiB\":409600,\"startMiB\":409601},{\"label\":\"part3\",\"number\":3,\"sizeMiB\":409600,\"startMiB\":819202},{\"label\":\"part4\",\"number\":4,\"sizeMiB\":409600,\"startMiB\":1228803}]}]}}","nodeNetwork":{"config":{"dns-resolv$ r":{"config":{"server":["fd00:4888:2000:1099::11"]}},"interfaces":[{"ipv6":{"address":[{"ip":"fd00:4888:2000:119c::100","prefix-length":64}],"dhcp":false,"enabled":true},"name":"ens1f0.304","state":"up","type":"vlan","vlan":{"base-iface":"ens1f0","id":304}}],"routes":{"config":[{"destination":"::/0","next-ho$ -address":"fd00:4888:2000:119c::","next-hop-interface":"ens1f0.304"}]}},"interfaces":[{"macAddress":"b4:96:91:a3:ab:94","name":"ens1f0"}]},"role":"master","rootDeviceHints":{"deviceName":"/dev/nvme0n1"}}],"numMasters":1,"serviceNetwork":["fd02::/112"]}],"pullSecretRef":{"name":"assisted-deployment-pull-secre$ -sno-1"},"sshPublicKey":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC26dCKccwEuxyYmStKzvRrWOjrBvpRbseiixKWAUJ/ZPulXjIR6oGZZNd7h/w6Rg0ETakgvSQRJjoMfsf49EXUawxAwKcnDmMXBytkLnsawAk+1raaKjrM27a46fEjFlcNDjuNMwgvE9PHTX3yUfEXqenJIhwHdod1w2GNE2jsKyZmqOF8Fey1Y/YIl/zK7STCSZn0+nc2iQx08NYsx/s/aaMgzr9Zggoq/9dD1BMljswFcu8YHhzV$ LWu5TY0m4PbzMJ8QjPxkfWXz3hYQMbCS20gdymskgoMGluDBYsqiZDBd933ff9K+yPwNdov7ZSoXuGKwWfSHPXfIm+L3dN2lNblz3gr2ncDcflNxpGiGbUwHEai/k5fxf8SiL6HydCSfsdtHB9hyxu14CNI5f+3QyRcO4q50251r2bPWzKiO7tpqXXStGSINc5SWZcp/ohb4eVvpRUwjWc2Uhezo/igkoc+rNmxpJG6VEo0BlOBE3p5ByN/II32M/VG/ReNBXE= root.lan"}}} ztp-hooks.watcher 2021-09-30 12:15:30 UTC [DEBUG] [watcher:177]: {"type":"DELETED","object":{"apiVersion":"ran.openshift.io/v1","kind":"SiteConfig","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"ran.openshift.io/v1\",\"kind\":\"SiteConfig\",\"meta$ ata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"clusters\"},\"name\":\"vz-wc-lab\",\"namespace\":\"vz-wc-lab\"},\"spec\":{\"baseDomain\":\"mgmt-hub-2.hqplan.lan\",\"clusterImageSetNameRef\":\"openshift-v4.8.11\",\"clusters\":[{\"additionalNTPSources\":[\"fd00:4888:2000:1099::11\"],\"cl$ sterLabels\":{\"common\":true,\"group-du-sno\":\"\",\"infra\":\"faregde\",\"mcp\":\"master\",\"pao\":\"true\",\"server\":\"HP\",\"siteName\":\"vz-wc-lab\",\"sno\":\"true\",\"sriov\":\"true\",\"test\":\"true\"},\"clusterName\":\"sno-1\",\"clusterNetwork\":[{\"cidr\":\"fd01::/48\",\"hostPrefix\":64}],\"cluster$ rofile\":\"du\",\"clusterType\":\"sno\",\"machineNetwork\":[{\"cidr\":\"fd00:4888:2000:119c::/64\"}],\"nodes\":[{\"bmcAddress\":\"redfish-virtualmedia://[fd00:4888:2000:1181::16]/redfish/v1/Systems/1\",\"bmcCredentialsName\":{\"name\":\"sno-1-master0\"},\"bootMACAddress\":\"b4:96:91:a3:ab:94\",\"bootMode\":\$ UEFI\",\"cpuset\":\"0-2,32-34\",\"hostName\":\"master-0.sno.hqlan.lan\",\"ignitionConfigOverride\":\"{\\\"ignition\\\":{\\\"version\\\":\\\"3.2.0\\\"},\\\"storage\\\":{\\\"disks\\\":[{\\\"device\\\":\\\"/dev/nvme1n1\\\",\\\"wipeTable\\\":true,\\\"partitions\\\":[{\\\"label\\\":\\\"part1\\\",\\\"number\\\":1,$ \\"sizeMiB\\\":409600,\\\"startMiB\\\":0},{\\\"label\\\":\\\"part2\\\",\\\"number\\\":2,\\\"sizeMiB\\\":409600,\\\"startMiB\\\":409601},{\\\"label\\\":\\\"part3\\\",\\\"number\\\":3,\\\"sizeMiB\\\":409600,\\\"startMiB\\\":819202},{\\\"label\\\":\\\"part4\\\",\\\"number\\\":4,\\\"sizeMiB\\\":409600,\\\"startM$ B\\\":1228803}]}]}}\",\"nodeNetwork\":{\"config\":{\"dns-resolver\":{\"config\":{\"server\":[\"fd00:4888:2000:1099::11\"]}},\"interfaces\":[{\"ipv6\":{\"address\":[{\"ip\":\"fd00:4888:2000:119c::100\",\"prefix-length\":64}],\"dhcp\":false,\"enabled\":true},\"name\":\"ens1f0.304\",\"state\":\"up\",\"type\":\"v lan\",\"vlan\":{\"base-iface\":\"ens1f0\",\"id\":304}}],\"routes\":{\"config\":[{\"destination\":\"::/0\",\"next-hop-address\":\"fd00:4888:2000:119c::\",\"next-hop-interface\":\"ens1f0.304\"}]}},\"interfaces\":[{\"macAddress\":\"b4:96:91:a3:ab:94\",\"name\":\"ens1f0\"}]},\"rootDeviceHints\":{\"deviceName\":\" /dev/nvme0n1\"}}],\"serviceNetwork\":[\"fd02::/112\"]}],\"pullSecretRef\":{\"name\":\"assisted-deployment-pull-secret-sno-1\"},\"sshPublicKey\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC26dCKccwEuxyYmStKzvRrWOjrBvpRbseiixKWAUJ/ZPulXjIR6oGZZNd7h/w6Rg0ETakgvSQRJjoMfsf49EXUawxAwKcnDmMXBytkLnsawAk+1raaKjrM27a46fEj FlcNDjuNMwgvE9PHTX3yUfEXqenJIhwHdod1w2GNE2jsKyZmqOF8Fey1Y/YIl/zK7STCSZn0+nc2iQx08NYsx/s/aaMgzr9Zggoq/9dD1BMljswFcu8YHhzVyLWu5TY0m4PbzMJ8QjPxkfWXz3hYQMbCS20gdymskgoMGluDBYsqiZDBd933ff9K+yPwNdov7ZSoXuGKwWfSHPXfIm+L3dN2lNblz3gr2ncDcflNxpGiGbUwHEai/k5fxf8SiL6HydCSfsdtHB9hyxu14CNI5f+3QyRcO4q50251r2bPWzKiO7tpqXXStG SINc5SWZcp/ohb4eVvpRUwjWc2Uhezo/igkoc+rNmxpJG6VEo0BlOBE3p5ByN/II32M/VG/ReNBXE= root.lan\"}}\n"},"creationTimestamp":"2021-09-30T08:49:03Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2021-09-30T12:15:12Z","finalizers":["foregroundDeletion"],"generation":2,"labels":{"app.kubernetes.io/i nstance":"clusters"},"managedFields":[{"apiVersion":"ran.openshift.io/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{}}},"f:spec":{".":{},"f:baseDomain":{},"f:clusterImage SetNameRef":{},"f:clusters":{},"f:pullSecretRef":{".":{},"f:name":{}},"f:sshPublicKey":{}}},"manager":"argocd-application-controller","operation":"Update","time":"2021-09-30T08:49:03Z"}],"name":"vz-wc-lab","namespace":"vz-wc-lab","resourceVersion":"24987309","uid":"b9f23f9b-135b-4db4-b7cd-09a4be899192"},"spec ":{"baseDomain":"mgmt-hub-2.hqplan.lan","clusterImageSetNameRef":"openshift-v4.8.11","clusters":[{"additionalNTPSources":["fd00:4888:2000:1099::11"],"clusterLabels":{"common":true,"group-du-sno":"","infra":"faregde","mcp":"master","pao":"true","server":"HP","siteName":"vz-wc-lab","sno":"true","sriov":"true"," test":"true"},"clusterName":"sno-1","clusterNetwork":[{"cidr":"fd01::/48","hostPrefix":64}],"clusterProfile":"du","clusterType":"sno","machineNetwork":[{"cidr":"fd00:4888:2000:119c::/64"}],"nodes":[{"bmcAddress":"redfish-virtualmedia://[fd00:4888:2000:1181::16]/redfish/v1/Systems/1","bmcCredentialsName":{"nam e":"sno-1-master0"},"bootMACAddress":"b4:96:91:a3:ab:94","bootMode":"UEFI","cpuset":"0-2,32-34","hostName":"master-0.sno.hqlan.lan","ignitionConfigOverride":"{\"ignition\":{\"version\":\"3.2.0\"},\"storage\":{\"disks\":[{\"device\":\"/dev/nvme1n1\",\"wipeTable\":true,\"partitions\":[{\"label\":\"part1\",\"num ber\":1,\"sizeMiB\":409600,\"startMiB\":0},{\"label\":\"part2\",\"number\":2,\"sizeMiB\":409600,\"startMiB\":409601},{\"label\":\"part3\",\"number\":3,\"sizeMiB\":409600,\"startMiB\":819202},{\"label\":\"part4\",\"number\":4,\"sizeMiB\":409600,\"startMiB\":1228803}]}]}}","nodeNetwork":{"config":{"dns-resolver ":{"config":{"server":["fd00:4888:2000:1099::11"]}},"interfaces":[{"ipv6":{"address":[{"ip":"fd00:4888:2000:119c::100","prefix-length":64}],"dhcp":false,"enabled":true},"name":"ens1f0.304","state":"up","type":"vlan","vlan":{"base-iface":"ens1f0","id":304}}],"routes":{"config":[{"destination":"::/0","next-hop- address":"fd00:4888:2000:119c::","next-hop-interface":"ens1f0.304"}]}},"interfaces":[{"macAddress":"b4:96:91:a3:ab:94","name":"ens1f0"}]},"role":"master","rootDeviceHints":{"deviceName":"/dev/nvme0n1"}}],"numMasters":1,"serviceNetwork":["fd02::/112"]}],"pullSecretRef":{"name":"assisted-deployment-pull-secret- sno-1"},"sshPublicKey":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC26dCKccwEuxyYmStKzvRrWOjrBvpRbseiixKWAUJ/ZPulXjIR6oGZZNd7h/w6Rg0ETakgvSQRJjoMfsf49EXUawxAwKcnDmMXBytkLnsawAk+1raaKjrM27a46fEjFlcNDjuNMwgvE9PHTX3yUfEXqenJIhwHdod1w2GNE2jsKyZmqOF8Fey1Y/YIl/zK7STCSZn0+nc2iQx08NYsx/s/aaMgzr9Zggoq/9dD1BMljswFcu8YHhzVyL Wu5TY0m4PbzMJ8QjPxkfWXz3hYQMbCS20gdymskgoMGluDBYsqiZDBd933ff9K+yPwNdov7ZSoXuGKwWfSHPXfIm+L3dN2lNblz3gr2ncDcflNxpGiGbUwHEai/k5fxf8SiL6HydCSfsdtHB9hyxu14CNI5f+3QyRcO4q50251r2bPWzKiO7tpqXXStGSINc5SWZcp/ohb4eVvpRUwjWc2Uhezo/igkoc+rNmxpJG6VEo0BlOBE3p5ByN/II32M/VG/ReNBXE= root.lan"}}} ztp-hooks.watcher 2021-09-30 12:15:30 UTC [DEBUG] [watcher:135]: Objects to delete are: ['vz-wc-lab'] ztp-hooks.watcher 2021-09-30 12:15:30 UTC [DEBUG] [watcher:137]: Objects to create/update are: ['vz-wc-lab'] ztp-hooks.watcher 2021-09-30 12:15:30 UTC [DEBUG] [watcher:95]: ['oc', 'apply', '-f', '/tmp/tmp321yvchy/update/customResource/vz-wc-lab/sno-1.yaml'] ztp-hooks.watcher 2021-09-30 12:15:31 UTC [DEBUG] [watcher:107]: OC wrapper error: --- apiVersion: v1 kind: Namespace metadata: labels: name: sno-1 name: sno-1 --- apiVersion: extensions.hive.openshift.io/v1beta1 kind: AgentClusterInstall ... ... ... ztp-hooks.watcher 2021-09-30 12:15:31 UTC [ERROR] [watcher:108]: namespace/sno-1 unchanged agentclusterinstall.extensions.hive.openshift.io/sno-1 unchanged nmstateconfig.agent-install.openshift.io/sno-1 unchanged klusterletaddonconfig.agent.open-cluster-management.io/sno-1 unchanged managedcluster.cluster.open-cluster-management.io/sno-1 unchanged infraenv.agent-install.openshift.io/sno-1 configured baremetalhost.metal3.io/sno-1 unchanged configmap/sno-1 unchanged The ClusterDeployment "sno-1" is invalid: spec.installed: Invalid value: false: cannot make uninstalled once installed Traceback (most recent call last): File "watcher.py", line 100, in __init__ check=True) File "/usr/lib64/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['oc', 'apply', '-f', '/tmp/tmp321yvchy/update/customResource/vz-wc-lab/sno-1.yaml']' returned non-zero exit status 1. ztp-hooks.watcher 2021-09-30 12:15:31 UTC [ERROR] [watcher:160]: Exception by ApiResponseParser: Failed to apply target manifests Traceback (most recent call last): File "watcher.py", line 100, in __init__ check=True) File "/usr/lib64/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['oc', 'apply', '-f', '/tmp/tmp321yvchy/update/customResource/vz-wc-lab/sno-1.yaml']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "watcher.py", line 148, in __init__ OcWrapper('apply', out_upd_path) File "watcher.py", line 109, in __init__ raise Exception(f"Failed to {action} target manifests") Exception: Failed to apply target manifests
If you executes the argocd client again you can see this: [root@bastion1 gitops-verizon]# argocd app sync clusters --prune TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE 2021-09-30T08:24:37-04:00 batch Job clusters-sub siteconfig-post Degraded 2021-09-30T08:24:49-04:00 batch Job clusters-sub siteconfig-pre Progressing 2021-09-30T08:24:51-04:00 batch Job clusters-sub siteconfig-pre Running Synced PreSync job.batch/siteconfig-pre created 2021-09-30T08:24:53-04:00 batch Job clusters-sub siteconfig-pre Succeeded Synced PreSync job.batch/siteconfig-pre created 2021-09-30T08:24:53-04:00 batch Job clusters-sub siteconfig-post Progressing 2021-09-30T08:24:59-04:00 batch Job clusters-sub siteconfig-post Running Synced PostSync job.batch/siteconfig-post created 2021-09-30T08:25:10-04:00 batch Job clusters-sub siteconfig-post Succeeded Synced PostSync job.batch/siteconfig-post created Name: clusters Project: default Server: https://kubernetes.default.svc Namespace: clusters-sub URL: https://openshift-gitops-server-openshift-gitops.apps.mgmt-hub-2.hqplan.lan/applications/clusters Repo: http://dummy:dummy@bastion1.mgmt-hub-2.hqplan.lan:3000/wc-vz-faredge/vz-faregde-wcl-ztp.git Target: rfe/argo_pipeline Path: gitops-verizon/site-configs SyncWindow: Sync Allowed Sync Policy: Automated (Prune) Sync Status: Synced to rfe/argo_pipeline (2768f45) Health Status: Healthy Operation: Sync Sync Revision: 2768f45acca32f7570709faf5935f6d942781ac0 Phase: Succeeded Start: 2021-09-30 08:25:08 -0400 EDT Finished: 2021-09-30 08:25:41 -0400 EDT Duration: 33s Message: successfully synced (no more tasks) GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE batch Job clusters-sub siteconfig-pre Succeeded PreSync job.batch/siteconfig-pre created batch Job clusters-sub siteconfig-post Succeeded PostSync job.batch/siteconfig-post created And this is the logs on the hook: [root@bastion1 gitops-verizon]# oc logs -n clusters-sub siteconfig-post-gk945 ztp-hooks.postsync Thu, 30 Sep 2021 12:24:15 +0000 INFO [post-sync-entrypoint] Retrieved RAN sites resourceVersion 24998948 ztp-hooks.watcher 2021-09-30 12:24:16 UTC [DEBUG] [watcher:217]: 24998948, siteconfigs ztp-hooks.watcher 2021-09-30 12:24:21 UTC [DEBUG] [watcher:135]: Objects to delete are: [] ztp-hooks.watcher 2021-09-30 12:24:21 UTC [DEBUG] [watcher:137]: Objects to create/update are: [] ztp-hooks.watcher 2021-09-30 12:24:21 UTC [DEBUG] [watcher:150]: No objects to update ztp-hooks.watcher 2021-09-30 12:24:21 UTC [DEBUG] [watcher:157]: No objects to delete
Note in the error log: The ClusterDeployment "sno-1" is invalid: spec.installed: Invalid value: false: cannot make uninstalled once installed This error was fixed by PR 720
Given the PR above was to master, this was tested against a 4.10 deployment. The siteconfig for a deployed test node was deleted and git pushed. The ArgoCD UI was monitored, and the node was eventually shown as needing pruning. A refresh was initiated with prune selected as an option, and the node was removed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.9.15 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0110