Created attachment 1855951 [details] openshift-gitops-repo-server-656fbcf646-fqf6t.log Description of problem: ZTP gitops apps report errors and do not recover when openshift-gitops-operator.v1.4.1 is used. Both clusters and policies apps report: rpc error: code = Unknown desc = `kustomize build /tmp/http___registry.kni-qe-0.lab.eng.rdu2.redhat.com_3000_kni-qe_ztp-site-configs/policygentemplates --enable-alpha-plugins` failed exit status 1: Error: loading generator plugins: unable to find plugin root - tried: ('<no value>'; homed in $KUSTOMIZE_PLUGIN_HOME), ('<no value>'; homed in $XDG_CONFIG_HOME), ('/.config/kustomize/plugin'; homed in default value of $XDG_CONFIG_HOME), ('/kustomize/plugin'; homed in home directory) Version-Release number of selected component (if applicable): OCP 4.10.0-0.nightly-2022-01-25-023600 openshift-gitops-operator.v1.4.1 cnf-features-deploy bf23d6508ca085ee414c8b7d60d4ebc897fd36f5 How reproducible: 100% Steps to Reproduce: 1. Create clusters and policies ZTP apps on a hub cluster with openshift-gitops-operator.v1.4.1 installed Actual results: clusters and policies apps report errors and do not recover as in BZ#2036729 rpc error: code = Unknown desc = `kustomize build /tmp/http___registry.kni-qe-0.lab.eng.rdu2.redhat.com_3000_kni-qe_ztp-site-configs/policygentemplates --enable-alpha-plugins` failed exit status 1: Error: loading generator plugins: unable to find plugin root - tried: ('<no value>'; homed in $KUSTOMIZE_PLUGIN_HOME), ('<no value>'; homed in $XDG_CONFIG_HOME), ('/.config/kustomize/plugin'; homed in default value of $XDG_CONFIG_HOME), ('/kustomize/plugin'; homed in home directory) Expected results: Apps do not report errors and proceed with cluster installation/configuration. Additional info: Attaching openshift-gitops-application-controller-0 and openshift-gitops-repo-server-656fbcf646-fqf6t logs.
Created attachment 1855970 [details] openshift-gitops-application-controller-0.log
Note: with openshift-gitops-operator.v1.3.2 the same error manifests when the apps are created for the first time but they eventually recover after 5 minutes or after trigerring Hard Refresh as described in BZ#2036729.
The current workaround is to pin the operator to v1.3.2. This CR can be used to install v1.3.2 of the operator: apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-gitops-operator #namespace: openshift-gitops namespace: openshift-operators spec: name: openshift-gitops-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: openshift-gitops-operator.v1.3.2 installPlanApproval: Manual You may need to manually approve the installplan created for the subscription: $ oc get installplan -n openshift-operators NAME CSV APPROVAL APPROVED install-7wdfv openshift-gitops-operator.v1.3.2 Manual false Note the name of the install plan will vary, but you are looking for the one with CSV v1.3.2 oc edit installplan -n openshift-operators install-7wdfv Set the "approved" field to true: spec: approval: Manual approved: true clusterServiceVersionNames: - openshift-gitops-operator.v1.3.2
Could we please get the fix backported to release-4.10 branch?
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days