Bug 1906428

Summary: [release-4.6]: When using webhooks in OCP 4.5 fails to rollout latest deploymentconfig
Product: OpenShift Container Platform Reporter: Lukasz Szaszkiewicz <lszaszki>
Component: openshift-apiserverAssignee: Lukasz Szaszkiewicz <lszaszki>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.6CC: akashem, akhaire, antonio.quintavalle, aos-bugs, bleanhar, hgomes, lszaszki, mfojtik, miminar, mtleilia, openshift-bugzilla-robot, rgangwar, rgregory, skrenger, sparpate, sttts, vjaypurk, xxia
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Specifies the correct version for the DeploymentConfig. Previously all requests targeting "deploymentconfigs/{name}/instantiate" subresource failed with "no kind DeploymentConfig is registered for version apps.openshift.io/"
Story Points: ---
Clone Of: 1867380
: 1906429 (view as bug list) Environment:
Last Closed: 2021-01-18 17:59:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1867380    
Bug Blocks: 1906429, 1909870    

Comment 1 Lukasz Szaszkiewicz 2020-12-11 08:27:22 UTC
*** Bug 1906640 has been marked as a duplicate of this bug. ***

Comment 2 Xingxing Xia 2020-12-18 04:54:48 UTC
I'm following JIRA issue DPTP-660 to do the pre-merge verification. Used the cluster-bot launching an env with the still open but Dev-approved PR(s).
The admission-controller.yaml and webhook-configuration.yaml files are copied from https://www.openpolicyagent.org/docs/v0.12.2/kubernetes-admission-control/
For admission-controller.yaml:
Directly creating it, the opa pod will be CrashLoopBackOff for "opa" container with pod logs that show err in binding 443 port. So I must add securityContext field:
         - name: opa
           image: openpolicyagent/opa:0.12.2
           securityContext:
             privileged: true
             runAsUser: 0

I must also add SCC (see the step below).

For webhook-configuration.yaml:
I must make some changes (see the step below) otherwise its creation not hit failure, e.g.:
ValidatingWebhookConfiguration is promoted to v1 from v1beta1, so I changed to v1;
And v1 requires sideEffects and admissionReviewVersions must be not empty, so I added them;
And I modified with "operator: In", and "scope: 'Namespaced'", because original link defines too destructively;
And lalbel the test namespace to ensure only the test namespace is affected, and other namespaces in the cluster are not destructed.

Verification steps:
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
cat >server.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF

openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf

oc create namespace opa
oc project opa
oc create secret tls opa-server --cert=server.crt --key=server.key
oc adm policy add-scc-to-user privileged -z default
oc apply -f admission-controller.yaml

oc get po
NAME                  READY   STATUS    RESTARTS   AGE
opa-fffdf4574-8pls4   2/2     Running   0          15s

cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    admissionReviewVersions:
    - v1beta1
    sideEffects: None
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: In
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
        scope: 'Namespaced'
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d '\n')
      service:
        namespace: opa
        name: opa
EOF

oc apply -f webhook-configuration.yaml
oc create ns test-ns
oc label ns test-ns openpolicyagent.org/webhook=ignore
oc create deploymentconfig mydc --image openshift/hello-openshift -n test-ns

oc rollout latest dc/mydc -n test-ns
deploymentconfig.apps.openshift.io/mydc rolled out
######## ^^ oc rollout works well for deploymentconfig without this bug's error

cat > dc-policy.rego << EOF
package kubernetes.admission

deny[msg] {
    input.request.kind.kind == "DeploymentConfig"
    msg:= "No entry for you"
}
EOF

oc create configmap dc-policy --from-file=dc-policy.rego

oc rollout latest dc/mydc -n test-ns
Error from server (No entry for you): admission webhook "validating-webhook.openpolicyagent.org" denied the request: No entry for you
oc create deploymentconfig mydc2 --image openshift/hello-openshift -n test-ns
Error from server (No entry for you): admission webhook "validating-webhook.openpolicyagent.org" denied the request: No entry for you
######## ^^ oc rollout works well for deploymentconfig without this bug's error, and the output is expected as the policy

So the bug is pre-merge verified. After the PR gets merged and bug becomes ON_QA, it will be moved to VERIFIED by the robot automatically (if not, I'll manually move at that time)

Comment 4 Xingxing Xia 2020-12-22 11:24:07 UTC
https://github.com/openshift/openshift-apiserver/pull/166 is merged 14 hours ago, but latest payloads are:
4.6.0-0.nightly-2020-12-21-185524 	Pending 	16 hours ago 			
4.6.0-0.nightly-2020-12-21-163117 	Accepted 	18 hours ago
...
No payloads contain the PR. So can't move to VERIFIED currently. Waiting for the payload.

Comment 6 Lukasz Szaszkiewicz 2021-01-11 10:43:11 UTC
*** Bug 1909870 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2021-01-18 17:59:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.12 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0037

Comment 10 Red Hat Bugzilla 2023-09-18 00:23:53 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days