Bug 1906429

Summary: [release-4.5]: When using webhooks in OCP 4.5 fails to rollout latest deploymentconfig
Product: OpenShift Container Platform Reporter: Lukasz Szaszkiewicz <lszaszki>
Component: openshift-apiserverAssignee: Lukasz Szaszkiewicz <lszaszki>
Status: CLOSED ERRATA QA Contact: Xingxing Xia <xxia>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.5CC: akashem, akhaire, antonio.quintavalle, aos-bugs, bleanhar, hgomes, lszaszki, ltitov, mfojtik, miminar, mtleilia, openshift-bugzilla-robot, rgregory, skrenger, sttts, vjaypurk, xxia
Target Milestone: ---Keywords: UpcomingSprint
Target Release: 4.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Specifies the correct version for the DeploymentConfig. Previously all requests targeting "deploymentconfigs/{name}/instantiate" subresource failed with "no kind DeploymentConfig is registered for version apps.openshift.io/"
Story Points: ---
Clone Of: 1906428 Environment:
Last Closed: 2021-01-26 16:32:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1906428    
Bug Blocks:    

Comment 1 Stefan Schimanski 2020-12-15 15:16:03 UTC
*** Bug 1907958 has been marked as a duplicate of this bug. ***

Comment 3 Xingxing Xia 2020-12-18 08:35:12 UTC
openshift/openshift-apiserver/pull/168 is still open, the bug should not be ON_QA, moving it back to POST.
Per the robot's workflow, 4.5 PR cannot be merged if the 4.6 bug isn't VERIFIED, 4.6 bug cannot be VERIFIED if the 4.6 PR isn't merged.
Now 4.6 PR is still open. Good news is I already pre-merge verified the 4.6 bug https://bugzilla.redhat.com/show_bug.cgi?id=1906428#c2 . Waiting for 4.6 PR's merging.

Comment 4 Lukasz Szaszkiewicz 2021-01-11 13:15:10 UTC
*** Bug 1909870 has been marked as a duplicate of this bug. ***

Comment 5 Xingxing Xia 2021-01-12 03:43:07 UTC
I'm following JIRA issue DPTP-660 to do the pre-merge verification. Used the cluster-bot launching an env with the still open but Dev-approved PR(s): launch openshift/openshift-apiserver#168

The admission-controller.yaml and webhook-configuration.yaml files are copied from https://www.openpolicyagent.org/docs/v0.12.2/kubernetes-admission-control/ . For webhook-configuration.yaml:
  I must make some changes (see the step below) otherwise its creation not hit failure, e.g.:
  ValidatingWebhookConfiguration is promoted to v1 from v1beta1, so I changed to v1;
  And v1 requires sideEffects and admissionReviewVersions must be not empty, so I added them;
  And I modified with "operator: In", and "scope: 'Namespaced'", because original link defines too destructively;
  And lalbel the test namespace to ensure only the test namespace is affected, and other namespaces in the cluster are not destructed.

Verification steps:
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
cat >server.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF

openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf

oc create namespace opa
oc project opa
oc create secret tls opa-server --cert=server.crt --key=server.key
oc adm policy add-scc-to-user anyuid -z default

oc apply -f admission-controller.yaml
oc get po
NAME                   READY   STATUS    RESTARTS   AGE
opa-75c7957bb7-dh9l8   2/2     Running   0          54s

cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    admissionReviewVersions:
    - v1beta1
    sideEffects: None
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: In
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
        scope: 'Namespaced'
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d '\n')
      service:
        namespace: opa
        name: opa
EOF

oc apply -f webhook-configuration.yaml

oc create ns test-ns
oc label ns test-ns openpolicyagent.org/webhook=ignore
oc create deploymentconfig mydc --image openshift/hello-openshift -n test-ns

oc rollout latest dc/mydc -n test-ns
deploymentconfig.apps.openshift.io/mydc rolled out
######## ^^ oc rollout works well for deploymentconfig without this bug's error

cat > dc-policy.rego << EOF
package kubernetes.admission

deny[msg] {
    input.request.kind.kind == "DeploymentConfig"
    msg:= "No entry for you"
}
EOF

oc create configmap dc-policy --from-file=dc-policy.rego
oc rollout latest dc/mydc -n test-ns
Error from server (No entry for you): admission webhook "validating-webhook.openpolicyagent.org" denied the request: No entry for you

oc create deploymentconfig mydc2 --image openshift/hello-openshift -n test-ns
Error from server (No entry for you): admission webhook "validating-webhook.openpolicyagent.org" denied the request: No entry for you
######## ^^ oc rollout works well for deploymentconfig without this bug's error, and the output is expected as the policy

So the bug is pre-merge verified. After the PR gets merged and bug becomes ON_QA, it will be moved to VERIFIED by the robot automatically (if not, I'll manually move at that time)

Comment 6 Lukasz Szaszkiewicz 2021-01-15 10:13:11 UTC
PR is in the merge queue.

Comment 10 errata-xmlrpc 2021-01-26 16:32:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.28 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0175