Bug 1906428 - [release-4.6]: When using webhooks in OCP 4.5 fails to rollout latest deploymentconfig
Summary: [release-4.6]: When using webhooks in OCP 4.5 fails to rollout latest deploym...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-apiserver
Version: 4.6
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.z
Assignee: Lukasz Szaszkiewicz
QA Contact: Xingxing Xia
URL:
Whiteboard:
: 1906640 (view as bug list)
Depends On: 1867380
Blocks: 1906429 1909870
TreeView+ depends on / blocked
 
Reported: 2020-12-10 13:53 UTC by Lukasz Szaszkiewicz
Modified: 2021-02-08 10:50 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Specifies the correct version for the DeploymentConfig. Previously all requests targeting "deploymentconfigs/{name}/instantiate" subresource failed with "no kind DeploymentConfig is registered for version apps.openshift.io/"
Clone Of: 1867380
: 1906429 (view as bug list)
Environment:
Last Closed: 2021-01-18 17:59:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-apiserver pull 166 0 None closed Bug 1906428: When using webhooks in OCP 4.5 fails to rollout latest deploymentconfig 2021-02-08 10:49:39 UTC
Red Hat Product Errata RHSA-2021:0037 0 None None None 2021-01-18 17:59:57 UTC

Comment 1 Lukasz Szaszkiewicz 2020-12-11 08:27:22 UTC
*** Bug 1906640 has been marked as a duplicate of this bug. ***

Comment 2 Xingxing Xia 2020-12-18 04:54:48 UTC
I'm following JIRA issue DPTP-660 to do the pre-merge verification. Used the cluster-bot launching an env with the still open but Dev-approved PR(s).
The admission-controller.yaml and webhook-configuration.yaml files are copied from https://www.openpolicyagent.org/docs/v0.12.2/kubernetes-admission-control/
For admission-controller.yaml:
Directly creating it, the opa pod will be CrashLoopBackOff for "opa" container with pod logs that show err in binding 443 port. So I must add securityContext field:
         - name: opa
           image: openpolicyagent/opa:0.12.2
           securityContext:
             privileged: true
             runAsUser: 0

I must also add SCC (see the step below).

For webhook-configuration.yaml:
I must make some changes (see the step below) otherwise its creation not hit failure, e.g.:
ValidatingWebhookConfiguration is promoted to v1 from v1beta1, so I changed to v1;
And v1 requires sideEffects and admissionReviewVersions must be not empty, so I added them;
And I modified with "operator: In", and "scope: 'Namespaced'", because original link defines too destructively;
And lalbel the test namespace to ensure only the test namespace is affected, and other namespaces in the cluster are not destructed.

Verification steps:
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
cat >server.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF

openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf

oc create namespace opa
oc project opa
oc create secret tls opa-server --cert=server.crt --key=server.key
oc adm policy add-scc-to-user privileged -z default
oc apply -f admission-controller.yaml

oc get po
NAME                  READY   STATUS    RESTARTS   AGE
opa-fffdf4574-8pls4   2/2     Running   0          15s

cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    admissionReviewVersions:
    - v1beta1
    sideEffects: None
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: In
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
        scope: 'Namespaced'
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d '\n')
      service:
        namespace: opa
        name: opa
EOF

oc apply -f webhook-configuration.yaml
oc create ns test-ns
oc label ns test-ns openpolicyagent.org/webhook=ignore
oc create deploymentconfig mydc --image openshift/hello-openshift -n test-ns

oc rollout latest dc/mydc -n test-ns
deploymentconfig.apps.openshift.io/mydc rolled out
######## ^^ oc rollout works well for deploymentconfig without this bug's error

cat > dc-policy.rego << EOF
package kubernetes.admission

deny[msg] {
    input.request.kind.kind == "DeploymentConfig"
    msg:= "No entry for you"
}
EOF

oc create configmap dc-policy --from-file=dc-policy.rego

oc rollout latest dc/mydc -n test-ns
Error from server (No entry for you): admission webhook "validating-webhook.openpolicyagent.org" denied the request: No entry for you
oc create deploymentconfig mydc2 --image openshift/hello-openshift -n test-ns
Error from server (No entry for you): admission webhook "validating-webhook.openpolicyagent.org" denied the request: No entry for you
######## ^^ oc rollout works well for deploymentconfig without this bug's error, and the output is expected as the policy

So the bug is pre-merge verified. After the PR gets merged and bug becomes ON_QA, it will be moved to VERIFIED by the robot automatically (if not, I'll manually move at that time)

Comment 4 Xingxing Xia 2020-12-22 11:24:07 UTC
https://github.com/openshift/openshift-apiserver/pull/166 is merged 14 hours ago, but latest payloads are:
4.6.0-0.nightly-2020-12-21-185524 	Pending 	16 hours ago 			
4.6.0-0.nightly-2020-12-21-163117 	Accepted 	18 hours ago
...
No payloads contain the PR. So can't move to VERIFIED currently. Waiting for the payload.

Comment 6 Lukasz Szaszkiewicz 2021-01-11 10:43:11 UTC
*** Bug 1909870 has been marked as a duplicate of this bug. ***

Comment 8 errata-xmlrpc 2021-01-18 17:59:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.6.12 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0037


Note You need to log in before you can comment on or make changes to this bug.