+++ This bug was initially created as a clone of Bug #1739085 +++ This bug is to establish that the issue in question is not present on 4.2 Description of problem: When multiple versions of a CRD is declared, and webhooks for that resource exist, sometimes create/update operations will fail with an error like: See upstream issue: https://github.com/kubernetes/kubernetes/issues/73752 A fix for this issue was recently merged upstream to the release-1.13 branch: https://github.com/kubernetes/kubernetes/pull/79495 We have applied this patch to the openshift release-4.1 branch and established that this appears to fix the issue on 4.1. It is critical in order to run Knative on OpenShift 4.1 that this bug be included in the next z-stream release. Version-Release number of selected component (if applicable): 4.1 How reproducible: Intermittent; required restart of API server pod in order to reliably trigger Steps to Reproduce: 1. Create a CRD with multiple versions 2. Create an admission webhook for the resource 3. Create/update the resource Actual results: Error from server (InternalError): Internal error occurred: no kind "Service" is registered for version "serving.knative.dev/v1beta1" in scheme "k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:62" Expected results: Updates should result in admission webhook being triggered / normal admission flow. Additional info:
Setting to ON_QA since QE will verify that this issue is not present on 4.2
First, tried to reproduce the issue in 4.1.0-0.nightly-2019-08-11-140905 which does not include the fix: Prepare test data: cat v.yaml apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration metadata: name: application-validation-webhook-configuration webhooks: - clientConfig: service: null url: https://127.0.0.1:8443/v1/validate-applications name: applications.shipper.booking.com failurePolicy: Ignore rules: - apiGroups: - '*' apiVersions: - '*' operations: - CREATE - UPDATE resources: - applications cat m.yaml apiVersion: admissionregistration.k8s.io/v1beta1 kind: MutatingWebhookConfiguration metadata: name: application-mutation-webhook-configuration webhooks: - clientConfig: service: null url: https://127.0.0.1:8443/v1/mutate-applications failurePolicy: Ignore name: applications.shipper.booking.com rules: - apiGroups: - '*' apiVersions: - '*' operations: - CREATE - UPDATE resources: - applications cat app-crd.yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: applications.shipper.booking.com spec: conversion: strategy: None group: shipper.booking.com names: kind: Application listKind: ApplicationList plural: applications shortNames: - app singular: application scope: Namespaced validation: openAPIV3Schema: properties: spec: properties: template: properties: values: type: object required: - values type: object required: - template type: object version: v1alpha1 versions: - name: v1alpha1 served: true storage: true - name: v1 served: true storage: false cat app-v1alpha1.yaml apiVersion: shipper.booking.com/v1alpha1 kind: Application metadata: name: super-server spec: template: values: replicaCount: 3 sed 's/v1alpha1/v1/' app-v1alpha1.yaml > app-v1.yaml --------------------------- Then do below steps: oc create -f v.yaml oc create -f m.yaml oc create -f app-crd.yaml oc adm new-project xxia-proj-1 oc adm new-project xxia-proj-2 oc create -f app-v1alpha1.yaml -n xxia-proj-1 oc create -f app-v1.yaml -n xxia-proj-2 oc patch kubeapiserver/cluster --type=json -p "[ {\"op\": \"replace\", \"path\": \"/spec/forceRedeploymentReason\", \"value\": \"`date` force rollout\" } ]" watch oc get po -l apiserver -n openshift-kube-apiserver # wait for kube-apiserver finish restarting # Update the CR oc -n xxia-proj-1 label application super-server foo=bar oc -n xxia-proj-2 label application super-server foo=bar Last two updates both reproduced the failure error: Error from server (InternalError): Internal error occurred: no kind "Application" is registered for version "shipper.booking.com/v1" in scheme "k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:62" # Do this apiVersion change operation as workwround, try to update the CR again, the update succeeds: oc -n xxia-proj-1 get application super-server -o json | jq '.apiVersion = "shipper.booking.com/v1alpha1"' | oc apply -f - oc -n xxia-proj-1 label application super-server foo=bar
Using above steps, verified 4.2.0-0.nightly-2019-08-10-002649 env, all CR updates succeed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922