Hide Forgot
Description of problem: Image registry operator goes to panic after add an invalid keyID Version-Release number of selected component (if applicable): 4.2.0-0.nightly-2019-08-06-195545 How reproducible: always Steps to Reproduce: 1.Add keyID with a invalid one, the correct is bf25b3200638758d72a1189d6041994818addf86. $oc patch config.imageregistry cluster -p '{"spec":{"storage":{"gcs":{"keyID":"bf25b3200638758d72a1189d6041994818addf"}}}}' --type=merge 2.Check image registry operator 3. Actual results: The pod goes to panic. $ oc logs -f cluster-image-registry-operator-5cc76d4449-45hzm I0807 08:54:19.292645 1 main.go:20] Cluster Image Registry Operator Version: v4.2.0-201908061419-dirty I0807 08:54:19.292981 1 main.go:21] Go Version: go1.11.6 I0807 08:54:19.292996 1 main.go:22] Go OS/Arch: linux/amd64 I0807 08:54:19.296900 1 controller.go:473] waiting for informer caches to sync I0807 08:54:20.900268 1 controller.go:482] started events processor E0807 08:54:21.080378 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76 /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:522 /opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:513 /opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:82 /opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/signal_unix.go:390 /go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:214 /go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:194 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/gcs/gcs.go:281 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:102 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:143 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:121 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:159 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:249 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:256 /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:480 /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x132de78] goroutine 256 [running]: github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108 panic(0x1779740, 0x2d97da0) /opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:513 +0x1b9 github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage.(*BucketHandle).newPatchCall(0x0, 0xc000cb7690, 0x1c75680, 0xc000747890, 0xc000cb7740) /go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:214 +0x48 github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage.(*BucketHandle).Update(0x0, 0x1c75600, 0xc00003e028, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:194 +0xe2 github.com/openshift/cluster-image-registry-operator/pkg/storage/gcs.(*driver).CreateStorage(0xc00074c000, 0xc0002fb600, 0xc0002b2d01, 0x1c83e40) /go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/gcs/gcs.go:281 +0x5d3 github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).syncStorage(0xc00000ba00, 0xc0002fb600, 0x0, 0x0) /go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:102 +0xcc github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).Apply(0xc00000ba00, 0xc0002fb600, 0x0, 0x0) /go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:143 +0x4d github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).createOrUpdateResources(0xc0000b8780, 0xc0002fb600, 0x7, 0xc0003ce201) /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:121 +0x16c github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).sync(0xc0000b8780, 0x1c8aef0, 0xc00000baa0) /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:159 +0x1102 github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor.func1(0xc0000b8780, 0x16d3060, 0x1c45220) /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:249 +0x8f github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor(0xc0000b8780) /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:256 +0x8e github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor-fm() /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:480 +0x2a github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00084eed0) /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54 github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00084eed0, 0x3b9aca00, 0x0, 0x1a82b01, 0xc00009a720) /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xbe github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00084eed0, 0x3b9aca00, 0xc00009a720) /go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).Run /go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:480 +0xf25 Expected results: Should prompt error in imageregsitry config and clusteroperator, but not operator go to crash. Additional info:
Do you get a panic when you add a valid encryption key? If so, this might be an issue with the google sdk. Let me know, thanks.
You are right, with correct keyID, the operator goes to panic too.
That KMS KeyID does not look like it is in the proper format. From the documentation: https://cloud.google.com/storage/docs/encryption/customer-managed-keys Key resources A Cloud KMS key resource has the following format: projects/[PROJECT_STORING_KEYS]/locations/[LOCATION]/keyRings/[KEY_RING_NAME]/cryptoKeys/[KEY_NAME] Where [VALUES_IN_BRACKETS] are values that depend on your key resource.
When add correct KMS KeyID, $oc describe config.image Last Transition Time: 2019-09-11T09:42:35Z Message: KMS encryption was successfully enabled on the GCS bucket Reason: Encryption Successful Status: True Type: StorageEncrypted Observed Generation: 8 Ready Replicas: 0 Storage: Gcs: Bucket: qe-xiu-jww6j-image-registry-us-central1-kiasctnmjqfmtoynijjprg Key ID: projects/openshift-qe/locations/global/keyRings/devexp-qe-test-0911/cryptoKeys/test Project ID: openshift-qe Region: us-central1 Storage Managed: true Events: <none> When add invaild keyid, could prompt reason. Last Transition Time: 2019-09-11T10:04:57Z Message: googleapi: Error 400: Bad Cloud KMS crypto key: 4b266a48081c1b48169148afb21c736edf51ce71, invalid Reason: InvalidStorageConfiguration Status: False Type: StorageEncrypted Observed Generation: 9 Ready Replicas: 0 Storage: Gcs: Bucket: qe-xiu-jww6j-image-registry-us-central1-kiasctnmjqfmtoynijjprg Key ID: 4b266a48081c1b48169148afb21c736edf51ce71 Project ID: openshift-qe Region: us-central1 Storage Managed: true Events: <none> Test in 4.2.0-0.nightly-2019-09-10-235718 version
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922