Bug 1738475 - [GCP]Image registry operator goes to panic after add keyID
Summary: [GCP]Image registry operator goes to panic after add keyID
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.2.0
Assignee: Corey Daley
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-07 09:25 UTC by XiuJuan Wang
Modified: 2019-10-16 06:35 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:35:02 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 366 0 None closed Bug 1738475: Bucket is nil if already exists during update 2020-04-09 06:05:23 UTC
Github openshift cluster-image-registry-operator pull 377 0 None closed Bug 1738475: Update condition reason code for invalid keyID 2020-04-09 06:05:23 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:35:27 UTC

Description XiuJuan Wang 2019-08-07 09:25:37 UTC
Description of problem:
Image registry operator goes to panic after add an invalid keyID

Version-Release number of selected component (if applicable):

4.2.0-0.nightly-2019-08-06-195545

How reproducible:
always

Steps to Reproduce:
1.Add keyID with a invalid one, the correct is bf25b3200638758d72a1189d6041994818addf86.

$oc patch config.imageregistry cluster -p '{"spec":{"storage":{"gcs":{"keyID":"bf25b3200638758d72a1189d6041994818addf"}}}}' --type=merge
2.Check image registry operator
3.

Actual results:
The pod goes to panic.

$ oc logs -f cluster-image-registry-operator-5cc76d4449-45hzm 
I0807 08:54:19.292645       1 main.go:20] Cluster Image Registry Operator Version: v4.2.0-201908061419-dirty
I0807 08:54:19.292981       1 main.go:21] Go Version: go1.11.6
I0807 08:54:19.292996       1 main.go:22] Go OS/Arch: linux/amd64
I0807 08:54:19.296900       1 controller.go:473] waiting for informer caches to sync
I0807 08:54:20.900268       1 controller.go:482] started events processor
E0807 08:54:21.080378       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:522
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:513
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:82
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/signal_unix.go:390
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:214
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:194
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/gcs/gcs.go:281
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:102
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:143
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:121
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:159
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:249
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:256
/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:480
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153
/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/asm_amd64.s:1333
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x132de78]

goroutine 256 [running]:
github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
panic(0x1779740, 0x2d97da0)
	/opt/rh/go-toolset-1.11/root/usr/lib/go-toolset-1.11-golang/src/runtime/panic.go:513 +0x1b9
github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage.(*BucketHandle).newPatchCall(0x0, 0xc000cb7690, 0x1c75680, 0xc000747890, 0xc000cb7740)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:214 +0x48
github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage.(*BucketHandle).Update(0x0, 0x1c75600, 0xc00003e028, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/cloud.google.com/go/storage/bucket.go:194 +0xe2
github.com/openshift/cluster-image-registry-operator/pkg/storage/gcs.(*driver).CreateStorage(0xc00074c000, 0xc0002fb600, 0xc0002b2d01, 0x1c83e40)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/storage/gcs/gcs.go:281 +0x5d3
github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).syncStorage(0xc00000ba00, 0xc0002fb600, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:102 +0xcc
github.com/openshift/cluster-image-registry-operator/pkg/resource.(*Generator).Apply(0xc00000ba00, 0xc0002fb600, 0x0, 0x0)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/resource/generator.go:143 +0x4d
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).createOrUpdateResources(0xc0000b8780, 0xc0002fb600, 0x7, 0xc0003ce201)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:121 +0x16c
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).sync(0xc0000b8780, 0x1c8aef0, 0xc00000baa0)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:159 +0x1102
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor.func1(0xc0000b8780, 0x16d3060, 0x1c45220)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:249 +0x8f
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor(0xc0000b8780)
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:256 +0x8e
github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).eventProcessor-fm()
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:480 +0x2a
github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00084eed0)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x54
github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00084eed0, 0x3b9aca00, 0x0, 0x1a82b01, 0xc00009a720)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xbe
github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00084eed0, 0x3b9aca00, 0xc00009a720)
	/go/src/github.com/openshift/cluster-image-registry-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/openshift/cluster-image-registry-operator/pkg/operator.(*Controller).Run
	/go/src/github.com/openshift/cluster-image-registry-operator/pkg/operator/controller.go:480 +0xf25

Expected results:
Should prompt error in imageregsitry config and clusteroperator, but not operator go to crash.

Additional info:

Comment 1 Corey Daley 2019-08-08 23:38:28 UTC
Do you get a panic when you add a valid encryption key?
If so, this might be an issue with the google sdk.
Let me  know, thanks.

Comment 2 XiuJuan Wang 2019-08-12 08:57:14 UTC
You are right, with correct keyID, the operator goes to panic too.

Comment 5 Corey Daley 2019-08-14 14:08:02 UTC
That KMS KeyID does not look like it is in the proper format.
From the documentation: https://cloud.google.com/storage/docs/encryption/customer-managed-keys

Key resources

A Cloud KMS key resource has the following format:

projects/[PROJECT_STORING_KEYS]/locations/[LOCATION]/keyRings/[KEY_RING_NAME]/cryptoKeys/[KEY_NAME]

Where [VALUES_IN_BRACKETS] are values that depend on your key resource.

Comment 8 XiuJuan Wang 2019-09-11 10:06:47 UTC
When add correct KMS KeyID,
$oc describe config.image

    Last Transition Time:  2019-09-11T09:42:35Z
    Message:               KMS encryption was successfully enabled on the GCS bucket
    Reason:                Encryption Successful
    Status:                True
    Type:                  StorageEncrypted
  Observed Generation:     8
  Ready Replicas:          0
  Storage:
    Gcs:
      Bucket:       qe-xiu-jww6j-image-registry-us-central1-kiasctnmjqfmtoynijjprg
      Key ID:       projects/openshift-qe/locations/global/keyRings/devexp-qe-test-0911/cryptoKeys/test
      Project ID:   openshift-qe
      Region:       us-central1
  Storage Managed:  true
Events:             <none>

When add invaild keyid, could prompt reason.

    Last Transition Time:  2019-09-11T10:04:57Z
    Message:               googleapi: Error 400: Bad Cloud KMS crypto key: 4b266a48081c1b48169148afb21c736edf51ce71, invalid
    Reason:                InvalidStorageConfiguration
    Status:                False
    Type:                  StorageEncrypted
  Observed Generation:     9
  Ready Replicas:          0
  Storage:
    Gcs:
      Bucket:       qe-xiu-jww6j-image-registry-us-central1-kiasctnmjqfmtoynijjprg
      Key ID:       4b266a48081c1b48169148afb21c736edf51ce71
      Project ID:   openshift-qe
      Region:       us-central1
  Storage Managed:  true
Events:             <none>

Test in 4.2.0-0.nightly-2019-09-10-235718 version

Comment 9 errata-xmlrpc 2019-10-16 06:35:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.