Bug 1733902 - [IPI on Azure]Report 409 error for azure storage container after remove spec.stroage.azure.container from imageregistry crd
Summary: [IPI on Azure]Report 409 error for azure storage container after remove spec....
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.2.0
Assignee: Corey Daley
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-29 07:29 UTC by XiuJuan Wang
Modified: 2019-10-16 06:34 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:33:51 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 334 None closed Bug 1733902: Retry container creation on correct ServiceCode 2020-04-21 07:53:52 UTC
Red Hat Product Errata RHBA-2019:2922 None None None 2019-10-16 06:34:05 UTC

Description XiuJuan Wang 2019-07-29 07:29:27 UTC
Description of problem:
Report 409 error for azure storage container after remove spec.stroage.azure.container from imageregistry crd.

Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-07-28-222114

How reproducible:
always

Steps to Reproduce:
1.Setup a ipi-azure cluster
2.Remove spec.stroage.azure.container from imageregistry crd
3.Watch image-registry clusteroperator and crd.

Actual results:
Container is added back with an empty value and crd report 409 error.

$ oc describe config.imageregistry 
Name:         cluster
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  imageregistry.operator.openshift.io/v1
Kind:         Config
Metadata:
  Creation Timestamp:  2019-07-29T05:57:50Z
  Finalizers:
    imageregistry.operator.openshift.io/finalizer
  Generation:        3
  Resource Version:  30533
  Self Link:         /apis/imageregistry.operator.openshift.io/v1/configs/cluster
  UID:               cc72db31-b1c5-11e9-9beb-000d3a94a1ba
Spec:
  Default Route:     false
  Http Secret:       1456f869ea210b3b8185d21c2aba618631ef9ed0ad5905144abc4f290197d511652206fb1c986ebd1c38dbae28a463f27bf8bc4e5ed6a1826c63a12d7a7686cc
  Logging:           2
  Management State:  Managed
  Proxy:
    Http:      
    Https:     
    No Proxy:  
  Read Only:   false
  Replicas:    1
  Requests:
    Read:
      Max In Queue:       0
      Max Running:        0
      Max Wait In Queue:  0s
    Write:
      Max In Queue:       0
      Max Running:        0
      Max Wait In Queue:  0s
  Storage:
    Azure:
      Account Name:  zhsun3gzhhs2p2zv
Status:
  Conditions:
    Last Transition Time:  2019-07-29T06:51:56Z
    Message:               Unable to create storage container: -> github.com/openshift/cluster-image-registry-operator/vendor/github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=ContainerAlreadyExists) =====
Description=The specified container already exists.
RequestId:a64ef3a8-f01e-013b-35da-45f51d000000
Time:2019-07-29T06:51:56.1779854Z, Details: 
   Code: ContainerAlreadyExists
   PUT https://zhsun3gzhhs2p2zv.blob.core.windows.net/image-registry?restype=container&timeout=61
   Authorization: REDACTED
   User-Agent: [openshift.io cluster-image-registry-operator/v4.2.0-201907281300-dirty Azure-Storage/0.7 (go1.11.6; linux)]
   X-Ms-Client-Request-Id: [ce363f4c-fe26-4f13-5b4e-095aaf6aa1ef]
   X-Ms-Date: [Mon, 29 Jul 2019 06:51:56 GMT]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 409 The specified container already exists.
   Content-Length: [230]
   Content-Type: [application/xml]
   Date: [Mon, 29 Jul 2019 06:51:55 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [ContainerAlreadyExists]
   X-Ms-Request-Id: [a64ef3a8-f01e-013b-35da-45f51d000000]
   X-Ms-Version: [2018-11-09]



    Reason:                AzureError
    Status:                Unknown
    Type:                  StorageExists
    Last Transition Time:  2019-07-29T06:04:48Z
    Message:               The registry is ready
    Reason:                Ready
    Status:                True
    Type:                  Available
    Last Transition Time:  2019-07-29T06:45:29Z
    Message:               Unable to apply resources: unable to sync storage configuration: unable to create storage container: -> github.com/openshift/cluster-image-registry-operator/vendor/github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /go/src/github.com/openshift/cluster-image-registry-operator/vendor/github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=ContainerAlreadyExists) =====
Description=The specified container already exists.
RequestId:a64ef3a8-f01e-013b-35da-45f51d000000
Time:2019-07-29T06:51:56.1779854Z, Details: 
   Code: ContainerAlreadyExists
   PUT https://zhsun3gzhhs2p2zv.blob.core.windows.net/image-registry?restype=container&timeout=61
   Authorization: REDACTED
   User-Agent: [openshift.io cluster-image-registry-operator/v4.2.0-201907281300-dirty Azure-Storage/0.7 (go1.11.6; linux)]
   X-Ms-Client-Request-Id: [ce363f4c-fe26-4f13-5b4e-095aaf6aa1ef]
   X-Ms-Date: [Mon, 29 Jul 2019 06:51:56 GMT]
   X-Ms-Version: [2018-11-09]
   --------------------------------------------------------------------------------
   RESPONSE Status: 409 The specified container already exists.
   Content-Length: [230]
   Content-Type: [application/xml]
   Date: [Mon, 29 Jul 2019 06:51:55 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Error-Code: [ContainerAlreadyExists]
   X-Ms-Request-Id: [a64ef3a8-f01e-013b-35da-45f51d000000]
   X-Ms-Version: [2018-11-09]



    Reason:                Error
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2019-07-29T06:01:23Z
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2019-07-29T06:01:23Z
    Status:                False
    Type:                  Removed
  Observed Generation:     3
  Ready Replicas:          0
  Storage:
    Azure:
      Account Name:  zhsun3gzhhs2p2zv
      Container:     
  Storage Managed:   true
Events:              <none>
Expected results:
Shouldn't prompt this error.


Additional info:
Met same error when add wrong parameter "containerName".

Comment 1 Corey Daley 2019-07-29 17:32:25 UTC
Looks like we just were not retrying on the correct error code / service code, submitted pull request to fix it.
https://github.com/openshift/cluster-image-registry-operator/pull/334

Comment 3 XiuJuan Wang 2019-08-01 06:59:27 UTC
The issue still could be reproduce with same error message in 4.2.0-0.nightly-2019-08-01-035705 version(which has include the fix).

Comment 4 Corey Daley 2019-08-02 19:04:29 UTC
Ok, it looks like the container name is static in the create code (image-registry), so let me work out what the flow should be and i'll throw together another pr.

Comment 5 Corey Daley 2019-08-09 00:23:52 UTC
The new pull request has merged, please check it again.

Comment 6 XiuJuan Wang 2019-08-09 06:30:52 UTC
@Corey
After remove spec.stroage.azure.container from imageregistry config, the container will be added back without error.
Check with 4.2.0-0.nightly-2019-08-09-000333 payload

Please set it on_qa, so I can verify it.

Comment 7 XiuJuan Wang 2019-08-09 12:25:35 UTC
Mark this as verified as comment #6.

Comment 8 errata-xmlrpc 2019-10-16 06:33:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.