Bug 1983935

Summary: [IBM Z]: Creating AWS and IBMCOS backing store with yaml reports INVALID_ENDPOINT
Product: [Red Hat Storage] Red Hat OpenShift Container Storage Reporter: Sravika <sbalusu>
Component: Multi-Cloud Object GatewayAssignee: Romy Ayalon <rayalon>
Status: CLOSED NOTABUG QA Contact: Raz Tamir <ratamir>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.8CC: etamir, ocs-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-28 10:45:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
aws backingstore description
none
openshift-console-screenshot
none
ibmcos backingstore description none

Description Sravika 2021-07-20 08:29:09 UTC
Created attachment 1803531 [details]
aws backingstore description

Description of problem (please be detailed as possible and provide log
snippests):

Creating an AWS backend store using yaml reports " BackingStore "myawsyamlbs" invalid external connection "INVALID_ENDPOINT" "

Create secret and aws backing store using the following yaml files:

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: myawsyamlbackingstore
  namespace: openshift-storage
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <AWS_ACCESS_KEY>
  AWS_SECRET_ACCESS_KEY: <AWS_SECRET_ACCESS_KEY>


awsbackingstore.yaml

apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  labels:
    app: noobaa
  name: myawsyamlbs
  namespace: openshift-storage
spec:
  awsS3:
    secret:
      name: myawsyamlbackingstore
      namespace: openshift-storage
    targetBucket: myawsyamlbucket
  type: aws-s3


Version of all relevant components (if applicable):

OCP: 4.8.0-rc.3
OCS: 4.8.0-450.ci
LSO: 4.8.0-202106291913
Noobaa: 5.7.0 

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

No

Is there any workaround available to the best of your knowledge?

It works with cli and openshift console

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
No

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install OCP, OCS
2. Create an AWS backing store using the secret.yaml and awsbackingstore.yaml 
provided in the description by supplying encoded AWS access key and secret access key.
3. oc create -f secret.yaml
4. oc create -f awsbackingstore.yaml


Actual results:

AWS backing store creation with yaml fails 

# oc get backingstore -n openshift-storage
NAME                           TYPE            PHASE      AGE
myaws-backingstore             aws-s3          Ready      101m
myawsmcgclibackingstore        aws-s3          Ready      89m
myawsyamlbs                    aws-s3          Rejected   21m
noobaa-default-backing-store   s3-compatible   Ready      4d21h



Expected results:

AWS backing store should be in READY state 

Additional info:

Must gather logs 
https://drive.google.com/file/d/1AlirOl71cWDIzd1UVSXV0SH5SYK3Jzol/view?usp=sharing

Comment 2 Sravika 2021-07-20 08:30:27 UTC
Created attachment 1803532 [details]
openshift-console-screenshot

Comment 3 Sravika 2021-07-20 11:33:32 UTC
Same error has been observed while creating ibmcos backing store with the following yaml files.

ibmcossecret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: ibmcosbs-secret
  namespace: openshift-storage
type: Opaque
data:
  IBM_COS_ACCESS_KEY_ID: <IBM COS ACCESS KEY ID ENCODED IN BASE64>
  IBM_COS_SECRET_ACCESS_KEY: <IBM COS SECRET ACCESS KEY ENCODED IN BASE64>


ibmcosbs.yaml:

apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
  finalizers:
  - noobaa.io/finalizer
  labels:
    app: noobaa
  name: ibmcosbsyaml
  namespace: openshift-storage
spec:
  ibmCos:
    endpoint: https://s3.eu-de.cloud-object-storage.appdomain.cloud
    secret:
      name: ibmcosbs-secret
      namespace: openshift-storage
    targetBucket: myibmcosbucket
  type: ibm-cos


Uploading the description of the backingstore as attachment

Comment 4 Sravika 2021-07-20 11:34:30 UTC
Created attachment 1803666 [details]
ibmcos backingstore description

Comment 5 Romy Ayalon 2021-07-21 13:30:43 UTC
Hi @sbalusu 

you have a trailing /n character in the secret key and access key you provided to the aws backingstore (probably the IBM has the same issue).

please try it again without the /n and let us know if we can close the bug.

Comment 6 Sravika 2021-07-22 07:42:14 UTC
@Rom Ayalon:  I see a need info flag set on my name, but I cannot see any comment. Can you please make the comment public.

Comment 7 Sravika 2021-07-22 12:47:13 UTC
@rayalon: I do not find any trailing character in both aws and ibm secret files. Please find the yamllint outputs of both the yaml files as follows:

# yamllint awssecret.yaml
awssecret.yaml
  1:1       warning  missing document start "---"  (document-start)
  9:81      error    line too long (81 > 80 characters)  (line-length)

yamllint ibmcossecret.yaml
ibmcossecret.yaml
  1:1       warning  missing document start "---"  (document-start)
  9:81      error    line too long (97 > 80 characters)  (line-length)


The line too log error is on the base64 encoded "AWS_SECRET_ACCESS_KEY"

Comment 8 Romy Ayalon 2021-07-22 14:12:28 UTC
try to check if you encoded it inside the base64, I specifically see in the logs that there are trailing /n in both secret key and access key.

Comment 9 Sravika 2021-07-26 08:09:38 UTC
@rayalon: The keys are base64 encoded only once and I verified it.

Comment 10 Sravika 2021-07-28 10:41:37 UTC
After discussing with @rayalon on chat , figured out that the base64 encoding has added a trailing newline character to the access key and secret access key.

The "-n" option while encoding (echo -n <key> | base64) has truncated the trailing line and the backingstore creation was successful.

Comment 11 Sravika 2021-07-28 10:43:13 UTC
The issue can be closed.