Note that this issue is a blocker for both testing and release of RHBQ 2.2.5 on OCP 4.9 and 4.10

I'm running the steps described by Pablo on a "4.9.0-0.nightly-2022-01-28-192738", and it works successfully. As per:

```
$ oc logs --tail=10 openshift-quickstart-1-build
Getting image source signatures
Copying blob sha256:16b78ed2e82205ceca54aa3a4205f029096a354d346b3bd2434a56b7f66e21c1
Copying blob sha256:131f1a26eef08c7274f7a115f461689819a4218e246ccacacc50ebcfd5e1c758
Copying blob sha256:8a53271b80c8bb6c008ffd7bc2a1c244014a8f5e576a4015dca62a7d6c9c9858
Copying blob sha256:678e7f2d7591fda2334cd3b41cca679ef1cab0f5665da594fcbf87d7c7f8cafb
Copying config sha256:fe705bffa1f57ea607a4606a4f4e1e55a1293f7b14dd5b1c35a2503bd9e08c21
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/bz-2047164/openshift-quickstart@sha256:02cf8f5eead94dcb3a9d7a3d71f8638337204862aaf98f7f313abbb0682b13e5
Push successful
```

The error message "kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" is related the storage layer, which is dependent of the node's Kernel version. Here on the cluster I've tested with, it's using "4.18.0-305.34.2.el8_4.x86_64".

https://github.com/containers/storage/blob/fafe869b23235791798a75e23bc91e7fe30fdf56/drivers/overlay/check.go#L110-L113

What's the type of nodes having this error, @pagonzal ?

Based on that I see in my ocp console we are running: `4.18.0-305.30.1.el8_4.x86_64` kernel version. 
Same config for master and workers nodes.

cmd:

oc get nodes -o wide

Version: v1.22.3+e790d7f
OS-IMAGE: Red Hat Enterprise Linux CoreOS 49.84.202201102104-0 (Ootpa)
KERNEL-VERSION: 4.18.0-305.30.1.el8_4.x86_64
CONTAINER-RUNTIME: cri-o://1.22.1-10.rhaos4.9.gitf1d2c6e.el8


So my storage layer version looks slightly different: 4.18.0-305.34.2.el8_4.x86_64 Vs 4.18.0-305.30.1.el8_4.x86_64

Do you think that this is the root of the problem? this storage layer "4.18.0-305.30.1.el8_4.x86_64" is not supported by ocp 4.9?

Also, note that we run on OpenStack via IPI (Installer-provisioned Infrastructure) and doesn't work on 4.10 nightlies either

Pablo - can you please provide the YAML for the BuildConfig that is generated by the mvn plugin?

Based on the symptoms described, it sounds like mvn is creating a BuildConfig and importing your ~/.docker/config.json file as an image pull secret. It appears that having https in the key is not valid for the current auth file specification. My recommendation is to either remove the https://index.docker.io/v1/ entry in the auth file, or log out/log back into docker hub using your local container runtime (if that is necessary for the build).

The problem is that by default docker adds this path to your ~/.docker/config.json so ocp4.9 doesn't work with a default docker config file (This is the issue), and is working with a previous ocp version (basically is a regression). 

What I mean with "by default docker adds this path to your "~/.docker/config.json" is that if a developer does a "docker login" into their prompt then a "https://index.docker.io/v1/" path is added, this is the standard docker behavior.  

On the other hand, if key "https://index.docker.io/v1/" is removed from "~/.docker/config.json" it's also failing with the same error, so maybe is cached or generated in somewhere. 

BuildConfig instance


```

kind: Build
apiVersion: build.openshift.io/v1
metadata:
  annotations:
    openshift.io/build-config.name: openshift-quickstart
    openshift.io/build.number: '1'
    openshift.io/build.pod-name: openshift-quickstart-1-build
  resourceVersion: '10101645'
  name: openshift-quickstart-1
  uid: 9dfa9510-7143-495c-874a-fc9a7e275df7
  creationTimestamp: '2022-02-01T09:26:46Z'
  generation: 2
  namespace: pablo-test
  ownerReferences:
    - apiVersion: build.openshift.io/v1
      kind: BuildConfig
      name: openshift-quickstart
      uid: 08868eca-baf8-4651-bccd-992d6b0e931d
      controller: true
  labels:
    app.kubernetes.io/name: openshift-quickstart
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.openshift.io/runtime: quarkus
    buildconfig: openshift-quickstart
    openshift.io/build-config.name: openshift-quickstart
    openshift.io/build.start-policy: Serial
spec:
  serviceAccount: builder
  source:
    type: Binary
    binary: {}
  strategy:
    type: Source
    sourceStrategy:
      from:
        kind: DockerImage
        name: >-
          registry.access.redhat.com/ubi8/openjdk-11@sha256:bef8578609a07ffc4b9c66ce6f9d7015a4f8639d82e4d9a482c58f802337b3d4
  output:
    to:
      kind: ImageStreamTag
      name: 'openshift-quickstart:1.0.0-SNAPSHOT'
    pushSecret:
      name: builder-dockercfg-46jnm
  resources: {}
  postCommit: {}
  nodeSelector: null
status:
  output: {}
  config:
    kind: BuildConfig
    namespace: pablo-test
    name: openshift-quickstart
  reason: GenericBuildFailed
  message: Generic Build failure - check logs for details.
  outputDockerImageReference: >-
    image-registry.openshift-image-registry.svc:5000/pablo-test/openshift-quickstart:1.0.0-SNAPSHOT
  duration: 22000000000
  startTimestamp: '2022-02-01T09:26:46Z'
  stages:
    - name: PullImages
      startTime: '2022-02-01T09:26:56Z'
      durationMilliseconds: 11745
      steps:
        - name: PullBaseImage
          startTime: '2022-02-01T09:26:56Z'
          durationMilliseconds: 11745
    - name: Build
      startTime: '2022-02-01T09:27:08Z'
      durationMilliseconds: 2
      steps:
        - name: DockerBuild
          startTime: '2022-02-01T09:27:08Z'
          durationMilliseconds: 2
  conditions:
    - type: New
      status: 'False'
      lastUpdateTime: '2022-02-01T09:26:46Z'
      lastTransitionTime: '2022-02-01T09:26:46Z'
    - type: Pending
      status: 'False'
      lastUpdateTime: '2022-02-01T09:26:50Z'
      lastTransitionTime: '2022-02-01T09:26:50Z'
    - type: Running
      status: 'False'
      lastUpdateTime: '2022-02-01T09:27:08Z'
      lastTransitionTime: '2022-02-01T09:27:08Z'
    - type: Failed
      status: 'True'
      lastUpdateTime: '2022-02-01T09:27:08Z'
      lastTransitionTime: '2022-02-01T09:27:08Z'
      reason: GenericBuildFailed
      message: Generic Build failure - check logs for details.
  logSnippet: >-
    Copying config
    sha256:106bcc7299ebfc09cb1cf049181d9796c7c2faabdf8ea486c3d904704aec4a33

    Writing manifest to image destination

    Storing signatures

    Generating dockerfile with builder image
    registry.access.r...78609a07ffc4b9c66ce6f9d7015a4f8639d82e4d9a482c58f802337b3d4

    error: build error: key https://index.docker.io/v1/ contains http[s]://
    prefix
  completionTimestamp: '2022-02-01T09:27:08Z'
  phase: Failed

```

So looks that this build is pointing to this secret `builder-dockercfg-46jnm`, that contains a .dockercfg as data

oc get secret builder-dockercfg-46jnm -o yaml | grep .dockercfg

This data has a base64 value, and if you decode it you will find a lot of service accounts, but I could not find the key "https://index.docker.io/v1/"

Anyway, my point is that changing the local configuration, which is the default one, is not a solution. I think that OCP 4.9 should handle internally the format of this URL and accept Https or URLs without protocol as "quay.io"


WDYT?

This seems to have been an issue with our cluster configuration from installation, where we added pull secret with the "https://index.docker.io/v1/" URL to cluster's pull secrets.

I am closing this as 'CLOSED,NOTABUG'.

Thanks for the information, Pablo and Michal. I missed the "~/.docker/config.json" issue at first, I'm sorry for that.