Bug 2004594

Description of problem:

Running s2i builds fail pulling from github.com

OpenShift release version:

Client Version: 4.6.6
Server Version: 4.8.10
Kubernetes Version: v1.21.1+9807387

Cluster Platform:

Openshift on oVirt

How reproducible:

Everytime

Steps to Reproduce (in detail):
1. Create new app pulling from public github repo
2. start build
3.


Actual results:

Cloning "https://github.com/user/repo" ...
error: fatal: unable to access 'https://github.com/user/repo/': SSL certificate problem: self signed certificate in certificate chain

Expected results:

Build succeeds


Impact of the problem:

Cannot update app.  App still runs on an older version


Additional info:

This build has worked, but is now failing and I'm not sure why.

Build Config:

```yaml
spec:
  nodeSelector: null
  output:
    to:
      kind: ImageStreamTag
      name: 'portfolio:latest'
  resources: {}
  successfulBuildsHistoryLimit: 5
  failedBuildsHistoryLimit: 5
  strategy:
    type: Source
    sourceStrategy:
      from:
        kind: ImageStreamTag
        namespace: openshift
        name: 'python:3.8-ubi7'
  postCommit: {}
  source:
    type: Git
    git:
      uri: 'https://github.com/<user>/<project>'
    contextDir: /
    sourceSecret:
      name: github-ssh-private-key
  triggers:
    - type: Generic
      generic:
        secretReference:
          name: portfolio-generic-webhook-secret
    - type: GitHub
      github:
        secretReference:
          name: portfolio-github-webhook-secret
    - type: ImageChange
      imageChange:
        lastTriggeredImageID: >-
          image-registry.openshift-image-registry.svc:5000/openshift/python@sha256:9caeaafa9409cae8a5ba72f7c02e94978c0fbed85753470d265777aa6d2281fa
    - type: ConfigChange
  runPolicy: Serial
```

Build Logs (--build-loglevel=5):

```
I0908 12:22:02.744117       1 builder.go:420] openshift-builder 4.8.0-202108130208.p0.git.b59f2b3.assembly.stream-b59f2b3
I0908 12:22:02.744217       1 builder.go:420] Powered by buildah v1.20.1
I0908 12:22:02.748729       1 builder.go:421] redacted build: {"kind":"Build","apiVersion":"build.openshift.io/v1","metadata":{"name":"portfolio-29","namespace":"homelab-main","uid":"6f1c30a2-adea-4972-9a3f-2da1e5f8e9b8","resourceVersion":"130395602","generation":1,"creationTimestamp":"2021-09-08T12:21:56Z","labels":{"app":"portfolio","app.kubernetes.io/component":"portfolio","app.kubernetes.io/instance":"portfolio","app.kubernetes.io/name":"python","app.kubernetes.io/part-of":"portfolio-app","app.openshift.io/runtime":"python","app.openshift.io/runtime-version":"3.8-ubi7","buildconfig":"portfolio","openshift.io/build-config.name":"portfolio","openshift.io/build.start-policy":"Serial"},"annotations":{"openshift.io/build-config.name":"portfolio","openshift.io/build.number":"29"},"ownerReferences":[{"apiVersion":"build.openshift.io/v1","kind":"BuildConfig","name":"portfolio","uid":"ffaf7264-6669-47e9-b54c-0c2f285e8259","controller":true}],"managedFields":[{"manager":"openshift-apiserver","operation":"Update"...
I0908 12:22:02.749793       1 scmauths.go:61] Finding auth for "..2021_09_08_12_21_58.218620292"
I0908 12:22:02.749825       1 scmauths.go:61] Finding auth for "..data"
I0908 12:22:02.749832       1 scmauths.go:61] Finding auth for "ssh-privatekey"
I0908 12:22:02.749840       1 scmauths.go:61] Found SCMAuth "ssh-privatekey" to handle "ssh-privatekey"
I0908 12:22:02.749849       1 scmauths.go:67] Setting up SCMAuth "ssh-privatekey"
I0908 12:22:02.750090       1 scmauths.go:46] source secret dir /var/run/secrets/openshift.io/source has file ..2021_09_08_12_21_58.218620292
I0908 12:22:02.750107       1 scmauths.go:46] source secret dir /var/run/secrets/openshift.io/source has file ..data
I0908 12:22:02.750111       1 scmauths.go:46] source secret dir /var/run/secrets/openshift.io/source has file ssh-privatekey
I0908 12:22:02.750119       1 scmauths.go:46] Adding Private SSH Auth:
#!/bin/sh
ssh -i /var/run/secrets/openshift.io/source/ssh-privatekey -o StrictHostKeyChecking=false "$@"

Cloning "https://github.com/<user>/<project>" ...
I0908 12:22:02.750285       1 source.go:237] git ls-remote --heads https://github.com/<user>/<project>
I0908 12:22:02.750312       1 repository.go:450] Executing git ls-remote --heads https://github.com/<user>/<project>
I0908 12:22:02.890618       1 repository.go:541] Error executing command: exit status 128
I0908 12:22:02.890729       1 source.go:237] fatal: unable to access 'https://github.com/<user>/<project>/': SSL certificate problem: self signed certificate in certificate chain
error: fatal: unable to access 'https://github.com/<user>/<project>/': SSL certificate problem: self signed certificate in certificate chain
```

I think it has something to do with dns or ndots because when I debug the current running pod (from a previous build with the same config), github does not resolve correctly until I change ndots.  More specifically, it returns  the default internal openshift 'Application is not available' page until I change ndots.

```
> oc debug deployment/portfolio --image=fedora
Starting pod/portfolio-debug ...
Pod IP: 10.129.2.35
If you don't see a command prompt, try pressing enter.
sh-5.1# curl github.com
<html>
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <style type="text/css">
      body {
        font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
        line-height: 1.66666667;
        font-size: 16px;
        color: #333;
        background-color: #fff;
        margin: 2em 1em;
      }
      h1 {
        font-size: 28px;
        font-weight: 400;
      }
      p {
        margin: 0 0 10px;
      }
      .alert.alert-info {
        background-color: #F0F0F0;
        margin-top: 30px;
        padding: 30px;
      }
      .alert p {
        padding-left: 35px;
      }
      ul {
        padding-left: 51px;
        position: relative;
      }
      li {
        font-size: 14px;
        margin-bottom: 1em;
      }
      p.info {
        position: relative;
        font-size: 20px;
      }
      p.info:before, p.info:after {
        content: "";
        left: 0;
        position: absolute;
        top: 0;
      }
      p.info:before {
        background: #0066CC;
        border-radius: 16px;
        color: #fff;
        content: "i";
        font: bold 16px/24px serif;
        height: 24px;
        left: 0px;
        text-align: center;
        top: 4px;
        width: 24px;
      }

      @media (min-width: 768px) {
        body {
          margin: 6em;
        }
      }
    </style>
  </head>
  <body>
    <div>
      <h1>Application is not available</h1>
      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>

      <div class="alert alert-info">
        <p class="info">
          Possible reasons you are seeing this page:
        </p>
        <ul>
          <li>
            <strong>The host doesn't exist.</strong>
            Make sure the hostname was typed correctly and that a route matching this hostname exists.
          </li>
          <li>
            <strong>The host exists, but doesn't have a matching path.</strong>
            Check if the URL path was typed correctly and that the route was created using the desired path.
          </li>
          <li>
            <strong>Route and path matches, but all pods are down.</strong>
            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
          </li>
        </ul>
      </div>
    </div>
  </body>
</html>
sh-5.1# cat /etc/resolv.conf
search homelab-main.svc.cluster.local svc.cluster.local cluster.local domain.com openshift.domain.com
nameserver 172.30.0.10
options ndots:5
sh-5.1# cp /etc/resolv.conf myResolv
sh-5.1# vi myResolv
sh-5.1# cat /etc/resolv.conf
search homelab-main.svc.cluster.local svc.cluster.local cluster.local domain.com openshift.domain.com
nameserver 172.30.0.10
options ndots:5
sh-5.1# cat /myResolv
search homelab-main.svc.cluster.local svc.cluster.local cluster.local domain.com openshift.domain.com
nameserver 172.30.0.10
options ndots:1
sh-5.1# cp myResolv /etc/resolv.conf
sh-5.1# cat /etc/resolv.conf
search homelab-main.svc.cluster.local svc.cluster.local cluster.local domain.com openshift.domain.com
nameserver 172.30.0.10
options ndots:1
sh-5.1# curl github.com
sh-5.1# curl https://raw.githubusercontent.com/<user>/<project>/master/README.md
<div align="center"><img src="/scrots/d.png" align="center"/></div>
# ...
# ... It returned the expected info
# ...
sh-5.1#
```



** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.