Bug 1880393 - Newline at the end of CA cert in security.tls.certificateAuthorities breaks Installation with "Unable to parse CA"
Summary: Newline at the end of CA cert in security.tls.certificateAuthorities breaks I...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.6.0
Assignee: Martin André
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-18 12:07 UTC by Robert Heinzmann
Modified: 2020-10-27 16:43 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:42:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 4201 0 None closed Bug 1880393: OpenStack UPI: Trim EOLs from the cacert trustbundle 2021-02-08 13:12:13 UTC
Github openshift installer pull 4203 0 None closed Bug 1880393: OpenStack UPI: Explain how to encode cert to base64 2021-02-08 13:12:13 UTC
Red Hat Knowledge Base (Solution) 5449881 0 None None None 2020-09-30 21:15:20 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:43:32 UTC

Description Robert Heinzmann 2020-09-18 12:07:54 UTC
Description of problem:

When using a CA certificate file (ca.pem) that contains a newline at the end in the ignition shim as security.tls.certificateAuthorities (as described in the OpenShift on OpenStack UPI documentation [1]), the ignition of the bootstrap machine aborts with:

~~~
------
Ignition has failed. Please ensure your config is valid. Note that only
Ignition spec v3.0.0+ configs are accepted.

A CLI validation tool to check this called ignition-validate can be
downloaded from GitHub:
    https://github.com/coreos/ignition/releases
------

Displaying logs from failed units: ignition-fetch.service
-- Logs begin at Fri 2020-09-18 08:36:55 UTC, end at Fri 2020-09-18 08:37:29 UTC. --
Sep 18 08:37:29 ignition[716]: Adding "Packstack" to list of CAs
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mUnable to decode CA (0xc000481950)[0m
Sep 18 08:37:29 systemd[1]: [0;1;39m[0;1;31m[0;1;39mignition-fetch.service: Main process exited, code=exited, status=1/FAILURE[0m
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mUnable to parse CA bundle: unable to decode PEM block[0m
Sep 18 08:37:29 systemd[1]: [0;1;39m[0;1;31m[0;1;39mignition-fetch.service: Failed with result 'exit-code'.[0m
Sep 18 08:37:29 ignition[716]: [0;1;39m[0;1;31m[0;1;39mfailed to fetch config: unable to decode PEM block[0m
Sep 18 08:37:29 systemd[1]: [0;1;31m[0;1;39m[0;1;31mFailed to start Ignition (fetch).[0m
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mfailed to acquire config: unable to decode PEM block[0m
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mIgnition failed: unable to decode PEM block[0m
Sep 18 08:37:29 systemd[1]: ignition-fetch.service: Triggering OnFailure= dependencies.
Press Enter for emergency shell or wait 5 minutes for reboot.                 Press Enter for emergency shell or wait 4 minutes 45 seconds for reboot.      Press Enter for emergency shell or wait 4 minutes 30 seconds for reboot.      Press Enter for emergency shell or wait 4 minutes 15 seconds for reboot.      Press Enter for emergency shell or wait 4 minutes for reboot.                 
~~~

[1] https://docs.openshift.com/container-platform/4.6/installing/installing_openstack/installing-openstack-user.html#installation-osp-converting-ignition-resources_installing-openstack-user


How reproducible:

Always when using a certificate with newline at the end.

Steps to Reproduce:

1. Follow the OpenShift on OpenStack UPI documentation to install OpenShift
2. In section "Preparing the bootstrap Ignition files" step 7 use a certificate file with a newline at the end (See https://docs.openshift.com/container-platform/4.6/installing/installing_openstack/installing-openstack-user.html#installation-osp-converting-ignition-resources_installing-openstack-user) 
3. Check the console output of the bootstrap machine

Actual results:

Bootstrap machine can not get the ignition file from the glace endpoint:

~~~
Displaying logs from failed units: ignition-fetch.service
-- Logs begin at Fri 2020-09-18 08:36:55 UTC, end at Fri 2020-09-18 08:37:29 UTC. --
Sep 18 08:37:29 ignition[716]: Adding "Packstack" to list of CAs
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mUnable to decode CA (0xc000481950)[0m
Sep 18 08:37:29 systemd[1]: [0;1;39m[0;1;31m[0;1;39mignition-fetch.service: Main process exited, code=exited, status=1/FAILURE[0m
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mUnable to parse CA bundle: unable to decode PEM block[0m
Sep 18 08:37:29 systemd[1]: [0;1;39m[0;1;31m[0;1;39mignition-fetch.service: Failed with result 'exit-code'.[0m
Sep 18 08:37:29 ignition[716]: [0;1;39m[0;1;31m[0;1;39mfailed to fetch config: unable to decode PEM block[0m
Sep 18 08:37:29 systemd[1]: [0;1;31m[0;1;39m[0;1;31mFailed to start Ignition (fetch).[0m
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mfailed to acquire config: unable to decode PEM block[0m
Sep 18 08:37:29 ignition[716]: [0;1;31m[0;1;39m[0;1;31mIgnition failed: unable to decode PEM block[0m
Sep 18 08:37:29 systemd[1]: ignition-fetch.service: Triggering OnFailure= dependencies.
Press Enter for emergency shell or wait 5 minutes for reboot.                 Press Enter for emergency shell or wait 4 minutes 45 seconds for reboot.      Press Enter for emergency shell or wait 4 minutes 30 seconds for reboot.      Press Enter for emergency shell or wait 4 minutes 15 seconds for reboot.      Press Enter for emergency shell or wait 4 minutes for reboot. 
~~~


Expected results:

Deployment starts, bootstrap machine fetches ignition file from glance. Bootstrapping completes. 


Additional info:

All relevant tools show the certificate as valid:

~~~
shell$ cat $OS_CACERT  | base64 -w0 | base64 -d | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:ff:1c:a0:fb:3a:d1:24:06:6e:44:dd:d2:cc:06:82:96:7d:3c:74
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = DE, ST = Bavaria, L = Germany, O = Elconaslab, OU = Munich, CN = Packstack
        Validity
            Not Before: Mar  9 06:16:00 2020 GMT
            Not After : Mar  8 06:16:00 2025 GMT
        Subject: C = DE, ST = Bavaria, L = Germany, O = Elconaslab, OU = Munich, CN = Packstack
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:ae:9d:18:56:45:c7:65:5c:35:f3:15:ac:0c:0f:
                    38:89:a3:87:17:2c:92:a7:4d:2e:f1:12:99:f0:24:
                    aa:5b:8b:a9:af:4f:3b:18:c7:a7:71:3e:31:f1:21:
                    ad:f9:27:1f:9e:45:a3:8c:cb:f4:9b:ae:35:7a:52:
                    ec:0d:67:68:06:6a:aa:1b:1e:ca:36:ee:a6:86:e8:
                    48:9c:67:a8:f3:db:19:fd:19:bf:76:ad:de:94:78:
                    de:c9:30:14:55:5a:60:69:cd:35:29:3d:bc:47:40:
                    ec:92:af:12:32:56:1b:67:1a:20:40:08:11:c8:fd:
                    1b:b8:38:0f:19:92:b1:aa:93:29:dc:7d:ac:69:90:
                    90:39:db:cc:fd:5d:ca:33:43:e8:b9:e0:10:d0:3b:
                    f3:a4:f8:c5:ed:b7:39:58:87:ba:59:30:cc:c8:6c:
                    a6:63:84:da:78:4c:1c:52:d5:5e:55:09:b9:66:c6:
                    67:0d:c4:84:7b:96:b4:a2:6c:7a:46:64:d3:0f:cc:
                    b4:7e:06:a1:d2:a1:62:81:ac:f0:b3:fa:68:be:90:
                    08:ef:57:09:cf:2c:12:36:a1:44:1e:72:ee:91:01:
                    c2:7b:c4:70:1a:48:14:53:97:31:01:d5:29:ab:d5:
                    1b:c9:90:b3:fd:2f:03:e9:55:80:d6:14:a6:07:66:
                    59:85
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                F7:3C:E7:F6:34:2B:EA:FE:85:72:60:89:A6:49:7D:9D:2E:10:59:58
    Signature Algorithm: sha256WithRSAEncryption
         25:c4:7c:bc:87:55:fe:28:b0:16:80:e8:ed:10:83:3b:2e:9c:
         1b:fc:39:0a:ef:df:68:70:99:f0:68:88:10:da:e4:74:e4:b9:
         c9:c7:be:61:dc:06:36:83:d7:ea:2d:14:f0:49:18:d6:46:c4:
         da:ce:a9:07:cf:c9:9e:76:39:85:4f:42:ee:4c:91:8f:ae:92:
         09:2e:51:e2:fd:56:0b:7c:7b:f4:05:b7:7c:2b:45:20:cb:26:
         8b:45:54:3c:94:d1:74:fc:b2:a7:93:c2:7c:2e:f7:dd:8e:02:
         7f:4a:1f:6f:10:82:96:41:38:63:8e:61:0a:c7:7d:c1:da:52:
         2c:6c:02:bd:7f:2b:e3:46:a5:69:f5:11:1c:32:5d:e2:66:a1:
         99:00:57:23:34:ef:9a:e8:ff:ec:5e:34:bf:b5:28:1b:75:f9:
         0f:82:cc:18:54:29:cd:75:47:bc:21:5c:04:c3:77:6d:46:2d:
         91:7b:a5:19:45:da:90:7e:6c:8e:08:3e:a3:0e:93:99:34:26:
         07:b0:31:85:dc:e0:fc:d2:a7:26:a7:ec:27:c4:3d:c6:b3:ce:
         ba:dc:37:d7:36:2a:0e:7b:0a:22:d1:73:dd:72:70:cd:33:dc:
         46:b2:06:de:33:e4:b4:94:09:39:df:1c:2c:80:f3:8e:f2:05:
         ce:5b:2f:db
-----BEGIN CERTIFICATE-----
MIIDpjCCAo6gAwIBAgIUQP8coPs60SQGbkTd0swGgpZ9PHQwDQYJKoZIhvcNAQEL
BQAwazELMAkGA1UEBhMCREUxEDAOBgNVBAgTB0JhdmFyaWExEDAOBgNVBAcTB0dl
cm1hbnkxEzARBgNVBAoTCkVsY29uYXNsYWIxDzANBgNVBAsTBk11bmljaDESMBAG
A1UEAxMJUGFja3N0YWNrMB4XDTIwMDMwOTA2MTYwMFoXDTI1MDMwODA2MTYwMFow
azELMAkGA1UEBhMCREUxEDAOBgNVBAgTB0JhdmFyaWExEDAOBgNVBAcTB0dlcm1h
bnkxEzARBgNVBAoTCkVsY29uYXNsYWIxDzANBgNVBAsTBk11bmljaDESMBAGA1UE
AxMJUGFja3N0YWNrMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArp0Y
VkXHZVw18xWsDA84iaOHFyySp00u8RKZ8CSqW4upr087GMencT4x8SGt+ScfnkWj
jMv0m641elLsDWdoBmqqGx7KNu6mhuhInGeo89sZ/Rm/dq3elHjeyTAUVVpgac01
KT28R0Dskq8SMlYbZxogQAgRyP0buDgPGZKxqpMp3H2saZCQOdvM/V3KM0PoueAQ
0DvzpPjF7bc5WIe6WTDMyGymY4TaeEwcUtVeVQm5ZsZnDcSEe5a0omx6RmTTD8y0
fgah0qFigazws/povpAI71cJzywSNqFEHnLukQHCe8RwGkgUU5cxAdUpq9UbyZCz
/S8D6VWA1hSmB2ZZhQIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/
BAUwAwEB/zAdBgNVHQ4EFgQU9zzn9jQr6v6FcmCJpkl9nS4QWVgwDQYJKoZIhvcN
AQELBQADggEBACXEfLyHVf4osBaA6O0QgzsunBv8OQrv32hwmfBoiBDa5HTkucnH
vmHcBjaD1+otFPBJGNZGxNrOqQfPyZ52OYVPQu5MkY+ukgkuUeL9Vgt8e/QFt3wr
RSDLJotFVDyU0XT8sqeTwnwu992OAn9KH28QgpZBOGOOYQrHfcHaUixsAr1/K+NG
pWn1ERwyXeJmoZkAVyM075ro/+xeNL+1KBt1+Q+CzBhUKc11R7whXATDd21GLZF7
pRlF2pB+bI4IPqMOk5k0JgewMYXc4PzSpyan7CfEPcazzrrcN9c2Kg57CiLRc91y
cM0z3EayBt4z5LSUCTnfHCyA847yBc5bL9s=
-----END CERTIFICATE-----


shell$ cat $OS_CACERT  | base64 -w0 | base64 -d > error.pem 
shell$ diff -u ok.pem error.pem 
--- ok.pem	2020-09-18 11:29:59.510870304 +0000
+++ error.pem	2020-09-18 11:31:45.023886343 +0000
@@ -20,3 +20,4 @@
 pRlF2pB+bI4IPqMOk5k0JgewMYXc4PzSpyan7CfEPcazzrrcN9c2Kg57CiLRc91y
 cM0z3EayBt4z5LSUCTnfHCyA847yBc5bL9s=
 -----END CERTIFICATE-----
+

~~~

Reproducer:

~~~
# Generate the ignition shim
export PEMFILE=error.pem
export MYTMP_IGNITION_HOST=$(openstack catalog show image | grep public | cut -d: -f2- | awk '{print $1}')
export MYTMP_IGNITION_URI=$(openstack image show ignition-${INFRA_ID}-boostrap -f value -c file)
export MYTMP_TOKEN=$(openstack token issue -c id -f value)
export MYTMP_CACERT_BASE64=$(cat $PREMFILE | base64 -w0)
[openstack ocp-proxy] [stack@osp16 upi]$ cat > $INFRA_ID-bootstrap-ignition.json <<EOF
{
  "ignition": {
    "config": {
      "merge": [{
        "source": "${MYTMP_IGNITION_HOST}${MYTMP_IGNITION_URI}", 
        "httpHeaders": [{
          "name": "X-Auth-Token", 
          "value": "${MYTMP_TOKEN}" 
        }]
      }]
    },
    "security": {
      "tls": {
        "certificateAuthorities": [{
          "source": "data:text/plain;charset=utf-8;base64,${MYTMP_CACERT_BASE64}" 
        }]
      }
    },
    "version": "3.1.0"
  }
}

EOF

shell$ ansible-playbook -i inventory.yaml 03_bootstrap.yaml 
shell$ openstack console log show "$INFRA_ID-bootstrap" | tail -30
[[0;32m  OK  [0m] Reached target Emergency Mode.
[   36.899668] systemd[1]: Started Journal Service.
[   36.899780] multipathd[772]: Sep 18 11:59:15 | /etc/multipath.conf does not exist, blacklisting all devices.
[   36.903808] multipathd[772]: Sep 18 11:59:15 | You can run "/sbin/mpathconf --enable" to create
[[0;32m  OK  [0m] Started Journal Service.
[   36.906446] multipathd[772]: Sep 18 11:59:15 | /etc/multipath.conf. See man mpathconf(8) for more details
[   36.908758] multipathd[772]: ok
[   36.909989] systemd[1]: Startup finished in 3.036s (kernel) + 0 (initrd) + 33.869s (userspace) = 36.906s.
------
Ignition has failed. Please ensure your config is valid. Note that only
Ignition spec v3.0.0+ configs are accepted.

A CLI validation tool to check this called ignition-validate can be
downloaded from GitHub:
    https://github.com/coreos/ignition/releases
------

Displaying logs from failed units: ignition-fetch.service
-- Logs begin at Fri 2020-09-18 11:58:41 UTC, end at Fri 2020-09-18 11:59:15 UTC. --
Sep 18 11:59:15 ignition[718]: Adding "Packstack" to list of CAs
Sep 18 11:59:15 ignition[718]: [0;1;31m[0;1;39m[0;1;31mUnable to decode CA (0xc00050a490)[0m
Sep 18 11:59:15 systemd[1]: [0;1;39m[0;1;31m[0;1;39mignition-fetch.service: Main process exited, code=exited, status=1/FAILURE[0m
Sep 18 11:59:15 ignition[718]: [0;1;31m[0;1;39m[0;1;31mUnable to parse CA bundle: unable to decode PEM block[0m
Sep 18 11:59:15 systemd[1]: [0;1;39m[0;1;31m[0;1;39mignition-fetch.service: Failed with result 'exit-code'.[0m
Sep 18 11:59:15 ignition[718]: [0;1;39m[0;1;31m[0;1;39mfailed to fetch config: unable to decode PEM block[0m
Sep 18 11:59:15 systemd[1]: [0;1;31m[0;1;39m[0;1;31mFailed to start Ignition (fetch).[0m
Sep 18 11:59:15 ignition[718]: [0;1;31m[0;1;39m[0;1;31mfailed to acquire config: unable to decode PEM block[0m
Sep 18 11:59:15 ignition[718]: [0;1;31m[0;1;39m[0;1;31mIgnition failed: unable to decode PEM block[0m
Sep 18 11:59:15 systemd[1]: ignition-fetch.service: Triggering OnFailure= dependencies.
Press Enter for emergency shell or wait 5 minutes for reboot.                 


.... now with ok.pem

# Generate the ignition shim
export PEMFILE=ok.pem
export MYTMP_IGNITION_HOST=$(openstack catalog show image | grep public | cut -d: -f2- | awk '{print $1}')
export MYTMP_IGNITION_URI=$(openstack image show ignition-${INFRA_ID}-boostrap -f value -c file)
export MYTMP_TOKEN=$(openstack token issue -c id -f value)
export MYTMP_CACERT_BASE64=$(cat $PREMFILE | base64 -w0)
[openstack ocp-proxy] [stack@osp16 upi]$ cat > $INFRA_ID-bootstrap-ignition.json <<EOF
{
  "ignition": {
    "config": {
      "merge": [{
        "source": "${MYTMP_IGNITION_HOST}${MYTMP_IGNITION_URI}", 
        "httpHeaders": [{
          "name": "X-Auth-Token", 
          "value": "${MYTMP_TOKEN}" 
        }]
      }]
    },
    "security": {
      "tls": {
        "certificateAuthorities": [{
          "source": "data:text/plain;charset=utf-8;base64,${MYTMP_CACERT_BASE64}" 
        }]
      }
    },
    "version": "3.1.0"
  }
}

EOF

shell$ ansible-playbook -i inventory.yaml 03_bootstrap.yaml 
shell$ openstack console log show "$INFRA_ID-bootstrap" | tail -30
[   83.301182] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   83.314623] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   84.135618] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   85.821226] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   85.825373] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   86.616417] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   87.739133] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   87.739787] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   87.743454] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   87.749200] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   87.888125] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   87.942578] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   88.119299] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   88.738561] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   88.743887] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)

Red Hat Enterprise Linux CoreOS 46.82.202009101640-0 (Ootpa) 4.6
SSH host key: SHA256:BBU2gNU8AUTCVqtNj/2NY15i8g9LBNLbt1pic1QIs50 (ECDSA)
SSH host key: SHA256:peD9L7rKXyK9vpJmUkYTnRPYm5DNXxAjM4+SUfCRNSk (ED25519)
SSH host key: SHA256:ccNoIRuNeWBFBK3mk+K9KytnrB8gJmmIRdw9i3Oc+SQ (RSA)
ens3: 192.168.150.105 fe80::f816:3eff:fe18:3c99
example-proxy-szkqv-bootstrap login: [   94.362674] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   94.382632] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   94.401596] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   95.621643] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   95.624142] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   95.802276] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   96.360331] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   96.503972] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[  119.234908] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)

shell$ diff -u ok.pem error.pem 
--- ok.pem	2020-09-18 11:29:59.510870304 +0000
+++ error.pem	2020-09-18 11:45:22.099448969 +0000
@@ -20,3 +20,4 @@
 pRlF2pB+bI4IPqMOk5k0JgewMYXc4PzSpyan7CfEPcazzrrcN9c2Kg57CiLRc91y
 cM0z3EayBt4z5LSUCTnfHCyA847yBc5bL9s=
 -----END CERTIFICATE-----
+

~~~

Comment 1 Martin André 2020-09-18 14:40:33 UTC
We thought we fixed this issue with https://bugzilla.redhat.com/show_bug.cgi?id=1813354. Pierre, do you mind have a look?

Comment 3 Martin André 2020-09-18 18:25:04 UTC
I missed this is UPI. We should update the script at [1]. Going to propose a fix.

[1] https://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md#edit-the-bootstrap-ignition

Comment 4 Robert Heinzmann 2020-09-18 20:13:46 UTC
Not sure if this is correct. The script in [1] is adding the CA Cert as a ignition FILE resource to the bootstrap.ign - this works. The error message is received when the bootstrap node is trying to download the real ignition via the ignition shim provided by userdata to the bootstrap instance. 

This means the user must fix the cert (documentation bug ?) or we fix ignition to be more robus.

[1] https://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md#edit-the-bootstrap-ignition

Comment 5 Martin André 2020-09-21 08:52:21 UTC
Indeed, we'll also need to tell users to have a valid certificate before they encode it to base64 in the ignition shim. I added some docs with https://github.com/openshift/installer/pull/4201/commits/e444f49c6a78f77d2f61d356cd1ed2255bfbe70c.

Comment 6 Robert Heinzmann 2020-09-21 09:00:49 UTC
Actually a simple way to do this is to pipe the cert through "openssl x509". 

Also, as the cert is injected into the bootstrap ignition shim with the Python script from the documentation for UPI, this script could also be fixed.

~~~
shell$ openssl x509 -in with_newline.pem -out without_newline.pem
shell$ diff -u with_newline.pem without_newline.pem 
--- with_newline.pem	2020-09-21 08:58:38.229545372 +0000
+++ without_newline.pem	2020-09-21 08:58:45.763903993 +0000
@@ -20,4 +20,3 @@
 pRlF2pB+bI4IPqMOk5k0JgewMYXc4PzSpyan7CfEPcazzrrcN9c2Kg57CiLRc91y
 cM0z3EayBt4z5LSUCTnfHCyA847yBc5bL9s=
 -----END CERTIFICATE-----
-
~~~

Comment 10 weiwei jiang 2020-09-23 07:01:45 UTC
Verified.


[   36.640590] ignition[700]: GET result: OK                                                                           
[   36.642723] ignition[700]: warning at $.networkd, line 28 col 3: Unused key networkd                           
[   36.644921] ignition[700]: warning at $.storage.files.0.mode, line 34 col 17: Unused key filesystem                                                                                                                                        
[   36.647279] ignition[700]: fetched user config from "openstack"                                                                                                                                                                            
[   36.648754] ignition[700]: warning at $.storage.files.1.mode, line 43 col 17: Unused key filesystem                                                                                                                                        
[   36.659790] ignition[700]: Adding "10.8.100.190" to list of CAs                                               
[   36.661887] ignition[700]: Adding "192.168.24.2" to list of CAs                                                                                                                                                                            
[   36.663663] ignition[700]: GET https://10.8.100.190:13292/v2/images/3c2eec62-8a71-4e69-8226-654d3c46037a/file: attempt #1                                                                                                                  
[   37.736731] ignition[700]: GET result: OK                                                                           
[   37.740548] ignition[700]: fetched referenced user config from "/v2/images/3c2eec62-8a71-4e69-8226-654d3c46037a/file"                                                                                                                      
[   37.768083] ignition[700]: Adding "10.8.100.190" to list of CAs                                                                                                                                                                            
[   37.769672] ignition[700]: Adding "192.168.24.2" to list of CAs                                            
[   37.771154] ignition[700]: Adding "10.8.100.190" to list of CAs                                               
[   37.772613] ignition[700]: Adding "192.168.24.2" to list of CAs                                                                                                                                                                            
[   37.779221] ignition[700]: fetch: fetch complete                                                                    
[   37.780615] ignition[700]: fetch: fetch passed

Comment 13 errata-xmlrpc 2020-10-27 16:42:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.