Bug 1876835

Summary: master ignitions fails in UPI install with Traceback (most recent call last): File "<string>", line 3, in <module> KeyError: 'storage'
Product: OpenShift Container Platform Reporter: Jatan Malde <jmalde>
Component: InstallerAssignee: Pierre Prinetti <pprinett>
Installer sub component: OpenShift on OpenStack QA Contact: David Sanz <dsanzmor>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: unspecified CC: pprinett, rheinzma
Version: 4.6   
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-08 15:05:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jatan Malde 2020-09-08 10:17:41 UTC
Description of problem:

Using the latest feature build and following this section for OpenShift on OpenStack UPI installation. 

https://docs.openshift.com/container-platform/4.6/installing/installing_openstack/installing-openstack-user.html#installation-osp-creating-control-plane-ignition_installing-openstack-user

(overcloud) [stack@undercloud-0 bugtest]$ ./openshift-install version
./openshift-install 4.6.0-fc.4
built from commit 878562e15ce6c31148b4cff758ea70e685e6b6e3
release image quay.io/openshift-release-dev/ocp-release@sha256:960ec73733150827076cbb5fa2c1f5aaa9a94bfbce1b4897e46432a56ac976c1
(overcloud) [stack@undercloud-0 bugtest]$
(overcloud) [stack@undercloud-0 bugtest]$ ./openshift-install create manifests
INFO Credentials loaded from file "/home/stack/shift/upi/bugtest/clouds.yaml" 
INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
INFO Manifests created in: manifests and openshift 
(overcloud) [stack@undercloud-0 bugtest]$ rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml openshift/99_openshift-cluster-api_worker-machineset-*.yaml
(overcloud) [stack@undercloud-0 bugtest]$ ./openshift-install create ignition-configs
INFO Consuming Worker Machines from target directory 
INFO Consuming Common Manifests from target directory 
INFO Consuming Master Machines from target directory 
INFO Consuming OpenShift Install (Manifests) from target directory 
INFO Consuming Openshift Manifests from target directory 
INFO Ignition-Configs created in: . and auth      
(overcloud) [stack@undercloud-0 bugtest]$ export INFRA_ID=$(jq -r .infraID metadata.json)
(overcloud) [stack@undercloud-0 bugtest]$ for index in $(seq 0 2); do
>     MASTER_HOSTNAME="$INFRA_ID-master-$index\n"
>     python -c "import base64, json, sys;
> ignition = json.load(sys.stdin);
> files = ignition['storage'].get('files', []);
> files.append({'path': '/etc/hostname', 'mode': 420, 'contents': {'source': 'data:text/plain;charset=utf-8;base64,' + base64.standard_b64encode(b'$MASTER_HOSTNAME').decode().strip()}});
> ignition['storage']['files'] = files;
> json.dump(ignition, sys.stdout)" <master.ign >"$INFRA_ID-master-$index-ignition.json"
> done
Traceback (most recent call last):
  File "<string>", line 3, in <module>
KeyError: 'storage'
Traceback (most recent call last):
  File "<string>", line 3, in <module>
KeyError: 'storage'
Traceback (most recent call last):
  File "<string>", line 3, in <module>
KeyError: 'storage'
(overcloud) [stack@undercloud-0 bugtest]$

The steps mention running the above command to create 3 master ignitions but since the above messages are reported the files which created are empty. 

Ideally the above error should be more self explanatory and the stacktrace is not helpful.

Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
ansible --version

How reproducible:

Steps to Reproduce:
1. Use openshift-install binary as listed above.
2. Follow the steps mentioned in the UPI documentation and run the command as mentioned to create 3 master ignition file.
3. 

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

From the build I see the ignitions created are of v3.1.0 and the python script does not fail for bootstrap since the storage section is available. 
Take a look at the master.ign file and I do not see the storage section and so is the above stacktrace. 

(overcloud) [stack@undercloud-0 bugtest]$ cat master.ign | jq .
{
  "ignition": {
    "config": {
      "merge": [
        {
          "source": "https://30.0.0.5:22623/config/master"
        }
      ]
    },
    "security": {
      "tls": {
        "certificateAuthorities": [
          {
            "source": "data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lJSXg4QWY4dUU0N1F3RFFZSktvWklodmNOQVFFTEJRQXdKakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1SQXdEZ1lEVlFRREV3ZHliMjkwTFdOaE1CNFhEVEl3TURrd09EQTVOVFV6TmxvWApEVE13TURrd05qQTVOVFV6Tmxvd0pqRVNNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUkF3RGdZRFZRUURFd2R5CmIyOTBMV05oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF2V2FQUGxmOWg4RWoKSmxEZjloaldCZ1pSVDRycDdXOEcyeS9PMXRabFBhdkErNkcxTHFmNFd4MVZ1OWVJeFBiQkl1bytVMkZ4MzNkWQp0SUx4T1BGTmlkRlcybTMrM3RzTUtiaHEvbnJ0Q0U5L0lBNUovYzhQSFRkaVhaV2RxUHZJYjBPQnFKbUl3Yk03CllIQUJNNnhsVjFqYUJ6YTczQ0E4SWVnallLcjZtVVlHV3pteXptanRjTTRDM2t1dXJ0WmhLZUZSSzBKK3FtM1oKVzZJZC9wazNSMmlsZytHYkl1NG9EakVPVjJXWWZzZ0hzenRQSkFQYm05T3VhbUN6VWhkSnpYWjBQU2RnbzhkQgpaT0NhSzRVZGJDRFVCbFo5SUxxemRaMGVpajJYWk1kM1NCdFNydTFsZnJSbk1WdFMwaExIa3A1V2MzVmdaTzFvCk5pS0JnVDkzendJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVWNsRHBlTHJxY3pzUzRoSFpxd3h1SmNHa093QXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUxXMzNRcC83N3crL1U2T1lHS1loUnBPenh1eWY2M280cVlFNUM5WU4wNVlsOGk5S2wydWo3UWxTZk41CnR6cWFlaU1CSHBhTHZzb1dSU0ZycWRqc0VsU2ZJM29aeElFSDN3VGJEaVB2ckxpU3NmSEc5SUxEZDNmNnY4b1kKU3dKMDNwY3hscGtwb24zdEpsQlV2NHlSTlJ2Nno4Q2x5V3BGOHZUengyZGdNdDlwTkVKOEpGSmphRW92bzdFLwozT1hmVHlmM0FNVVRUS0swTzBLd2VHQ0hyS0pROVdiMEZyVmtreDVPVUI0c1JBVngxTGpMWjZvMGRYczVpUGwvCk9wWk9QM0hTcXgyVnI0S210Mk1jTndZcGQrZmhOM0xIZFUwcitKTlYxcW1FMjEvQ2hLaFRtWlFMK0IwcnhWY2kKWmFCR0x4WUFVTGtwUmxwUTZmNXh4VXRTRlY0PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
          }
        ]
      }
    },
    "version": "3.1.0"
  }
}

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Pierre Prinetti 2020-09-08 15:05:25 UTC
This should be fixed with the current upstream documentation[1].


[1]: https://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md#master-ignition

*** This bug has been marked as a duplicate of bug 1868439 ***