Bug 2037626

This looks like there is an error in the Machine Config Server (it is returning a 500 error when querying for bootstrap.ign) can you attach a must gather log bundle or the machine config server logs?

Tang disk encryption is only supported starting in ignition config version 3.2.0. The request coming from the ansible playbook is not requesting an explicit ignition config version. The default served by the machine-config-server is something older than version 3.2.0, so the machine-config-server is balking at creating the ignition config. It seems to me that the machine-config-server should serve a ignition config using the necessary config version when the request is not explicitly requesting a config version.

As such, this seems like a machine-config-server bug to me. Feel free to send it back if you feel that the requester should instead need to know the details of what ignition config version is required.

Hi Zack,

sorry that seems I didn't describe the issue clearly in Description.

In task of "Fetch bootstrap ignition file locally", it is as below in product code, there is no any Accept headers defined, and got the error in Comment 4.

- name: Fetch bootstrap ignition file locally
  uri:
    url: "{{ openshift_node_bootstrap_endpoint }}"
    dest: "{{ temp_dir.path }}/bootstrap.ign"
    validate_certs: false
    http_agent: "Ignition/0.35.0"
  delay: 10
  retries: 60
  register: bootstrap_ignition
  until:
  - bootstrap_ignition.status is defined
  - bootstrap_ignition.status == 200

After adding Accept headers as below, it works.

- name: Fetch bootstrap ignition file locally
  uri:
    url: "{{ openshift_node_bootstrap_endpoint }}"
    dest: "{{ temp_dir.path }}/bootstrap.ign"
    validate_certs: false
    headers:
      Accept: application/vnd.coreos.ignition+json; version=3.2.0 
    http_agent: "Ignition/0.35.0"
  delay: 10
  retries: 60
  register: bootstrap_ignition
  until:
  - bootstrap_ignition.status is defined
  - bootstrap_ignition.status == 200

Ansible debug log has been attached in comment 1, please check.

So a few things here:

1. The expected behaviour (MCO serving spec 2) is in place for historical reasons to ease transition to spec 3. We have opened up a new card for changing the default https://issues.redhat.com/browse/MCO-154.

2. I think generally, we would like to have anything calling ignition/MCS directly to provide the necessary accept header. One thing is this can help prevent bugs in the future for version bumps bringing in unexpected behaviour, when the caller is not ready to switch to the new version. (A guess a counterpoint is that, without specifying the version, the ansible repo will not need updates when the MCO changes spec versions, which may or may not be ideal. In any case, all versions should always be supported. In this case, tang only works on 3.2+, so a system that still uses spec 2 would not be able to parse it).

I am going to send back to the installer team for consideration of adding that header to the ansible playbooks to fix this for existing versions, and the MCO will work on changing the default for future releases.

For (2), it is acceptable to me that your preference is that the caller provide an accept header. But, in the absence of that, as a consumer of your API, I take on the risk of receiving whichever version you choose to send to me. What I do not expect is that, when I tell your API that I do not care which version I get, your server responds that it cannot give me a response because the server cannot generate the version that the server itself chose to use. I expect the server to generate the config using whichever version that it has to use. If that is a version that the client cannot ultimately use, then it is the responsibility of the client to look at the version in the response and determine that.

Nevertheless, we will look into making the ansible more robust, although it is not clear to me yet how.

@jerzhang Can you remind me which versions of ignition config are supported in which versions of OpenShift?

I am marking this as a non-blocker since there is a workaround whereby the user running the ansible playbook edits the playbook to pass accept headers when requesting the ignition config from the MCO.

@jerzhang What is the behavior of the MCS when the accept headers include both v2.2 and v3.2? Would the MCS serve v3.2 (1) always, (2) only when v3.2 is required, or (3) still try to serve v2.2 and fail?

> I take on the risk of receiving whichever version you choose to send to me. What I do not expect is that, when I tell your API that I do not care which version I get, your server responds that it cannot give me a response because the server cannot generate the version that the server itself chose to use. I expect the server to generate the config using whichever version that it has to use. If that is a version that the client cannot ultimately use, then it is the responsibility of the client to look at the version in the response and determine that.

That is also an improvement we can look to make (if the spec contains fields that are not in prior versions - in this case, tang encryption, serve the latest)

> Can you remind me which versions of ignition config are supported in which versions of OpenShift?

OCP Version 4.1->4.4 supports 2.2
OCP Version 4.5 supports 2.2, 3.0
OCP Version 4.6 supports 2.2. 3.0, 3.1
OCP Version 4.7+ supports 2.2, 3.0, 3.1, 3.2 <-added tang

For details, please see: 1. Ignition (spec) version to OCP version of https://docs.google.com/document/d/1HfGU-kZdogPb2EBnEVWU1ZLLsX05vkBhwI4JrQxUtkE/edit#

> What is the behavior of the MCS when the accept headers include both v2.2 and v3.2? Would the MCS serve v3.2 (1) always, (2) only when v3.2 is required, or (3) still try to serve v2.2 and fail?

I think we just match ignition behaviour on this. We don't have that kind of nuance. If you for some reason provide both, it just takes the first section it sees that has "vnd.coreos.ignition+json" and serves it, meaning that if your header lists 2.2 first, it will serve 2.2, and 3.2 if it's first.

> I think we just match ignition behaviour on this. We don't have that kind of nuance. If you for some reason provide both, it just takes the first section it sees that has "vnd.coreos.ignition+json" and serves it, meaning that if your header lists 2.2 first, it will serve 2.2, and 3.2 if it's first.

@jerzhang If we list the versions in decreasing order, will the MCS skip versions that it does not recognize? Or will the MCS balk on unrecognized versions?

In other words, can we add the accept header "application/vnd.coreos.ignition+json;version=3.2.0, application/vnd.coreos.ignition+json; version=2.2.0" and let MCS pick 3.2.0 on OpenShift 4.7+ and 2.2.0 on earlier versions?

This wouldn't work. In 4.6 for example, this would error because only 3.1/2.2 is supported, and any other headers would cause an error I am pretty sure

MCS has to serve up archaic Ignition due to the fact that we don't lifecycle RHCOS boot media however openshift-ansible always runs an MCD that matches the cluster version[1], therefore we know that it should always support the latest Ignition spec available in that version of MCS and MCD. How about hardcoding the version for now but extending the MCD accept an argument and emit its Ignition spec version so that we can then request the correct version without updating code into the future?

1 - https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_node/tasks/apply_machine_config.yml#L40-L51

The installer team can resolve this bz by implementing the solution in comment 14, by hardcoding the value and await future work to determine the appropriate ignition version programmatically.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069