Bug 1934176 - Machine Config Operator degrades during cluster update with failed to convert Ignition config spec v2 to v3
Summary: Machine Config Operator degrades during cluster update with failed to convert...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Zack Zlotnik
QA Contact: Michael Nguyen
Depends On:
Blocks: 1956462
TreeView+ depends on / blocked
Reported: 2021-03-02 16:27 UTC by Christian Passarelli
Modified: 2021-06-15 18:00 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2540 0 None open Bug 1934176: ensure SSH key uniqueness 2021-04-30 00:22:38 UTC
Red Hat Knowledge Base (Solution) 5723941 0 None None None 2021-03-12 10:00:50 UTC

Description Christian Passarelli 2021-03-02 16:27:24 UTC
Description of problem:
Upgrading from 4.5.16 to 4.6.8, if duplicated sshAuthorizedKeys are present for infra nodes that are already present for worker nodes, MCO is not able to update infra nodes degrading with following error in the Machine Config Pool:

machineconfiguration.openshift.io/reason: 'can''t reconcile config rendered-infra-9151dbbd7f4d73a6f9963622d9114e14
      with rendered-infra-4e01833bf0ec9fe78ccd807b17886b30: parsing old Ignition config
      failed with error: failed to convert Ignition config spec v2 to v3: unable to
      convert Ignition spec v2 config to v3: error at $.passwd.users.0.sshAuthorizedKeys.1:
      duplicate entry defined

      : unreconcilable'

Version-Release number of selected component (if applicable):

How reproducible:
Not sure.

Steps to Reproduce:
1. Install 4.5.16 cluster.
2. Create new Machine Config Pool for infra nodes adding a duplicate MC for core sshkey.
3. Update the cluster to 4.6.8

Actual results:
Infra nodes are not updated.

Expected results:
Machine Config Operator should be able to continue the update successfully.

Additional info:

Comment 3 Yu Qi Zhang 2021-03-03 01:37:34 UTC
Ah, ok the inspect has the info needed. I think the error is a variation of https://bugzilla.redhat.com/show_bug.cgi?id=1880603 that still exists today, where the ignition parser doesn't like the fact that there are two instances of the exact same ssh key.

For the time being, there are a few fixes that should work, first I think we should delete the "99-infra-ssh" machineconfig, as the 99-worker-ssh machineconfig has the exact same key and also applies to infra nodes. https://access.redhat.com/solutions/5723941 actually should work for this, but in general we should try not to promote modifying the rendered-config in any way. Try modifying the config as follows:

oc edit mc/rendered-infra-9151dbbd7f4d73a6f9963622d9114e14.yaml

edit the config such that

      version: 2.2.0
      - name: core
        - ssh-rsa ...
      - name: core
        - ssh-rsa ...


      version: 2.2.0
      - name: core
        - ssh-rsa ...

The MCD should do the rest. I will try to see the feasibility of backporting a fix to 4.6

Comment 5 Yu Qi Zhang 2021-03-05 22:37:31 UTC
Setting blocker- but will consider fixing and backporting to 4.6

Note You need to log in before you can comment on or make changes to this bug.