Description of the problem: As a follow-up to https://bugzilla.redhat.com/show_bug.cgi?id=2065288, it is observed that ignition overrides for day-2 node deployments appear to have problems when storage is overridden. It works sometimes for the node, other times it does not where RHACM 2.5 is driving the day-2 worker node addition. This is for use cases where this is only one block device in the node, and the user wishes to reduce the size of the root partition and create additional partitions for LSO (local storage operator) using the resize function in ignition 3.2.0. More background is in the comments in Jira ticket here: https://issues.redhat.com/browse/MGMTBUGSM-201. Release version: Operator snapshot version: advanced-cluster-management.v2.5.0 OCP version: ACM running on 4.9.38 Cluster adding day 2 node on 4.8.44 Steps to reproduce: 1. Add host to existing cluster using Secret and BareMetalHost CRDs using ignition-config-overrides 2. Observe node to boot off the generated ISO and write to disk 3. First boot exhibits the error often, but not ever time. Expected results: Host disk config is overridden and works everytime. Additional info: I will attach logs and AgentClusterInstall and other relevant CRDs to this BZ.
Some of my manifests are here as requested: Override specifically is here: https://github.com/novacain1/carslab-public/blob/main/rhocp-clusters/caravan.cars.lab/ztp/10-bmh-cu2.yaml#L24 BMH & Secret CRD: https://github.com/novacain1/carslab-public/blob/main/rhocp-clusters/caravan.cars.lab/ztp/10-bmh-cu2.yaml ACI CRD: https://github.com/novacain1/carslab-public/blob/main/rhocp-clusters/caravan.cars.lab/ztp/02-aci.yaml
I managed to capture the logs from the assisted-service pod for the same node where this worked (and the node was added to the cluster Day-2 with the storage modifications), as well as log output for where this action failed. Attaching them here as requested by @mko.
Hey Mat. Ok, reproduced this again on my first attempt with a fresh cluster deployed from ACM 2.5.1. Target cluster version is 4.10.22. From the assisted-service pod in the open-cluster-management namespace in my cluster hosting ACM, I managed to capture this which you asked for: $ oc rsh assisted-service-7df9d4c6b9-hsvz4 $ cat /data/c53c55fc-c6f8-4a31-9a87-1efece4f81ec/worker-81acb6aa-f73c-a5f7-a3c1-fa760c4fb43f.ign {"ignition":{"config":{"merge":[{"source":"http://10.50.0.98:22624/config/worker","verification":{}}],"replace":{"verification":{}}},"proxy":{},"security":{"tls":{"certificateAuthorities":[{"source":"data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lJTUp4T1ZNTWtia1V3RFFZSktvWklodmNOQVFFTEJRQXdKakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1SQXdEZ1lEVlFRREV3ZHliMjkwTFdOaE1CNFhEVEl5TURjeU5USXdOREkwTWxvWApEVE15TURjeU1qSXdOREkwTWxvd0pqRVNNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUkF3RGdZRFZRUURFd2R5CmIyOTBMV05oTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUE0SEVhNmsxNlM0bFkKcENUcnczaElJQVpUR3hBY056MVdyZWU2M1lkUVF6MW1zVWFzRDE0aEpjdVRqT0tmNXNkMy9tTlN6ZjVGVnpidwpEUnBpMzNNQ0ZBYkpzWTQzQ29yejBoSlNPdFRUN2Fza2dhSkEzUFZqTXA4RDlWZlVOUkFza2NVZjNKRUVMQmJoCmQwY1RxWGFJSVlZcGxXSm9SbUQzUEZ3VEN4dUQ0NE9tbE9vS1YrOHRKNk8vdmZnUENsTWh4SXc1UEtrQ0c0REYKY3plandMbUE5Q0duaHRBaTVUUXVnN0ljaVU3MU1vck14RXBRaG1sbFBYZ05xWGJBVmRKbzd6SUczRk5PSXFjMApvTmUwbUxtVXFyV1BUbWZMdmU3ZElJL2RLY3N3SHVXSnV0SkhjbGVJMGhuV01aaFhNazhkRHU3UTRQcTdiYU9oCkliWlVSNjdaVHdJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVVNCT3hVSkx6K1VRR1J4Q2ZZSXVZcEpBaHc0b3dEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUkxMzBEY01FSi9LQXg4am8zWUZjUS8zbXdTVlZUMkErZ2VsTWlYY2xNTTc5eDJMS015Q0o0QUlnYmJRCm9NN1UvQnpGeWlnT2JWd3hwODMwQ1AwMDU3Z2xFdk5EL1dUbnNmSGFkRXdWZTg4UUNTSjEvaEtIVlZ1cU81RUgKckloMVlQbjMwVW1keWZHd3A2SHc3WHp6UGZBVmRZQWVrL2RNdnpaUklvWVl3a0RGWmd3dm56alU0S0JOQk56MApOVWc2ZFBMc21zd0lMNjZFWmV3c0tlRkdocHRneXdkdVpsc3BrN3dydm8xd2wyRVFWVmZmaGxkQWgwRUhpOUo2CkVyV1h0eVVHcE0yNm9rK1BjemV0S1dJeEQzVURHM1kwdS84Nk1iK0F6L0NpSWdaazUveGJkanJRMUtWeW51eS8KM2drWlNBOC95MEJvV3IyaUZWdXJ0Z1Z6SlBJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==","verification":{}}]}},"timeouts":{},"version":"3.1.0"},"passwd":{},"storage":{"disks":[{"device":"/dev/sdb","partitions":[{"label":"root","number":4,"sizeMiB":204800},{"label":"lso_20G_1","sizeMiB":20000},{"label":"lso_20G_2","sizeMiB":20000},{"label":"lso_20G_3","sizeMiB":20000}],"wipeTable":false}],"files":[{"group":{},"overwrite":true,"path":"/etc/hostname","user":{"name":"root"},"contents":{"source":"data:,cu2","verification":{}},"mode":420}]},"systemd":{}} I can provide access to my environment if that would be helpful, you'll need to give me a few pieces of information in a form (that I can send you) as my lab is outside of Red Hat. I'm also asking my colleagues Hamza and Sarp (Cced here) to recreate this as well, where his lab is accessible inside the Red Hat network. Let me know if you need anything else.
Thanks! From what I see here it looks like Assisted Service correctly generated a worker ignition with your overrides. Access to the lab will be definitely helpful now as the file you shown me is kind of killing all the working theories I had till now. What direction I would be taking now is to understand whether the node pulled this ignition (with overrides) and for what reason it failed to apply it as expected or whether the node pulled another file (if so, which and why). Please send me the form for whatever info you need from me.
Posting an update as last days lots of work was happening via emails. We have managed to find a race inside BMAC that got its own BZ (https://bugzilla.redhat.com/show_bug.cgi?id=2112321). Some PRs that address it are already on the way, https://github.com/openshift/assisted-service/pull/4201 being the most important. There still needs to be some testing to see how it changes the behaviour from this BZ here, but in principle we want to backport it (potentially all the way down to 2.4). For the sake of completeness, whenever we have a reproducer or just an attempt that gives an interesting result that needs investigation, the following is a set of commands/resources that are needed to have a picture of what happened * oc logs --tail=-1 -n open-cluster-management assisted-service-7d[...]lc -c assisted-service * oc get bmh -n caravan-cars-lab cu2-caravan-cars-lab -oyaml * oc get agent -n caravan-cars-lab 81a[...]43f -oyaml * oc exec -n open-cluster-management assisted-service-7d[...]lc -it -- cat /data/daa[...]68d/worker-81a[...]43f.ign
To give an update, I have managed to reproduce the issue and get a proper dump from the serial console of the machine affected. Please note this is before 4201 got merged, but is still worth discussing. What can be observed here is 1) ignition override provided by the user to the AI is not malformed and makes it to the worker node as-is 2) RHCOS fails to process the ignition even though its format is correct - the issue comes from the `sgdisk` command Below some of the serial log indicating what happened ``` [ 4.023865] systemd[1]: Starting Ignition (disks)... [ 4.038201] ignition[864]: Ignition 2.13.0 [ 4.038987] ignition[864]: Stage: disks [ 4.039634] ignition[864]: reading system config file "/usr/lib/ignition/base.d/00-core.ign" [ 4.040529] ignition[864]: no config dir at "/usr/lib/ignition/base.platform.d/metal" [ 4.073779] ignition[864]: disks: createPartitions: op(1): [started] waiting for devices [/dev/sda] [ 4.076157] ignition[864]: disks: createPartitions: op(1): [finished] waiting for devices [/dev/sda] [ 4.077727] ignition[864]: disks: createPartitions: created device alias for "/dev/sda": "/run/ignition/dev_aliases/dev/sda" -> "/dev/sda" [ 4.079318] ignition[864]: disks: createPartitions: op(2): [started] partitioning "/run/ignition/dev_aliases/dev/sda" [ 4.079907] sda: sda1 sda2 sda3 sda4 [ 4.081072] ignition[864]: disks: createPartitions: op(2): op(3): [started] reading partition table of "/run/ignition/dev_aliases/dev/sda" [ 4.083164] ignition[864]: disks failed [ 4.083762] ignition[864]: disks: createPartitions: op(2): op(3): [finished] reading partition table of "/run/ignition/dev_aliases/dev/sda" [ 4.085525] ignition[864]: disks: createPartitions: op(2): running sgdisk with options: [--pretend --delete=4 --new=4:1050624:+209715200 --new=0:0:+20971520 --new=0:0:+20971520 --new=0:0:+20971520 /run/ignition/dev_aliases/dev/sda] [ 4.087971] ignition[864]: disks: createPartitions: op(2): [failed] partitioning "/run/ignition/dev_aliases/dev/sda": Failed to pretend to create partitions. Err: exit status 4. Stderr: Could not create partition 4 from 1050624 to 210765823 [ 4.087971] Error encountered; not saving changes. [ 4.153815] ignition[864]: Full config: [ 4.155724] ignition[864]: { [ 4.156320] ignition[864]: "ignition": { [ 4.156976] ignition[864]: "config": { [ 4.157652] ignition[864]: "merge": [ [ 4.158313] ignition[864]: { [ 4.168996] ignition[864]: "verification": {} [ 4.169742] ignition[864]: } [ 4.170347] ignition[864]: ], [ 4.170937] ignition[864]: "replace": { [ 4.171619] ignition[864]: "verification": {} [ 4.172315] ignition[864]: } [ 4.172867] ignition[864]: }, [ 4.173420] ignition[864]: "proxy": {}, [ 4.174070] ignition[864]: "security": { [ 4.174660] ignition[864]: "tls": {} [ 4.175289] ignition[864]: }, [ 4.175796] ignition[864]: "timeouts": {}, [ 4.176578] ignition[864]: "version": "3.4.0-experimental" [ 4.177282] ignition[864]: }, [ 4.177855] ignition[864]: "kernelArguments": {}, [ 4.178492] ignition[864]: "passwd": { [ 4.179054] ignition[864]: "users": [ [ 4.179610] ignition[864]: { [ 4.180070] ignition[864]: "gecos": "CoreOS Admin", [ 4.180684] ignition[864]: "groups": [ [ 4.181283] ignition[864]: "adm", [ 4.181793] ignition[864]: "sudo", [ 4.182377] ignition[864]: "systemd-journal", [ 4.182970] ignition[864]: "wheel" [ 4.183492] ignition[864]: ], [ 4.183975] ignition[864]: "name": "core", [ 4.184552] ignition[864]: "sshAuthorizedKeys": [ [ 4.185154] ignition[864]: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsLJKEMcZ+i+sHFpHH8s5pjv8ThEdrP/vlXvpiLdb13hvwzxw0/eV9zmeWREqndr9c5+bvOpmgSVX1f6sV++DsLu3y0Ilgn8bnRLe5ttC0hjCVSScev634UTZZ89GleNybEnedkDBp+k5ZXhEBx+ofCtlzQYScTL0En0Fj9+OJmGETr7J+itqBjhGpB036k5gAvqmEb80wFdhv0CTfEqEEs9jxL0gx0QB90FcM4XZ6K43ZVd74Xq2U/oMiGHhr9Sn/IMiIOjHFAVrnSuCQvvV8gCq/vZDL0C7H9+c0vnhnBZF0PY0Z4afKPNKWPWDxptK0rLbPpvR/xbFN+ZTFgeu23Tz6ZS8mop2MCzoFxfKYC0LUHLd0EpDAqY7WZOs/PhdEdexs+GOOsKutRcMJkvrEpMjwTy4d34TKItOah5eRtKJDJ8CteolFFBsdjC/AtRQR+JyC3sZioyJzwXGzfgei+u5E2KfcVbNPrYYwDu89kOfqppKZOwYV+BQpJyO6MO0= root@ipi-ci-op-63nkki56-a74b6-1553100853160710144" [ 4.190435] ignition[864]: ] [ 4.190888] ignition[864]:} [ 4.191354] ignition[864]: ] [ 4.191778] ignition[864]: }, [ 4.191785] ignition[864]: "sto rage": { [ 4.192674] ignition[864]: "disks": [ [ 4.193199] ignition[864]: { [ 4.193635] ignition[864]: "device": "/dev/sda", [ 4.193642] ignition[864]: "partitions": [ [ 4.194723] ignition[864]: { [ 4.195239] ignition[864]: "label": "root", [ 4.195805] ignition[864]: "number": 4, [ 4.196343] ignition[864]: "resize": true, [ 4.196900] ignition[864]: "sizeMiB": 102400 [ 4.197550] ignition[864]: }, [ 4.198030] ignition[864]: { [ 4.198489] ignition[864]: "label": "lso_10G_1", [ 4.199145] ignition[864]: "sizeMiB": 10240 [ 4.199718] ignition[864]: }, [ 4.200183] ignition[864]: { [ 4.200697] ignition[864]: "label": "lso_10G_2", [ 4.201296] ignition[864]: "sizeMiB": 10240 [ 4.201907] ignition[864]: }, [ 4.202395] ignition[864]: { [ 4.202858] ignition[864]: "label": "lso_10G_3", [ 4.203460] ignition[864]: "sizeMiB": 10240 [ 4.204031] ignition[864]: } [ 4.204504] ignition[864]: ], [ 4.204954] ignition[864]: "wipeTable": false [ 4.205514] ignition[864]: } [ 4.205935] ignition[864]: ], [ 4.206351] ignition[864]: "files": [ [ 4.206803] ignition[864]: { [ 4.207213] ignition[864]: "group": {}, [ 4.207745] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/kubeconfig", [ 4.208494] ignition[864]: "user": {}, [ 4.208973] ignition[864]: "contents": { [ 4.209452] ignition[864]: "compression": "gzip", [ 4.210024] ignition[864]: "verification": {} [ 4.210544] ignition[864]: } [ 4.210939] ignition[864]: }, [ 4.210946] ignition[864]: { [ 4.211699] ignition[864]: "group": {}, [ 4.212194] ignition[864]: "path": "/etc/kubernetes/manifests/etcd-pod.yaml", [ 4.212904] ignition[864]: "user": {}, [ 4.213366] ignition[864]: "contents": { [ 4.213890] ignition[864]: "compression": "gzip", [ 4.214466] ignition[864]: "verification": {} [ 4.215009] ignition[864]: } [ 4.215481] ignition[864]: }, [ 4.215947] ignition[864]: { [ 4.216579] ignition[864]: "group": {}, [ 4.217087] ignition[864]: "path": "/etc/kubernetes/manifests/kube-apiserver-pod.yaml", [ 4.217833] ignition[864]: "user": {}, [ 4.218324] ignition[864]: "contents": { [ 4.218828] ignition[864]: "compression": "gzip", [ 4.219401] ignition[864]: "verification": {} [ 4.219946] ignition[864]: } [ 4.220379] ignition[864]: }, [ 4.220780] ignition[864]: { [ 4.221175] ignition[864]: "group": {}, [ 4.221182] ignition[864]: "path": "/etc/kubernetes/manifests/kube-controller-manager-pod.yaml", [ 4.222676] ignition[864]: "user": {}, [ 4.223192] ignition[864]: "contents": { [ 4.223697] ignition[864]: "compression": "gzip", [ 4.225915] ignition[864]: "verification": {} [ 4.226495] ignition[864]: } [ 4.226926] ignition[864]: }, [ 4.227420] ignition[864]: { [ 4.227833] ignition[864]: "group": {}, [ 4.228343] ignition[864]: "path": "/etc/kubernetes/manifests/kube-scheduler-pod.yaml", [ 4.229231] ignition[864]: "user": {}, [ 4.229758] ignition[864]: "contents": { [ 4.230289] ignition[864]: "compression": "gzip", [ 4.230911] ignition[864]: "source": "data:;base64,H4sIAAAAAAAC/6RTwY7bOAy95ysEX+ayjhNn4kkMBNjDXvawswEK9FL0QEu0I0QWDYpOkRb990JO6jjTGQzQHik+Pr4nktDZj8jBki/VaTk7Wm9KtScza1HAgEA5U8pDi6WqiCQIQ5ce+wrToA9oeod8BYQONJbqkjsHwXamlIMKXYgcSlGHPhxsLXNLmSYvTC7tHHgsVSLcY/IarO3Io5dSJWPDiAPvSUAs+dfYufepwxO6UiWLC60ANyjzL8RHR2Dmd/gWPDTYDn0eviVY16glKVWyZ6yRGc0/PVvffLhIsL5Jvj/MQoc6No9ewHrkQUp6/a5fPkkp20KDpWJsbBA+z6veOrPI59pO5BA3mbYpdWmx8sejXRdZEKgc/h0OkK+LsoZNXsPmsarqYq3zzWpZrzbr/KmoK5MXha7MY/0IT6vtolos12azXBe40YtqlRfLrdlub1L2vXN7clafS/Vv/UyyZwzoZUBoaluI2/ApOZw75Ogn+Usl976SzwMYuLnOIVXpsB+afG2bXYaisxizR8GQBdSMErIbZKxyCAY5RYdadnEfxoxGltRY3mUn4DjcCeMI6ohltxhD6OWAXqweluQ3JUUSYvv1TzgcNUJBDDLvanDhZgtcoEl2fD/t8ml1WluHF+uOmux2hnc3lN3PZe7oIuFErm/xP+q9jBNqY7QHOZTqDSMD8Ofh378xgvnfu3OpJjOaMr4jdHbH7aiJxAcK8owSr3PkvSi/3lQEDPzX6u5V9beWU80vXbzJ9p7yF7J/BAAA//+BDVRNPAUAAA==", [ 4.238737] ignition[864]: "verification": {} [ 4.239394] ignition[864]: } [ 4.239955] ignition[864]: }, [ 4.240490] ignition[864]: { [ 4.240993] ignition[864]: "group": {}, [ 4.241627] ignition[864]: "path": "/usr/local/bin/bootstrap-in-place-post-reboot.sh", [ 4.242548] ignition[864]: "user": {}, [ 4.243143] ignition[864]: "contents": { [ 4.243771] ignition[864]: "compression": "gzip", [ 4.244475] ignition[864]: "verification": {} [ 4.245153] ignition[864]: }, [ 4.245697] ignition[864]: "mode": 365 [ 4.246296] ignition[864]: }, [ 4.246811] ignition[864]: { [ 4.247363] ignition[864]: "group": {}, [ 4.247945] ignition[864]: "path": "/var/log/log-bundle-bootstrap.tar.gz", [ 4.248807] ignition[864]: "user": {}, [ 4.249392] ignition[864]: "contents": { [ 4.249978] ignition[864]: "compression": "gzip", [ 4.273563] ignition[864]: "verification": {} [ 4.274239] ignition[864]: } [ 4.274802] ignition[864]: }, [ 4.275360] ignition[864]: { [ 4.275894] ignition[864]: "group": {}, [ 4.276509] ignition[864]: "path": "/usr/local/bin/installer-masters-gather.sh", [ 4.277394] ignition[864]: "user": {}, [ 4.278018] ignition[864]: "contents": { [ 4.278635] ignition[864]: "compression": "gzip", [ 4.279347] ignition[864]: "verification": {} [ 4.280049] ignition[864]: }, [ 4.280601] ignition[864]: "mode": 365 [ 4.280613] ignition[864]: }, [ 4.282121] ignition[864]: { [ 4.282657] ignition[864]: "group": {}, [ 4.283246] ignition[864]: "path": "/usr/local/bin/installer-gather.sh", [ 4.283254] ignition[864]: "user": {}, [ 4.285013] ignition[864]: "contents": { [ 4.285651] ignition[864]: "compression": "gzip", [ 4.286331] ignition[864]: "verification": {} [ 4.286928] ignition[864]: }, [ 4.287448] ignition[864]: "mode": 365 [ 4.287962] ignition[864]: }, [ 4.287971] ignition[864]: { [ 4.289138] ignition[864]: "group": {}, [ 4.289648] ignition[864]: "path": "/etc/kubernetes/bootstrap-configs/cluster-policy-controller-config.yaml", [ 4.290814] ignition[864]: "user": {}, [ 4.291324] ignition[864]: "contents": { [ 4.291839] ignition[864]: "compression": "gzip", [ 4.292448] ignition[864]: "source": "data:;base64,H4sIAAAAAAAC/6yOQUsDMRCF7/sr8gey2yIi5FYKggf1IHjPJm/XIXESJtNK/700XcSrIHOa93gfn6/0DmlU2JlSwe2DFg2FVUqu2TPGUHihdfwpRyrTeT8k4ujMawW/XdPjbZIhz579Cjn22dAgZ+L1iZfiBmNm4niIUdCaM7uxn9vv7u4ftvIF+lUkOaOhDsYEiD5ShjMTNEzpNEMYijY1BIG2HtlN2HZj+zvKEPt5M7IhE1jHIHol9+d4+BM5eDufOGZskITLf9slXIbvAAAA//+c84T4lQEAAA==", [ 4.295462] ignition[864]: "verification": {} [ 4.296050] ignition[864]: }, [ 4.296575] ignition[864]: "mode": 420 [ 4.297101] ignition[864]: }, [ 4.297544] ignition[864]: { [ 4.298079] ignition[864]: "group": {}, [ 4.298669] ignition[864]: "path": "/etc/kubernetes/bootstrap-configs/kube-apiserver-config.yaml", [ 4.299831] ignition[864]: "user": {}, [ 4.300366] ignition[864]: "contents": { [ 4.300988] ignition[864]: "compression": "gzip", [ 4.301637] ignition[864]: "verification": {} [ 4.302234] ignition[864]: }, [ 4.302709] ignition[864]: "mode": 420 [ 4.303316] ignition[864]: }, [ 4.303778] ignition[864]: { [ 4.304228] ignition[864]: "group": {}, [ 4.304775] ignition[864]: "path": "/etc/kubernetes/bootstrap-configs/kube-controller-manager-config.yaml", [ 4.305940] ignition[864]: "user": {}, [ 4.306525] ignition[864]: "contents": { [ 4.307079] ignition[864]: "compression": "gzip", [ 4.307703] ignition[864]: "source": "data:;base64,H4sIAAAAAAAC/6RUTY/TMBC991dEvayENEnaZVm0N7RHxJW7a7+mVl3bjMfRll+PHCcQykos4tTU896br2eraL+Ckw3+qTnnA3TwwsFFpzxaHfzRDm2I8Olkj9La0I27DV4E3sB84iFf4CU9bZpGORe0EpAPBqSt4emYmu1RuYRtgWQ5wYvVSmzwNKc72qECO4juyiF7CFKXoBmSul+4WSOw/f4fEhosZCzPlFFxx9mvaAXjchLw1EfF7fp2t//Y9m3f7d6vEMkO3vqBJtWjdfh7JQ4y0cCtZnlFy2Se+qtSj/v+9ArojOu/5zvjWqSmUWQGaReyIQ5Z8Me6Zis4LJu8e3c3/ZKIqx+HECQJq1jl52g4w2sHVU/g1cGBzNWri9UUOYy2+M36eWlb4TwlPDq80BhcvoCiy4P1qzXd9Lf0VXGpq7QOL9CbZnIyqWjpkDnJnOa+77fr2Le4tLx7mCIOyoAJDlpuSluHiJFCZg1yQZ8rsE70omJyUGmy0A1F+EoRbIOpjPuCiSMx9FU7MMVgSHCJrlyistio5ESnkKR8zPVs30Tyx7TCcwhCWr3RLGU0CTwW84dLdBDQ8l/RIXvjMNs2QRcPxcDLuHb9/uFxO4V4tBqktA7ZC0W2Yynxbaa9Yc+uXU6Xi2AjsfLDrLV73Lf3fb2fHzZNk1Ote12FZpjyACmXftvv2Xrz1HzOBzz/9PwX5dUAfq6vxo8AAAD//xFPgmQnBQAA", [ 4.314964] ignition[864]: "verification": {} [ 4.315638] ignition[864]: }, [ 4.316179] ignition[864]: "mode": 420 [ 4.316772] ignition[864]: }, [ 4.317295] ignition[864]: { [ 4.317801] ignition[864]: "group": {}, [ 4.318509] ignition[864]: "path": "/etc/kubernetes/bootstrap-configs/kube-scheduler-config.yaml", [ 4.319720] ignition[864]: "user": {}, [ 4.320339] ignition[864]: "contents": { [ 4.320959] ignition[864]: "compression": "gzip", [ 4.321675] ignition[864]: "source": "data:;base64,H4sIAAAAAAAC/0yPQU7DMBBF9z7FXCAxoRIgb1tWIISExN6xf6mV1I5mxiBuj5KUpjvr+f2ZP35Kn2BJJTsaag8JJ8Q6gttQ8jF9tcOTtKnY766H+p0JY0LWfckZQeeUoSW32o4sNNgZcIZCrKjXFJqpxIYhpXKA2FU++0nsdV+zTbHb0wwpR0cvtcfHv7lffir7eb8Z4SP4edzq3BBHyhUrExwuGUfd7lEMESPj5wAfx5ThqLu70LXnawmDo63rMuNWePNnyOQDHJUJWU7pqMsZzfWqxVb+fQenEh3dP4j5CwAA//9kxa1BcwEAAA==", [ 4.325279] ignition[864]: "verification": {} [ 4.325975] ignition[864]: }, [ 4.326599] ignition[864]: "mode": 420 [ 4.327222] ignition[864]: }, [ 4.327776] ignition[864]: { [ 4.328329] ignition[864]: "group": {}, [ 4.328964] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/admin-kubeconfig-ca-bundle.crt", [ 4.330253] ignition[864]: "user": {}, [ 4.330882] ignition[864]: "contents": { [ 4.331594] ignition[864]: "verification": {} [ 4.332365] ignition[864]: }, [ 4.332929] ignition[864]: "mode": 420 [ 4.333537] ignition[864]: }, [ 4.334064] ignition[864]: { [ 4.334573] ignition[864]: "group": {}, [ 4.335157] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/aggregator-ca-bundle.crt", [ 4.336337] ignition[864]: "user": {}, [ 4.336919] ignition[864]: "contents": { [ 4.337504] ignition[864]: "verification": {} [ 4.338109] ignition[864]: }, [ 4.338589] ignition[864]: "mode": 420 [ 4.339132] ignition[864]: }, [ 4.339600] ignition[864]: { [ 4.340063] ignition[864]: "group": {}, [ 4.340606] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/aggregator-ca.crt", [ 4.341494] ignition[864]: "user": {}, [ 4.342043] ignition[864]: "contents": { [ 4.342592] ignition[864]: "compression": "gzip", [ 4.343239] ignition[864]: "verification": {} [ 4.343827] ignition[864]: }, [ 4.344329] ignition[864]: "mode": 420 [ 4.344863] ignition[864]: }, [ 4.345316] ignition[864]: { [ 4.345745] ignition[864]: "group": {}, [ 4.346231] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/aggregator-ca.key", [ 4.347023] ignition[864]: "user": {}, [ 4.347521] ignition[864]: "contents": { [ 4.348075] ignition[864]: "compression": "gzip", [ 4.348673] ignition[864]: "verification": {} [ 4.349277] ignition[864]: }, [ 4.349724] ignition[864]: "mode": 420 [ 4.350215] ignition[864]: }, [ 4.350635] ignition[864]: { [ 4.351047] ignition[864]: "group": {}, [ 4.351537] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/aggregator-client.crt", [ 4.352543] ignition[864]: "user": {}, [ 4.353227] ignition[864]: "contents": { [ 4.353753] ignition[864]: "compression": "gzip", [ 4.354345] ignition[864]: "verification": {} [ 4.354892] ignition[864]: }, [ 4.355323] ignition[864]: "mode": 420 [ 4.355797] ignition[864]: }, [ 4.356206] ignition[864]: { [ 4.356604] ignition[864]: "group": {}, [ 4.357086] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/aggregator-client.key", [ 4.358053] ignition[864]: "user": {}, [ 4.358544] ignition[864]: "contents": { [ 4.359050] ignition[864]: "compression": "gzip", [ 4.360650] ignition[864]: "verification": {} [ 4.361206] ignition[864]: }, [ 4.361643] ignition[864]: "mode": 420 [ 4.362137] ignition[864]: }, [ 4.362556] ignition[864]: { [ 4.362966] ignition[864]: "group": {}, [ 4.363467] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/aggregator-signer.crt", [ 4.364472] ignition[864]: "user": {}, [ 4.364975] ignition[864]: "contents": { [ 4.365507] ignition[864]: "verification": {} [ 4.366063] ignition[864]: }, [ 4.366506] ignition[864]: "mode": 420 [ 4.366996] ignition[864]: }, [ 4.367430] ignition[864]: { [ 4.367845] ignition[864]: "group": {}, [ 4.368346] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/aggregator-signer.key", [ 4.369359] ignition[864]: "user": {}, [ 4.369864] ignition[864]: "contents": { [ 4.370496] ignition[864]: "compression": "gzip", [ 4.371102] ignition[864]: "verification": {} [ 4.371659] ignition[864]: }, [ 4.372104] ignition[864]: "mode": 420 [ 4.372601] ignition[864]: }, [ 4.373035] ignition[864]: { [ 4.373457] ignition[864]: "group": {}, [ 4.373955] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/apiserver-proxy.crt", [ 4.374957] ignition[864]: "user": {}, [ 4.375474] ignition[864]: "contents": { [ 4.375986] ignition[864]: "compression": "gzip", [ 4.376599] ignition[864]: "verification": {} [ 4.377161] ignition[864]: }, [ 4.377605] ignition[864]: "mode": 420 [ 4.378114] ignition[864]: }, [ 4.378543] ignition[864]: { [ 4.378957] ignition[864]: "group": {}, [ 4.379458] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/apiserver-proxy.key", [ 4.380446] ignition[864]: "user": {}, [ 4.380951] ignition[864]: "contents": { [ 4.381479] ignition[864]: "compression": "gzip", [ 4.382084] ignition[864]: "verification": {} [ 4.382634] ignition[864]: }, [ 4.383070] ignition[864]: "mode": 420 [ 4.383565] ignition[864]: }, [ 4.384013] ignition[864]: { [ 4.384445] ignition[864]: "group": {}, [ 4.384931] ignition[864]: "path": "/etc/kubernetes/bootstrap-secrets/bound-service-account-signing-key.key", ------ Ignition has failed. Please ensure your config is valid. Note that only Ignition spec v3.0.0+ configs are accepted. A CLI validation tool to check this called ignition-validate can be downloaded from GitHub: https://github.com/coreos/ignition/releases ------ Displaying logs from failed units: ignition-disks.service -- Logs begin at Fri 2022-07-29 22:37:24 UTC, end at Fri 2022-07-29 22:37:27 UTC. -- Jul 29 22:37:27 ignition[864]: "name": "zincati.service" Jul 29 22:37:27 ignition[864]: }, Jul 29 22:37:27 ignition[864]: { Jul 29 22:37:27 ignition[864]: "contents": "[Unit]\nDescription=Run dnsmasq to provide local dns for Single Node OpenShift\nBefore=kubelet.service crio.service\nAfter=network.target\n\n[Service]\nExecStart=/usr/sbin/dnsmasq -k\n\n[Install]\nWantedBy=multi-user.target\n", Jul 29 22:37:27 ignition[864]: "enabled": true, Jul 29 22:37:27 ignition[864]: "name": "dnsmasq.service" Jul 29 22:37:27 ignition[864]: } Jul 29 22:37:27 ignition[864]: ] Jul 29 22:37:27 ignition[864]: } Jul 29 22:37:27 ignition[864]: } Press Enter for emergency shell or wait 5 minutes for reboot. ```
Hi all, The bug has been ultimately traced back to the fact that one of the functions used to merge ignition configs was not order-agnostic and it did have an impact whether we do `merge(v32,v31)` or `merge(v31,v32)`. It's been our oversight when implementing the feature and it's now being fixed as https://github.com/openshift/assisted-service/pull/4234. Further updates will be tracked in https://bugzilla.redhat.com/show_bug.cgi?id=2113954
Thanks for the updates Mat. Once things settle and merge here across the board, what version of RHACM will contain all of the cumulative fixes to address the original problem (storage overrides)? Just looking through this BZ (2099929) I see three other issues: rhacm 2.7 - https://bugzilla.redhat.com/show_bug.cgi?id=2113954 - Ignition override v3.2.0 renders as v3.1 in day2 flow rhacm 2.6 - https://bugzilla.redhat.com/show_bug.cgi?id=2112321 - BMAC reconcile loop never stops after changes rhacm 2.5 - https://bugzilla.redhat.com/show_bug.cgi?id=2065288 - Day-2 ignition override is ignored
Right, to sum up what's what 1) https://bugzilla.redhat.com/show_bug.cgi?id=2113954 (https://github.com/openshift/assisted-service/pull/4234) 2) https://bugzilla.redhat.com/show_bug.cgi?id=2112321 (https://github.com/openshift/assisted-service/pull/4201) Once (1) is merged to master branch, I will backport it to 2.5 and then (most probably) to 2.4. Currently fix on master is still pending. The issue (2) is already merged on master branch. The backport to 2.5 is already merged. The backport to 2.4 is still pending. I'm not 100% familiar with the z-stream releases for RHACM so I can't say exactly what and when, but in principle once we merge backports to 2.5 it's the next z-stream release in RHACM that will contain those patches. Of course in the meantime let me know if you want me to build you an image (whether from master or from backports-2.5).
Coming back to this. This behavior seems to be working in RHACM 2.5.2 for me. Do we know if that version definitely got the fixes for this? image: registry.redhat.io/multicluster-engine/agent-service-rhel8@sha256:459c722b51021d2302cf17e62582bc8773c4eca29509768bec56d6ef352908ae --- apiVersion: metal3.io/v1alpha1 kind: BareMetalHost metadata: ... bmac.agent-install.openshift.io/ignition-config-overrides: '{"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/nvme0n1","partitions":[{"label":"root","number":4,"resize":true,"sizeMiB":102400},{"label":"lso_50G_1","sizeMiB":51200},{"label":"lso_50G_2","sizeMiB":51200},{"label":"lso_50G_3","sizeMiB":51200},{"label":"lso_50G_4","sizeMiB":51200},{"label":"lso_50G_5","sizeMiB":51200}],"wipeTable":false}]}}' $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1.1G 0 rom nvme0n1 259:0 0 953.9G 0 disk ├─nvme0n1p1 259:1 0 1M 0 part ├─nvme0n1p2 259:2 0 127M 0 part ├─nvme0n1p3 259:3 0 384M 0 part /boot ├─nvme0n1p4 259:4 0 100G 0 part /sysroot ├─nvme0n1p5 259:5 0 50G 0 part ├─nvme0n1p6 259:6 0 50G 0 part ├─nvme0n1p7 259:7 0 50G 0 part ├─nvme0n1p8 259:8 0 50G 0 part └─nvme0n1p9 259:18 0 50G 0 part I'm continuing to test, but this looks promising two runs of this, with my intended configuration.
Last fix was merged to the 2.5 branch on August 7th; anything cut after this date contains the fix
The build referenced by Dave earlier - registry.redhat.io/multicluster-engine/agent-service-rhel8@sha256:459c722b51021d2302cf17e62582bc8773c4eca29509768bec56d6ef352908ae was built early September: "Labels": { "architecture": "x86_64", "build-date": "2022-09-06T02:45:14.123173", And it contains the fix: commit edac97acfcc8fc9f1173935eb6ae8421a5b84cdb Author: OpenShift Cherrypick Robot <openshift-cherrypick-robot> Date: Sun Aug 7 03:32:10 2022 -0700 [..] Fixes: Bug-2113954 Fixes: Bug-2099929 Closes: MGMTBUGSM-447 Closes: MGMTBUGSM-499 Based on the earlier comment I am going to consider this verified and shall close.