Description of problem: Version-Release number of the following components: 4.6.0-0.nightly-2020-08-01-215144 How reproducible: Always Steps to Reproduce: 1. Following https://github.com/openshift/installer/blob/master/docs/user/azure/install_upi.md to run a 4.6 upi install 2. 3. Actual results: Failed at "Launch the permanent control plane" steps. $ az group deployment create --debug --name qeci-6882_05_masters -g qeci-6882-v6lh2-rg --template-file '/home/jenkins/workspace/Launch Environment Flexy/private-openshift-misc/v3-launch-templates/functionality-testing/aos-4_6/hosts/upi_on_azure-arm-templates/05_masters.json' --parameters masterIgnition='eyJpZ25pdGlvbiI6eyJjb25maWciOnsibWVyZ2UiOlt7InNvdXJjZSI6Imh0dHBzOi8vYXBpLWlu dC5xZWNpLTY4ODIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tOjIyNjIzL2NvbmZp Zy9tYXN0ZXIifV19LCJzZWN1cml0eSI6eyJ0bHMiOnsiY2VydGlmaWNhdGVBdXRob3JpdGllcyI6 W3sic291cmNlIjoiZGF0YTp0ZXh0L3BsYWluO2NoYXJzZXQ9dXRmLTg7YmFzZTY0LExTMHRMUzFD UlVkSlRpQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENrMUpTVVJGUkVORFFXWnBaMEYzU1VKQlowbEpa RWhOV2pZNVZITjJiREIzUkZGWlNrdHZXa2xvZG1OT1FWRkZURUpSUVhkS2FrVlRUVUpCUjBFeFZV VUtRM2hOU21JelFteGliazV2WVZkYU1FMVNRWGRFWjFsRVZsRlJSRVYzWkhsaU1qa3dURmRPYUUx Q05GaEVWRWwzVFVSbmQwMTZRVEZOUkVWNFRqRnZXQXBFVkUxM1RVUm5kMDFVUVRGTlJFVjRUakZ2 ZDBwcVJWTk5Ra0ZIUVRGVlJVTjRUVXBpTTBKc1ltNU9iMkZYV2pCTlVrRjNSR2RaUkZaUlVVUkZk MlI1Q21JeU9UQk1WMDVvVFVsSlFrbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGUk9F Rk5TVWxDUTJkTFEwRlJSVUYzYjBWa1RXeFBURTVDY1VnS01EQmthbmwxVFhNMWRYQkViV1ZxZVha c1YyRnZWa1FyVGs5c1QwSnlRVzk2YVhaUk9ERmlPVFUwTTNGclVFVjVhekZ6Y0dsQ1FuZFRRVWhq TDBwSVF3cE5SR2xrVERCT2NHb3lXaXRzTmt0bk9VOUNTMk5PUjNZMVRXUTJja3RuTkZSaFlWVjJX bEoyTkVadFF6UnpRWG80TTFWT1RGWk9iM05PVFVnelJXdDVDbnBTVm5kWGFVWTRNMjg0Y2xNMlNF MUZlR3hGVmxwdE1EbG5VVlZwUlZnNFpHa3ZVWE5ZYUZWdVYyTkVhMHhhZGxCWFVITlZNekJ6UjBa eVNrNDNkVllLV25NclFtUktRWEI2TDJKcFZWUXJla0ZpVTNOSVRYVkZkMWhJVVV0dU5ETk5UR2dy TkhGQ1lXbDVVVGxSTUZsM1QwcHVhMmxpVm5oSlNWcFdaMkV3VHdwNEszaGlVWGhWYzBKbU0ya3ZU VkZvYmtkeGNta3hWVGhPYlRKWGNFbEtOVzE2U21ZemJXdFZTVlpSWkRoVGQxbG1ha2hzZWxSU2Nt ZFhWMlo2YjJKVkNtdDZlbkpPTVhKbmRWRkpSRUZSUVVKdk1FbDNVVVJCVDBKblRsWklVVGhDUVdZ NFJVSkJUVU5CY1ZGM1JIZFpSRlpTTUZSQlVVZ3ZRa0ZWZDBGM1JVSUtMM3BCWkVKblRsWklVVFJG Um1kUlZXbFlVbUZoYm5WSlpuTkNXVGh5YkhOeloyRTFOemh6Y0dSa1ZYZEVVVmxLUzI5YVNXaDJZ MDVCVVVWTVFsRkJSQXBuWjBWQ1FVRnRiV0l3TUdkSFJHcE1SMGt6ZFZaRldVdE9Za2RXZEdka1dF bGpPVU5STlhkaU1VSmxaVUZXVEZaS1NITXdhVVp4V1dkNFpHSllTblZrQ25OVk9FbE9Ta0VyUzNW dGFtTmlkbVpEZUhkNmFFZHRUMVpHWkRjeGVVcHlRMDgyZDIxU2IwNXBaSFJ3V0VGSWRGVlBRMGRH WlUxamRsSTBibkpSTTNJS1ZsRlpTaTlLUmt4NmVEazJOVkpRZGpsTVlrSnBabXQ1TVdaYVdXZEJN M0J4UW1kYVMxVlFkbmRvTlhKeFUzUktiazEwUTB0bmNuTndTVU5hWTNCckt3cDRSRTUyU1VacFdr TnROV05KTnpkNlRUTkJPSGd4Vm5SVk9YcElieXM1Tm1zeE1UUXZhRXhMYTJ0eFVFaHdjVnBHWmt4 akwyNWxVQzl1Y25kSVNWWlFDazVvUTFZMmNHcEpaR3R3ZW1sa1dsUm5VemgyUmpsdVJIVndVMlZ0 VXpnMFpWcHJVRXRJZFhKNFYwaEJPWGR5T1dGaU1VRXlRMUZKWTNWdlJtMDNPVUlLWkZrek5Vc3hR a2hrZDBkMGFYQnZUelpRYVdjclMzQXllVzV2UFFvdExTMHRMVVZPUkNCRFJWSlVTVVpKUTBGVVJT MHRMUzB0Q2c9PSJ9XX19LCJ2ZXJzaW9uIjoiMy4xLjAifX0=' --parameters sshKeyData='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa37P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D openshift-qe' --parameters privateDNSZoneName='qeci-6882.qe.azure.devcluster.openshift.com' --parameters baseName='qeci-6882-v6lh2' <--snip--> DEBUG: msrest.http_logger : Response content: DEBUG: msrest.http_logger : {"status":"Failed","error":{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"code\": \"InvalidParameter\",\r\n \"message\": \"Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters.\",\r\n \"target\": \"customData\"\r\n }\r\n}"},{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"code\": \"InvalidParameter\",\r\n \"message\": \"Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters.\",\r\n \"target\": \"customData\"\r\n }\r\n}"},{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"code\": \"InvalidParameter\",\r\n \"message\": \"Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters.\",\r\n \"target\": \"customData\"\r\n }\r\n}"}]}} DEBUG: msrest.exceptions : At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details. ERROR: cli.azure.cli.core.util : Deployment failed. Correlation ID: 7f4c7adb-eef8-456a-a5a1-ea21ed26ace2. { "error": { "code": "InvalidParameter", "message": "Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters.", "target": "customData" } } ERROR: Deployment failed. Correlation ID: 7f4c7adb-eef8-456a-a5a1-ea21ed26ace2. { "error": { "code": "InvalidParameter", "message": "Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters.", "target": "customData" } } Expected results: Installation get passed. Additional info: 1. 4.5 does not hit such problems. 2. remove newline in base64 code, the master deployment get passed. Change export WORKER_IGNITION=`cat <installation_directory>/worker.ign | base64` to export WORKER_IGNITION=`cat <installation_directory>/worker.ign | base64 -w0`
I think the error is because of the newlined base64 instead of a continuous bas64 string. This might be causing the Cannot_Parse style errors that show up as `Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters` error. Can you retry with, ``` --parameters masterIgnition='eyJpZ25pdGlvbiI6eyJjb25maWciOnsibWVyZ2UiOlt7InNvdXJjZSI6Imh0dHBzOi8vYXBpLWludC5xZWNpLTY4ODIucWUuYXp1cmUuZGV2Y2x1c3Rlci5vcGVuc2hpZnQuY29tOjIyNjIzL2NvbmZpZy9tYXN0ZXIifV19LCJzZWN1cml0eSI6eyJ0bHMiOnsiY2VydGlmaWNhdGVBdXRob3JpdGllcyI6W3sic291cmNlIjoiZGF0YTp0ZXh0L3BsYWluO2NoYXJzZXQ9dXRmLTg7YmFzZTY0LExTMHRMUzFDUlVkSlRpQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENrMUpTVVJGUkVORFFXWnBaMEYzU1VKQlowbEpaRWhOV2pZNVZITjJiREIzUkZGWlNrdHZXa2xvZG1OT1FWRkZURUpSUVhkS2FrVlRUVUpCUjBFeFZVVUtRM2hOU21JelFteGliazV2WVZkYU1FMVNRWGRFWjFsRVZsRlJSRVYzWkhsaU1qa3dURmRPYUUxQ05GaEVWRWwzVFVSbmQwMTZRVEZOUkVWNFRqRnZXQXBFVkUxM1RVUm5kMDFVUVRGTlJFVjRUakZ2ZDBwcVJWTk5Ra0ZIUVRGVlJVTjRUVXBpTTBKc1ltNU9iMkZYV2pCTlVrRjNSR2RaUkZaUlVVUkZkMlI1Q21JeU9UQk1WMDVvVFVsSlFrbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGUk9FRk5TVWxDUTJkTFEwRlJSVUYzYjBWa1RXeFBURTVDY1VnS01EQmthbmwxVFhNMWRYQkViV1ZxZVhac1YyRnZWa1FyVGs5c1QwSnlRVzk2YVhaUk9ERmlPVFUwTTNGclVFVjVhekZ6Y0dsQ1FuZFRRVWhqTDBwSVF3cE5SR2xrVERCT2NHb3lXaXRzTmt0bk9VOUNTMk5PUjNZMVRXUTJja3RuTkZSaFlWVjJXbEoyTkVadFF6UnpRWG80TTFWT1RGWk9iM05PVFVnelJXdDVDbnBTVm5kWGFVWTRNMjg0Y2xNMlNFMUZlR3hGVmxwdE1EbG5VVlZwUlZnNFpHa3ZVWE5ZYUZWdVYyTkVhMHhhZGxCWFVITlZNekJ6UjBaeVNrNDNkVllLV25NclFtUktRWEI2TDJKcFZWUXJla0ZpVTNOSVRYVkZkMWhJVVV0dU5ETk5UR2dyTkhGQ1lXbDVVVGxSTUZsM1QwcHVhMmxpVm5oSlNWcFdaMkV3VHdwNEszaGlVWGhWYzBKbU0ya3ZUVkZvYmtkeGNta3hWVGhPYlRKWGNFbEtOVzE2U21ZemJXdFZTVlpSWkRoVGQxbG1ha2hzZWxSU2NtZFhWMlo2YjJKVkNtdDZlbkpPTVhKbmRWRkpSRUZSUVVKdk1FbDNVVVJCVDBKblRsWklVVGhDUVdZNFJVSkJUVU5CY1ZGM1JIZFpSRlpTTUZSQlVVZ3ZRa0ZWZDBGM1JVSUtMM3BCWkVKblRsWklVVFJGUm1kUlZXbFlVbUZoYm5WSlpuTkNXVGh5YkhOeloyRTFOemh6Y0dSa1ZYZEVVVmxLUzI5YVNXaDJZMDVCVVVWTVFsRkJSQXBuWjBWQ1FVRnRiV0l3TUdkSFJHcE1SMGt6ZFZaRldVdE9Za2RXZEdka1dFbGpPVU5STlhkaU1VSmxaVUZXVEZaS1NITXdhVVp4V1dkNFpHSllTblZrQ25OVk9FbE9Ta0VyUzNWdGFtTmlkbVpEZUhkNmFFZHRUMVpHWkRjeGVVcHlRMDgyZDIxU2IwNXBaSFJ3V0VGSWRGVlBRMGRHWlUxamRsSTBibkpSTTNJS1ZsRlpTaTlLUmt4NmVEazJOVkpRZGpsTVlrSnBabXQ1TVdaYVdXZEJNM0J4UW1kYVMxVlFkbmRvTlhKeFUzUktiazEwUTB0bmNuTndTVU5hWTNCckt3cDRSRTUyU1VacFdrTnROV05KTnpkNlRUTkJPSGd4Vm5SVk9YcElieXM1Tm1zeE1UUXZhRXhMYTJ0eFVFaHdjVnBHWmt4akwyNWxVQzl1Y25kSVNWWlFDazVvUTFZMmNHcEpaR3R3ZW1sa1dsUm5VemgyUmpsdVJIVndVMlZ0VXpnMFpWcHJVRXRJZFhKNFYwaEJPWGR5T1dGaU1VRXlRMUZKWTNWdlJtMDNPVUlLWkZrek5Vc3hRa2hrZDBkMGFYQnZUelpRYVdjclMzQXllVzV2UFFvdExTMHRMVVZPUkNCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2c9PSJ9XX19LCJ2ZXJzaW9uIjoiMy4xLjAifX0=' ```
> I think the error is because of the newlined base64 instead of a continuous bas64 string. This might be causing the Cannot_Parse style errors that show up as `Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters` error. Can you retry with, Yeah, I already tried that. See #2 in "Additional info" of comment 0. export WORKER_IGNITION=`cat <installation_directory>/worker.ign | base64 -w0` will help remove newlined in base64 to make it become a continuous bas64 string. It works.
So if `-w0` works , what is the expectation here from the installer team?
Should we update doc in https://github.com/openshift/installer/blob/master/docs/user/azure/install_upi.md accordingly based on your double confirm about `-w0`. right?
Sure we can update the docs to use `-w0` for base64
This the templates haven't in a while. We should make sure there is no host level setup that can affect the output of base64 - for example, we were testing on mac before and that didn't need -w0 or something like that.
Verified this bug with fixed PR. works well. New line is stripped. $ az group deployment create --debug --name jialiu-7588_05_masters -g jialiu-7588-zqmtk-rg --template-file '/home/installer3/workspace/Launch Environment Flexy/private-templates/functionality-testing/aos-4_6/hosts/upi_on_azure-arm-templates/05_masters.json' --parameters masterIgnition='eyJpZ25pdGlvbiI6eyJjb25maWciOnsibWVyZ2UiOlt7InNvdXJjZSI6Imh0dHBzOi8vYXBpLWludC5qaWFsaXUtNzU4OC5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb206MjI2MjMvY29uZmlnL21hc3RlciJ9XX0sInNlY3VyaXR5Ijp7InRscyI6eyJjZXJ0aWZpY2F0ZUF1dGhvcml0aWVzIjpbeyJzb3VyY2UiOiJkYXRhOnRleHQvcGxhaW47Y2hhcnNldD11dGYtODtiYXNlNjQsTFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVUkZSRU5EUVdacFowRjNTVUpCWjBsSlZucDNSV1ZNWjJKaE1XZDNSRkZaU2t0dldrbG9kbU5PUVZGRlRFSlJRWGRLYWtWVFRVSkJSMEV4VlVVS1EzaE5TbUl6UW14aWJrNXZZVmRhTUUxU1FYZEVaMWxFVmxGUlJFVjNaSGxpTWprd1RGZE9hRTFDTkZoRVZFbDNUVVJuZVU1cVFUQk5SRTB4VGxadldBcEVWRTEzVFVSbmVVNUVRVEJOUkUweFRsWnZkMHBxUlZOTlFrRkhRVEZWUlVONFRVcGlNMEpzWW01T2IyRlhXakJOVWtGM1JHZFpSRlpSVVVSRmQyUjVDbUl5T1RCTVYwNW9UVWxKUWtscVFVNUNaMnR4YUd0cFJ6bDNNRUpCVVVWR1FVRlBRMEZST0VGTlNVbENRMmRMUTBGUlJVRjJSVVpPYzI1SFNtUm1VRVVLVERkc1JuVnFVbXBNTjFCRldsRTFaRmRqVkVFd2VsRjNaRUphY0hCREx6TXdVSHBwYldoelZGcE5ObGxHVUVvMVEzZ3hhbVEyVDI1SU9WVlJaRW8xTndvMFkyWmFiRTFTU25jeFkwcDZjaXRhVm1KdFpXeGFTRGhMT0VOTVVqVTRNa2hCU0RFM09UVldRemROTDFBNGRTdDFTVVk1VEZCNGFGcFlXR1Z2WW5CSENrTmFUR2xHUTJ0bFMxZHZXbkpvVURGSmRuSnJOV1pLYmpOVVZ6WjJjMjlyZDJ0a04yNUJiMEp0WVUwMFdtUndUbXN5ZWpOMU4wUXhTMHhvVDBnd2ExSUtUbFY0YUdveFIzQTNMMVJ1TDNJcmNXSkdkalZvSzBWb2FrMHJhV1JzU0RSRFRrUXZRa2M1VGsxb1pHMU9TelYzUjJkeFYyaFRNMFp2VlhGcGRuTjBiUW92VGpCM1JYcGFiek42VkRJNVNXSTNPR0pWVW5nMmFrZERlbWxVZW5RdlV6WTVTRFJESzB4SVFVNUpORE5VSzB0SWFVbHZibFl4U0ZWTGFXVjBZekF3Q2xaS1JGaFhVMGxzUlZGSlJFRlJRVUp2TUVsM1VVUkJUMEpuVGxaSVVUaENRV1k0UlVKQlRVTkJjVkYzUkhkWlJGWlNNRlJCVVVndlFrRlZkMEYzUlVJS0wzcEJaRUpuVGxaSVVUUkZSbWRSVlZselFuWm9kV3BuUlZrekswdzNOMHB4VEN0U1pqRjZOM1l3UVhkRVVWbEtTMjlhU1doMlkwNUJVVVZNUWxGQlJBcG5aMFZDUVVnM2QycE5URmxaSzNOclptcFNkRTFzUW1wRlExTkxaSEpzTUZoRU1XTkZjVmhTU0dkWFJVdzFaVlJYYm5WYWRsVm1hRk52UlRod1R6aElDbmRyVTBGYVNYSkNia2N5WTBkdmNUUmFhSFp4V1ZBeEsyTk1MM2RXTVZveVRVdHpZM0UyYW5oV1JFSTRTVXMzVmt4Uk1rTm1XVXB1WmtreVNHNXlaM1lLVUZGVWVIUkpOWEptVURWeFZFRjNZa2N2VUc5SGVYQjNMM1ZXUld4c1RGTjFSVWhGVm10WWNrTmpTMUo1Wm5JMEx6VlVWMVpaTVZOVmFsZGlSMFpDU2dwclIwTlBMMnAwYzNKTFZHSXpRMngzUzAwclJIZERSVTk0U0VWMmExYzJka2xMV2sxeGJERTBWVXBaYUZsTVFpdHRSVTFVWlcwMFJIbEpTVlJ1WWpKTENsa3dSMHMxY2tabUx6VXliV2xOT0hwMVYyaFZlRGh1WjFSa09HVkZVbGhxTnpWYVQwNDRaVFp1TURoeksxSjJaV3d2YlZsb2JEaFdWa2xuT0hWdVF6QUtkbFZGY0M5NWEyTTJjbXN2SzJWemVXMHhTbTV4TmxJMVprOUJQUW90TFMwdExVVk9SQ0JEUlZKVVNVWkpRMEZVUlMwdExTMHRDZz09In1dfX0sInZlcnNpb24iOiIzLjEuMCJ9fQ==' --parameters sshKeyData='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa37P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D openshift-qe' --parameters privateDNSZoneName='jialiu-7588.qe.azure.devcluster.openshift.com' --parameters baseName='jialiu-7588-zqmtk'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196