Bug 2088529

Summary: ForemanCustomScript in Host provisioned on Azure CR fails with `command not found`
Product: Red Hat Satellite Reporter: Jitendra Yejare <jyejare>
Component: Compute Resources - AzureAssignee: Chris Roberts <chrobert>
Status: CLOSED ERRATA QA Contact: addubey
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.11.0CC: bbuckingham, chrobert, gtalreja, pcreech, pmendezh, rlavi
Target Milestone: 6.13.0Keywords: AutomationBlocker, PrioBumpQA, Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: https://projects.theforeman.org/issues/35481
Whiteboard:
Fixed In Version: rubygem-foreman_azure_rm-2.2.7-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-03 13:21:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jitendra Yejare 2022-05-19 15:54:49 UTC
Created attachment 1881401 [details]
[Screenshots] Azure Portal VM

Description of problem:
When Satellite provisions host on Azure CR, a `ForemanCustomScript` Extension that  Satellite adds to the Azure host is throwing an error `Enable succeeded: [stout] [stderr] /bin/sh: azurerootpasswd: command not found`.

This failure probably delayed the VM read operation from Azure python client and failed all QE tests for Azure Provisioning.

Version-Release number of selected component (if applicable):
Satellite 6.11 | 6.10 | 6.9

How reproducible:
Always

Steps to Reproduce:
1. Create an Azure Compute Resource.
2. Create an Image in Azure CR with marketplace image `marketplace://RedHat:RHEL:7-RAW:latest` (BTW, the issue is reproducible with all the images that Satellite supports to provision the host from)
3. Using the image above provision an azure host, and make sure the azure host has the following compute attributes:

script_command = 'touch /var/tmp/text.txt',
script_uris = `https://raw.githubusercontent.com/SatelliteQE/robottelo/master/tests/foreman/data/uri.sh`

Actual results:
1. The VM is being provisioned to Azure but the ForemanCustomScript extention post VM deployment is failing with the error `Enable succeeded: [stout] [stderr] /bin/sh: azurerootpasswd: command not found. Note `azurerootpasswd` is the password set as a password in compute attributes.
2. The script command in compute attributes was executed successfully.
3. But, The custom script in `script_uris` in compute attributes is not executed and seems the failure is for that reason only.

Expected results:
The VM provisions successfully with the post-deployment script (both script command and script uris) executed successfully.

Additional info:
Attachments with screenshots from Azure portal showing failure from VMs `ForemanCustomScript` extention.

Comment 2 Jitendra Yejare 2022-05-24 13:43:19 UTC
@bbuckingham No, It's something working in earlier z-versions of 6.9 - 6.10 and especially seeing from earlier snaps of 6.11 and the same issue in 6.9 and 6.10 at the same time.

Comment 3 Jitendra Yejare 2022-05-24 14:05:51 UTC
Hey Brad,

Just run the 6.11 automation and this issue is not reproducible now :)  TBH, I have seen it passing earlier as well once in 6.11 snaps and again starts failing.

Not sure what's the case in 6.10 and 6.9, but I will post to you on that!

Comment 4 Ron Lavi 2022-05-26 11:44:46 UTC
Hey @jyejare@redhat.com, any news with the other versions?
I am suggesting to remove it from 6.11.z as it wasn't reproduced anymore.

Comment 5 Jitendra Yejare 2022-05-26 14:14:57 UTC
@rlavi It reproduced again yesterday on snap 22 of Sat 6.11. The passing is very infrequent/intermittent but the failure is very frequent in the last 22 snaps!

So I will still keep it open for 6.11 and rest of the versions!

Comment 6 Chris Roberts 2022-05-26 15:36:16 UTC
Are you setting the password in the virtual machine tab or when you create an image in the compute resource?

Comment 7 Jitendra Yejare 2022-05-27 08:24:16 UTC
@chrobert I am setting the password from Virtual Machine, but not in the image in the Azure compute resource!

Comment 12 Chris Roberts 2022-08-31 19:11:25 UTC
Made it past the /bin/sh: azurerootpasswd: command not found`

It looks like 5 years ago we hard coded this line in the plugin:

https://github.com/theforeman/foreman_azure_rm/blob/master/app/models/foreman_azure_rm/azure_rm.rb#L297

Looking at the if/else block, even if script was blank we would make it true which is a another bug in itself(my pr will fix this) because each vm would have the linux script extension added to it.

Looking at how Azure does things now they do not use that file anymore instead in /etc/sudoers.d we have this

[root@kari-elofson ~]# cat /etc/sudoers.d/90-cloud-init-users 
# Created by cloud-init v. 19.4 on Tue, 03 Aug 2021 14:30:54 +0000

# User rules for azureuser
azureuser ALL=(ALL) NOPASSWD:ALL

# User rules for azureuser
azureuser ALL=(ALL) NOPASSWD:ALL

I tested removing that line of code and we do not create the linux script extension on vms now and I do not see the error posted in the BZ when looking at /var/lib/waagent//custom-script/download/0/stderr/out

The issue I am seeing now is that it's downloading the uri.sh but it's not actually echoing anything into /var/tmp/text.txt. We are making the file but it's not doing the echo. When I try to do it as the normal user when logging into the Azure vm, I don't get any errors and it echos fine to the file.

I am going to keep debugging and see if that is an issue with us or if something else changed on the Azure side. First issue is solved at least and will make a PR when I am done looking at the 2nd part of the issue.

Comment 13 Chris Roberts 2022-08-31 19:18:00 UTC
Looking at this we have to actually call the script we are downloading:

https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux#property-values (see commandtoexecute) Since this was working before I am guessing something prob changed in this extension.

Going to try and adjust my command to create the /var/tmp/text and then run the script. If I can get that to work I will post my command here and in the pr testing steps. Hopefully that is the last piece to this puzzle.

Comment 14 Chris Roberts 2022-08-31 21:16:54 UTC
Able to get the QE uri.sh script running with this as the command:

sudo sh /var/lib/waagent/custom-script/download/0/uri.sh

Will start to work on a PR and link it up when it's done.

Comment 15 Bryan Kearney 2022-09-08 20:04:08 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35481 has been resolved.

Comment 21 errata-xmlrpc 2023-05-03 13:21:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097