Description of problem:

When executing the leapp preupgrade or upgrade from Satellite WebUi using the predefined job templates and the remote_execution_ssh_user is a non-root user, Then after running the job for a while, Leapp fails to find grubby command.

Summary : {"details": "Command ['grubby', '--info', 'ALL'] failed with exit code 1.", "stderr": "Process Process-237:\nTraceback (most recent call last):\n  File \"/usr/lib64/python2.7/multiprocessing/process.py\", line 258, in _bootstrap\n    self.run()\n  File \"/usr/lib64/python2.7/multiprocessing/process.py\", line 114, in run\n    self._target(*self._args, **self._kwargs)\n  File \"/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py\", line 72, in _do_run\n    actor_instance.run(*args, **kwargs)\n  File \"/usr/lib/python2.7/site-packages/leapp/actors/__init__.py\", line 289, in run\n    self.process(*args)\n  File \"/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/sourcebootloaderscanner/actor.py\", line 18, in process\n    scan_source_boot_loader_configuration()\n  File \"/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/sourcebootloaderscanner/libraries/sourcebootloaderscanner.py\", line 87, in scan_source_boot_loader_configuration\n    entries=scan_boot_entries()\n  File \"/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/actors/sourcebootloaderscanner/libraries/sourcebootloaderscanner.py\", line 41, in scan_boot_entries\n    grubby_output = run(CMD_GRUBBY_INFO_ALL, split=True)\n  File \"/usr/lib/python2.7/site-packages/leapp/libraries/stdlib/__init__.py\", line 181, in run\n    stdin=stdin, env=env, encoding=encoding)\n  File \"/usr/lib/python2.7/site-packages/leapp/libraries/stdlib/call.py\", line 217, in _call\n    os.execvpe(command[0], command, env=environ)\n  File \"/usr/lib64/python2.7/os.py\", line 353, in execvpe\n    _execvpe(file, args, env)\n  File \"/usr/lib64/python2.7/os.py\", line 380, in _execvpe\n    func(fullname, *argrest)\nOSError: [Errno 2] No such file or directory\n"}


Version-Release number of selected component (if applicable):

Satellite 6.10/6.11/6.12

How reproducible:

In the customer's environment 


Steps to Reproduce: ( This may or may not work )

1. Setup a Satellite 6.12 for REX + Leapp plugin ( prepare it for performing RHEL 7 - 8 conversion on client hosts )

2. On an RHEL 7.9 system, create a non-root user, give it sudo privileges and ensure that it has a PATH env variable set where /usr/sbin/ is not the PATH.

3. Register that system with the satellite.

4. Remote Execution SSH User should be set to that non-root user and the effective user should be root.

5. Run the Preupgrade job on that host from Satellite UI --> Hosts --> All Hosts --> Select Actions dropdown

   If it succeeds, then run the Upgrade job in the same way.


Actual results:

On either the Preupgrade or the Upgrade job, leapp will fail to find grubby binary to list the boot information. 


Expected results:

No such issues.


Additional info:

After doing an strace on sshd of client system, we found that, when the SSH user becomes effective user, It's inheriting the wrong PATH variable in the environment and hence leapp cannot find out grubby command later.

The way we worked it around:

For "Check Leapp" template
--------------------------

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
if ! command -v leapp > /dev/null
then
  echo "Leapp is not installed."
  exit 1
fi

----------------------------

Above automatically fixes the execution for "Run preupgrade via Leapp " template

For "Run upgrade via Leapp" template
------------------------------------

---
- hosts: all
  tasks:
    - name: Run Leapp Upgrade
      environment:
        PATH: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin'
      command: leapp upgrade
<%- if input('Reboot') == "true" -%>
    - name: Reboot the machine
      reboot:
        reboot_timeout: 1800
<%- end -%>


------------------------------------

The analysis of strace:

The PATH variable used during leapp preupgrade is "PATH=/usr/local/bin:/usr/bin" as a result unable to find grubby binary (available in /usr/sbin) and hence leapp preupgrade fails with error Failed to call grubby 


83547 10:46:18.547227 execve("/usr/bin/leapp", ["leapp", "preupgrade"], ["SMIT_QUOTE=n", "SHELL=/bin/bash", "TERM=xterm", "USER=root", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg"..., "SUDO_USER=decoy", "SUDO_UID=1011", "USERNAME=root", "MAIL=/var/mail/root", "PATH=/usr/local/bin:/usr/bin", "SMIT_SEMI_COLON=n", "PWD=/home/decoy", "LANG=en_US.UTF-8", "SHLVL=1", "SUDO_COMMAND=/var/tmp/foreman-ssh-cmd-56a92adc-0cfb-44a2-a49f-f78411d29de0/script", "HOME=/root", "LOGNAME=root", "SMIT_SHELL=n", "SUDO_GID=1011", "_=/usr/bin/leapp"]) = 0 <0.002122>

This is where it all starts:


83463 10:46:16.764980 execve("/bin/bash", ["bash", "-c", "mkdir -p /var/tmp/foreman-ssh-cmd-56a92adc-0cfb-44a2-a49f-f78411d29de0"], ["LANG=en_US.UTF-8", "USER=decoy", "LOGNAME=decoy", "HOME=/home/decoy", "PATH=/usr/local/bin:/usr/bin", "MAIL=/var/mail/decoy", "SHELL=/bin/bash", "SSH_CLIENT=10.13.XX.XX 53502 22", "SSH_CONNECTION=10.13.XX.XX 53502 10.241.XX.XX 22", "XDG_SESSION_ID=1900", "XDG_RUNTIME_DIR=/run/user/1011"]) = 0 <0.001512>



I personally never was able to reproduce the issue but we have seen it happening 100% with two users without the workaround mentioned