Bug 2049143

TLDR; this is simple ansible failure that should be very easy to debug, but Satellite makes it very hard.

This can be caused for two reasons (that I did not managed to investingate yet):
1) capsule terminates ansible-runner too soon before it can report what went wrong properly
2) ansible-runner fails too fast itself

In the `job_events` I have found a file `d16f1694-b480-444a-b44c-74c6e8e36e02-partial.json` with following content that finally gave away what happend:

{
   "parent_uuid":"94188218-2466-5ab3-79d8-00000000023e",
   "created":"2022-02-01T15:24:58.295828",
   "pid":42264,
   "event":"runner_on_failed",
   "event_data":{
      "play":"all",
      "event_loop":null,
      "task_args":"",
      "play_uuid":"94188218-2466-5ab3-79d8-000000000006",
      "task_uuid":"94188218-2466-5ab3-79d8-00000000023e",
      "playbook_uuid":"d29f8cd9-2458-450c-9615-f2af26596621",
      "playbook":"playbook.yml",
      "task_action":"redhat.satellite.subscription_manifest",
      "duration":0.011068,
      "ignore_errors":null,
      "task_path":"/usr/share/ansible/collections/ansible_collections/redhat/satellite/roles/manifest/tasks/main.yml:10",
      "play_pattern":"all",
      "task":"Upload Subscription Manifest to Foreman",
      "end":"2022-02-01T15:24:58.295642",
      "uuid":"d16f1694-b480-444a-b44c-74c6e8e36e02",
      "remote_addr":"<redacted_hostname>",
      "res":{
         "msg":"The task includes an option with an undefined variable. The error was: 'satellite_organization' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/redhat/satellite/roles/manifest/tasks/main.yml': line 10, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Upload Subscription Manifest to Foreman\n  ^ here\n",
         "_ansible_no_log":false
      },
      "host":"<redacted_hostname>",
      "start":"2022-02-01T15:24:58.284574",
      "role":"manifest"
   },
   "uuid":"d16f1694-b480-444a-b44c-74c6e8e36e02"
}

Tho this does not appear in any final output, only in the ansible runner "partial" outputs which get ignored because it is supposed to get into the final output.

Running `ansible-runner` manually results in a properly populated stdout and job_events/* so it's our fault and we terminate the runner before it manages to finish what it is doing.

Version-Release number of selected component (if applicable):
6.11 - 3 

Steps to Reproduce:
1. Import Ansible roles
2. Create `satellite_organization` variable for `convert2rhel` role with value "Default Organization"
3. Assign "redhat.satellite.convert2rhel" to the Satellite host
4. Start "Run Ansible roles"
5. Check the output of the job

Actual results:
Role is run, and doesn't end with a false success as in original case, any failures are properly reported

PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [dhcp-3-152.vms.sat.rdu2.redhat.com]
TASK [Display all parameters known for the Foreman host] ***********************
ok: [dhcp-3-152.vms.sat.rdu2.redhat.com] => {
  ...
TASK [Apply roles] *************************************************************
TASK [Upload Subscription Manifest and sync RHEL repository] *******************
TASK [Subscription Manifest] ***************************************************
TASK [redhat.satellite.manifest : Download Subscription Manifest from Red Hat Customer Portal] ***
skipping: [dhcp-3-152.vms.sat.rdu2.redhat.com]
TASK [redhat.satellite.manifest : Upload Subscription Manifest to Foreman] *****
 161:
fatal: [dhcp-3-152.vms.sat.rdu2.redhat.com]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'satellite_manifest_path' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/redhat/satellite/roles/manifest/tasks/main.yml': line 10, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Upload Subscription Manifest to Foreman\n  ^ here\n"}
 162:
PLAY RECAP *********************************************************************
 163:
dhcp-3-152.vms.sat.rdu2.redhat.com : ok=2    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0
 164:
Exit status: 2

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498