Bug 1912941

Summary: Verbose log outputs for Ansible jobs are reported to all Hosts present on the Job.
Product: Red Hat Satellite Reporter: Rudnei Bertol Jr. <rbertolj>
Component: Ansible - Configuration ManagementAssignee: Ondřej Ezr <oezr>
Status: CLOSED ERRATA QA Contact: Sam Bible <sbible>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: aruzicka, chrism, kdixon, lstejska, myoder, oezr, pcreech, rlavi, zhunting
Target Milestone: 6.12.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: tfm-rubygem-smart_proxy_ansible-3.4.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-16 13:32:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rudnei Bertol Jr. 2021-01-05 16:47:42 UTC
Description of problem:

When enabling debug mode on Satellite, the task is generating an output log for all hosts present on the Job.

Version-Release number of selected component (if applicable):

Red Hat Satellite (build: 6.8.2)

How reproducible:

Enable the debug mode on settings and run the Ansible Default Template on more than one host.

Steps to Reproduce:
1. Enable Ansible Debug.
2. Select two or more hosts.
3. Run ansible roles on those hosts.
4. Check the host execution logs.

Actual results:

The host execution log is showing the debug log for all hosts present on the inventory instead of just the expected host.

It's causing a huge delay on the task and makes the time 10 times slower.

Expected results:

Just the debug output log for the host that the output is being checked.

Additional info:

Following a small snippet for the reproducer.

~~~
  32:
TASK [Gathering Facts] *********************************************************
  33:
task path: /tmp/d20210105-14561-c8qt93/project/playbook.yml:2
  34:
<cap68.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root
  35:
<cap68.redhat.com> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="root"' -o ConnectTimeout=10 cap68.redhat.com '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
  36:
<sat68.redhat.com> ESTABLISH SSH CONNECTION FOR USER: root
  37:
<sat68.redhat.com> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="root"' -o ConnectTimeout=10 sat68.redhat.com '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
~~~

Comment 1 Ondřej Ezr 2021-03-23 23:48:45 UTC
Unfortunatelly ansible-runner that is running the ansible playbook is not providing host related data for the verbose debug events, so we don't know what host to send this to, that is why it gets send to all of them.
I'll try to reach out to them and ask for our options here.

Comment 2 Ondřej Ezr 2021-03-24 00:19:29 UTC
Created redmine issue https://projects.theforeman.org/issues/32188 from this bug

Comment 3 Ondřej Ezr 2021-03-24 00:35:48 UTC
I found a possible solution at least for the connection events - that is most of the verbose events, so it should significantly improve the situation

Comment 4 Bryan Kearney 2021-03-24 04:02:14 UTC
Upstream bug assigned to oezr

Comment 5 Bryan Kearney 2021-03-24 04:02:16 UTC
Upstream bug assigned to oezr

Comment 6 Ondřej Ezr 2021-06-11 11:17:01 UTC
*** Bug 1943347 has been marked as a duplicate of this bug. ***

Comment 10 Bryan Kearney 2022-06-06 16:02:49 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/32188 has been resolved.

Comment 11 Sam Bible 2022-08-17 19:54:34 UTC
Version: Satellite 6.12.0 - 6.1

Steps to Reproduce:
1. Enable Ansible log debugging at -vvv level
2. Select two hosts.
3. Run ansible roles on those hosts.
4. Check the Dynflow Console Log on the hosts.

Actual results:
The host execution log is showing the debug log for all hosts present on the inventory instead of just the expected host.

Expected results:
Just the debug output log for the host that the output is being checked.

Additional info:
version of smart_proxy on the satellite in question
rubygem-smart_proxy_ansible.noarch                    3.4.0-1.el8sat 

Some output from the run's Dynflow Console log (The host here is 3-58, but you will see an inverted log if you check 2-183 ):


    timestamp: 1660765430.3873568
  - output_type: stdout
    output: "<dhcp-2-183.vms.sat.rdu2.redhat.com> ESTABLISH SSH CONNECTION FOR USER:
      root\n"
    timestamp: 1660765430.387411
  - output_type: stdout
    output: !ruby/string:Sequel::SQL::Blob '<dhcp-2-183.vms.sat.rdu2.redhat.com> SSH:
      EXEC ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s
      -o StrictHostKeyChecking=no -o Port=22 -o ''IdentityFile="/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy"''
      -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
      -o PasswordAuthentication=no -o ''User="root"'' -o ConnectTimeout=10 -o ''ControlPath="/var/lib/foreman-proxy/ansible/cp/f2058edf75"''
      dhcp-2-183.vms.sat.rdu2.redhat.com ''/bin/sh -c ''"''"''( umask 77 && mkdir
      -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1660765425.1873856-18588-142048714011140
      `" && echo ansible-tmp-1660765425.1873856-18588-142048714011140="` echo /root/.ansible/tmp/ansible-tmp-1660765425.1873856-18588-142048714011140
      `" ) && sleep 0''"''"''''

'
    timestamp: 1660765430.3874888
  - output_type: stdout
    output: "<dhcp-3-58.vms.sat.rdu2.redhat.com> (0, b'/root\\n', b'')\n"
    timestamp: 1660765430.3875659
  - output_type: stdout
    output: "<dhcp-3-58.vms.sat.rdu2.redhat.com> ESTABLISH SSH CONNECTION FOR USER:
      root\n"
    timestamp: 1660765430.387669
  - output_type: stdout

Comment 12 Adam Ruzicka 2022-09-01 11:10:18 UTC
Unless the fix was cherrypicked in downstream packaging, and I don't think it was, this had no chance to get into 6.12 yet. If I'm reading things right this was not released in any version of smart_proxy_ansible yet.

Comment 14 Sam Bible 2022-09-12 15:29:49 UTC
Version: Satellite 6.12.0 - 10

Steps to Reproduce:
1. Enable Ansible log debugging at -vvv level
2. Select two hosts.
3. Run ansible roles on those hosts.
4. Check the Dynflow Console Log on the hosts.

Actual results:
The host execution log is showing the debug log for all hosts present on the inventory instead of just the expected host.

Expected results:
The host execution log is showing the debug log for all hosts present on the inventory instead of just the expected host.

Comment 18 errata-xmlrpc 2022-11-16 13:32:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506