Description of problem:

The ansible playbook executed for "check for upgrade" is not showing the update of "redhat-virtualization-host-image-update". However, if we execute it manually, it shows the update.

The one executed manually shows redhat-virtualization-host-image-update and rhvm-appliance...

# ANSIBLE_STDOUT_CALLBACK=hostupgradeplugin ansible-playbook --check --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa --inventory=/tmp/inventory_rh /usr/share/ovirt-engine/playbooks/ovirt-host-upgrade.yml
["python-perf", "redhat-virtualization-host-image-update", "libgomp", "libvirt-lock-sanlock", "rhvm-appliance", "libgcc"]


However, if the engine is executing the same, it's not showing the redhat-virtualization-host-image-update and rhvm-appliance.

2019-03-15 09:14:44,447+03 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-hostUpdatesChecker-Thread-3) [] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Check for available updates on host  was completed successfully with message 'found updates for packages libvirt-lock-sanlock, python-perf, libgcc, libgomp'.

There is no way to get logs of the playbook as we are passing LEVEL0 as per https://gerrit.ovirt.org/#/c/86124/. Since the output is temporarily written in "/tmp", I read it using the below script.

# while true; do if ls /tmp/playbook* 1> /dev/null 2>&1; then cat /tmp/playbook*; fi;sleep 0.001; done
["libvirt-lock-sanlock", "python-perf", "libgcc", "libgomp"]

So for the exact commands, ansible is giving one output when it's run through command line and a different when it's executed by the engine which looks very strange.

To get more insight and to get the verbose output of Ansible, I tried to troubleshoot with byteman with below code.


--------- cut here -------
RULE trace logging entry
CLASS org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder
METHOD enableLogging
AT ENTRY
IF true
DO
    traceln("::::::::: invoke" + $1);
    $1 = true;
ENDRULE
RULE trace verbose entry
CLASS org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder
METHOD verboseLevel
AT ENTRY
IF true
DO
    traceln("::::::::: invoke" + $1);
    $1 = AnsibleVerbosity.LEVEL4;
ENDRULE
--------- cut here -------


I was able to get the verbose output in my test server however not working for the customer server. Also, it's not working in another test server which I have tested. Getting below message. Looks like byteman is loaded twice.

===
2019-03-18 22:09:38,336-04 INFO  [stdout] (Weld Thread Pool -- 8) RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder.verboseLevel(org.ovirt.engine.core.common.utils.ansible.AnsibleVerbosity) org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder for rule trace verbose entry

2019-03-18 22:09:38,350-04 INFO  [stdout] (Weld Thread Pool -- 9) RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder.verboseLevel(org.ovirt.engine.core.common.utils.ansible.AnsibleVerbosity) org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder for rule trace verbose entry

2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute called for trace logging entry_2
2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute for decommissioned key trace logging entry_2
2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute called for trace verbose entry_0
2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute for decommissioned key trace verbose entry_0
====

Without logs, I am unable to troubleshoot this further.

Version-Release number of selected component (if applicable):

rhvm-4.2.8.5

How reproducible:

All 3 servers of customers.

Steps to Reproduce:

When doing "check for upgrade" in customer's environment.

Actual results:

"Check for upgrade" not showing the update for redhat-virtualization-host-image

Expected results:

"Check for upgrade" should show update for redhat-virtualization-host-image

Additional info: