Bug 1690328

Summary: "Check for upgrade" not showing the update for redhat-virtualization-host-image
Product: Red Hat Enterprise Virtualization Manager Reporter: nijin ashok <nashok>
Component: ovirt-engineAssignee: Ondra Machacek <omachace>
Status: CLOSED ERRATA QA Contact: Petr Matyáš <pmatyas>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2.8-4CC: gveitmic, lleistne, lsvaty, mgoldboi, mperina, omachace, Rhev-m-bugs
Target Milestone: ovirt-4.3.4Keywords: ZStream
Target Release: 4.3.1   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-engine-4.3.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-20 14:48:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description nijin ashok 2019-03-19 09:55:48 UTC
Description of problem:

The ansible playbook executed for "check for upgrade" is not showing the update of "redhat-virtualization-host-image-update". However, if we execute it manually, it shows the update.

The one executed manually shows redhat-virtualization-host-image-update and rhvm-appliance...

# ANSIBLE_STDOUT_CALLBACK=hostupgradeplugin ansible-playbook --check --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa --inventory=/tmp/inventory_rh /usr/share/ovirt-engine/playbooks/ovirt-host-upgrade.yml
["python-perf", "redhat-virtualization-host-image-update", "libgomp", "libvirt-lock-sanlock", "rhvm-appliance", "libgcc"]


However, if the engine is executing the same, it's not showing the redhat-virtualization-host-image-update and rhvm-appliance.

2019-03-15 09:14:44,447+03 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-hostUpdatesChecker-Thread-3) [] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Check for available updates on host  was completed successfully with message 'found updates for packages libvirt-lock-sanlock, python-perf, libgcc, libgomp'.

There is no way to get logs of the playbook as we are passing LEVEL0 as per https://gerrit.ovirt.org/#/c/86124/. Since the output is temporarily written in "/tmp", I read it using the below script.

# while true; do if ls /tmp/playbook* 1> /dev/null 2>&1; then cat /tmp/playbook*; fi;sleep 0.001; done
["libvirt-lock-sanlock", "python-perf", "libgcc", "libgomp"]

So for the exact commands, ansible is giving one output when it's run through command line and a different when it's executed by the engine which looks very strange.

To get more insight and to get the verbose output of Ansible, I tried to troubleshoot with byteman with below code.


--------- cut here -------
RULE trace logging entry
CLASS org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder
METHOD enableLogging
AT ENTRY
IF true
DO
    traceln("::::::::: invoke" + $1);
    $1 = true;
ENDRULE
RULE trace verbose entry
CLASS org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder
METHOD verboseLevel
AT ENTRY
IF true
DO
    traceln("::::::::: invoke" + $1);
    $1 = AnsibleVerbosity.LEVEL4;
ENDRULE
--------- cut here -------


I was able to get the verbose output in my test server however not working for the customer server. Also, it's not working in another test server which I have tested. Getting below message. Looks like byteman is loaded twice.

===
2019-03-18 22:09:38,336-04 INFO  [stdout] (Weld Thread Pool -- 8) RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder.verboseLevel(org.ovirt.engine.core.common.utils.ansible.AnsibleVerbosity) org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder for rule trace verbose entry

2019-03-18 22:09:38,350-04 INFO  [stdout] (Weld Thread Pool -- 9) RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder.verboseLevel(org.ovirt.engine.core.common.utils.ansible.AnsibleVerbosity) org.ovirt.engine.core.common.utils.ansible.AnsibleCommandBuilder for rule trace verbose entry

2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute called for trace logging entry_2
2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute for decommissioned key trace logging entry_2
2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute called for trace verbose entry_0
2019-03-18 22:12:57,578-04 INFO  [stdout] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) Rule.execute for decommissioned key trace verbose entry_0
====

Without logs, I am unable to troubleshoot this further.

Version-Release number of selected component (if applicable):

rhvm-4.2.8.5

How reproducible:

All 3 servers of customers.

Steps to Reproduce:

When doing "check for upgrade" in customer's environment.

Actual results:

"Check for upgrade" not showing the update for redhat-virtualization-host-image

Expected results:

"Check for upgrade" should show update for redhat-virtualization-host-image

Additional info:

Comment 5 Ondra Machacek 2019-04-17 14:40:55 UTC
This is most likely issue with locale, bash has different then java proccess I guess. I will verify it on my env.

Comment 6 nijin ashok 2019-04-25 08:57:35 UTC
(In reply to Ondra Machacek from comment #5)
> This is most likely issue with locale, bash has different then java proccess
> I guess. I will verify it on my env.

Hello Ondra,

Did you get a chance to test/verify this?

Thanks.

Comment 7 Ondra Machacek 2019-05-03 11:59:57 UTC
Yes, I did.

Can the customer try to modify /usr/share/ovirt-engine/playbooks/ovirt-host-upgrade.yml and put there after 'remote_user: root' section following two lines:

remote_user: root
environment:
  LANG: "en_US.utf8"

The try to run check-for-upgrade. I've tried it on my env and sucessfully verified, but I would like to confirm it before sending proper patch. Thanks.

Comment 8 nijin ashok 2019-05-07 15:14:00 UTC
(In reply to Ondra Machacek from comment #7)
> Can the customer try to modify
> /usr/share/ovirt-engine/playbooks/ovirt-host-upgrade.yml and put there after
> 'remote_user: root' section following two lines:
> 
> remote_user: root
> environment:
>   LANG: "en_US.utf8"
> 
> The try to run check-for-upgrade. I've tried it on my env and sucessfully
> verified, but I would like to confirm it before sending proper patch. Thanks.

Thank you. It's working good now for the customer.

Comment 10 Petr Matyáš 2019-06-03 11:39:41 UTC
Verified on ovirt-engine-4.3.4.2-0.1.el7.noarch

Check for upgrade succeeded to find redhat-virtualization-host-image.
I've used en_US.UTF-8 and cs_CZ.utf8 locales on RHVH.

Comment 12 errata-xmlrpc 2019-06-20 14:48:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1566