Bug 1411783 - Update of the HE VM does not work
Summary: Update of the HE VM does not work
Alias: None
Product: ovirt-hosted-engine-ha
Classification: oVirt
Component: Agent
Version: 2.1.0
Hardware: x86_64
OS: Linux
Target Milestone: ovirt-4.1.0-rc
Assignee: Jenny Tokar
QA Contact: Artyom
Depends On:
TreeView+ depends on / blocked
Reported: 2017-01-10 13:26 UTC by Artyom
Modified: 2017-02-15 14:51 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-02-15 14:51:20 UTC
oVirt Team: SLA
rule-engine: ovirt-4.1+
rule-engine: blocker+
rule-engine: planning_ack+
msivak: devel_ack+
mavital: testing_ack+

Attachments (Terms of Use)
logs(you can start looking from the date "Tue Jan 10 15:15:00") (638.24 KB, application/zip)
2017-01-10 13:26 UTC, Artyom
no flags Details

System ID Private Priority Status Summary Last Updated
oVirt gerrit 70112 0 master MERGED Do not replace the ovf configuration with the backup vm.conf 2017-01-12 13:49:06 UTC
oVirt gerrit 70115 0 v2.1.z MERGED Do not replace the ovf configuration with the backup vm.conf 2017-01-12 13:55:13 UTC
oVirt gerrit 71163 0 v2.1.z MERGED When refreshing vm.conf use dedicated method 2017-01-25 13:56:56 UTC

Description Artyom 2017-01-10 13:26:32 UTC
Created attachment 1239107 [details]
logs(you can start looking from the date "Tue Jan 10 15:15:00")

Description of problem:
Update of the HE VM does not work

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Deploy HE and add master storage domain to the engine to initiate auto-import process(HE VM has 4GB)
2. Enable global maintenance
3. Update OvfUpdateIntervalInMinutes to 1 minute (# engine-config -s OvfUpdateIntervalInMinutes=1 && systemctl restart ovirt-engine)
4. Update HE VM memory to 6GB
5. Wait 5 minutes(to be sure that OVF updated)
6. Restart HE VM(on the host # hosted-engine --vm-poweroff && hosted-engine --vm-start)
7. Check amount of the memory on the HE VM guest OS

Actual results:
Guest OS has 4GB of the memory

Expected results:
Guest OS has 8GB of the memory

Additional info:
I also tried to reduce number of CPU's and it also does not work

Comment 1 Doron Fediuck 2017-01-11 11:33:13 UTC
Looking at the agent log below shows an overrun.
Martin, care to review?

1. 4GiB before the change:

MainThread::DEBUG::2017-01-10 15:16:30,408::config::448::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_file_content_from_shared_storage) Reading 'vm.conf' from '/rhev/data-center/mnt/'
MainThread::DEBUG::2017-01-10 15:16:30,409::heconflib::73::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_dd_pipe_tar) executing: 'dd if=/rhev/data-center/mnt/ bs=4k'
MainThread::DEBUG::2017-01-10 15:16:30,425::heconflib::74::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_dd_pipe_tar) executing: 'tar -xOf - vm.conf'
MainThread::DEBUG::2017-01-10 15:16:30,438::heconflib::92::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_dd_pipe_tar) stdout: vmId=d53e5737-22ac-44f9-bb10-3006dee22b05

2. Then we see the update:

MainThread::INFO::2017-01-10 15:16:44,004::config::409::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::DEBUG::2017-01-10 15:16:44,009::ovf2VmParams::243::root::(confFromOvf) conf is cpuType=Conroe

3. Then it's back to 4GiB:

MainThread::DEBUG::2017-01-10 15:16:44,025::heconflib::142::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(extractConfFile) extracting 'vm.conf' from '/rhev/data-center/mnt/'
MainThread::DEBUG::2017-01-10 15:16:44,025::heconflib::74::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_dd_pipe_tar) executing: 'tar -xOf - vm.conf'
MainThread::DEBUG::2017-01-10 15:16:44,038::heconflib::92::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_dd_pipe_tar) stdout: vmId=d53e5737-22ac-44f9-bb10-3006dee22b05

Comment 2 Doron Fediuck 2017-01-11 11:40:29 UTC
looking at the time diff (0.021 sec) suggests it may be a race.
What is the frequency of reproduction?

Comment 3 Martin Sivák 2017-01-11 11:54:30 UTC
Artyom, you do not need to do the step no. 3 anymore:

3. Update OvfUpdateIntervalInMinutes to 1 minute (# engine-config -s OvfUpdateIntervalInMinutes=1 && systemctl restart ovirt-engine)

But even if you do, what do you see in the webadmin UI? And does this happen when you leave the update interval set to 60s?

Comment 4 Artyom 2017-01-11 13:54:34 UTC
To Doron:
I tried it 3 times on different setups and all time have the same issue so for it 100%

To Martin:
1. Under the UI I can see that HE VM has updated configuration(like memory equal to 6GB)
2. I tried it first with default OVF update interval, when it did not work I changed OVF update interval to 1 minute.

Comment 5 Martin Sivák 2017-01-12 12:35:37 UTC
Doron, your comment talks about two different files. The vm.conf and the ovf stores. We have both in the shared storage and only the OVF store is updated (vm.conf is the original configuration as used by setup).

The OVF store contains the proper value since the update and never reverts back.

So the bug is probably only in the part that decides which file will be used to start the VM.

Comment 6 Martin Sivák 2017-01-12 13:00:10 UTC
And that is here:

MainThread::INFO::2017-01-10 15:16:44,009::config::414::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE

Here we know that the OVF value was properly extracted. And yet the if not content: returns true and continues to read the fallback config file.


15:16:44,010::config::448::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_file_content_from_shared_storage) Reading 'vm.conf' from '/rhev/data-center/mnt/'

Comment 7 Martin Sivák 2017-01-12 13:33:08 UTC
The code I cited was a bit old, but the new one had the issue slightly better hidden. We properly used the OVF content, but then rewrote it when we tried to publish it to the /var/run cache file.

Comment 8 Artyom 2017-01-24 11:02:23 UTC
The problem still exists under ovirt-hosted-engine-ha-2.1.0-1.el7ev.noarch

Comment 9 Red Hat Bugzilla Rules Engine 2017-01-24 11:02:29 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 10 Artyom 2017-02-05 14:30:54 UTC
Verified on ovirt-hosted-engine-ha-

1) Memory update - PASS
2) CPU update - PASS
3) Add additional nic - PASS

Note You need to log in before you can comment on or make changes to this bug.