Bug 1566111 - Failed to auto-import HE-VM after vintage deployment of SHE.
Summary: Failed to auto-import HE-VM after vintage deployment of SHE.
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.HostedEngine
Version: 4.2.2.6
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ovirt-4.2.3
: ---
Assignee: Doron Fediuck
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks: 1560666
TreeView+ depends on / blocked
 
Reported: 2018-04-11 14:26 UTC by Nikolai Sednev
Modified: 2018-04-22 11:35 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-20 09:18:29 UTC
oVirt Team: Integration
ylavi: ovirt-4.2+
ykaul: blocker+


Attachments (Terms of Use)
engine logs (9.44 MB, application/x-xz)
2018-04-11 14:26 UTC, Nikolai Sednev
no flags Details
sosreport from alma04 (11.88 MB, application/x-xz)
2018-04-11 14:28 UTC, Nikolai Sednev
no flags Details

Description Nikolai Sednev 2018-04-11 14:26:43 UTC
Created attachment 1420358 [details]
engine logs

Description of problem:
Failed to auto-import HE-VM after vintage deployment of SHE.
Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "name" violates not-null constraint
2018-04-11 17:14:39,502+03 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineS
cheduled-Thread-96) [d43f1d3] Trying to release exclusive lock which does not exist, lock key: 'HostedEngineVM_NAME'
2018-04-11 17:14:39,502+03 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-eng
ineScheduled-Thread-96) [d43f1d3] Lock freed to object 'EngineLock:{exclusiveLocks='[d182a2da-3d05-4b46-8231-fb1451dc6
69f=VM, HostedEngine=VM_NAME]', sharedLocks='[d182a2da-3d05-4b46-8231-fb1451dc669f=REMOTE_VM]'}'
2018-04-11 17:14:39,519+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-96) [d43f1d3] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm HostedEngine to Data Center Default, Cluster Default
2018-04-11 17:14:39,519+03 WARN  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-96) [d43f1d3] Command 'ImportVm' id: '6ee413c4-5306-4032-a1cb-6864665e58ac' end method execution failed, as the command isn't marked for endAction() retries silently ignoring

Forth to Simone, the root issue is here: https://paste.fedoraproject.org/paste/UI4Wqyh2RvqpO9lp8z6gJg

Version-Release number of selected component (if applicable):
Engine:
ovirt-engine-setup-4.2.2.6-0.1.el7.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Host:
rhvm-appliance-4.2-20180410.0.el7.noarch
ovirt-hosted-engine-setup-2.2.16-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.10-1.el7ev.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

How reproducible:
100%

Steps to Reproduce:
1.Deploy SHE over any type of storage, in my case it was iSCSI and NFS was used for data storage domain.
2.Add data storage domain and wait for auto-import of SHE-VM to take place.

Actual results:
Auto-import fails.

Expected results:
Auto import should succeed.

Additional info:
Logs from engine and host attached.

Comment 1 Nikolai Sednev 2018-04-11 14:28:30 UTC
Created attachment 1420359 [details]
sosreport from alma04

Comment 2 Yaniv Kaul 2018-04-11 15:09:14 UTC
Why would you use vintage deployment? (Just to try it out as plan B to node zero deployment?)

Comment 3 Nikolai Sednev 2018-04-11 15:46:22 UTC
(In reply to Yaniv Kaul from comment #2)
> Why would you use vintage deployment? (Just to try it out as plan B to node
> zero deployment?)

Because of the verification of other bugs requires it. This one for example: https://bugzilla.redhat.com/show_bug.cgi?id=1560666

Comment 4 Simone Tiraboschi 2018-04-12 07:13:08 UTC
(In reply to Nikolai Sednev from comment #3)
> (In reply to Yaniv Kaul from comment #2)
> > Why would you use vintage deployment? (Just to try it out as plan B to node
> > zero deployment?)
> 
> Because of the verification of other bugs requires it. This one for example:
> https://bugzilla.redhat.com/show_bug.cgi?id=1560666

And by the way some specif flows, like migrating to hosted-engine, still relies on that to let the user restore a engine backup before executing engine-setup.

Comment 5 Martin Sivák 2018-04-12 07:27:36 UTC
Yaniv, we still have it as a fallback in 4.2.

Comment 6 Simone Tiraboschi 2018-04-12 08:17:59 UTC
Upstream on 4.2 on OST is still working:

http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_he-basic-suite-4.2/96/artifact/exported-artifacts/test_logs/he-basic-suite-4.2/post-006_network_by_label.py/lago-he-basic-suite-4-2-engine/_var_log/ovirt-engine/engine.log

2018-04-11 21:48:06,131-04 INFO  [org.ovirt.engine.core.bll.HostedEngineImporter] (EE-ManagedThreadFactory-engine-Thread-20) [5bbf633c] Try to import the Hosted Engine VM 'VM [HostedEngine]'
2018-04-11 21:48:06,260-04 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[37b940bd-e487-4036-a3a2-d40f2afb9fd5=VM, HostedEngine=VM_NAME]', sharedLocks='[37b940bd-e487-4036-a3a2-d40f2afb9fd5=REMOTE_VM]'}'
2018-04-11 21:48:06,328-04 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] START, GetImageInfoVDSCommand( GetImageInfoVDSCommandParameters:{storagePoolId='4175b61e-3df2-11e8-9a08-5452c0a8c863', ignoreFailoverLimit='false', storageDomainId='97b3ec99-c788-48d2-a8f2-85c5cfd48b4f', imageGroupId='f5b81a12-db53-4671-87a1-3ba17be698a7', imageId='39be3abc-3e66-4e9d-8ebe-972b05fb4ffa'}), log id: 6674c459
2018-04-11 21:48:06,330-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] START, GetVolumeInfoVDSCommand(HostName = lago-he-basic-suite-4-2-host-0, GetVolumeInfoVDSCommandParameters:{hostId='328c5290-7939-474e-894f-b5f766366435', storagePoolId='4175b61e-3df2-11e8-9a08-5452c0a8c863', storageDomainId='97b3ec99-c788-48d2-a8f2-85c5cfd48b4f', imageGroupId='f5b81a12-db53-4671-87a1-3ba17be698a7', imageId='39be3abc-3e66-4e9d-8ebe-972b05fb4ffa'}), log id: 75a14e5a
2018-04-11 21:48:06,353-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] FINISH, GetVolumeInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@9a66e055, log id: 75a14e5a
2018-04-11 21:48:06,353-04 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage.DiskImage@9a66e055, log id: 6674c459
2018-04-11 21:48:06,842-04 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] Running command: ImportVmCommand internal: true. Entities affected :  ID: 97b3ec99-c788-48d2-a8f2-85c5cfd48b4f Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN,  ID: 97b3ec99-c788-48d2-a8f2-85c5cfd48b4f Type: StorageAction group IMPORT_EXPORT_VM with role type ADMIN
2018-04-11 21:48:07,012-04 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] EVENT_ID: MAC_ADDRESS_IS_EXTERNAL(925), VM HostedEngine has MAC address(es) 54:52:c0:a8:c8:63, which is/are out of its MAC pool definitions.
2018-04-11 21:48:07,088-04 WARN  [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] VM '37b940bd-e487-4036-a3a2-d40f2afb9fd5' doesn't have active snapshot in export domain
2018-04-11 21:48:07,225-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{vmId='37b940bd-e487-4036-a3a2-d40f2afb9fd5', status='Down', exitStatus='Normal'}), log id: 35da80e5
2018-04-11 21:48:07,247-04 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] FINISH, SetVmStatusVDSCommand, log id: 35da80e5
2018-04-11 21:48:07,567-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] EVENT_ID: IMPORTEXPORT_STARTING_IMPORT_VM(1,165), Starting to import Vm HostedEngine to Data Center Default, Cluster Default
2018-04-11 21:48:07,604-04 INFO  [org.ovirt.engine.core.bll.HostedEngineImporter] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] Successfully imported the Hosted Engine VM
2018-04-11 21:48:07,610-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-20) [200277e4] EVENT_ID: HOSTED_ENGINE_VM_IMPORT_SUCCEEDED(10,456), Hosted Engine VM was imported successfully
2018-04-11 21:48:08,959-04 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-79) [200277e4] Command 'ImportVm' id: '8d03deda-b36c-45c4-8f39-8ac4164a003b' child commands '[]' executions were completed, status 'SUCCEEDED'
2018-04-11 21:48:10,014-04 INFO  [org.ovirt.engine.core.bll.exportimport.ImportVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [200277e4] Ending command 'org.ovirt.engine.core.bll.exportimport.ImportVmCommand' successfully.

Comment 7 Martin Sivák 2018-04-12 08:23:15 UTC
Can this be another case of 7.4 vs 7.5?

Nikolai, is there anything specific about your machine? Some strange networking or something?

Comment 8 Simone Tiraboschi 2018-04-12 08:24:46 UTC
The nic device on Nikolai test:

    <interface type='bridge'>
      <mac address='00:16:3e:7b:b8:54'/>
      <source bridge='ovirtmgmt'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

and on OST one:

    <interface type='bridge'>
      <mac address='54:52:c0:a8:c8:63'/>
      <source bridge='ovirtmgmt'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

looks absolutely comparable.

Comment 9 Nikolai Sednev 2018-04-12 08:26:31 UTC
(In reply to Martin Sivák from comment #7)
> Can this be another case of 7.4 vs 7.5?
> 
> Nikolai, is there anything specific about your machine? Some strange
> networking or something?

Nothing.
I'm running on RHEL7.5 from scratch over the same untagged native VLAN topology, using the same host that works just fine with the new Node 0 appliance, but fails with the vintage.

Comment 10 Martin Sivák 2018-04-12 14:08:30 UTC
I just tested this using pure host with no vlans (just eth0 and the ovirtmgmt) and all went fine.

Comment 11 Nikolai Sednev 2018-04-12 15:35:46 UTC
(In reply to Martin Sivák from comment #10)
> I just tested this using pure host with no vlans (just eth0 and the
> ovirtmgmt) and all went fine.

Please try doing so while deploying SHE over iSCSI instead of NFS, as that what I did differently on my environment.

Comment 12 Martin Sivák 2018-04-16 07:52:19 UTC
Still does not reproduce, I can try once more in case it is a race condition, but please do the same.

Comment 13 Sandro Bonazzola 2018-04-20 09:03:13 UTC
Setting conditional NAK on reproducer. We can't do much without being able to reproduce the issue.

Comment 14 Simone Tiraboschi 2018-04-20 09:18:29 UTC
CLOSING as WORKSFORME,
please reopen if we found a reproducer

Comment 15 Nikolai Sednev 2018-04-22 11:35:54 UTC
Works for me on these components:
ovirt-hosted-engine-setup-2.2.18-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.10-1.el7ev.noarch
rhvm-appliance-4.2-20180420.0.el7.noarch
Linux 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Successfully deployed vintage over iSCSI and also added NFS data storage domain, then received hosted-storage as auto-import without any issues.


Note You need to log in before you can comment on or make changes to this bug.