Bug 1483309

Summary: startVM fails with OVN network
Product: [oVirt] vdsm Reporter: Mor <mkalfon>
Component: CoreAssignee: Francesco Romani <fromani>
Status: CLOSED CURRENTRELEASE QA Contact: Mor <mkalfon>
Severity: high Docs Contact:
Priority: unspecified    
Version: ---CC: ahadas, bugs, danken, edwardh, fromani, mburman, myakove, tjelinek
Target Milestone: ovirt-4.2.0Keywords: Automation, Regression
Target Release: ---Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-20 10:42:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 988285    
Attachments:
Description Flags
vdsm supervdsm and engine logs none

Description Mor 2017-08-20 10:57:48 UTC
Created attachment 1315927 [details]
vdsm supervdsm and engine logs

Description of problem:
VM with OVN network attached fails to start.

vdsm.log shows the following error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 117, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 301, in destroy
    res = self.vm.destroy(gracefulAttempts)
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 129, in vm
    raise exception.NoSuchVM(vmId=self._UUID)
NoSuchVM: Virtual machine does not exist: {'vmId': '0cb1b7a9-c7b2-401e-9c0f-3f46257e5154'}

Version-Release number of selected component (if applicable):
vdsm-api-4.20.2-90.git6511af5.el7.centos.noarch
vdsm-hook-vmfex-dev-4.20.2-90.git6511af5.el7.centos.noarch
vdsm-http-4.20.2-90.git6511af5.el7.centos.noarch
vdsm-yajsonrpc-4.20.2-90.git6511af5.el7.centos.noarch
vdsm-client-4.20.2-90.git6511af5.el7.centos.noarch
vdsm-jsonrpc-4.20.2-90.git6511af5.el7.centos.noarch
vdsm-python-4.20.2-90.git6511af5.el7.centos.noarch
vdsm-4.20.2-90.git6511af5.el7.centos.x86_64
openvswitch-ovn-common-2.7.2-3.1fc27.el7.x86_64
ovirt-provider-ovn-driver-1.1-2.20170818071729.git634477f.el7.centos.noarch
openvswitch-ovn-host-2.7.2-3.1fc27.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Use engine-setup and answer-file to install oVirt engine and OVN.
2. Install ovirt-provider-ovn-driver on host.
3. Create OVN network and attach the network to a new vNIC on VM.
4. Start the VM.

Actual results:
VM fails to start, engine and vdsm report an error.

Expected results:
Should start normally.

Comment 1 Francesco Romani 2017-08-31 09:57:46 UTC
plain OVS works with
vdsm-hook-vmfex-dev-4.20.2-122.git87f120b.el7.centos.noarch
vdsm-client-4.20.2-122.git87f120b.el7.centos.noarch
vdsm-python-4.20.2-122.git87f120b.el7.centos.noarch
vdsm-jsonrpc-4.20.2-122.git87f120b.el7.centos.noarch
vdsm-api-4.20.2-122.git87f120b.el7.centos.noarch
vdsm-4.20.2-122.git87f120b.el7.centos.x86_64
vdsm-yajsonrpc-4.20.2-122.git87f120b.el7.centos.noarch
vdsm-http-4.20.2-122.git87f120b.el7.centos.noarch


Vdsm produces:

        <interface type="bridge">
            <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci" />
            <mac address="$MAC" />
            <model type="virtio" />
            <source bridge="vdsmbr_6gnlz2dB" />
            <virtualport type="openvswitch" />
            <filterref filter="vdsm-no-mac-spoofing" />
        </interface>

virsh dumpxml:

    <interface type='bridge'>
      <mac address='$MAC'/>
      <source bridge='vdsmbr_6gnlz2dB'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='e74663f0-b181-4a53-a5a8-117ee26f0d24'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

ovs-vsctl show:
ac17af3c-1a10-4740-a920-fbc3c20c9205
    Bridge "vdsmbr_6gnlz2dB"
        Port "vnet0"
            Interface "vnet0"
        Port "vdsmbr_6gnlz2dB"
            Interface "vdsmbr_6gnlz2dB"
                type: internal
        Port ovirtmgmt
            Interface ovirtmgmt
                type: internal
        Port "eth0"
            Interface "eth0"
    ovs_version: "2.7.0"

so looks like the issue is related to ovirt-provider-ovn-driver?
On the host on which I tried, we have:

# rpm -qa | grep ovn
(nothing)
# rpm -qa | grep openvswitch
openvswitch-2.7.0-1.el7.x86_64

Dan/Edward: I'm not sure how to proceed further. Maybe just installing the missing packages and try again? any configuration needed on the host?

Comment 2 Dan Kenigsberg 2017-08-31 10:44:48 UTC
Let us leave switchType=ovs aside, for a while. I do not think it is anyway related.

I don't see in the attached vdsm.log that ovirt-provider-ovn-driver's hook was actually executed. Do you?

Comment 3 Francesco Romani 2017-08-31 11:34:14 UTC
(In reply to Dan Kenigsberg from comment #2)
> Let us leave switchType=ovs aside, for a while. I do not think it is anyway
> related.
> 
> I don't see in the attached vdsm.log that ovirt-provider-ovn-driver's hook
> was actually executed. Do you?

Nope, I don't see the hook being called.
I reviewed the code to doublecheck we still call the hooks in the engine xml flow, and we do. That part is not supposed to change.

Comment 4 Francesco Romani 2017-09-01 07:52:42 UTC
(In reply to Francesco Romani from comment #3)
> (In reply to Dan Kenigsberg from comment #2)
> > Let us leave switchType=ovs aside, for a while. I do not think it is anyway
> > related.
> > 
> > I don't see in the attached vdsm.log that ovirt-provider-ovn-driver's hook
> > was actually executed. Do you?
> 
> Nope, I don't see the hook being called.
> I reviewed the code to doublecheck we still call the hooks in the engine xml
> flow, and we do. That part is not supposed to change.

I was half-wrong.
The code to call the hook is indeed there, but that code is triggered only if the device has some custom properties. And currently we have none, because Vdsm doesn't know where to find them

Patch https://gerrit.ovirt.org/#/c/81286/1 will make Vdsm consume the custom properties, but to fix this bug, Engine needs to send them.

Comment 5 Mor 2017-09-07 06:12:57 UTC
The patches are not merged yet.

Comment 6 Arik 2017-09-07 06:28:11 UTC
(In reply to Mor from comment #5)
> The patches are not merged yet.

I wonder if a test actually failed or is it only because of the tracking of the patches here in bugzilla... the patches are in and should be available for you in the nightly build already, this bz has not been updated with the right status of patches for some reason.

Comment 7 Francesco Romani 2017-09-11 07:57:05 UTC
regression in pre-alpha. Customers should never seen this, hence no need for doc_text.

Comment 8 Mor 2017-09-11 10:41:06 UTC
Verified:
4.2.0-0.0.master.20170906185835.gitcee3e58.el7.centos
ovirt-provider-ovn-1.1-2.20170905145914.gitaf4fe87.el7.centos.noarch
ovirt-provider-ovn-driver-1.1-2.20170905145914.gitaf4fe87.el7.centos.noarch

Comment 9 Sandro Bonazzola 2017-12-20 10:42:00 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.