Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1597290

Summary: ovs-vsctl database connection failed (No such file or directory)
Product: Red Hat OpenStack Reporter: Noam Manos <nmanos>
Component: opendaylightAssignee: lpeer <lpeer>
Status: CLOSED WORKSFORME QA Contact: Noam Manos <nmanos>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: aadam, mkolesni, nmanos, nyechiel, tfreger
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
N/A
Last Closed: 2018-07-09 08:32:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Console output
none
sosreport controller-0 none

Description Noam Manos 2018-07-02 13:45:07 UTC 
Description of problem:
After deploying OSP13 with ODL - all ODL agents on Controller and Compute are down.
Their ovs-vsctl shows: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

Version-Release number of selected component (if applicable):
OSP 13 - 2018-06-13.1
RHEL 7.5 - Linux 3.10.0-862.6.3.el7.x86_64


How reproducible:
On certain environment.

Steps to Reproduce:

1) Install OSP13 with ODL as in:
http://staging-jenkins2-qe-playground.usersys.redhat.com/view/DFG/view/opendaylight/view/odl-netvirt/job/DFG-opendaylight-odl-netvirt-13_director-rhel-virthost-1cont_2comp-ipv4-vxlan-tempest-itzikb-poc/4/consoleFull

2) ssh into controllers and computes and check ovs status:
[root@controller-0 heat-admin]# ovs-vsctl show

3) Restart ovs:
[root@controller-0 heat-admin]# systemctl restart openvswitch

4) Go back to undercloud and check neutron agents:
[stack@undercloud-0 ~]$ . overcloudrc
(overcloud) [stack@undercloud-0 ~]$ neutron agent-list


Actual results:
[root@controller-0 heat-admin]# ovs-vsctl show
ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)

After restarting controller OVS, we saw ODL L2 agents are all down:
(overcloud) [stack@undercloud-0 ~]$ neutron agent-list

+--------------------------------------+----------------+--------------------------+-------------------+-------+----------------+------------------------------+
| id                                   | agent_type     | host                     | availability_zone | alive | admin_state_up | binary                       |
+--------------------------------------+----------------+--------------------------+-------------------+-------+----------------+------------------------------+
| 06959a19-5cdb-4e83-8984-a8ea56a7aecb | ODL L2         | controller-0.localdomain |                   | :-)   | True           | neutron-odlagent-portbinding |
| 388411b7-284f-49f5-8181-65ab89014467 | ODL L2         | compute-1.localdomain    |                   | xxx   | True           | neutron-odlagent-portbinding |
| ac44d6a5-27c0-4c36-9a5b-5f82e710c3d1 | DHCP agent     | controller-0.localdomain | nova              | :-)   | True           | neutron-dhcp-agent           |
| b23df8cd-e0a7-4f99-9bc5-a4b55ff78a81 | ODL L2         | compute-0.localdomain    |                   | xxx   | True           | neutron-odlagent-portbinding |
| d25fcff3-8864-4502-b6c5-a266eb9d9435 | Metadata agent | controller-0.localdomain |                   | :-)   | True           | neutron-metadata-agent       |
+--------------------------------------+----------------+--------------------------+-------------------+-------+----------------+------------------------------+


Expected results:
All ODL L2 agents should be alive.

Additional info:
Attaching console output and sosreport from controller.
Comment 1 Noam Manos 2018-07-02 13:46:41 UTC 
Created attachment 1455963 [details]
Console output
Comment 2 Noam Manos 2018-07-02 13:48:26 UTC 
Created attachment 1455964 [details]
sosreport controller-0
Comment 3 Mike Kolesnik 2018-07-04 12:34:26 UTC 
Is this with GA puddle?
The GA puddle is 2018-06-21.2 but you specified 2018-06-13.1
Comment 4 Noam Manos 2018-07-04 16:21:16 UTC 
(In reply to Mike Kolesnik from comment #3)
> Is this with GA puddle?
> The GA puddle is 2018-06-21.2 but you specified 2018-06-13.1

The puddle that came with this Jenkins IR deployment was 2018-06-13.1 (Gerrit Branch rhos-13.0-patches). I'm redploying to verify what's the puddle is:

https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/opendaylight/view/odl-netvirt/job/DFG-opendaylight-odl-netvirt-13_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ha-tempest/237/consoleFull
Comment 5 Noam Manos 2018-07-09 08:32:07 UTC 
Cannot reproduce on puddle: 2018-07-03.3