This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 818754

Summary: Provide native systemd unit files and fix startup dependency
Product: [Fedora] Fedora Reporter: Sven Lankes <sven>
Component: openvswitchAssignee: Flavio Leitner <fleitner>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: chrisw, collett.ian, ipilcher, jose.p.oliveira.oss, markmc, mfuruta, mgrepl, rolf, tgraf
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openvswitch-1.10.0-2.fc19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-23 23:39:14 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
Proposed patch
none
auto-start with dependencies
none
strace of "hung" systemctl command (systemctl start openvswitch.service)
none
/etc/sysconfig/network-scripts/ifup-ovs
none
/etc/sysconfig/openvswitch
none
/lib/systemd/system/openvswitch.service
none
/lib/systemd/system/openvswitch-nonetwork.service none

Description Sven Lankes 2012-05-03 16:44:20 EDT
On a machine that has it's OVS-Setup configured through
old-fashioned network-scripts (no NetworkManager) the network
doesn't come up after boot because openvswitch isn't started
until *after* the network scripts try to configure the ovswitch.

changing:

After=syslog.target network.target

to:

Before=network.target
After=syslog.target

in the openvswitch.service-File.

fixes the issue for me.
Comment 1 Chris Wright 2012-06-19 19:45:00 EDT
Unfortunately there is a chicken and egg issue here.  A common ovs configuration uses a controller which is only accessible via the network.  So starting ovs before network.target would break those configurations.
Comment 2 Ian Pilcher 2012-11-12 16:07:37 EST
Editting the unit file in-place is obviously undesirable; changes will be overwritten whenever the openvswitch RPM is upgraded.  Copying it to /etc/systemd/system and editting it there is only slightly less ugly.  (Changes such as the new systemd macros in F18 won't be effective, for example.)  Additionally, moving the initialization of *all* network interfaces around doesn't really solve the problem.

What about something like this?

/usr/lib/systemd/system/ovsif@.service:

  [Unit]
  Description=Open vSwitch Interface %I
  Requires=openvswitch.service
  After=openvswitch.service

  [Service]
  Type=oneshot
  RemainAfterExit=yes
  ExecStart=/sbin/ifup %I
  ExecStop=/sbin/ifdown %I
  StandardOutput=syslog+console

  [Install]
  WantedBy=multi-user.target

To use this:

1.  Set "ONBOOT=no" in the ifcfg file for the Open vSwitch bridge and all
    interfaces that are attached to it.
2.  Enable it with "systemctl enable ovsif@<INTERFACE>.service"

For example, given the following ifcfg files:

  DEVICE=eth0
  ONBOOT=no
  DEVICETYPE=ovs
  TYPE=OVSPort
  IPV6INIT=no
  HWADDR=00:22:4d:4f:c7:ed
  OVS_BRIDGE=ovs0

  DEVICE=ovs0
  ONBOOT=no
  TYPE=OVSBridge
  DEVICETYPE=ovs
  IPV6INIT=no
  IPADDR=172.31.250.1
  GATEWAY=172.31.250.254
  NETMASK=255.255.255.0

Run "systemctl enable ovsif@eth0.service" and the network comes up properly at boot.
Comment 3 Cong Wang 2012-12-11 05:02:51 EST
Hmm, so users have to use "systemctl enable ovsif@eth0.service" instead of "ifup eth0" to enable all ovs ports and bridges?

This seems not convenient at all, but we don't have any better solution...
Comment 4 Ian Pilcher 2012-12-11 07:59:06 EST
(In reply to comment #3)
> Hmm, so users have to use "systemctl enable ovsif@eth0.service" instead of
> "ifup eth0" to enable all ovs ports and bridges?
> 
> This seems not convenient at all, but we don't have any better solution...

Actually, you need to enable one "port" per bridge.  So in the case of a bridge that you want connected to a physical interface, you do "systemctl enable ovsif@<INTERFACE>.service".  In the case of a bridge that will be used for a disconnected or NATed network, you do "systemctl enable ovsif@<BRIDGE>.service".

After using this for a a while, I've seen a couple of drawbacks to this approach:

1) The Open vSwitch interfaces come up late, often a couple of seconds after I've logged in.  Depending on the services that are running on the box, some additional "sequencing" may be required.

2) If the system does not shut down cleanly (kernel panic, etc.), the Open vSwitch database does not get cleaned up, and a bunch of vnetX ports need to be manually cleaned up before I can start my VMs.  (Actually, I don't think that this is specifically related to the unit file.)

I still don't have a better idea, however.
Comment 5 Cong Wang 2012-12-12 04:13:38 EST
I am wondering if we could first start ovs-vswitchd and ovsdb-server without network (that is, without connecting to controllers), and then after network is started, use ovs-vsctl to active the network connection to controllers. This requires to change the source code of ovs-vswitchd, if not implemented.

Fortunately, ovs-vswitchd talks with ovsdb-server via unix socket in openvswitch service file.
Comment 6 Thomas Graf 2012-12-12 04:37:07 EST
The bridge may be connected to an OpenFlow controller so we may need network connectivity as soon as the first port is attached to the bridge.

The only real solution to this problem seems to be to start the openvswitch service as soon as an ovs interface needs it _and_ to allow ovs ifcfg files to define dependencies on other interfaces that are required to be up as a pre condition if openflow is in use.

ifcfg-ovsbr:
  TYPE=OVSbridge
  DEVICE=ovsbr
  REQUIRES=p20p1

ifcfg-p20p1:
  [...]

ifcfg-port0:
  TYPE=OVSPort
  [...]

What do you think?
Comment 7 Cong Wang 2012-12-13 06:09:22 EST
So, if I understand correctly, in ifup-ovs we need to bring up the REQUIRES interface first? No other additional changes?
Comment 8 Thomas Graf 2012-12-13 12:37:20 EST
Exactly, maybe we can come up with something smarter later on but this solves the dependency issue in a reliable easy to understand way.
Comment 9 Rolf Fokkens 2012-12-26 05:57:04 EST
I followed the directions of the Description (#c0) and have openvswitch running now.

On https://fedoraproject.org/wiki/Features/Open_vSwitch it is stated that openvswitch is now 100% completed. I intended to move my bridge config to openvswitch, but it seems 100% is not accurate. Should I wait?
Comment 10 Thomas Graf 2013-01-02 06:14:59 EST
(In reply to comment #9)
> I followed the directions of the Description (#c0) and have openvswitch
> running now.
> 
> On https://fedoraproject.org/wiki/Features/Open_vSwitch it is stated that
> openvswitch is now 100% completed. I intended to move my bridge config to
> openvswitch, but it seems 100% is not accurate. Should I wait?

The 100% completion represents the goal to integrate Open vSwitch into Fedora with a specified list of features. That goal has been reached so the 100% is accurate.

There may still be bugs though such as the one tracked by this BZ. However, we have been running Open vSwitch on Fedora heavily and it is absolutely reliable in general.
Comment 11 Cong Wang 2013-01-23 02:54:57 EST
Something like below?

diff --git a/ifup-ovs b/ifup-ovs
index c427879..3b4a159 100644
--- a/ifup-ovs
+++ b/ifup-ovs
@@ -36,6 +36,7 @@ fi
 
 case "$TYPE" in
        OVSBridge)
+               [ -n "${OVSRequires}" ] && /sbin/ifup "${OVSRequires}"
                ovs-vsctl -t ${TIMEOUT} -- --may-exist add-br "$DEVICE" $OVS_OPTIONS ${OVS_EXTRA+-- $OVS_EXTRA}
                if [ "${OVSBOOTPROTO}" = "dhcp" ] && [ -n "${OVSINTF}" ]; then
                        case " ${OVSDHCPINTERFACES} " in


I probably miss something...
Comment 12 Thomas Graf 2013-01-23 05:50:30 EST
(In reply to comment #11)
> Something like below?
> 
> diff --git a/ifup-ovs b/ifup-ovs
> index c427879..3b4a159 100644
> --- a/ifup-ovs
> +++ b/ifup-ovs
> @@ -36,6 +36,7 @@ fi
>  
>  case "$TYPE" in
>         OVSBridge)
> +               [ -n "${OVSRequires}" ] && /sbin/ifup "${OVSRequires}"

That's exactly what I meant but I would make it a list of interfaces and iterate over it. I realize it's most likely to be a single interface in most use cases but having the flexibility does not hurt.
Comment 13 Cong Wang 2013-01-25 04:25:46 EST
Agreed.


diff --git a/ifup-ovs b/ifup-ovs
index c427879..e06404b 100644
--- a/ifup-ovs
+++ b/ifup-ovs
@@ -36,6 +36,9 @@ fi
 
 case "$TYPE" in
        OVSBridge)
+               [ -n "${OVSRequires}" ] && for _i in "${OVSRequires}"; do
+                       /sbin/ifup "$_i"
+               done
                ovs-vsctl -t ${TIMEOUT} -- --may-exist add-br "$DEVICE" $OVS_OPTIONS ${OVS_EXTRA+-- $OVS_EXTRA}
                if [ "${OVSBOOTPROTO}" = "dhcp" ] && [ -n "${OVSINTF}" ]; then
                        case " ${OVSDHCPINTERFACES} " in
Comment 14 Cong Wang 2013-01-25 04:29:17 EST
Created attachment 687347 [details]
Proposed patch
Comment 15 Thomas Graf 2013-01-25 11:08:34 EST
Created attachment 687525 [details]
auto-start with dependencies

We need to ifup the dependencies before starting the openvswitch service. I also added a recursion checker to be on the safe side.

What do you think?
Comment 17 Fedora Update System 2013-01-25 12:46:59 EST
openvswitch-1.7.3-7.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/openvswitch-1.7.3-7.fc18
Comment 18 Cong Wang 2013-01-26 09:07:01 EST
Nice work! Thanks, Thomas!
Comment 19 Fedora Update System 2013-01-26 11:04:52 EST
Package openvswitch-1.7.3-7.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openvswitch-1.7.3-7.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-1545/openvswitch-1.7.3-7.fc18
then log in and leave karma (feedback).
Comment 20 Fedora Update System 2013-02-04 22:13:40 EST
openvswitch-1.7.3-7.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 21 Ian Pilcher 2013-02-05 13:21:56 EST
After updating to openvswitch-1.7.3-7.fc18 (and disabling the ovsif@ hackery that I was previously using), my Open vSwitch bridges are not being started at boot (or they are taking a *very* long time to come up -- on the order of 5 minutes or more).

The slightly good news is that I am able to reproduce the problem in runlevel 3 by booting with the network service disabled (chkconfig network off) and running "service network start".  Looking at the output of "ps fax", I can see that the network service is stuck trying to start the openvswitch service:

 1152 ?        Ss     0:00 login -- root     
 1282 tty1     Ss     0:00  \_ -bash
 1379 tty1     S+     0:00      \_ /bin/sh /sbin/service network start
 1387 tty1     S+     0:00          \_ /bin/bash /etc/init.d/network start
 1393 tty1     S+     0:00              \_ /bin/systemctl start network.service
 1394 ?        S      0:00                  \_ /usr/bin/systemd-tty-ask-password-agent --watch

 1395 ?        Ss     0:00 /bin/bash /etc/rc.d/init.d/network start
 1482 ?        S      0:00  \_ /bin/bash /etc/sysconfig/network-scripts/ifup-ovs ifcfg-eth0 boot
 1499 ?        S      0:00      \_ /bin/systemctl start openvswitch.service

Eventually, systemd gives up on the network service and kills it, at which point the openvswitch service starts and brings up some of the interfaces/bridges.  journalctl shows this:

Feb 05 11:31:11 ian.icp.selfip.net systemd[1]: Starting LSB: Bring up/down networking...
Feb 05 11:31:11 ian.icp.selfip.net network[1395]: Bringing up loopback interface:  [  OK  ]
Feb 05 11:31:11 ian.icp.selfip.net network[1395]: Bringing up interface eth0:  Redirecting to /bin/systemctl start  openvswitch.service
Feb 05 11:36:11 ian.icp.selfip.net systemd[1]: network.service operation timed out. Terminating.
Feb 05 11:36:11 ian.icp.selfip.net systemd[1]: Failed to start LSB: Bring up/down networking.
Feb 05 11:36:11 ian.icp.selfip.net systemd[1]: Unit network.service entered failed state
Feb 05 11:36:11 ian.icp.selfip.net systemd[1]: Starting Network.
Feb 05 11:36:11 ian.icp.selfip.net systemd[1]: Reached target Network.
Feb 05 11:36:11 ian.icp.selfip.net systemd[1]: Starting Open vSwitch...
Feb 05 11:36:11 ian.icp.selfip.net openvswitch.init[1559]: Inserting openvswitch module [  OK  ]
Feb 05 11:36:11 ian.icp.selfip.net kernel: openvswitch: Open vSwitch switching datapath
Feb 05 11:36:11 ian.icp.selfip.net openvswitch.init[1559]: Starting ovsdb-server [  OK  ]
Feb 05 11:36:11 ian.icp.selfip.net ovs-vsctl[1678]: 00001|vsctl|INFO|Called as ovs-vsctl --no-wait --timeout=5 -- init -- set Open_vSwitch . db-version=6.9.3
Feb 05 11:36:12 ian.icp.selfip.net ovs-vsctl[1683]: 00001|vsctl|INFO|Called as ovs-vsctl --no-wait --timeout=5 set Open_vSwitch . ovs-version=1.7.3 "external-ids:system-id=\"dddaff8a-9df9-4d4a-b4e8-c943848a7b66\"" "system-type=\"Fedora\"" "system-version=\"18-SphericalCow\""
Feb 05 11:36:12 ian.icp.selfip.net openvswitch.init[1559]: Configuring Open vSwitch system IDs [  OK  ]
Feb 05 11:36:12 ian.icp.selfip.net kernel: device ovs0 entered promiscuous mode
Feb 05 11:36:12 ian.icp.selfip.net kernel: device nfs0 entered promiscuous mode
Feb 05 11:36:12 ian.icp.selfip.net kernel: device eth0 entered promiscuous mode
Feb 05 11:36:12 ian.icp.selfip.net openvswitch.init[1559]: Starting ovs-vswitchd [  OK  ]
Feb 05 11:36:12 ian.icp.selfip.net systemd-udevd[1670]: error changing net interface name ovs0 to eth0: File exists
Feb 05 11:36:12 ian.icp.selfip.net systemd[1]: Started Open vSwitch.
Feb 05 11:36:12 ian.icp.selfip.net ovs-vsctl[1702]: 00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br ovs0
Feb 05 11:36:14 ian.icp.selfip.net kernel: e1000e 0000:00:19.0: irq 56 for MSI/MSI-X
Feb 05 11:36:14 ian.icp.selfip.net kernel: e1000e 0000:00:19.0: irq 56 for MSI/MSI-X
Feb 05 11:36:14 ian.icp.selfip.net kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Feb 05 11:36:14 ian.icp.selfip.net ovs-vsctl[1803]: 00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port ovs0 eth0
Feb 05 11:36:14 ian.icp.selfip.net ovs-vsctl[1815]: 00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br ovs0
Feb 05 11:36:18 ian.icp.selfip.net kernel: e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Feb 05 11:36:18 ian.icp.selfip.net kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

I've tried the following, with no success.

* Setting ONBOOT=no if ifcfg-eth0 (the only physical interface).
* Putting SELinux into permissive mode.

Neither change enabled the network service to start properly.

Any ideas?
Comment 22 Thomas Graf 2013-02-05 14:00:17 EST
(In reply to comment #21)
> The slightly good news is that I am able to reproduce the problem in
> runlevel 3 by booting with the network service disabled (chkconfig network
> off) and running "service network start".  Looking at the output of "ps
> fax", I can see that the network service is stuck trying to start the
> openvswitch service:

What happens if you instead of bring up the openvswitch service manually instead of the network service.

> Feb 05 11:36:12 ian.icp.selfip.net systemd-udevd[1670]: error changing net
> interface name ovs0 to eth0: File exists

This looks very wrong.
Comment 23 Ian Pilcher 2013-02-05 14:14:43 EST
(In reply to comment #22)
> What happens if you instead of bring up the openvswitch service manually
> instead of the network service.

I forgot to mention that.  "systemctl start openvswitch.service" works just fine (and then "service network start" works).

> > Feb 05 11:36:12 ian.icp.selfip.net systemd-udevd[1670]: error changing net
> > interface name ovs0 to eth0: File exists
> 
> This looks very wrong.

Yeah, I saw that just now and assumed that it was due to a 70-persistent-net.rules rule that was matching on the MAC address, but that file doesn't exist any more.

I do not see this message in the log when I use the ovsif@ "hackery" from comment #2.

I have no idea what is causing that or if it is related to the problem starting the openvswitch service.  I do see that the message occurs 1 second after systemd has killed the network service, so while it may be related to the problem, I doubt that it's the root cause.
Comment 24 Thomas Graf 2013-02-05 14:17:44 EST
(In reply to comment #23)
> Yeah, I saw that just now and assumed that it was due to a
> 70-persistent-net.rules rule that was matching on the MAC address, but that
> file doesn't exist any more.
> 
> I do not see this message in the log when I use the ovsif@ "hackery" from
> comment #2.
> 
> I have no idea what is causing that or if it is related to the problem
> starting the openvswitch service.  I do see that the message occurs 1 second
> after systemd has killed the network service, so while it may be related to
> the problem, I doubt that it's the root cause.

Could you make your ifcfg files available somewhere?
Comment 25 Ian Pilcher 2013-02-05 14:20:44 EST
DEVICE=eth0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSPort
IPV6INIT=no
HWADDR=00:22:4d:4f:c7:ed
OVS_BRIDGE=ovs0

DEVICE=ovs0
ONBOOT=yes
TYPE=OVSBridge
DEVICETYPE=ovs
IPV6INIT=no
DELAY=0
IPADDR=172.31.250.1
GATEWAY=172.31.250.254
NETMASK=255.255.255.0

DEVICE=nfs0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
IPADDR=192.168.2.254
NETMASK=255.255.255.0
MTU=9000
Comment 26 Thomas Graf 2013-02-06 05:52:39 EST
(In reply to comment #25)

I copied the ifcfg files you provided onto my F18 and adjusted the MAC address. The network service starts up without any delays.

Could you provide the output of systemctl status network.service while it is hanging?
Comment 27 Ian Pilcher 2013-02-06 08:16:41 EST
(In reply to comment #26)
> Could you provide the output of systemctl status network.service while it is
> hanging?

network.service - LSB: Bring up/down networking
          Loaded: loaded (/etc/rc.d/init.d/network)
          Active: activating (start) since Wed 2013-02-06 06:49:24 CST; 17s ago
         Control: 1382 (network)
          CGroup: name=systemd:/system/network.service
                  ├─1382 /bin/bash /etc/rc.d/init.d/network start
                  ├─1469 /bin/bash /etc/sysconfig/network-scripts/ifup-ovs if...
                  └─1486 /bin/systemctl start openvswitch.service

Feb 06 06:49:24 ian.icp.selfip.net systemd[1]: Starting LSB: Bring up/down networking...
Feb 06 06:49:24 ian.icp.selfip.net network[1382]: Bringing up loopback interface:  [  OK  ]
Feb 06 06:49:24 ian.icp.selfip.net network[1382]: Bringing up interface eth0:  Redirecting to /bin/systemctl start  openvswitch.service

openvswitch.service - Open vSwitch
          Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled)
          Active: inactive (dead)

This looks to me like some sort of deadlock within systemd, where the in-process "service network start" is blocking the "systemctl start openvswitch.service".
Comment 28 Ian Pilcher 2013-02-06 15:39:32 EST
Created attachment 694138 [details]
strace of "hung" systemctl command (systemctl start openvswitch.service)

I edited /etc/sysconfig/network-scripts/ifup-ovs and replaced "/sbin/service openvswitch start" with "/bin/strace -t -o /var/tmp/systemctl-strace.txt /bin/systemctl start openvswitch.service".  The attached fileis the result.

I looks like systemctl is waiting for a response from systemd.
Comment 29 Ian Pilcher 2013-02-06 16:05:50 EST
So the Google fairy finally visited, and I managed to stumble on this:

  http://www.mail-archive.com/systemd-devel@lists.freedesktop.org/msg08044.html

Sure enough, I changed ifup-ovs to start Open vSwitch with:

  /bin/systemctl --ignore-dependencies start openvswitch.service

And everything seems to be working as intended.

I'm going to try removing network.target from the Before line of openvswitch.service.  I'm guessing that change will also get rid of the deadlock.
Comment 30 Ian Pilcher 2013-02-06 17:36:25 EST
(In reply to comment #29)
> I'm going to try removing network.target from the Before line of
> openvswitch.service.  I'm guessing that change will also get rid of the
> deadlock.

Booyah!

So changing openvswitch.service this way isn't a viable solution, but this does seem to pretty much pin down the problem.  The one thing that I'm still puzzled by is how this ever worked for anyone.

Thomas/Cong - Did you by chance do anything to your systems that got rid of the dependency loop?  (Or is there something else about your systems, that would make systemd think that the network.target had been reached when the network service was running?)

I've asked whether this deadlock is the expected behavior on the systemd-devel list.  Unless someone there tells me that my analysis is incorrect, I will reopen this bug.
Comment 31 Thomas Graf 2013-02-06 18:08:25 EST
(In reply to comment #30)
> Thomas/Cong - Did you by chance do anything to your systems that got rid of
> the dependency loop?  (Or is there something else about your systems, that
> would make systemd think that the network.target had been reached when the
> network service was running?)
> 
> I've asked whether this deadlock is the expected behavior on the
> systemd-devel list.  Unless someone there tells me that my analysis is
> incorrect, I will reopen this bug.

I think we need to reopen this one. I'm not sure why it does not trigger on my machine. I just tried it again and it works fine.

Removing the network.target dependency in the openvswitch.service seems to be the right fix here.
Comment 32 Ian Pilcher 2013-02-06 18:23:09 EST
(In reply to comment #31)
> I think we need to reopen this one. I'm not sure why it does not trigger on
> my machine. I just tried it again and it works fine.

Did you reboot with no active network interfaces first?  systemd has some weird heuristics around the network.target, and as long as it thinks that it has reached that target swystemctl won't hang when starting openvswitch.

> Removing the network.target dependency in the openvswitch.service seems to
> be the right fix here.

See comment #1.  If it were that easy, we never would have gotten here.  :-)

I suspect that the "correct" way to do this, from a systemd point of view, is to have a separate unit file (openvwitch-early.service or somesuch) that doesn't have the dependency, but that seems like a maintenance nightmare.

We could do something like this (in ifup-ovs):

  if [ ! -f /var/lock/subsys/openvswitch ]; then
      if [ -x /usr/bin/systemctl ]; then
          /usr/bin/systemctl --ignore-dependencies start openvswitch.service
      else
          /sbin/service openvswitch start
      fi
  fi

(Assuming that we want to support non-systemd distributions.)

The only other possibility I can think of is to change network.target from Before to Requires in the openvswitch service file.  I have no idea if it would actually work, but if it did it would probably be the most idiomatic way to express the actual requirement.
Comment 33 Thomas Graf 2013-02-06 18:42:06 EST
(In reply to comment #32)
> Did you reboot with no active network interfaces first?  systemd has some
> weird heuristics around the network.target, and as long as it thinks that it
> has reached that target swystemctl won't hang when starting openvswitch.

This explains it.

> > Removing the network.target dependency in the openvswitch.service seems to
> > be the right fix here.
> 
> See comment #1.  If it were that easy, we never would have gotten here.  :-)

Well, since we introduced a method to specify interfaces dependencies in the ifcfg files we no longer need the before network.target

> I suspect that the "correct" way to do this, from a systemd point of view,
> is to have a separate unit file (openvwitch-early.service or somesuch) that
> doesn't have the dependency, but that seems like a maintenance nightmare.
> 
> We could do something like this (in ifup-ovs):
> 
>   if [ ! -f /var/lock/subsys/openvswitch ]; then
>       if [ -x /usr/bin/systemctl ]; then
>           /usr/bin/systemctl --ignore-dependencies start openvswitch.service
>       else
>           /sbin/service openvswitch start
>       fi
>   fi
> 
> (Assuming that we want to support non-systemd distributions.)

.. I think the above is actually more desirable. We only backported this fix to F18 so far so at least for F18 we can get away with just switching to systemctl but we'd need a construct like the above for the upstream patch.

> The only other possibility I can think of is to change network.target from
> Before to Requires in the openvswitch service file.  I have no idea if it
> would actually work, but if it did it would probably be the most idiomatic
> way to express the actual requirement.

Let me read up on systemd a bit more to figure out what's the right way to fix this.
Comment 34 Ian Pilcher 2013-02-06 18:51:16 EST
(In reply to comment #33)
> > > Removing the network.target dependency in the openvswitch.service seems to
> > > be the right fix here.
> > 
> > See comment #1.  If it were that easy, we never would have gotten here.  :-)
> 
> Well, since we introduced a method to specify interfaces dependencies in the
> ifcfg files we no longer need the before network.target

Consider the case where Open vSwitch needs to talk to an external controller (such as Quantum) over the network, but all of its bridges and ports are used exclusively for VM traffic, so they don't get IP addresses at the host level.  It would be nice if the sysadmin could configure the management network with traditional ifcfg files and just "systemctl enable openvswitch.service".  For this to work, openvswitch.service needs to require (in some sense) the network.
Comment 35 Thomas Graf 2013-02-08 09:14:54 EST
(In reply to comment #34)
> Consider the case where Open vSwitch needs to talk to an external controller
> (such as Quantum) over the network, but all of its bridges and ports are
> used exclusively for VM traffic, so they don't get IP addresses at the host
> level.  It would be nice if the sysadmin could configure the management
> network with traditional ifcfg files and just "systemctl enable
> openvswitch.service".  For this to work, openvswitch.service needs to
> require (in some sense) the network.

This is basically why I have added OVSREQUIRES which allows to add a specific dependency on one or more interfaces if a controller is used.

However, I agree that if possible, openvswitch.service should have a dependency on network.target but it should be possible to auto start openvswitch.service as well if someone brings up an individual interface without the whole network.target as-is.

I think the --ignore-dependencies is the way to go here.
Comment 36 Ian Collett 2013-02-24 05:53:41 EST
I have just reload OVS with the RPM openvswitch-1.7.3-8.fc19.i686, when I check the info with yum (yum info openvswith) it still shows the Release as 7.fc18, but the file as the repo.   

However when I enter the command "systemctl enable ovsif@em1.service", it reports that if "failed to issue method call: No such file or directory".    Looking in /usr/lib/sysemd/system there dose NOT seem to be an "ovsif@.service file", so I have copied the file from post 2 above.  Rebooted, and still have the same error/issue.

I also have ifcfg-em1 file for the interface em1 and the bridge in this case ifcfg-of-switch.


Regards
IAN
Comment 37 Ian Collett 2013-02-25 05:39:13 EST
Update - this morning I disabled networkmanger, and in each ifcfg file set boot to yes and the config worked.   Not yet tried to delete the ovsif@service file I added.


Regards
IAN
Comment 38 Thomas Graf 2013-05-17 12:26:00 EDT
The current workaround looks like this:
if [ ! -f /var/lock/subsys/openvswitch ]; then
        if [ -x /usr/bin/systemctl ]; then
                /usr/bin/systemctl --ignore-dependencies start openvswitch.service
        else
                 /sbin/service openvswitch start
        fi
fi

This is not ideal. We would want native systemd unit files to handle the dependencies correctly and we want to get rid of the old init file.
Comment 39 Flavio Leitner 2013-06-19 21:50:45 EDT
Created attachment 763189 [details]
/etc/sysconfig/network-scripts/ifup-ovs

ifup-ovs modified to start openvswitch-nonetwork unit.
Comment 40 Flavio Leitner 2013-06-19 21:53:08 EDT
Created attachment 763190 [details]
/etc/sysconfig/openvswitch

The openvswitch parameters are now readable by systemd units.
Comment 41 Flavio Leitner 2013-06-19 21:54:46 EDT
Created attachment 763191 [details]
/lib/systemd/system/openvswitch.service

This is the frontend/admin systemd service for openvswitch.
Comment 42 Flavio Leitner 2013-06-19 21:55:56 EDT
Created attachment 763192 [details]
/lib/systemd/system/openvswitch-nonetwork.service

This is the backend/system internal systemd service for openvswitch.
Comment 43 Flavio Leitner 2013-06-19 22:07:02 EDT
I've attached 4 files replacing the old way of starting openvswitch.

Basically there is the openvswitch-nonetwork service without network dependency which does the job and the openvswitch service that runs after network service and requires the openvswitch-nonetwork.

The openvswitch.init is not used anymore and the parameters are used directly by the backend openvswitch-nonetwork service.

If the system has a ovs bridge, then ifup-ovs will start openvswitch-nonetwork which starts the daemons. This service has no network dependency and shouldn't be manipulated by the admin. It is the service that actually does the job.

However, if you manually stop or restart the openvswitch-nonetwork, the action is propagated to the frontend openvswitch service, so its status is accurate.

The openvswitch service can be enabled/disabled or started/stopped/restarted. The start/stop/restart actions are propagated to the openvswitch-nonetwork which does the job.

Could someone please test this and verify it does work?

Thanks!
Comment 44 Ian Pilcher 2013-07-03 13:15:20 EDT
(In reply to Flavio Leitner from comment #43)
> Could someone please test this and verify it does work?

I just tested this on Fedora 19.  It appears to work with my setup:

cc926b32-41d5-48ba-b170-0db64e0bdbdc
    Bridge "ovs0"
        Port "eth0"
            Interface "eth0"
        Port "ovs0"
            Interface "ovs0"
                type: internal
    Bridge "nfs0"
        Port "nfs0"
            Interface "nfs0"
                type: internal
    ovs_version: "1.10.0"

I did have to put SELinux in permissive mode to get things working.  In enforcing mode, openvswitch-nonetwork.service fails to start with the following error:

type=SYSCALL msg=audit(1372886824.343:88): arch=c000003e syscall=269 success=no exit=-13 a0=ffffffffffffff9c a1=1a23160 a2=1 a3=8 items=0 ppid=1 pid=2681 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="ovs-ctl" exe="/usr/bin/bash" subj=system_u:system_r:openvswitch_t:s0 key=(null)

type=AVC msg=audit(1372886824.343:88): avc:  denied  { execute } for  pid=2681 comm="ovs-ctl" name="plymouth" dev="dm-1" ino=1838383 scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file
Comment 45 Flavio Leitner 2013-07-03 14:13:15 EDT
Ian,

Yeah, it did work for me on F18 but I can reproduce the same SELinux issue on F19 here. I think we need to update SELinux default policies, so let me exchange some ideas with selinux folks first.

Thanks for testing it!
Comment 46 Fedora End Of Life 2013-07-04 01:49:10 EDT
This message is a reminder that Fedora 17 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 17. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '17'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 17's end of life.

Bug Reporter:  Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 17 is end of life. If you 
would still like  to see this bug fixed and are able to reproduce it 
against a later version  of Fedora, you are encouraged  change the 
'version' to a later Fedora version prior to Fedora 17's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 47 Miroslav Grepl 2013-07-04 02:18:06 EDT
Could you guys re-test it with the latest F19 builds.

http://koji.fedoraproject.org/koji/buildinfo?buildID=431242
Comment 48 Thomas Graf 2013-07-04 04:38:57 EDT
(In reply to Miroslav Grepl from comment #47)
> Could you guys re-test it with the latest F19 builds.
> 
> http://koji.fedoraproject.org/koji/buildinfo?buildID=431242

I confirm that the above build fixes the issue:

[tgraf@lsx ~]$ cat /etc/fedora-release 
Fedora release 19 (Schrödinger’s Cat)

[tgraf@lsx ~]$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

[tgraf@lsx ~]$ sudo systemctl start openvswitch
[tgraf@lsx ~]$ sudo systemctl status openvswitch
openvswitch.service - Open vSwitch Unit
   Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; disabled)
   Active: active (exited) since Thu 2013-07-04 10:37:10 CEST; 17s ago
  Process: 5844 ExecStart=/bin/true (code=exited, status=0/SUCCESS)

Jul 04 10:37:10 lsx.localdomain systemd[1]: Started Open vSwitch Unit.

[tgraf@lsx ~]$ sudo systemctl status openvswitch-nonetwork
openvswitch-nonetwork.service - Open vSwitch Internal Unit
   Loaded: loaded (/usr/lib/systemd/system/openvswitch-nonetwork.service; static)
   Active: active (exited) since Thu 2013-07-04 10:37:10 CEST; 35s ago
  Process: 5841 ExecStartPost=/usr/bin/touch /var/lock/subsys/openvswitch (code=exited, status=0/SUCCESS)
  Process: 5717 ExecStart=/usr/share/openvswitch/scripts/ovs-ctl start --system-id=random $FORCE_COREFILES $OVSDB_SERVER_PRIORITY $VSWITCHD_PRIORITY $VSWITCHD_MLOCKALL $BRCOMPAT $OVS_CTL_OPTS (code=exited, status=0/SUCCESS)
   CGroup: name=systemd:/system/openvswitch-nonetwork.service
           ├─5825 ovsdb-server: monitoring pid 5826 (healthy)
           ├─5826 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsysl...
           ├─5837 ovs-vswitchd: monitoring pid 5838 (healthy)
           ├─5838 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:em...
           └─5839 ovs-vswitchd: worker process for pid 5838

Jul 04 10:37:10 lsx.localdomain systemd[1]: Starting Open vSwitch Internal U....
Jul 04 10:37:10 lsx.localdomain ovs-ctl[5717]: Starting ovsdb-server [  OK  ]
Jul 04 10:37:10 lsx.localdomain ovs-vsctl[5827]: 00001|vsctl|INFO|Called as ...0
Jul 04 10:37:10 lsx.localdomain ovs-vsctl[5832]: 00001|vsctl|INFO|Called as ..."
Jul 04 10:37:10 lsx.localdomain ovs-ctl[5717]: Configuring Open vSwitch syst...]
Jul 04 10:37:10 lsx.localdomain ovs-ctl[5717]: Inserting openvswitch module ...]
Jul 04 10:37:10 lsx.localdomain ovs-ctl[5717]: Starting ovs-vswitchd [  OK  ]
Jul 04 10:37:10 lsx.localdomain systemd[1]: Started Open vSwitch Internal Unit
Comment 49 Fedora Update System 2013-07-04 05:22:50 EDT
openvswitch-1.10.0-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/openvswitch-1.10.0-2.fc19
Comment 50 Flavio Leitner 2013-07-04 09:02:08 EDT
I can confirm Thomas results on comment#48 on my system as well.
Thanks everyone!
Comment 51 Ian Pilcher 2013-07-04 12:29:08 EDT
Can anyone explain *why* openvswitch needs to run plymouth?
Comment 52 Ian Pilcher 2013-07-04 14:00:29 EDT
Piling on that the combination of openvswitch-1.10.0-2.fc19.x86_64 and selinux-policy-3.12.1-59.fc19.noarch works here.
Comment 53 Fedora Update System 2013-07-04 22:09:32 EDT
Package openvswitch-1.10.0-2.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openvswitch-1.10.0-2.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-12358/openvswitch-1.10.0-2.fc19
then log in and leave karma (feedback).
Comment 54 Fedora Update System 2013-07-23 23:39:14 EDT
openvswitch-1.10.0-2.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.