Bug 1399448

Summary: Upgrading to Rhel 7.3 breaks network.service and service network restart
Product: Red Hat Enterprise Linux 7 Reporter: jnikolak
Component: initscriptsAssignee: David Kaspar // Dee'Kej <deekej>
Status: CLOSED NOTABUG QA Contact: qe-baseos-daemons
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.3CC: aloughla, atragler, bgalvani, deekej, fgiudici, initscripts-maint-list, jnikolak, lnykryn, lrintel, rkhan, sukulkar, thaller
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 23:38:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description jnikolak 2016-11-29 05:36:33 UTC
Description of problem:

After upgrading Networking manager

  ModemManager.x86_64 0:1.6.0-2.el7                                                                             
  ModemManager-glib.x86_64 0:1.6.0-2.el7                                                                        
  NetworkManager.x86_64 1:1.4.0-12.el7                                                                          
  NetworkManager-adsl.x86_64 1:1.4.0-12.el7                                                                     
  NetworkManager-config-server.x86_64 1:1.4.0-12.el7                                                            
  NetworkManager-glib.x86_64 1:1.4.0-12.el7                                                                     
  NetworkManager-libnm.x86_64 1:1.4.0-12.el7                                                                    
  NetworkManager-libreswan.x86_64 0:1.2.4-1.el7                                                                 
  NetworkManager-libreswan-gnome.x86_64 0:1.2.4-1.el7                                                           
  NetworkManager-team.x86_64 1:1.4.0-12.el7                                                                     
  NetworkManager-tui.x86_64 1:1.4.0-12.el7                                                                      
  NetworkManager-wifi.x86_64 1:1.4.0-12.el7

service network restart and
systemctl restart network.service stops working

The service is inactive and cannot be started.
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
   Active: inactive (dead) since Tue 2016-11-29 14:20:38 AEST; 8min ago

The issue only seems to occur if the network device is connected.

nmclu dev status
--> connected

It seems the symbolic links pointing /etc/init.d/network to use systemd network restart commands are misconfigured.

In Rhel 7.2 having Networking Manager and network.service seemed to run without issue.

[root@rhel7b network-scripts]# nmcli dev status
ens3    ethernet  connected  ens3       
lo      loopback  unmanaged  --         
[root@rhel7b network-scripts]# service network restart
Restarting network (via systemctl):                        [  OK  ]

Comment 1 Thomas Haller 2016-11-29 11:26:04 UTC
the linked case from the portal is a bit hard to follow, because it talks about various things and there are some incorrect assumptions. In general,

 - enabling both (legacy) network.service and NetworkManager.service together 
   is  supported and possible.
 - note that network.service (aka initscripts) will delegate commands to 
   NetworkManager, if:
     - NetworkManager is running
     - NetworkManager claims to manage the ifcfg-file (e.g. because it doesn't
       contain "NM_CONTROLLED=no"
   So very often, `ifup eth0` ends up calling:
      nmcli connection load /etc/sysconfig/network-scripts/ifcfg-eth0 &&
      nmcli connection up $UUID
   That is, initscripts call out to nmcli.

Anyway. A failure of network.service usually means that one of the ifup commands fails. In that case, the output of the following commands would be helpful:
  journalctl -u network.service
  journalctl -u NetworkManager.service
  ls -la /etc/sysconfig/network-scripts/
  grep ^ /etc/sysconfig/network-scripts/ifcfg-*

also, for more helpful logfiles, please enable debug-logging of NetworkManager before reproducing the issue. In short: edit /etc/NetworkManager/NetworkManager.conf and add there [logging] level=TRACE, followed by `systemctl restart NetworkManager.service` (for more details, see https://cgit.freedesktop.org/NetworkManager/NetworkManager/plain/contrib/fedora/rpm/NetworkManager.conf?id=c90ec2d8c8a12b44c908bf7f80b23059c29f68fa )

Thank you.

Comment 3 jnikolak 2016-12-13 01:31:22 UTC
Hello thanks for that info and the misconception of Network Manager.

I'd like to find out why the following occurs after an upgrade.

systemctl status network (checking status of legacy network)
We can see in new versions we get bad;vendor preset;disabled

--> Rhel 7.3
 network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)  < --- bad vendor
   Active: active (exited) since Mon 2016-11-21 19:15:16 CST; 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11107 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS)
  Process: 11294 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)

Previous versions showed it as the following:
--> Rhel 7.2
● network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network)
   Active: active (exited) since Tue 2016-12-13 09:22:49 AEST; 1h 55min ago
     Docs: man:systemd-sysv-generator(8)

Why does the new version show this status.

Comment 4 Thomas Haller 2016-12-13 14:12:30 UTC
(In reply to jnikolak from comment #3)

> I'd like to find out why the following occurs after an upgrade.

I don't know that.

But as the exit-status seems to be 0, it is likely unrelated to NetworkManager.

Reassigning to initscripts.

Comment 5 Lukáš Nykrýn 2016-12-13 15:14:32 UTC
--> Rhel 7.3
>  network.service - LSB: Bring up/down networking
>    Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled) 
> < --- bad vendor

This part is fine, network does not have a unit files ("bad"), and it is statically enabled (so not enabled in presets), we should fix both of those things, but it is just a cosmetical issue.

So is there any other bug here?

Comment 6 jnikolak 2016-12-14 00:56:01 UTC

Thanks, for letting me know that the status "bad" is nothing to be concerned about. 

For comparison, I can see what you mean by the unit causing the error.

in 7.2 ---->
systemctl list-unit-files | grep network
dbus-org.freedesktop.network1.service       invalid 

in 7.3 ---->
systemctl list-unit-files  | grep network
dbus-org.freedesktop.network1.service         bad  

       |"bad"             | Unit file is invalid or | > 0       |
       |                  | another error occurred. |           |
       |                  | Note that is-enabled    |           |
       |                  | will not actually       |           |
       |                  | return this state, but  |           |
       |                  | print an error message  |           |
       |                  | instead. However the    |           |
       |                  | unit file listing       |           |
       |                  | printed by              |           |
       |                  | list-unit-files might   |           |
       |                  | show it.                |           |

Both versions have the symlink pointing to the networkd.service

In 7.2 -->
ls -la /usr/lib/systemd/system/dbus-org.freedesktop.network1.service
1 root root 24 Jul 10 11:43 /usr/lib/systemd/system/dbus-org.freedesktop.network1.service -> systemd-networkd.service
[root@rhel7b network-scripts]# 

In 7.3 --> 
ls -la /usr/lib/systemd/system/dbus-org.freedesktop.network1.service 
lrwxrwxrwx. 1 root root 24 Nov 21 21:27 /usr/lib/systemd/system/dbus-org.freedesktop.network1.service -> systemd-networkd.service

systemctl --all | grep networkd
In 7.2 --> empty

The different being is that Rhel 7.2 doesnt have this service.

But in 7.3 --> We can see the service is the issue.

● systemd-networkd.service                                                                                       not-found inactive dead      systemd-networkd.service
  systemd-networkd.socket                                                                                        loaded    inactive dead      networkd rtnetlink socket

git clone https://github.com/systemd/systemd.git
I had a look at the systemd code and the vendor preset is called on an 
--> else if condition to determine if an array is empty.

3896         if (i->load_error != 0)
3897                 printf("   Loaded: %s%s%s (Reason: %s)\n",
3898                        on, strna(i->load_state), off, i->load_error);
3899         else if (path && !isempty(i->unit_file_state) && !isempty(i->unit_file_preset))
3900                 printf("   Loaded: %s%s%s (%s; %s; vendor preset: %s)\n",

It cannot find the service, but I cannot track down where the network is looking for the file.
cat ./system-preset/90-systemd.preset
enable systemd-networkd.service

Comment 7 Lukáš Nykrýn 2016-12-14 09:38:01 UTC
dbus-org.freedesktop.network1.service this is not related to the classic network initscript in any way. It is a dbus interface for systemd-networkd and again a cosmetic error. We need to move /usr/lib/systemd/system/dbus-org.freedesktop.network1.service out of the systemd package and we already have a bug for that https://bugzilla.redhat.com/show_bug.cgi?id=1404444

Comment 8 jnikolak 2016-12-14 23:38:43 UTC
Hello Lukas, thanks for that info and clearing up the issue, 

closing bug...