Bug 896135

Summary: network fails to start after Fedora 17 -> Fedora 18 upgrade because of network device rename failure
Product: [Fedora] Fedora Reporter: Jonathan Kamens <jik>
Component: systemdAssignee: Michal Schmidt <mschmidt>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: franta, johannbg, lars, lnykryn, metherid, michal, mschmidt, msekleta, notting, plautrba, systemd-maint, trever, vpavlin
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-25 21:43:08 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Jonathan Kamens 2013-01-16 12:15:31 EST
I had these rules in /etc/udev/rules.d/70-persistent-net.rules:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="90:e6:ba:ba:47:2e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0a:cd:13:69:55", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="e0:91:f5:20:34:41", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

I did a yum upgrade from Fedora 17 to 18. After the upgrade, neither eth1 nor eth2 started properly. These errors showed up in /var/log/messages:

Jan 16 11:22:35 jik2 network[907]: Bringing up interface eth1:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device eth1 has different MAC address than expected, ignoring.
Jan 16 11:22:35 jik2 /etc/sysconfig/network-scripts/ifup-eth: Device eth1 has different MAC address than expected, ignoring.
Jan 16 11:22:38 jik2 network[907]: Bringing up interface eth2:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device eth2 has different MAC address than expected, ignoring.
Jan 16 11:22:38 jik2 /etc/sysconfig/network-scripts/ifup-eth: Device eth2 has different MAC address than expected, ignoring.

And indeed, when I checked the MAC addresses of the two interfaces, they were reversed, i.e., udev failed to properly rename the interfaces as specified in the rules file.

When I did "udevadm test" on the relevant devices, udev complained that it couldn't rename eth1 to eth2 because eth2 already existed. Surely that was also the case in F17; I'm not sure why it's a problem now but wasn't a problem then.

I worked around this by editing the rules file to rename one of the interfaces to "ethlan" and the other to "ethwan", to names which were otherwise unused, and then I had to go find all the config files all over the place referring to eth1 and eth2 and change them to refer to ethwan and ethlan. Ugh.

I don't know how many other people are going to run into this when upgrading to F18, but I can't imagine that I'll be the only one.
Comment 1 Michal Schmidt 2013-01-16 13:27:46 EST
(In reply to comment #0)
> When I did "udevadm test" on the relevant devices, udev complained that it
> couldn't rename eth1 to eth2 because eth2 already existed. Surely that was
> also the case in F17; I'm not sure why it's a problem now but wasn't a
> problem then.

This comment by Kay should explain it:
https://bugs.freedesktop.org/show_bug.cgi?id=56929#c3
Comment 2 Jonathan Kamens 2013-01-16 14:10:25 EST
That does explain the breakage in F18, but I'm not sure what to make of the fact that this means there's a good chance that the network is going to break for people who upgrade from F17 to F18 who aren't using NetworkManager for everything.
Comment 3 Michal Schmidt 2013-01-22 04:04:25 EST
*** Bug 902604 has been marked as a duplicate of this bug. ***
Comment 4 Michal Schmidt 2013-01-22 04:33:46 EST
Perhaps this udev change was not announced loudly enough. The configurations with swapping names appear common.
I'm inclined to re-add the renaming hack to udev in F18, emitting a deprecation warning message when it triggers. And for F19 we could add a release note.
Comment 5 Jóhann B. Guðmundsson 2013-01-22 05:00:31 EST
(In reply to comment #4)
> Perhaps this udev change was not announced loudly enough. The configurations
> with swapping names appear common.
> I'm inclined to re-add the renaming hack to udev in F18, emitting a
> deprecation warning message when it triggers. And for F19 we could add a
> release note.

I do believe these changes are to fix workarounds like Johathan has implement and I think systemd 197 release announcement hit major newssite which in turns mention this pages [1] along with this workaround "ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules" if you want to get the old behaviour back. 

http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames

I personally would prefer that we do not re-add the naming hack due to the fact that they are oddball cases ( majority of users wont notice this change ) and point those that need it ( for some odd reason ) the old behavior back to "ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules"
Comment 6 Michal Schmidt 2013-01-22 05:44:34 EST
(In reply to comment #5)
The PredictableNetworkInterfaceNames feature is not related to this BZ.
80-net-name-slot.rules is not even present in F18.
Comment 7 Jóhann B. Guðmundsson 2013-01-22 06:22:36 EST
(In reply to comment #6)
> (In reply to comment #5)
> The PredictableNetworkInterfaceNames feature is not related to this BZ.
> 80-net-name-slot.rules is not even present in F18.

hmm 
Dont we ship systemd 197 in F18 which introduced this feature? [1]
( My workstation is on F17 and my test laptop is on rawhide ) 

http://koji.fedoraproject.org/koji/buildinfo?buildID=377626
Comment 8 Michal Schmidt 2013-01-22 06:24:46 EST
We ship v197, but with some incompatible changes backed out. See this update description:
https://admin.fedoraproject.org/updates/FEDORA-2013-0655/systemd-197-1.fc18.1
Comment 9 Michal Schmidt 2013-01-28 07:52:05 EST
*** Bug 904678 has been marked as a duplicate of this bug. ***
Comment 10 Michal Jaegermann 2013-01-28 12:47:39 EST
(In reply to comment #5)

> ... that they are oddball cases

These "oddball cases" include every installation which required a custom iptables configuration with filter rules which depend on interface names.  There is where this breakage _really_ hits.

> ( majority of users wont notice this change )

Your basic tablet in a default configuration indeed should be OK.  So what?

At https://bugs.freedesktop.org/show_bug.cgi?id=56929#c3 Kay writes:
"Biosdevname *should* work fine".  Great, but if you are upgrading then what they are?  Nothing in dmesg reveals their presence. Oh, and there is some utility, not installed by a defalt, which may or may not help here.

Apparently all preparatory renaming work should be done _before_ one starts on an upgrade in order to minimize the damage but upgrade notes are silent on this issue.

BTW - I have on hands one machine with an "on the board" network interface which looks just fine to kernels only is broken in that sense that it is not sending or receiving any packets and no amount of mucking with NM will help here.  So what it gets named and how is quite important here.
Comment 11 Michal Schmidt 2013-01-28 12:51:57 EST
Michal, there was no need to react to comment #5. As you can see from the following comments, Jóhann misunderstood what this bug was about.
Comment 12 Jóhann B. Guðmundsson 2013-01-28 13:07:31 EST
(In reply to comment #11)
> Michal, there was no need to react to comment #5. As you can see from the
> following comments, Jóhann misunderstood what this bug was about.

Yes I misunderstood what this bug was about I linked it to the upcoming changes in F19
Comment 13 Fedora Update System 2013-01-28 14:12:23 EST
systemd-197-1.fc18.2 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/systemd-197-1.fc18.2
Comment 14 Fedora Update System 2013-01-29 19:33:29 EST
Package systemd-197-1.fc18.2:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-197-1.fc18.2'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-1590/systemd-197-1.fc18.2
then log in and leave karma (feedback).
Comment 15 Fedora Update System 2013-02-22 20:01:17 EST
Package initscripts-9.42.2-1.fc18, systemd-197-1.fc18.2:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing initscripts-9.42.2-1.fc18 systemd-197-1.fc18.2'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-1590/initscripts-9.42.2-1.fc18,systemd-197-1.fc18.2
then log in and leave karma (feedback).
Comment 16 Michal Schmidt 2013-02-25 20:02:10 EST
*** Bug 915528 has been marked as a duplicate of this bug. ***
Comment 17 Fedora Update System 2013-02-25 21:43:10 EST
initscripts-9.42.2-1.fc18, systemd-197-1.fc18.2 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.