Bug 916233

Summary: latest upgrade breaks netboot
Product: [Fedora] Fedora Reporter: Marcus Moeller <marcus.moeller>
Component: NetworkManagerAssignee: Dan Williams <dcbw>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: dcbw, jklimes, notting, rhbgs.10.bigi_gigi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-10 01:26:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
NetworkManager log none

Description Marcus Moeller 2013-02-27 15:29:20 UTC
We are using fedora-readonly.service in a netboot environment where we boot via NFS.

With the latest upgrade of NetworkManager to 1: the setup does not start correctly anymore.

With NetworkManager.x86_64 1: everything worked fine.

Comment 1 Marcus Moeller 2013-02-28 10:06:26 UTC
I am trying to find out if it's one of the fedora specific problems or the general update to 0.9.8 which causes the problem. Sadly I did not find a way to debug, yet as the system simply freezes during boot, with the latest version of NetworkManager.

Comment 2 Jirka Klimes 2013-03-14 17:14:43 UTC
Yeah, that's not easy to debug things when the system doesn't boot.
But, you may look at http://freedesktop.org/wiki/Software/systemd/Debugging that could help finding out what's going on, capture logs, etc.

Comment 3 Dan Williams 2013-03-14 19:50:56 UTC
It may be that the Dracut scripts which move the DHCP lease to the real system after switchroot don't move them to the new location that NM 0.9.8 looks for them in.  Also, NetworkManager doesn't look in the old location for leasefiles either.

Posted a patch to the NM mailing list that may solve this, and also did this scratch build.  Are you able to install this scratch build and test whether it fixes the issue?


Comment 4 Bengt Giger 2013-03-15 11:08:09 UTC
I'm handling the case for Marcus. I installed NetworkManager, but the freeze remains. I managed to boot single user mode and start NetworkManager manually. I will try to get some debugging output to provide more information.

Comment 5 Bengt Giger 2013-03-15 11:12:05 UTC
Created attachment 710595 [details]
NetworkManager log

NetworkManager --no-daemon --log-level=DEBUG

After starting NetworkManager, system freezes

Comment 6 Dan Williams 2013-03-15 19:34:17 UTC
Yes, NM is taking the interface down, which it shouldn't be doing.  I'll post a new scratch build with more debugging if you would be so kind as to repeat the testing.  Thanks!

Comment 7 Dan Williams 2013-03-15 19:36:52 UTC
Actually, here's one issue:

NetworkManager[738]: <warn> Failed to copy leasefile /var/lib/dhclient/dhclient-6af563d3-2c30-4e67-ab90-3a353f852b09-eth0.lease to /var/lib/NetworkManager/dhclient-6af563d3-2c30-4e67-ab90-3a353f852b09-eth0.lease: (21) Error opening file '/var/lib/NetworkManager/dhclient-6af563d3-2c30-4e67-ab90-3a353f852b09-eth0.lease': Read-only file system

is /var/lib/ really read-only at this point?  That seems odd; /var/lib/ is usually supposed to be read/write like /var/run, I think.  In any case, I'll see if I can modify the patch to just use the old location instead of copying it.

Comment 8 Bengt Giger 2013-03-18 10:44:46 UTC
In our current stable netboot environment, /var/lib is r/o. Some of the subdirectories are replaced with tmpfs and made r/w, when booting properly. Inclunding /var/lib/dhclient, but not yet /var/lib/NetworkManager. 

I could not yet figure out from single user mode if during normal boot, these tmpfs mounts are completed when NetworkManager wants to do the copy job. And I am still searching the code that triggers the mounts, if I find it I could test with an additional tmpfs mount for /var/lib/NetworkManager.

Comment 9 Bengt Giger 2013-03-18 14:45:27 UTC
Added /var/lib/NetworkManager to rwtab, system boots nicely, no locking anymore.

Comment 10 Dan Williams 2013-03-20 16:15:00 UTC
Proposed patch for review upstream in associated GNOME bug.  That patch should fall back correctly if it cannot copy the leasefile.

Comment 11 Marcus Moeller 2013-03-21 07:00:31 UTC
@Bill: could we please add:

files	/var/lib/NetworkManager

to /etc/rwtab?

Comment 12 Bill Nottingham 2013-03-21 18:06:36 UTC
Added on git master.

Comment 13 Fedora Update System 2013-03-28 01:23:55 UTC
network-manager-applet-,NetworkManager- has been submitted as an update for Fedora 18.

Comment 14 Fedora Update System 2013-03-29 01:33:05 UTC
Package network-manager-applet-, NetworkManager-
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing network-manager-applet- NetworkManager-'
as soon as you are able to, then reboot.
Please go to the following url:
then log in and leave karma (feedback).

Comment 15 Fedora Update System 2013-04-10 01:26:48 UTC
network-manager-applet-, NetworkManager- has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.