Bug 607921

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

NM should not remove the current IP address of eth0 *if*:

1) there's a valid ifcfg file for the configuration with 'BOOTPROTO=ibft'
2) 'iscsiadm -m fw' returns the configuration that the device has currently applied

If there's any possible way we can get the 'syslog' output from this machine, and also any /etc/sysconfig/network-scripts/ifcfg-* files, as well as 'iscsiadm -m fw' output, that would be very useful in diagnosing the issue, and then we can also update the testcases.

Hi Dan,

(In reply to comment #2)
> NM should not remove the current IP address of eth0 *if*:
> 
> 1) there's a valid ifcfg file for the configuration with 'BOOTPROTO=ibft'

I'm afraid that that is a bit limited iscsi booting can happen without ibft too, for example using pxeboot.

NetworkManager should not down an interface (or remove its ip address) when initially configuring it (*), this is what the old network scripts do. Talking about the old networkscripts they also automatically pick up any
/dev/.initramfs/net.${DEVICE}.lease file, it would be nice if NetworkManager could do that too.

*) Or at least it should not do that for specially marked interfaces, see below.

Actually currently anaconda never writes ifcfg files with BOOTPROTO=ibft, not
even when it got the information from ibft, as the old network scripts and dracut don't know how to get the necessary information from ibft. There are plans to change this in the future.

What anaconda does is it simply writes a normal ifcfg file with the information it got from ibft, so it writes either BOOTPROTO=dhcp or BOOTPROTO=none plus
static IP configuration settings. I've not tested what happens when using a static IP yet.

> 2) 'iscsiadm -m fw' returns the configuration that the device has currently
> applied
> 

See above, I'm afraid that the iscsiadm -m fw output is not relevant at this time as ifcfg files with BOOTPROTO=ibft in them are never written. The support for BOOTPROTO=ibft in NetworkManager is good to have though as we are planning on switching to that in the future.

I'm not sure who has been talking with you before about what is needed to allow network booting to work with NetworkManager, but I'm afraid he did not involve me which is rather unfortunate as I'm the iscsi installation and boot specialist in the anaconda team.

Note that is not really urgent, currently anaconda has code which puts
NM_CONTROLLED=no into the ifcfg file for NIC's over which iscsi booting is done which works ok. We could change this code to write NETBOOT=yes to the relevant ifcfg files to hint to NM not to down these / remove ip addresses from them.

I think it would be good to discuss this more interactively, you can find me on irc in #anaconda@freenode my nick is hansg.

(In reply to comment #3)
> Hi Dan,
> 
> (In reply to comment #2)
> > NM should not remove the current IP address of eth0 *if*:
> > 
> > 1) there's a valid ifcfg file for the configuration with 'BOOTPROTO=ibft'
> 
> I'm afraid that that is a bit limited iscsi booting can happen without ibft
> too, for example using pxeboot.
> 
> NetworkManager should not down an interface (or remove its ip address) when
> initially configuring it (*), this is what the old network scripts do. Talking
> about the old networkscripts they also automatically pick up any
> /dev/.initramfs/net.${DEVICE}.lease file, it would be nice if NetworkManager
> could do that too.
> 
> *) Or at least it should not do that for specially marked interfaces, see
> below.

So the issue is that NM needs to match up the configuration of the device with some system configuration that's specified somewhere (ie, an ifcfg file).  We can't really have devices with configuration that's not backed by an ifcfg file since then there are a number of guarantees that are broken (like applications not able to get information about the configuration over dbus, not able to get connection UUIDs or the human-readable connection name, we're not able to assign security domains based on that UUID and other stuff we want to do in the future).

I've got various plans to handle this issue more elegantly in the future, but at the moment if the device does not have backing configuration and is set to be managed by NM, then NM will understandably take over management of the interface.

Note that this also works for plain ethernet; as long as the device has configuration backing it, NM will take that connection over when it starts, and will not down the device when it quits.

> Actually currently anaconda never writes ifcfg files with BOOTPROTO=ibft, not
> even when it got the information from ibft, as the old network scripts and
> dracut don't know how to get the necessary information from ibft. There are
> plans to change this in the future.

Ok, I was basing most of the code that I wrote last August for this on configuration guides and existing configs from Dell and IBM, I think.

> What anaconda does is it simply writes a normal ifcfg file with the information
> it got from ibft, so it writes either BOOTPROTO=dhcp or BOOTPROTO=none plus
> static IP configuration settings. I've not tested what happens when using a
> static IP yet.
>
> > 2) 'iscsiadm -m fw' returns the configuration that the device has currently
> > applied
> > 
> 
> See above, I'm afraid that the iscsiadm -m fw output is not relevant at this
> time as ifcfg files with BOOTPROTO=ibft in them are never written. The support
> for BOOTPROTO=ibft in NetworkManager is good to have though as we are planning
> on switching to that in the future.

Ok, let me know when this happens.

> I'm not sure who has been talking with you before about what is needed to allow
> network booting to work with NetworkManager, but I'm afraid he did not involve
> me which is rather unfortunate as I'm the iscsi installation and boot
> specialist in the anaconda team.

Yeah, we should talk more :)  Like I said, I did most of the work last summer based on iSCSI documentation from IBM.  I supposed I'd assumed that if people were using it, Anaconda was writing it.

> Note that is not really urgent, currently anaconda has code which puts
> NM_CONTROLLED=no into the ifcfg file for NIC's over which iscsi booting is done
> which works ok. We could change this code to write NETBOOT=yes to the relevant
> ifcfg files to hint to NM not to down these / remove ip addresses from them.

Again, it might actually be easier if anaconda just wrote out BOOTPROTO=ibft :)  But whatever, sounds like we have a way to make this work for now and we can concentrate on fixing up the other stuff later.

I was talking with Warren about various dracut things (not specifically iscsi) before he left, and the idea was that *something* (dracut, whatever) would copy out the configuration and DHCP lease files that were generated by the DHCP client in the initrd into the proper locations after switchroot.

I'd rather not add yet another location for DHCP lease information to NM that only exists in the pre-boot environment, because I assume that lease needs to exist after switchroot too, and it should probably go into the standard location.  So after switchroot, is it possible to copy any configuration that existed before into the correct system locations?

To clarify, until anaconda writes out BOOTPROTO=ibft, the correct way to handle this is in anaconda is to:

1) write out the ifcfg with all the config currently applied to the interface and stuff the output of `uuidgen` into a UUID=<uuid> variable, and copy the leasefile to /var/lib/dhclient/dhclient-<uuid>-<iface>.lease; at which point NM should just take the interface over without downing it

or

2) write out NM_CONTROLLED=no

Hi,

(In reply to comment #5)
> To clarify, until anaconda writes out BOOTPROTO=ibft, the correct way to handle
> this is in anaconda is to:
> 

Thanks for the answers.

> 1) write out the ifcfg with all the config currently applied to the interface
> and stuff the output of `uuidgen` into a UUID=<uuid> variable, and copy the
> leasefile to /var/lib/dhclient/dhclient-<uuid>-<iface>.lease; at which point NM
> should just take the interface over without downing it
> 

To be clear my testcase as described in the original bug report already had an ifcfg in place with a more or less complete configuration for the device in question, and as described in the original bug report, there was a valid
/var/lib/dhclient/dhclient-<iface>.lease
file in place, but I see that NM expects:
/var/lib/dhclient/dhclient-<uuid>-<iface>.lease

I'm not sure if my test case even had an UUID= in the ifcfg file, and I'm sure that the dhclient.lease file did not have the uuid of the iface in its name.

I'll re-try my test case with the above 2 fixed.

Now as for the copying of the leases file from the initrd to /var/lib/dhclient. This cannot be done in dracut / in the initrd as at that time the root filesystem is still mounted read-only so this copying needs to be happening in some initscript. In the case of the old network scripts the copying gets done from: /etc/sysconfig/network-scripts/ifup-eth. This is not helpful in the NM case as this script then does not run (and it does not put the uuid in the lease filename).

So in the NM case some initscript should copy these files over. It seems to me that /etc/rc.d/init.d/NetworkManager is the logical place to do this.

I've tried again using the method described in comment#0, but this time first copying the leases file to /var/lib/dhclient/dhclient-<uuid>-<iface>.lease before starting NetworkManager, and the results were the same (eth0 no longer having an ip address after starting NM.

After fixing the ifcfg file to have NM_CONTROLLED=no in rescue mode and starting the system again I noticed that /var/lib/dhclient/dhclient-<uuid>-<iface>.lease has been truncated to 0.

Re-producing a second time, this time first doing an ls -l in /var/lib/dhclient/, so that I could see what would happen there after starting NetworkManager (everything I want to run to test needs to be in the cache) I noticed that this time around NM decided to create a /var/lib/dhclient/dhclient-<uuid>-<iface>.lease file with a new UUID, even though the ifcfg file contains an UUID= line.

I'll attach the dhclient lease file and the ifcfg file from one of the test runs.

Created attachment 428309 [details]
ifcfg used in testing

Created attachment 428313 [details]
dhclient lease file

Any chance I could get /var/log/messages from when NM downs the interface?

(In reply to comment #10)
> Any chance I could get /var/log/messages from when NM downs the interface?    

I'm afraid that is going to be rather tricky, the only thing I can think of is to set rsyslog up for remote logging and then hope that, that keeps running long enough to get some interesting messages to the other side.

The problem is that as soon as NM "downs" the interface in question the system is pretty much dead as any access to the root filesystem will hang.

Note: I'm going with PTO for the coming 2 weeks. If you want me to try the rsyslog remote logging that will have to wait till I get back from my PTO.

Note that for 6.0 we are writing NM_CONTROLLED=no, so there is no hurry in fixing this.

*** Bug 608723 has been marked as a duplicate of this bug. ***

EMC has experienced this in duped bug 608723. Will add them here.

I would think that anything that caused the RHEL 6.0 server to panic whenever an iscsi array and disk was attached would be of high priority. In this state I cannot test or use any iscsi.

Hi all,

Reading bug 608723 I'm not sure it is a duplicate of this bug at all.

Don, I'll put a comment in bug 608723 with a request for more information, can you please answer that there, then once we've figured out what is going on, we can see if we need to re-open 608723.

Regards,

Hans

EMC: This has been deferred to RHEL 6.1 given there is a workaround in 6.0.

Hi all,

I'm moving from the anaconda team to the spice team, so I will no longer be involved in handling installation / iscsi issues. The anaconda / boot iscsi handling has been taken over by Ales Kozumplik whom I've added to the CC.

Ales,

This bug needs some input from our side, see comment #10.

Regards,

Hans

Created attachment 457511 [details]
logs and NM settings

Hi,

On an updated rhel6 I can only partially reproduce this: when changing NM_CONTROLLED to 'yes' and then starting NM, the device really gets shutdown and then restarted but that doesn't seem to affect the iscsi session (system runs on, ip addr reports the correct IP address). Maybe that's because NM doesn't try to read something from the disk which is not in cache (i.e. I  was lucky).

Nevertheless, attaching forwarded rsyslog messages that show what is going on as well as ifcfg-eth0 and a zip file with the leases after NM went up.

Ales

Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

When NM starts up and finds interfaces, it'll try to match the existing configuration of those devices up with an existing ifcfg file, and if there's a match then it will "take over" the interface and not change anything.  This happens for both iSCSI (TYPE=ibft) and normal ethernet connections.  If DHCP is used, there  must be an existing valid leasefile in /var/lib/dhclient/ with a name following the format dhclient-<uuid>-<iface>.lease.  If the leasefile is created during initrd time it should be copied from the in-memory location that the initrd creates it to /var/lib/dhclient/ so that NetworkManager can match it up with the correct ifcfg file and that file's UUID.  The UUID must be part of the name of the leasefile because otherwise there's no way to  match it up with a specific ifcfg file's connection information.

I realize that might make it sound complicated, but it's really just:

1) for every ifcfg file that might work for the interface (ie, no HWADDR or HWADDR specifies the interface's MAC address)
   a) if BOOTPROTO=dhcp then look for a dhclient leasefile in the format dhclient-<UUID>-<iface>.lease
        1) if any of the valid leases from the leasefile match an IP address that exists on the interface, we've got a match
   b) if BOOTPROTO=static or missing, but IPADDR matches an IP address on the interface, we've got a match

if any if the ifcfg files matches, then NM will "take over" the interface instead of changing any configuration.

Based on Dan's comment from December it seems to me like the simplest thing to do is modify the SysV network config to change the lease file name to add in the $UUID value.  If that were done, if I'm reading Dan's email correctly, then NM would silently adopt the already-existing interface because the lease file would already exist.

Is this reasonable?  Is something going to happen with this bug?  I filed bug #817556 which has a different result (my normal network, not iSCSI, is messed up because the two different invocations of dhclient get back different IP addresses) but the same root cause (I think).

(In reply to comment #26)
> Based on Dan's comment from December it seems to me like the simplest thing
> to do is modify the SysV network config to change the lease file name to add
> in the $UUID value.  If that were done, if I'm reading Dan's email
> correctly, then NM would silently adopt the already-existing interface
> because the lease file would already exist.

That should be all we need, yes.  The scripts just need to generate a UUID using 'uuidgen' and stuff that into the ifcfg file, and then use that same UUID when constructing the leasefile path to pass to dhclient with "-lf".

*** Bug 817556 has been marked as a duplicate of this bug. ***

OK.  I've modified the /etc/sysconfig/network-scripts/ifup-eth and ifdown-eth files (patch attached) to add the UUID value into the lease file name (also the extension had to be changed from ".leases" to just ".lease").

After applying this patch and rebooting, I no longer see the problem I had originally where I would get two different IP addresses assigned to my system (one assigned when sysv network started and a different one assigned when NetworkManager started).

So, that's a good thing.

However, I still do see two different dhclient invocations running, one from sysv network (/etc/sysconfig/network-scripts/ifup-eth) and one from NetworkManager.  And in fact now I have two different instances of dhclient running, both listing the same lease filename with -lf!!  That seems like a good way to get a corrupted lease file, in some situations.

So it appears that Dan's understanding (in comment #24) that NetworkManager will "adopt" the existing interface if it already has a lease file with the proper UUID is not quite right.  In any event, it will certainly still start its own dhclient even if there's already one running for that interface.

FYI, here's the "ps" for my system after making those changes:

root      2138     1  0 17:44 ?        00:00:00 /sbin/dhclient -H psmithrhel -1 -q -lf /var/lib/dhclient/dhclient-294ca0c4-a1a0-45cc-becc-0638604afe17-eth0.lease -pf /var/run/dhclient-eth0.pid eth0
root      2472  2453  0 17:44 ?        00:00:00 /sbin/dhclient -d -4 -sf /usr/libexec/nm-dhcp-client.action -pf /var/run/dhclient-eth0.pid -lf /var/lib/dhclient/dhclient-294ca0c4-a1a0-45cc-becc-0638604afe17-eth0.lease -cf /var/run/nm-dhclient-eth0.conf eth0

Created attachment 608007 [details]
ifX-eth changes to share lease file

See also bug #817660

That contains a patch that modifies NetworkManager so that if a non-UUID lease is detected it will copy it over and start with that.

Hm.  Actually, the double-dhclient thing might be an artifact of some debugging I was doing.  After I cleaned it all up and restarted again, I only have one dhclient running.

There's nothing inherently *incorrect* about changing the lease file name that initscripts uses, but it is a behavior change and could cause some surprises in RHEL.

Also, the above patch means that "ifup ; upgrade-to-new-initscritps ; ifdown" will use the wrong lease file on downing the interface.

See generate_lease_file_name() in the current upstream (Fedora) network-functions.

Hi Bill; thanks very much for your comment.  We're having a major issue with this problem; it's causing probably 15-20% of our site to have unstable DNS entries assigned to their DHCP-enabled systems.  I'm really looking for some short-term workaround I can apply, which is at least not completely incompatible with a potential official fix.

So, can you elaborate on your comment a bit: do you feel that the model of changing the initscripts (as I did) is the right way to go ultimately, and you're just raising potential issues that have to be dealt with in the RPM scripts?  Or do you think it's NOT the right way to go and we should do something different?

The alternative, from bug #817660 is to modify NetworkManager to recognize the initscripts lease file and copy it over.

A third option, I guess, is to disable the sysv "network" service completely and let NetworkManager handle bringing up the interfaces.

I need to make some kind of decision on a workaround today... any thoughts about what shape of the "official" fix might take?

(In reply to comment #34)
> Hi Bill; thanks very much for your comment.  We're having a major issue with
> this problem; it's causing probably 15-20% of our site to have unstable DNS
> entries assigned to their DHCP-enabled systems.  I'm really looking for some
> short-term workaround I can apply, which is at least not completely
> incompatible with a potential official fix.
> 
> So, can you elaborate on your comment a bit: do you feel that the model of
> changing the initscripts (as I did) is the right way to go ultimately, and
> you're just raising potential issues that have to be dealt with in the RPM
> scripts?  Or do you think it's NOT the right way to go and we should do
> something different?

Changing anything means that it's a behavior change - there could be could that expects the old initscripts lease file name that would break if we changed initscripts, or code that expects the NM lease file name if we changed NM.

What we did upstream in Fedora was change initscripts, see:

http://git.fedorahosted.org/cgit/initscripts.git/diff/?id=60b84547c03ec7d82633f3c9d6939af4747a6bff

It uses the old name as long as the lease file exists, otherwise it uses a NM-compatible name.

Does that not work for you? (It still has the caveats above in that it's a behavior change.)

Thanks for the link Bill.  I think your change will work, AS LONG AS I am sure to remove all existing lease files then reboot so that when the system starts again it will create the new lease file name.

Other than that I believe the patch I posted and the change in Fedora have equivalent functionality.

So given comment #34, if we have both initscripts fix from comment #35, and the NetworkManager change from bug 817660, we have an effective fix for this problem?

Also, should this bug be marked a duplicate of bug 817660, since that's tracking the actual patch and fix?

I think that Bill's change in comment #35 is all that's needed.  If you have that you don't really need the NetworkManager change.  If you start with a clean install that has Bill's change and has never obtained a DHCP lease, then you will have no problems.

If you have an install which has previously done obtained a DHCP lease using the old scripts, so that now you have two lease files, then you'll have to delete the one you don't want (the non-UUID one) and reboot (or at least restart networking).  Of course you can just delete all your lease files and reboot, for the same effect.

Ok, closing as a duplicate of 817660 then as we should have the fix for leasefile names (UUID vs. non-UUID) in RHEL 6.4.

*** This bug has been marked as a duplicate of bug 817660 ***