Bug 692230

Created attachment 488861 [details]
/var/log/messages

Created attachment 488862 [details]
/etc/fstab

What's the output of 'chkconfig --list network'?

(In reply to comment #3)
> What's the output of 'chkconfig --list network'?

# chkconfig --list network

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overriden by native
      systemd configuration.

network         0:off   1:off   2:off   3:off   4:off   5:off   6:off

What happens if you enable network?

Created attachment 489065 [details]
/var/log/messages

(In reply to comment #5)
> What happens if you enable network?

# systemctl enable network.service 
network.service is not a native service, redirecting to /sbin/chkconfig.
Executing /sbin/chkconfig network on

# chkconfig --list network

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overriden by native
      systemd configuration.

network         0:off   1:off   2:on    3:on    4:on    5:on    6:off

However, after reboot.  /opt is still not mounted.  See attached updated /var/log/messages.

If I attempt to mount /opt manually, I get ...

# mount /opt
mount: special device UUID=1f7631bc-e441-4e1e-8cf8-f91469b2f454 does not exist

To get it mounted properly, I have to...
 1) start iscsid
 2) iscsiadm -m discovery -t st -p 10.10.9.219
 3) iscsiadm -m node -T $TARGET_IQN -p $IP --login
 4) mount /opt

This was tested with updated packages, including:

  dracut.noarch 0:009-3.fc15              dracut-network.noarch 0:009-3.fc15
  fedora-logos.noarch 0:15.0.0-2.fc15     libudev.x86_64 0:167-1.fc15
  systemd.x86_64 0:21-2.fc15              systemd-units.x86_64 0:21-2.fc15
  udev.x86_64 0:167-1.fc15

Are the iscsi/iscsid services properly running? The fact that you had to redo target discovery implies something else weird is going on.

(In reply to comment #7)
> Are the iscsi/iscsid services properly running? The fact that you had to redo
> target discovery implies something else weird is going on.

After a fresh install ... 

# service network start
Starting network (via systemctl):  Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:
Determining IP information for eth0.../etc/init.d/functions: line 58: /dev/stderr: Permission denied
/etc/rc.d/init.d/functions: line 58: /dev/stderr: Permission denied
/etc/init.d/functions: line 58: /dev/stderr: Permission denied
/etc/rc.d/init.d/functions: line 58: /dev/stderr: Permission denied
 done.
[  OK  ]
[  OK  ]

# ps -wef | grep iscsi
root       666     2  0 15:46 ?        00:00:00 [iscsi_eh]
root       704     1  0 15:46 ?        00:00:00 iscsid
root       705     1  0 15:46 ?        00:00:00 iscsid

# systemctl restart iscsi.service
Starting iscsi: [  434.884895] scsi2 : iSCSI Initiator over TCP/IP
[  435.425646] scsi 2:0:0:0: Direct-Access     IBM      DS300    S320    7.01 PQ: 0 ANSI: 4
[  435.430418] sd 2:0:0:0: Attached scsi generic sg0 type 0
[  435.433392] sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[  435.436572] sd 2:0:0:0: [sda] Write Protect is off
[  435.441813] sd 2:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  435.465492]  sda: sda1

[  435.475578] sd 2:0:0:0: [sda] Attached SCSI disk

# mount /opt
[  448.687674] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)


To attempt to summarize, networking isn't activating (or enabled).  Once I enable network (using chkconfig network on) by default ... all works well.

The network service is not enabled by default, and has not been for multiple releases. So, I'm confused - what's the issue here?  The network scripts should work, but they're not enabled by default. NM is (it's in base.)

(In reply to comment #9)
> The network service is not enabled by default, and has not been for multiple
> releases. So, I'm confused - what's the issue here?  The network scripts should
> work, but they're not enabled by default. NM is (it's in base.)

I meant networking in the abstract sense, not specific to /etc/init.d/network or NetworkManager.  Networking should be enabled after performing an iSCSI installation.  I have a suspicion that there is a change in post-install network configuration now that the initrd.img contains the install.img.

I'll retest *forcing* network enablement during install.  If that works, I'll close out this bug and explore that behavior change with anaconda.

Adding clumens for installer perspective.

(In reply to comment #10)
> I'll retest *forcing* network enablement during install.  If that works, I'll
> close out this bug and explore that behavior change with anaconda.

Okay, question on networking.  In Fedora 14, when doing an installation where install.img was a remote source, networking was enabled by default.  In Fedora 15, since there is no install.img, setting up networking by default requires an extra step (askmethod+remote or [X] connect automatically in GUI).  I'm not sure if th

Checking "[X] Connect Automatically" during F-15-Beta install, and doing a minimal package install does enable networking in network-scripts/ifcfg-eth0.  However, NetworkManager is *not* installed and the network service is disabled.

How should networking be enabled in this case?  Should NetworkManager be installed, or should 'network' be enabled?  I believe this issue is the core problem in the iSCSI use case.

Historically, anaconda does not enable or disable services based on package selection; in prior releases, if NM is not installed, network has not been enabled.

(In reply to comment #12)
> Historically, anaconda does not enable or disable services based on package
> selection; in prior releases, if NM is not installed, network has not been
> enabled.

So adding a 'chkconfig network on' whenever NetworkManager isn't installed is a bad hack, right?

What's the recommendation here ... just install NetworkManager?

(In reply to comment #13)
> (In reply to comment #12)
> > Historically, anaconda does not enable or disable services based on package
> > selection; in prior releases, if NM is not installed, network has not been
> > enabled.
> 
> So adding a 'chkconfig network on' whenever NetworkManager isn't installed is a
> bad hack, right?

A bad hack ... where? It would be a bit of an odd special case to add in anaconda.

It's certainly the 'proper' solution if you want to use the network service instead of NetworkManager; can be done in %post, or on booting the system.

(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > Historically, anaconda does not enable or disable services based on package
> > > selection; in prior releases, if NM is not installed, network has not been
> > > enabled.
> > 
> > So adding a 'chkconfig network on' whenever NetworkManager isn't installed is a
> > bad hack, right?
> 
> A bad hack ... where? It would be a bit of an odd special case to add in
> anaconda.

I'm thinking about when the user selects "[X] Connect automatically", installs @Minimal ... and networking isn't connected automatically as requested.  I think that's a valid bug.

> It's certainly the 'proper' solution if you want to use the network service
> instead of NetworkManager; can be done in %post, or on booting the system.

This would certainly "do the right thing"

diff --git a/pyanaconda/storage/devices.py b/pyanaconda/storage/devices.py
index 045b0d3..1aa2eee 100644
--- a/pyanaconda/storage/devices.py
+++ b/pyanaconda/storage/devices.py
@@ -3486,7 +3486,7 @@ class LoopDevice(StorageDevice):
 class iScsiDiskDevice(DiskDevice, NetworkStorageDevice):
     """ An iSCSI disk. """
     _type = "iscsi"
-    _packages = ["iscsi-initiator-utils", "dracut-network"]
+    _packages = ["iscsi-initiator-utils", "dracut-network", "NetworkManager"]
 
     def __init__(self, device, **kwargs):
         self.node = kwargs.pop("node")

Well, anaconda definitely has the capability to enable services and install packages based upon the storage technologies used.  We already ensure multipathd is running in that case.  It would be trivial to ensure the old school "network" service is enabled on iSCSI installs.

We could do that, but it only *needs* to be enabled for iscsi if NM isn't enabled.

(We could also see what happens if we just have both enabled all the time - with systemd, it may not make much difference.)

Yeah anaconda does not have that kind of logic built in and like you said earlier, we historically haven't done those kinds of package installation decisions.  I'd hate to start now.

The best I can hope for is that enabling both services doesn't cause any interference.  It'd also be nice if we didn't have the two partially overlapping services.  That would make this all much easier.

In summary, I don't know where to go with this bug.

network.service and NetworkManager.service should work fine if enabled at the same time, as far as I know. Chris, is that what you are wondering about?

That would certainly make this a lot easier.

Discussed at 2011/04/15 blocker review meeting. jlaska had left at that point and we're pretty unclear on the exact impact of this bug. For proper blocker review purposes, can jlaska+clumens provide a summary of the exact impact of this bug and related criteria? Thanks!



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Discussed at 2011-04-21 blocker review meeting (http://meetbot.fedoraproject.org/fedora-bugzappers/2011-04-21/f15-blocker-review.2011-04-21-17.00.html) ...

Fell off jlaska's radar, he (me) agreed to retest and provide an updated impact summary.

Created attachment 494736 [details]
dmesg (systemd.log_level=debug systemd.log_target=kmsg)

[  OK  ] - Installing F15-Beta with local *and* iSCSI drives, accepting defaults, works fine and boots without error.  In this case, the iSCSI drive is partitioned as a PV in the LVM volume group that makes up '/'.  Dracut connects to the iSCSI drive on boot and all is right in the world.  Systemd doesn't really get involved with discovering and activating any iSCSI volumes in this case.

[FAILED] - Installing F15-Beta with local *and* iSCSI drives, with the iSCSI drive mounted as /opt.  In this case, /opt is not critical for system bring-up, therefore dracut does not setup that mount.  Instead, systemd->netfs should be discovering and activating the device.  However, the iSCSI drive is *not* activated and connected during boot.

Attaching log with "systemd.log_level=debug systemd=log_target=kmsg"

A quick scan of the logs makes me think the netfs.service isn't functioning properly.

(In reply to comment #23)
> A quick scan of the logs makes me think the netfs.service isn't functioning
> properly.

With nottings help, we identified that /etc/init.d/iscsi script is exiting because it thinks that networking is not active, even though it is active.

From /etc/init.d/iscsi ...

  # if the network isn't up yet exit cleanly, NetworkManager will call us
  # again when the network is up
  [ ! -f /var/lock/subsys/network -a ! -f /var/lock/subsys/NetworkManager ] && exit 0

Notting informs me that NetworkManager no longer uses /var/log/subsys to denote status.  Perhaps something different needs to be used (like service NetworkManager status)?

Removing that line properly starts and mounts (netfs) iscsi volumes on boot.  Reassigning to iscsi-initiator-utils who is responsible for that initscript.

Created attachment 494762 [details]
patch

Here's a patch to the initscript - it should be backwards-compatible.

You could, of course, change both checks to use status.

Created attachment 494763 [details]
/etc/init.d/iscsi patch

(In reply to comment #24)
> Removing that line properly starts and mounts (netfs) iscsi volumes on boot. 
> Reassigning to iscsi-initiator-utils who is responsible for that initscript.

The attached patch against /etc/init.d/iscsi resolves the issue.

Patch looks ok to me. I will merge it up and build it.

(In reply to comment #26)
> You could, of course, change both checks to use status.

Erm, don't change the check for /var/lock/subsys/network to use status. The network script's status() function is pretty stupid. The patch in comment #5 should still be fine, though.

Merged patch in comment #25 from Bill in comment iscsi-initiator-utils-6.2.0.872-10.fc15.

Build is here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=3024857

Thanks.

*** Bug 681275 has been marked as a duplicate of this bug. ***

Hi Mike et all,

Thanks for taking care of the initsccript systemd issue. I had fixing this
on my to do for a while, but have not gotten around to it. Actually I was
planning on fixing it today, but you beat me to it :)

Note we also had another bug open for the same issue, bug 681275, I've just dupped that one to this bug.

There also is another issue, when you've a system log in to iscsi targets
it itself serves (for sharing disks for virtual machine migration testing),
during boot the system will hang for approx a minute (and during shutdown
forever). This is caused by the starting of tgtd and the iscsi service
racing. I've a fix for this in the form of adding:

Should-Start:      tgtd
Should-Stop:       tgtd

To the LSB header for the iscsi service, this makes systemd waits for tgtd
to fully start before starting the iscsi service.

I also have a minor tweak to fix the status printing when doing
"service iscsi stop", and to fix the autostarting of iscsid by iscsiadm
when upgrading from an older iscsi-initiator-utils with an iscsid.conf
without a iscsid.startup entry.

I've had this setting on my disk for a while now, waiting for me to get
around to fix the bit you just fixed before pushing an update :)

I also need to take a look at bug 656605, and then iscsi-initiator-utils should be fully systemd ready. I'll do so to today, and do a new build with all these fixes in, and then also create an update in bodhi for it.

Thanks & Regards,

Hans

iscsi-initiator-utils-6.2.0.872-11.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/iscsi-initiator-utils-6.2.0.872-11.fc15

Package iscsi-initiator-utils-6.2.0.872-11.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing iscsi-initiator-utils-6.2.0.872-11.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/iscsi-initiator-utils-6.2.0.872-11.fc15
then log in and leave karma (feedback).

To summarize ... due to a bug in the iscsi initscript, non-root iSCSI volumes were not discovered and activated on bootup.  Based on our understanding of this issue, I think it still qualifies as a release blocker impacting the following criteria ...

  "The installer must be able to complete an installation using IDE, SATA, SCSI and iSCSI storage devices"

It's a moot point considering an update is proposed to resolve this issue, but I think it still warrants a decision to ensure this update is pulled into F15.

Tested and confirmed fix.  

In one of many reboot attempts, it seems that the iSCSI non-root volume was not properly mounted.  I'm investigating whether there is a timing issue with regards to booting, activating the network and discovering iSCSI devices.  The frequency and nature of that issue makes me think it's worthy of a different bug report.  I'll follow-up in a new bug if I'm able to gather more information.

One issue may be that checking for the status of NM means you're checking that NM is active - it does not necessarily mean that NM has configured a connection yet.

A test for that would be something like 'nm-online -x', instead of 'status NetworkManager'. Don't know if that's what iscsi needs, though.

Checking for a connection is best (That is what iscsi needs). If we just check for NM being active it could result in what James is seeing in comment #36.

OK, see /etc/init.d/netfs for an example here.

(In reply to comment #39)
> OK, see /etc/init.d/netfs for an example here.

I'm going to move this issue back to ASSIGNED, and we can track the revised fix here.  Shout if you'd prefer having this tracked as a new bug.

Discussed at 2011-04-29 blocker review meeting. Accepted as a blocker: affects criterion "# The installer must be able to complete an installation using IDE, SATA, SCSI and iSCSI storage devices " (regarding the functioning of the installed system) - non-root iSCSI partitions will not mount correctly at boot time.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

I've prepared a new iscsi-initiators-utils update making the check to see if networking is available the same as in netfs. Expect a bodhi update for this soon.

iscsi-initiator-utils-6.2.0.872-12.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/iscsi-initiator-utils-6.2.0.872-12.fc15

Package iscsi-initiator-utils-6.2.0.872-12.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing iscsi-initiator-utils-6.2.0.872-12.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/iscsi-initiator-utils-6.2.0.872-12.fc15
then log in and leave karma (feedback).

(In reply to comment #43)
> iscsi-initiator-utils-6.2.0.872-12.fc15 has been submitted as an update for
> Fedora 15.
> https://admin.fedoraproject.org/updates/iscsi-initiator-utils-6.2.0.872-12.fc15

Tested and confirmed that my non-root iSCSI volume gets mounted everytime on boot using iscsi-initiator-utils-6.2.0.872-12.fc15.  Thanks!

iscsi-initiator-utils-6.2.0.872-12.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

the fixes aren't in the RPM package.


[root@talon packages]# pwd
/var/cache/yum/x86_64/15/fedora/packages
[root@talon packages]# ls -al
iscsi-initiator-utils-6.2.0.872-12.fc15.x86_64.rpm 
-rw-r--r--. 1 root root 331704 Apr 30 20:51
iscsi-initiator-utils-6.2.0.872-12.fc15.x86_64.rpm

#mkdir iscsirpm
#cp iscsi-initiator-utils-6.2.0.872-12.fc15.x86_64.rpm iscsirpm/
#cd iscsirpm

[root@talon iscsirpm]# rpm2cpio
iscsi-initiator-utils-6.2.0.872-12.fc15.x86_64.rpm | cpio -idmv


./etc/NetworkManager/dispatcher.d/04-iscsi
./etc/iscsi
./etc/iscsi/iscsid.conf
./etc/rc.d/init.d/iscsi
./etc/rc.d/init.d/iscsid
./sbin/iscsi-iname
./sbin/iscsiadm
./sbin/iscsid
./sbin/iscsistart
./usr/lib64/libiscsi.so.0
./usr/lib64/python2.7/site-packages/libiscsimodule.so
./usr/share/doc/iscsi-initiator-utils-6.2.0.872
./usr/share/doc/iscsi-initiator-utils-6.2.0.872/README
./usr/share/man/man8/iscsi-iname.8.gz
./usr/share/man/man8/iscsiadm.8.gz
./usr/share/man/man8/iscsid.8.gz
./usr/share/man/man8/iscsistart.8.gz
./var/lib/iscsi
./var/lib/iscsi/ifaces
./var/lib/iscsi/isns
./var/lib/iscsi/nodes
./var/lib/iscsi/send_targets
./var/lib/iscsi/slp
./var/lib/iscsi/static
3693 blocks
[root@talon iscsirpm]# cd etc/rc.d/init.d/
[root@talon init.d]# pwd
/var/cache/yum/x86_64/15/fedora/packages/iscsirpm/etc/rc.d/init.d

[root@talon init.d]# grep "isn't" iscsi --context=5

start() {
    [ -x $exec ] || exit 5
    [ -f $config ] || exit 6

    # if the network isn't up yet exit cleanly, NetworkManager will call us
    # again when the network is up
    [ ! -f /var/lock/subsys/network ] && ! nm-online -x >/dev/null 2>&1 && exit
0

    # if no nodes are setup to startup automatically exit cleanly
    grep -qrs "node.startup = automatic" /var/lib/iscsi/nodes




[root@talon init.d]# rpm -qi iscsi-initiator-utils
Name        : iscsi-initiator-utils
Version     : 6.2.0.872
Release     : 12.fc15
Architecture: x86_64
Install Date: Wed 05 Oct 2011 10:28:53 PM EDT
Group       : System Environment/Daemons
Size        : 1887005
License     : GPLv2+
Signature   : RSA/SHA256, Sat 30 Apr 2011 02:00:12 PM EDT, Key ID b4ebf579069c8460
Source RPM  : iscsi-initiator-utils-6.2.0.872-12.fc15.src.rpm
Build Date  : Sat 30 Apr 2011 04:35:52 AM EDT
Build Host  : x86-07.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://www.open-iscsi.org
Summary     : iSCSI daemon and utility programs
Description :
The iscsi package provides the server daemon for the iSCSI protocol,
as well as the utility programs used to manage it. iSCSI is a protocol
for distributed disk access using SCSI commands sent over Internet
Protocol networks.



#repoquery iscsi-initiator-utils --location
http://mirrors.servercentral.net/fedora/releases/15/Everything/x86_64/os/Packages/iscsi-initiator-utils-6.2.0.872-12.fc15.i686.rpm
http://mirrors.servercentral.net/fedora/releases/15/Everything/x86_64/os/Packages/iscsi-initiator-utils-6.2.0.872-12.fc15.x86_64.rpm

Please take ownership of this issue which contains more details on my previous comment:

Bug 743740 - fstab iscsi mount point no longer mounts in F15 after systemd update

nevermind, I see that the right patch is already in the fix.  

# if the network isn't up yet exit cleanly, NetworkManager will call us
    # again when the network is up
    [ ! -f /var/lock/subsys/network ] && ! nm-online -x >/dev/null 2>&1 && exit
0


This means I haven't really figured out what the root cause was for the
behavior I was experiencing re: Bug report 743740