Description of problem: The _netdev option is used in /etc/fstab to mark devices that depend on the network for operation. This causes the initscripts to defer mounting file systems located on these devices until the network is brought up and the netfs initscript runs. Device nodes for these devices (nbd, gnbd, iscsi etc.) are created by udev once the appropriate subsystem is started. Processing of the udev backlog may not have completed by the time the netfs script runs causing failures to occur when attempting to mount file systems. This can happen when there are a large number of devices or when device initialisation is slow. This is fixed by adding a udevsettle to the start() function in netfs. Version-Release number of selected component (if applicable): initscripts-8.45.17.EL-1 How reproducible: 100% Steps to Reproduce: 1. grep udev /etc/init.d/netfs Actual results: No results. Expected results: An invocation of udevsettle to cause the script to sync up with udev processing. Additional info: Reproducing actual errors is harder; need to create for e.g. a very large number of iSCSI devices or arrange for the udev processing to take longer than normal (maybe adding a sleep in a udev rule would work for simulating this). This has been seen in production environments with large numbers of devices.
Created attachment 310267 [details] Add a udevsettle before mounting _netdev devices
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
As described in comment #0; when you have a system that exhibits this behavior it will fail 100% of the time. This normally involves very large numbers of devices or devices that are for some reason slow for udev to process. If using modified udev scripts is an acceptable way to reproduce for testing purposes then just adding a delay of a couple of seconds should be adequate to trigger a mount failure in the netfs script; just create an entry in fstab labelled with _netdev that attempts to mount such a "slow" device.
Yes, I think might be acceptable and probably easier to setup then many iscsi drives (even though possible). Thanks for the info Bryn!
please test: http://people.redhat.com/harald/downloads/initscripts/initscripts-8.45.20-0.1.el5/
Upon re-review... why isn't this patch in the iscsi script instead?
+1, seeing this issue here on multiple systems. Confirming that adding /sbin/udevsettle --timeout=30 to init.d/iscsi startup, just after loginall, solves it nicely. Solid across 20 reboots.
Bryn, Harald, Can you please add more clear/step-by-step reproducer for QE? Thank you in advance.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2009-0245.html