Bug 2100668

Summary:	"Apply all sysctl settings when NFS-related modules are loaded" causes error messages in early boot of images, prevents nfs module loading (so breaks e.g. 'inst.updates=nfs', 'inst.stage2=nfs'...)
Product:	[Fedora] Fedora	Reporter:	Adam Williamson <awilliam>
Component:	nfs-utils	Assignee:	Steve Dickson <steved>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	luk.claes, matt.fagnani, pvalena, red, robatino, steved
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:	openqa
Fixed In Version:	nfs-utils-2.6.1-3.rc8.fc37 nfs-utils-2.6.1-2.rc8.fc36	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-07-01 01:07:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2009537

Description Adam Williamson 2022-06-23 23:20:50 UTC

With recent Rawhide installer and live images, if you watch carefully during early boot (right after picking an option from the boot menu), you briefly see several error messages:

dracut-pre-udev[NNN]: sh: line 1: /sbin/sysctl: No such file or directory
dracut-pre-udev[NNN]: modprobe: ERROR: libkmod/libkmod-module.c:990 command_do() Error running install command '/sbin/modprobe --ignore-install sunrpc && /sbin/sysctl -q --pattern sunrpc --system' for module sunrpc: retcode 127
dracut-pre-udev[NNN]: modprobe: ERROR: could not insert 'sunrpc': Invalid argument

This looks to me like it's caused by the modprobe config file added in "Apply all sysctl settings when NFS-related modules are loaded" - https://lore.kernel.org/all/165335769457.22265.5383162992195478413@noble.neil.brown.name/T/#r3ad8e3996fcfa83f3ae3973a5b9b1ada14d92644 - which specifies:

install sunrpc /sbin/modprobe --ignore-install sunrpc $CMDLINE_OPTS && /sbin/sysctl -q --pattern sunrpc --system

it's obviously not safe to assume `/sbin/sysctl` will always be available, apparently it is not in this early boot initramfs environment. Not sure what the best 'fix' is, but it'd be nice to not trigger such an error.

Comment 1 Adam Williamson 2022-06-23 23:49:02 UTC

Oh, this is actually *way* worse than I thought. It prevents the nfs module loading at all, which prevents any use of NFS from the kernel command line for anaconda, e.g. "inst.updates=nfs://somewhere" or "inst.stage2=nfs://somewhere".

That makes this a Beta blocker, probably. Adding "|| :" to the end of the sysctl lines in the config file, so they don't fail if sysctl fails, seems to allow nfs to load, so I think I'll do that as a quick fix.

Comment 2 Adam Williamson 2022-06-24 00:11:01 UTC

https://koji.fedoraproject.org/koji/taskinfo?taskID=88666982 should address this as a quick fix, anyhow. Leaving the bug open for Steve to see what he thinks of the patch and if he wants to go a different way, but if openQA confirms with tomorrow's compose that this makes things work, I'll drop the blocker proposal.

Comment 3 Adam Williamson 2022-06-24 00:12:49 UTC

BTW, to reproduce the problem, grab https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20220623.n.0/compose/Everything/x86_64/iso/Fedora-Everything-netinst-x86_64-Rawhide-20220623.n.0.iso and try to pass `inst.updates=nfs://somethingvalid` or `inst.stage2=nfs://somethingvalid`. It doesn't have to be a server that actually exists and works, just syntactically valid. You'll eventually get dumped to a dracut console from which you can verify that the nfs module isn't loaded, and running `modprobe nfs` shows the same errors and does not load it.

Comment 4 Steve Dickson 2022-06-24 19:13:56 UTC

(In reply to Adam Williamson from comment #2)
> https://koji.fedoraproject.org/koji/taskinfo?taskID=88666982 should address
> this as a quick fix, anyhow. Leaving the bug open for Steve to see what he
> thinks of the patch and if he wants to go a different way, but if openQA
> confirms with tomorrow's compose that this makes things work, I'll drop the
> blocker proposal.

There is already an upstream fix for this posted 2 days
ago.... which I was looking at... and seems to fix the problem

So no.. I don't want this fix... I want the upstream fix... 

PLEASE!!! before you do a commit/push/build, PLEASE contact me via
my inbox! because this is just making more work for me.

Comment 5 Steve Dickson 2022-06-24 19:20:39 UTC

(In reply to Steve Dickson from comment #4)
> (In reply to Adam Williamson from comment #2)
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=88666982 should address
> > this as a quick fix, anyhow. Leaving the bug open for Steve to see what he
> > thinks of the patch and if he wants to go a different way, but if openQA
> > confirms with tomorrow's compose that this makes things work, I'll drop the
> > blocker proposal.
> 
> There is already an upstream fix for this posted 2 days
> ago.... which I was looking at... and seems to fix the problem
> 
> So no.. I don't want this fix... I want the upstream fix... 
> 
> PLEASE!!! before you do a commit/push/build, PLEASE contact me via
> my inbox! because this is just making more work for me.

Even better.. post patches upstream linux-nfs <linux-nfs.org>
which is the right way to do things... PLEASE!!!

Comment 6 Adam Williamson 2022-06-24 19:20:59 UTC

Sorry about that. The patches do effectively the same thing, AFAICS. I didn't catch that the problem had already been seen as it's not in the same thread. It's *much* harder to follow things in mailing list archives than it is in proper issue trackers :(

Comment 7 Adam Williamson 2022-06-24 19:21:34 UTC

Mailing lists are also extremely inconvenient for this kind of drive-by fix. It would be way easier if I could submit a patch to a forge.

Comment 8 Matt Fagnani 2022-06-24 19:44:54 UTC

*** Bug 2098418 has been marked as a duplicate of this bug. ***

Comment 9 Fedora Update System 2022-06-28 15:06:13 UTC

FEDORA-2022-38325154c4 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-38325154c4

Comment 10 Pavel Valena 2022-06-28 16:49:54 UTC

Hello, does this fix https://bugzilla.redhat.com/show_bug.cgi?id=2099256#c3 ?

Any idea where does the 50-nfs.conf live?

I could only find: https://lore.kernel.org/all/165335769457.22265.5383162992195478413@noble.neil.brown.name/T/

Comment 11 Steve Dickson 2022-06-28 17:35:36 UTC

(In reply to Pavel Valena from comment #10)
> Hello, does this fix https://bugzilla.redhat.com/show_bug.cgi?id=2099256#c3 ?
> 
> Any idea where does the 50-nfs.conf live?
> 
> I could only find:
> https://lore.kernel.org/all/165335769457.22265.5383162992195478413@noble.
> neil.brown.name/T/

/usr/lib/modprobe.d/50-nfs.conf

Comment 12 Adam Williamson 2022-06-28 19:50:01 UTC

"Hello, does this fix https://bugzilla.redhat.com/show_bug.cgi?id=2099256#c3 ?"

Yes, it does. Both the patch I initially applied downstream and the one Steve took from upstream make the `sysctl` calls non-fatal, so module loading will still happen even if they fail.

Comment 13 Fedora Update System 2022-06-29 01:33:38 UTC

FEDORA-2022-38325154c4 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-38325154c4`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-38325154c4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2022-07-01 01:07:35 UTC

FEDORA-2022-38325154c4 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 15 Adam Williamson 2022-07-13 15:28:27 UTC

*** Bug 2102903 has been marked as a duplicate of this bug. ***