Bug 1666868 - NFS install fails with kernel "invalid creds" traceback
Summary: NFS install fails with kernel "invalid creds" traceback
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: rawhide
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa AcceptedBlocker
Depends On:
Blocks: F30BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2019-01-16 19:48 UTC by Adam Williamson
Modified: 2019-02-13 01:30 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-13 01:30:54 UTC


Attachments (Terms of Use)
rdsosreport.txt from a failed test (87.59 KB, text/plain)
2019-01-16 19:51 UTC, Adam Williamson
no flags Details

Description Adam Williamson 2019-01-16 19:48:30 UTC
Since about Fedora-Rawhide-20190106.n.0, openQA NFS install test seems to be regularly failing with a kernel trace about "invalid creds". The setup for the test is that there's a "support server" (running Fedora 29) which sets up an /etc/exports with this content:

/export 10.0.2.0/24(ro)
/repo 10.0.2.0/24(ro)

and starts nfs-server.service (with appropriate firewall config). The VM running the actual test boots an installer image with this kernel arg:

inst.stage2=nfs:nfsvers=4:10.0.2.110:/repo

which basically tells anaconda's dracut module to mount the NFS share 10.0.2.110:/repo and get the installer from that share. As you can see from the server config, there are no 'credentials' required, so it's not clear why the kernel would compain about 'invalid creds'.

What actually happens is that the installer boot process drops to the dracut emergency shell, as this NFS install stuff is implemented in dracut so when it fails that's all we have.

Here's an extract from the log:

[   15.545181] localhost dracut-initqueue[829]: anaconda mounting NFS repo at nfs:nfsvers=4:10.0.2.110:/repo
[   15.590747] localhost kernel: FS-Cache: Loaded
[   15.624752] localhost kernel: FS-Cache: Netfs 'nfs' registered for caching
[   15.632601] localhost kernel: Key type dns_resolver registered
[   15.763512] localhost kernel: NFS: Registering the id_resolver key type
[   15.763541] localhost kernel: Key type id_resolver registered
[   15.763548] localhost kernel: Key type id_legacy registered
[   15.790450] localhost kernel: CRED: Invalid credentials
[   15.790459] localhost kernel: CRED: At include/linux/cred.h:253
[   15.790461] localhost kernel: CRED: Specified credentials: 00000000221ef78f 
[   15.790463] localhost kernel: CRED: ->magic=0, put_addr=          (null)
[   15.790464] localhost kernel: CRED: ->usage=1, subscr=0
[   15.790466] localhost kernel: CRED: ->*uid = { 0,0,0,0 }
[   15.790467] localhost kernel: CRED: ->*gid = { 0,0,0,0 }
[   15.790469] localhost kernel: CRED: ->security is           (null)
[   15.790479] localhost kernel: ------------[ cut here ]------------
[   15.790481] localhost kernel: kernel BUG at kernel/cred.c:825!
[   15.790533] localhost kernel: invalid opcode: 0000 [#1] SMP PTI
[   15.790553] localhost kernel: CPU: 1 PID: 1077 Comm: mount.nfs Not tainted 5.0.0-0.rc2.git1.1.fc30.x86_64 #1
[   15.790579] localhost kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180724_192412-buildhw-07.phx2.fedoraproject.org-1.fc29 04/01/2014
[   15.790623] localhost kernel: RIP: 0010:__invalid_creds+0x48/0x4a
[   15.790641] localhost kernel: Code: 44 89 e2 48 89 ee 48 c7 c7 59 07 30 9c e8 a0 2d 05 00 48 c7 c6 6b 07 30 9c 48 89 df 65 48 8b 14 25 00 5e 01 00 e8 34 fe ff ff <0f> 0b 48 c7 c7 98 08 30 9c e8 7a 2d 05 00 44 89 e2 48 89 ee 48 c7
[   15.790695] localhost kernel: RSP: 0018:ffffb79fc0923a30 EFLAGS: 00010293
[   15.790715] localhost kernel: RAX: 0000000000000000 RBX: ffffffffc0661760 RCX: 0000000000000006
[   15.790746] localhost kernel: RDX: 0000000000000000 RSI: ffffa096e3c80cf0 RDI: ffffa096fbbd6c20
[   15.790776] localhost kernel: RBP: ffffffffc09409c9 R08: 0000003a7e5cbd26 R09: 0000000000000000
[   15.790806] localhost kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000000fd
[   15.790829] localhost kernel: R13: ffffa096e5e89800 R14: ffffffffc0661760 R15: ffffb79fc0923d90
[   15.790852] localhost kernel: FS:  00007ff263021880(0000) GS:ffffa096fba00000(0000) knlGS:0000000000000000
[   15.790877] localhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   15.790897] localhost kernel: CR2: 00007f85730521b8 CR3: 0000000063d8c000 CR4: 00000000000006e0
[   15.790927] localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   15.790950] localhost kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   15.790973] localhost kernel: Call Trace:
[   15.791004] localhost kernel:  nfs4_discover_server_trunking+0x28f/0x2b0 [nfsv4]
[   15.791033] localhost kernel:  nfs4_init_client+0x16e/0x240 [nfsv4]
[   15.791053] localhost kernel:  ? kvm_sched_clock_read+0x14/0x30
[   15.791071] localhost kernel:  ? sched_clock+0x5/0x10
[   15.791088] localhost kernel:  ? sched_clock_cpu+0xc/0xb0
[   15.791106] localhost kernel:  ? _raw_spin_unlock+0x24/0x30
[   15.791130] localhost kernel:  ? nfs_get_client+0x4c9/0x5d0 [nfs]
[   15.791686] localhost kernel:  nfs4_set_client+0xb2/0x100 [nfsv4]
[   15.792193] localhost kernel:  nfs4_create_server+0xff/0x290 [nfsv4]
[   15.792656] localhost kernel:  nfs4_remote_mount+0x28/0x50 [nfsv4]
[   15.793129] localhost kernel:  mount_fs+0x4f/0x167
[   15.793563] localhost kernel:  vfs_kern_mount.part.36+0x54/0x160
[   15.794023] localhost kernel:  nfs_do_root_mount+0x7f/0xc0 [nfsv4]
[   15.794462] localhost kernel:  nfs4_try_mount+0x43/0x70 [nfsv4]
[   15.794901] localhost kernel:  ? get_nfs_version+0x21/0x80 [nfs]
[   15.795340] localhost kernel:  nfs_fs_mount+0x673/0xaa0 [nfs]
[   15.795771] localhost kernel:  ? nfs_clone_super+0x70/0x70 [nfs]
[   15.796182] localhost kernel:  ? nfs_parse_mount_options+0xae0/0xae0 [nfs]
[   15.796594] localhost kernel:  mount_fs+0x4f/0x167
[   15.797004] localhost kernel:  vfs_kern_mount.part.36+0x54/0x160
[   15.797414] localhost kernel:  do_mount+0x217/0xd50
[   15.797828] localhost kernel:  ksys_mount+0xba/0xd0
[   15.798230] localhost kernel:  __x64_sys_mount+0x21/0x30
[   15.798659] localhost kernel:  do_syscall_64+0x60/0x1f0
[   15.799075] localhost kernel:  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[   15.799526] localhost kernel: RIP: 0033:0x7ff2636fe01e
[   15.799943] localhost kernel: Code: 48 8b 0d 6d fe 0b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 3a fe 0b 00 f7 d8 64 89 01 48
[   15.800925] localhost kernel: RSP: 002b:00007ffc01647ab8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a5
[   15.801434] localhost kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff2636fe01e
[   15.801893] localhost kernel: RDX: 000055d04aef2b90 RSI: 000055d04aef2b70 RDI: 000055d04aef2bb0
[   15.802333] localhost kernel: RBP: 00007ffc01647c40 R08: 000055d04aef4c40 R09: 00007ff2637bed00
[   15.802763] localhost kernel: R10: 0000000000000001 R11: 0000000000000202 R12: 00007ffc01647c40
[   15.803181] localhost kernel: R13: 000055d04aef4a40 R14: 0000000000000010 R15: 000055d04939d279
[   15.803598] localhost kernel: Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bochs_drm drm_kms_helper ttm virtio_blk virtio_scsi virtio_console drm 8021q garp serio_raw mrp stp llc virtio_net ata_generic net_failover pata_acpi failover qemu_fw_cfg sunrpc sha256_ssse3 dm_crypt dm_round_robin dm_multipath linear raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_intel raid1 raid0 iscsi_ibft floppy squashfs zstd_decompress cramfs be2iscsi bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi edd
[   15.806361] localhost kernel: ---[ end trace 73c81ba3e4d13a89 ]---
[   15.806821] localhost kernel: RIP: 0010:__invalid_creds+0x48/0x4a
[   15.807298] localhost kernel: Code: 44 89 e2 48 89 ee 48 c7 c7 59 07 30 9c e8 a0 2d 05 00 48 c7 c6 6b 07 30 9c 48 89 df 65 48 8b 14 25 00 5e 01 00 e8 34 fe ff ff <0f> 0b 48 c7 c7 98 08 30 9c e8 7a 2d 05 00 44 89 e2 48 89 ee 48 c7
[   15.808270] localhost kernel: RSP: 0018:ffffb79fc0923a30 EFLAGS: 00010293
[   15.808746] localhost kernel: RAX: 0000000000000000 RBX: ffffffffc0661760 RCX: 0000000000000006
[   15.809279] localhost kernel: RDX: 0000000000000000 RSI: ffffa096e3c80cf0 RDI: ffffa096fbbd6c20
[   15.809772] localhost kernel: RBP: ffffffffc09409c9 R08: 0000003a7e5cbd26 R09: 0000000000000000
[   15.810275] localhost kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000000fd
[   15.810767] localhost kernel: R13: ffffa096e5e89800 R14: ffffffffc0661760 R15: ffffb79fc0923d90
[   15.811272] localhost kernel: FS:  00007ff263021880(0000) GS:ffffa096fba00000(0000) knlGS:0000000000000000
[   15.811768] localhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   15.812290] localhost kernel: CR2: 00007f85730521b8 CR3: 0000000063d8c000 CR4: 00000000000006e0
[   15.813114] localhost kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   15.813943] localhost kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   15.815224] localhost kernel: mount.nfs (1077) used greatest stack depth: 11656 bytes left
[   15.803190] localhost dracut-initqueue[829]: Warning: Couldn't mount nfs:nfsvers=4:10.0.2.110:/repo

I will attach the full dracut rdsosreport.txt.

Proposing as a Beta blocker: "When using the dedicated installer images, the installer must be able to use HTTP, FTP and NFS repositories as package sources." - https://fedoraproject.org/wiki/Fedora_30_Beta_Release_Criteria#Remote_package_sources

Comment 1 Adam Williamson 2019-01-16 19:51:10 UTC
Created attachment 1521110 [details]
rdsosreport.txt from a failed test

Comment 2 Chris Murphy 2019-02-11 17:43:54 UTC
Looks like upstream bug
https://bugzilla.kernel.org/show_bug.cgi?id=202377

And looks like it got fixed by the time rc3 rolled out.

Comment 3 Adam Williamson 2019-02-11 17:49:05 UTC
we'll see if it works in openQA today, then.

Comment 4 František Zatloukal 2019-02-12 14:22:56 UTC
Discussed during the 2019-02-11 blocker review meeting: [1]

The decision to classify this bug as an AcceptedBlocker was made:

"accepted as a clear violation of "When using the dedicated installer images, the installer must be able to use HTTP, FTP and NFS repositories as package sources""

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2019-02-11/f30-blocker-review.2019-02-11-17.13.log.txt

Comment 5 Adam Williamson 2019-02-13 01:30:54 UTC
All NFS tests have passed in the last few composes, so looks like this is good now.


Note You need to log in before you can comment on or make changes to this bug.