Bug 1099761

Summary: mount -o remount,rw mountpoint NULL pointer error when the source is nfs with ipv6
Product: [Fedora] Fedora Reporter: arthur <zzou>
Component: kernelAssignee: Mateusz Guzik <mguzik>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: gansalmon, itamar, jonathan, kernel-maint, kzak, madhu.chinakonda, mchehab, mguzik, mluscon, ruyang, zzou
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-3.14.7-100.fc19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1099793 (view as bug list) Environment:
Last Closed: 2014-06-13 22:49:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 821620, 1099793    

Description arthur 2014-05-21 07:12:28 UTC
Description of problem:
Using mount to remount a mountpoint failed if the mountpoint is nfs with ipv6 address.

Version-Release number of selected component (if applicable):
util-linux-2.24-2.fc20.x86_64

How reproducible:
100%

Steps to Reproduce:
1. add a entry in /etc/fstab like "[fe80::5054:ff:fe48:ca80%eth0]:/mnt     /mnt    nfs     defaults        0 0" 
2.reboot
3.remount the mountpoint /mnt using "mount -o remount,ro /mnt"

Actual results:
remount failed

Expected results:
remount succeed

Additional info:
Filesystem                          1K-blocks    Used Available Use% Mounted on
/dev/mapper/fedora-root               8649736 3285568   4901732  41% /
devtmpfs                               952308       0    952308   0% /dev
tmpfs                                  960236       0    960236   0% /dev/shm
tmpfs                                  960236     604    959632   1% /run
tmpfs                                  960236       0    960236   0% /sys/fs/cgroup
tmpfs                                  960236       4    960232   1% /tmp
/dev/vda1                              487652   95988    361968  21% /boot
[fe80::5054:ff:fe48:ca80%eth0]:/mnt   8649984 3589120   4598528  44% /mnt
[root@localhost ~]# mount -o remount,ro /mnt
[   40.842524] BUG: unable to handle kernel NULL pointer dereference at 00000000000000d8
[   40.843025] IP: [<ffffffff81546dcf>] dev_get_by_name_rcu+0x2f/0x90
[   40.843025] PGD 0 
[   40.843025] Oops: 0000 [#1] SMP 
[   40.843025] Modules linked in: nfsv4 dns_resolver nfs fscache nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE bnep bluetooth cfg80211 rfkill ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw joydev i2c_piix4 i2c_core ppdev parport_pc parport virtio_balloon microcode serio_raw mperf nfsd auth_rpcgss nfs_acl lockd sunrpc virtio_blk virtio_net virtio_pci virtio_ring ata_generic virtio pata_acpi
[   40.843025] CPU: 1 PID: 1259 Comm: mount.nfs Not tainted 3.11.10-301.fc20.x86_64 #1
[   40.843025] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   40.843025] task: ffff88007a981e80 ti: ffff88007bc1c000 task.ti: ffff88007bc1c000
[   40.843025] RIP: 0010:[<ffffffff81546dcf>]  [<ffffffff81546dcf>] dev_get_by_name_rcu+0x2f/0x90
[   40.843025] RSP: 0018:ffff88007bc1dd30  EFLAGS: 00010a87
[   40.843025] RAX: 00000000881b7465 RBX: 0000000000000000 RCX: 0000000000000020
[   40.843025] RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000030687465
[   40.843025] RBP: ffff88007bc1dd40 R08: 0000000000016e80 R09: ffff88007d001e00
[   40.843025] R10: 0000000000000004 R11: ffff88007bc1dd10 R12: ffff88007a811a60
[   40.843025] R13: ffff88007a3650f8 R14: 0000000000000000 R15: ffff88007a365100
[   40.843025] FS:  00007f80252a18c0(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[   40.843025] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   40.843025] CR2: 00000000000000d8 CR3: 000000007ab2f000 CR4: 00000000000006e0
[   40.843025] Stack:
[   40.843025]  ffff88007a811a60 ffff88007ac2b620 ffff88007bc1dd50 ffffffff81546e3e
[   40.843025]  ffff88007bc1dda0 ffffffffa0056f1e ffff88007ac2b637 ffff88007ac2b637
[   40.843025]  0000000000000000 ffff88007a365000 0000000000000000 0000000000000000
[   40.843025] Call Trace:
[   40.843025]  [<ffffffff81546e3e>] dev_get_by_name+0xe/0x20
[   40.843025]  [<ffffffffa0056f1e>] rpc_pton+0x10e/0x1d0 [sunrpc]
[   40.843025]  [<ffffffffa0319218>] nfs_parse_mount_options+0x398/0xc70 [nfs]
[   40.843025]  [<ffffffff8118ea67>] ? kmem_cache_alloc_trace+0x1d7/0x230
[   40.843025]  [<ffffffffa031bc27>] ? nfs_remount+0x67/0x350 [nfs]
[   40.843025]  [<ffffffffa031bd2a>] nfs_remount+0x16a/0x350 [nfs]
[   40.843025]  [<ffffffff811acb0a>] do_remount_sb+0x7a/0x1a0
[   40.843025]  [<ffffffff811c86a9>] do_mount+0x689/0xa20
[   40.843025]  [<ffffffff811c7ea6>] ? copy_mount_options+0x36/0x170
[   40.843025]  [<ffffffff811c8ac3>] SyS_mount+0x83/0xc0
[   40.843025]  [<ffffffff816533d9>] system_call_fastpath+0x16/0x1b
[   40.843025] Code: 00 55 48 89 e5 41 54 49 89 f4 be 10 00 00 00 53 48 89 fb 4c 89 e7 e8 c1 b3 db ff 4c 89 e7 89 c6 e8 97 cb c6 ff 69 c0 01 00 37 9e <48> 8b 93 d8 00 00 00 31 db c1 e8 18 48 8d 04 c2 48 8b 00 48 8d 
[   40.843025] RIP  [<ffffffff81546dcf>] dev_get_by_name_rcu+0x2f/0x90
[   40.843025]  RSP <ffff88007bc1dd30>
[   40.843025] CR2: 00000000000000d8
[   40.891754] ---[ end trace 4bc36db59be31d9a ]---
[root@localhost ~]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Wed May 21 03:34:42 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/fedora-root /                       ext4    defaults        1 1
UUID=e271aea5-ed08-4c3d-9732-07daa6f00f22 /boot                   ext4    defaults        1 2
/dev/mapper/fedora-swap swap                    swap    defaults        0 0
[fe80::5054:ff:fe48:ca80%eth0]:/mnt     /mnt    nfs     defaults        0 0

Comment 1 Josh Boyer 2014-05-21 15:48:44 UTC
Can you recreate this with the latest F20 kernel update (3.14.4)?  3.11.10 is the GA kernel for F20 and is rather old at this point.

Comment 2 arthur 2014-05-22 02:22:34 UTC
(In reply to Josh Boyer from comment #1)
> Can you recreate this with the latest F20 kernel update (3.14.4)?  3.11.10
> is the GA kernel for F20 and is rather old at this point.

Hi Josh Boyer,
   remount still failed with the latest kernle 3.14.4-200.fc20.x86_64 

[   42.537784] BUG: unable to handle kernel NULL pointer dereference at 00000000000000d8
[   42.538027] IP: [<ffffffff815e243f>] dev_get_by_name_rcu+0x2f/0x90
[   42.538027] PGD 0 
[   42.538027] Oops: 0000 [#1] SMP 
[   42.538027] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bnep bluetooth 6lowpan_iphc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE cfg80211 rfkill ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ppdev i2c_piix4 microcode parport_pc virtio_balloon parport pvpanic serio_raw i2c_core nfsd auth_rpcgss nfs_acl lockd sunrpc virtio_blk virtio_net virtio_pci virtio_ring virtio ata_generic pata_acpi
[   42.538027] CPU: 1 PID: 1271 Comm: mount.nfs Not tainted 3.14.4-200.fc20.x86_64 #1
[   42.538027] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   42.538027] task: ffff88003731cc00 ti: ffff88007a584000 task.ti: ffff88007a584000
[   42.538027] RIP: 0010:[<ffffffff815e243f>]  [<ffffffff815e243f>] dev_get_by_name_rcu+0x2f/0x90
[   42.538027] RSP: 0018:ffff88007a585d08  EFLAGS: 00010a87
[   42.538027] RAX: 00000000881b7465 RBX: 0000000000000000 RCX: 0000000000000020
[   42.538027] RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000030687465
[   42.538027] RBP: ffff88007a585d18 R08: 0000000000017380 R09: ffff88007d001e00
[   42.538027] R10: 0000000000000004 R11: ffff88007a585ce8 R12: ffff88007a772e00
[   42.538027] R13: ffff88007b27df28 R14: 0000000000000000 R15: ffff88007b27df30
[   42.538027] FS:  00007fa15a4b88c0(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[   42.538027] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   42.538027] CR2: 00000000000000d8 CR3: 000000007c3db000 CR4: 00000000000006e0
[   42.538027] Stack:
[   42.538027]  ffff88007a772e00 ffff8800370341c0 ffff88007a585d28 ffffffff815e24ae
[   42.538027]  ffff88007a585d80 ffffffffa004de32 ffff8800370341d7 ffff8800370341d7
[   42.538027]  0000000000000000 00000000320eb197 ffff88007b27de00 0000000000000000
[   42.538027] Call Trace:
[   42.538027]  [<ffffffff815e24ae>] dev_get_by_name+0xe/0x20
[   42.538027]  [<ffffffffa004de32>] rpc_pton+0x132/0x1f0 [sunrpc]
[   42.538027]  [<ffffffffa03350e8>] nfs_parse_mount_options+0x3a8/0xc90 [nfs]
[   42.538027]  [<ffffffff811ccd06>] ? kmem_cache_alloc_trace+0x1d6/0x200
[   42.538027]  [<ffffffffa0337e03>] nfs_remount+0x1d3/0x3f0 [nfs]
[   42.538027]  [<ffffffff811ed34a>] do_remount_sb+0x7a/0x1a0
[   42.538027]  [<ffffffff8120b221>] do_mount+0x6c1/0xae0
[   42.538027]  [<ffffffff811796a4>] ? __get_free_pages+0x14/0x50
[   42.538027]  [<ffffffff8120b976>] SyS_mount+0x96/0xf0
[   42.538027]  [<ffffffff816ff569>] system_call_fastpath+0x16/0x1b
[   42.538027] Code: 00 55 48 89 e5 41 54 49 89 f4 be 10 00 00 00 53 48 89 fb 4c 89 e7 e8 11 98 d7 ff 4c 89 e7 89 c6 e8 d7 24 c1 ff 69 c0 01 00 37 9e <48> 8b 93 d8 00 00 00 31 db c1 e8 18 48 8d 04 c2 48 8b 00 48 8d 
[   42.538027] RIP  [<ffffffff815e243f>] dev_get_by_name_rcu+0x2f/0x90
[   42.538027]  RSP <ffff88007a585d08>
[   42.538027] CR2: 00000000000000d8
[   42.565834] ---[ end trace 163129b3911cbefc ]---

Thanks
zzou

Comment 3 Mateusz Guzik 2014-06-10 10:21:39 UTC
The bug is that ->net is not popoulated when remounting.

This fixes the problem for me:
diff --git a/fs/nfs/super.c b/fs/nfs/super.c
index 2cb5694..104ef01 100644
--- a/fs/nfs/super.c
+++ b/fs/nfs/super.c
@@ -2248,6 +2248,7 @@ nfs_remount(struct super_block *sb, int *flags, char *raw_data)
        data->nfs_server.addrlen = nfss->nfs_client->cl_addrlen;
        data->version = nfsvers;
        data->minorversion = nfss->nfs_client->cl_minorversion;
+       data->net = current->nsproxy->net_ns;
        memcpy(&data->nfs_server.address, &nfss->nfs_client->cl_addr,
                data->nfs_server.addrlen);

As for reproducer, it is enough to:
mount -t nfs '[fe80::5054:ff:fe10:223a%eth0]':/mnt/export /mnt/tmp
mount -t nfs -o remount,ro '[fe80::5054:ff:fe10:223a%eth0]':/mnt/export /mnt/tmp

Comment 4 Mateusz Guzik 2014-06-10 10:46:11 UTC
Patch sent upstream:
http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg660865.html

Comment 5 Mateusz Guzik 2014-06-11 05:07:59 UTC
Patch got committed with id a914722f333b3359d2f4f12919380a334176bb89 .

Do I have to do anything else w.r.t. fedora?

Comment 6 Josh Boyer 2014-06-11 13:43:55 UTC
(In reply to Mateusz Guzik from comment #5)
> Patch got committed with id a914722f333b3359d2f4f12919380a334176bb89 .
> 
> Do I have to do anything else w.r.t. fedora?

No, it's CC'd to stable and we'll either pick it up when it hits 3.14.y or 3.15.y, if not before then.  Thanks!

Comment 7 Josh Boyer 2014-06-11 20:23:20 UTC
Picked this up early.  Thanks again.

Comment 8 Fedora Update System 2014-06-12 12:13:56 UTC
kernel-3.14.7-200.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/kernel-3.14.7-200.fc20

Comment 9 Fedora Update System 2014-06-12 12:15:52 UTC
kernel-3.14.7-100.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/kernel-3.14.7-100.fc19

Comment 10 Fedora Update System 2014-06-13 05:26:56 UTC
Package kernel-3.14.7-200.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-3.14.7-200.fc20'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-7313/kernel-3.14.7-200.fc20
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2014-06-13 22:49:50 UTC
kernel-3.14.7-200.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 12 Fedora Update System 2014-06-16 23:29:05 UTC
kernel-3.14.7-100.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.