This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 968587 - instant and remote reliable crash - kernel 3.9.xx
instant and remote reliable crash - kernel 3.9.xx
Status: CLOSED DUPLICATE of bug 954181
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
18
i686 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Kernel Maintainer List
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-29 18:17 EDT by Laurent Jacquot
Modified: 2013-06-03 04:43 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-03 04:43:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Laurent Jacquot 2013-05-29 18:17:38 EDT
when using the two last kernels kernel-PAE-3.9.2-200.fc18.i686 and kernel-PAE-3.9.4-200.fc18.i686, I can reproduce an instant and remote crash:

Guest and host are uptodate, selinux is enforcing on both, I'm using the default virtualisation techonology through virt{sh,-manager}: qemu-kvm. 

Guest runs a vsftpd server allowing read-only access to a "Musique" and a "Vidéos" directory

Host export to the guest those "Musique" and "Vidéos" directories through 9p.

As soon as (a single click) a ftp client tries to access the "Musique" directory, the host severely crashes, a reboot is required, and I did not manage to capture any traces.
NOTE: the "Vidéos" directory works as expected!

Some config data on the guest:

<root@guest>#cat /etc/fstab|grep virtio
/Musique /home/greg/Musique 9p trans=virtio,version=9p2000.L,access=500	0 0
/Videos	 /home/greg/Vidéos 9p trans=virtio,version=9p2000.L,access=500	0 0

<root@guest>#grep greg /etc/passwd
greg:x:500:500:greg:/home/greg:/bin/bash

<greg@guest>$ll -Z |egrep  Musique\|Vidéos
drwxr-xr-x. greg greg system_u:object_r:nfs_t:s0       Musique
drwxr-xr-x. greg greg system_u:object_r:nfs_t:s0       Vidéos


Some config on the host:
On the Host the exports are defined like this in the xml guest definition:
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/srv/mediatomb/Vidéos/Divx'/>
      <target dir='/Videos'/>
      <readonly/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </filesystem>
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/srv/mediatomb/Musique'/>
      <target dir='/Musique'/>
      <readonly/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </filesystem>

/srv/mediatomb/{Musique,Vidéos} are bind mounted read-only from /home:
<root@host>#cat /usr/bin/pre-exec-mediatomb.sh
#!/bin/bash

ROOT="/srv/mediatomb"
MNT="Images Vidéos Vidéos/Divx Musique"

for i in $MNT; do
	if grep -qs "$ROOT/$i" /proc/mounts; then
		echo $i already bind mounted
	else
		mount -o bind /home/alex/$i $ROOT/$i
		mount -o remount,ro,bind $ROOT/$i
		echo bind mounting $i
	fi
done

<root@host>#ll -Z /srv/mediatomb/ |egrep Musique\|Vidéos
drwxr-xr-x. alex      alex      unconfined_u:object_r:user_home_t:s0 Musique
drwxr-xr-x. alex      alex      unconfined_u:object_r:user_home_t:s0 Vidéos

<root@host>#grep alex /etc/passwd
alex:x:500:500:alex:/home/alex:/bin/bash

<root@host>#df -h|egrep Musique\|Vidéos
/dev/mapper/Store-Rushs       385G  132G  234G  37% /home/alex/Vidéos
/dev/mapper/Store-Music       385G  143G  223G  40% /home/alex/Musique
/dev/mapper/Store-Videos      385G  270G   96G  74% /home/alex/Vidéos/Divx

<root@host># ll /home/alex/Vidéos/Divx/|wc -l
30

<root@host>#ll /home/alex/Musique/|wc -l
95

I really do not understand why access to "Musique" crashes and "Vidéos" does not... maybe a one page stack overflow?

Booting an older kernel (3.8 and before) works as expected.
Feel free to ask for any information, but this is my main home workstation, so crashing is frown upon.
Comment 1 Laurent Jacquot 2013-05-29 18:29:28 EDT
Some more data on the filesystem config:

<root@host>#df -h|egrep Musique\|Vidéos
/dev/mapper/Store-Rushs       385G  132G  234G  37% /home/alex/Vidéos
/dev/mapper/Store-Music       385G  143G  223G  40% /home/alex/Musique
/dev/mapper/Store-Videos      385G  270G   96G  74% /home/alex/Vidéos/Divx

<root@host># lvs |egrep Videos\|Music\|Rushs
  Music      Store -wi-ao--- 390,62g
  Rushs      Store -wi-ao--- 390,62g        
  Videos     Store -wi-ao--- 390,62g 

<root@host># pvs |grep Store
  /dev/md1   Store lvm2 a--  509,97g  21,69g
  /dev/md2   Store lvm2 a--  599,97g  14,03g
  /dev/md3   Store lvm2 a--  599,97g 129,34g
  /dev/md4   Store lvm2 a--  599,97g 209,34g

<root@host># cat /proc/mdstat |grep md[1-6]
Personalities : [raid6] [raid5] [raid4] [raid1] 
md3 : active raid5 sdb7[1] sda7[0] sdd7[4] sdc7[2]
      629140992 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md2 : active raid5 sda6[0] sdd6[4] sdb6[1] sdc6[2]
      629140992 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md1 : active raid5 sdd5[4] sda5[0] sdb5[1] sdc5[2]
      534769152 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 1/2 pages [4KB], 65536KB chunk

md5 : active raid5 sda9[0] sdb9[1] sdd9[4] sdc9[2]
      413683200 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md4 : active raid5 sdd8[4] sdb8[1] sda8[0] sdc8[2]
      629140992 blocks super 1.1 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/2 pages [0KB], 65536KB chunk
Comment 2 Andrew Jones 2013-05-30 04:05:05 EDT
Thanks for all the details. The piece that looks the most likely is the 9p use. Please enable kdump on the host, reproduce the issue, and then grab the backtraces from each cpu. If the backtrace confirms 9p, then it might be possible to identify which of the 30 or so changes to it since v3.8 introduced this regression by code analysis alone. Either way, please paste the backtraces here.

Also, if a backtrace points at 9p, but more debugging is necessary, then you may want to consider building a kernel with CONFIG_NET_9P_DEBUG, reproducing with that (still with kdump), and then extracting the kernel's ring buffer from the core. Actually, either way, extracting the kernel's ring buffer could be useful, particularly if you boot your kernel with the ignore_loglevel kernel parameter.
Comment 3 Laurent Jacquot 2013-05-31 15:32:03 EDT
Ok I managed to capture the vmcore (BTW I'm impressed by the kdump feature!) here is my attemps to provide some info:

<root@host># crash /var/crash/var/crash/127.0.0.1-2013.05.31-19\:02\:59/vmcore /usr/lib/debug/lib/modules/3.9.4-200.fc18.i686.PAE/vmlinux
...snip...
      KERNEL: /usr/lib/debug/lib/modules/3.9.4-200.fc18.i686.PAE/vmlinux
    DUMPFILE: /var/crash/var/crash/127.0.0.1-2013.05.31-19:02:59/vmcore  [PARTIAL DUMP]
        CPUS: 4
        DATE: Fri May 31 21:02:49 2013
      UPTIME: 00:03:05
LOAD AVERAGE: 0.49, 0.51, 0.23
       TASKS: 306
    NODENAME: jack
     RELEASE: 3.9.4-200.fc18.i686.PAE
     VERSION: #1 SMP Fri May 24 20:24:58 UTC 2013
     MACHINE: i686  (2942 Mhz)
      MEMORY: 8 GB
       PANIC: "Oops: 0000 [#1] SMP " (check log for details)
         PID: 1936
     COMMAND: "vhost-1935"
        TASK: e7243fc0  [THREAD_INFO: e9f04000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)
crash> log
...snip dmesg output....[  184.960203] BUG: unable to handle kernel NULL pointer dereference at 0000000c
[  184.962813] IP: [<c08b9c72>] dev_queue_xmit+0xb2/0x3c0
[  184.965471] *pdpt = 0000000027077001 *pde = 0000000000000000 
[  184.968183] Oops: 0000 [#1] SMP 
[  184.970984] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc bnep bluetooth rfkill raid1 snd_hda_codec_via snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm acpi_cpufreq mperf coretemp iTCO_wdt iTCO_vendor_support snd_page_alloc snd_timer snd soundcore uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core microcode videodev media i2c_i801 lpc_ich r8169 mfd_core mii asus_atk0110 vhost_net tun macvtap macvlan kvm_intel kvm uinput binfmt_misc raid456 async_raid6_recov async_memcpy async_pq raid6_pq async_xor xor async_tx ata_generic pata_acpi nouveau mxm_wmi wmi video crc32c_intel i2c_algo_bit drm_kms_helper ttm drm
[  184.981413]  firewire_ohci firewire_core i2c_core crc_itu_t pata_jmicron
[  184.988163] Pid: 1936, comm: vhost-1935 Not tainted 3.9.4-200.fc18.i686.PAE #1 System manufacturer System Product Name/P7P55D LE
[  184.991761] EIP: 0060:[<c08b9c72>] EFLAGS: 00010202 CPU: 0
[  184.995422] EIP is at dev_queue_xmit+0xb2/0x3c0
[  184.999052] EAX: e7195f40 EBX: e71c5800 ECX: 00000000 EDX: 000016e2
[  185.002683] ESI: 18e6c000 EDI: e9dabe00 EBP: e9f05de4 ESP: e9f05dbc
[  185.006356]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  185.010214] CR0: 8005003b CR2: 0000000c CR3: 2713f000 CR4: 000027f0
[  185.013992] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[  185.017773] DR6: ffff0ff0 DR7: 00000400
[  185.021578] Process vhost-1935 (pid: 1936, ti=e9f04000 task=e7243fc0 task.ti=e9f04000)
[  185.025596] Stack:
[  185.029521]  e71956e2 000016e2 00000000 0000000c e9f05e00 ee916000 f6abcb00 e71c5800
[  185.033508]  f6bed800 0000000c e9f05e00 f7fcf2f9 c08af530 000016e2 e71c5800 00000001
[  185.037542]  0000000c e9f05e90 f7fd8b17 000016ee 00000000 006860ee e9f05f0c 0000000c
[  185.041768] Call Trace:
[  185.045813]  [<f7fcf2f9>] macvlan_start_xmit+0x59/0x110 [macvlan]
[  185.049917]  [<c08af530>] ? memcpy_fromiovecend+0x50/0x80
[  185.054053]  [<f7fd8b17>] macvtap_get_user+0x407/0x610 [macvtap]
[  185.058420]  [<f7fd8d5a>] macvtap_sendmsg+0x3a/0x50 [macvtap]
[  185.062708]  [<f80231bf>] handle_tx+0x27f/0x5f0 [vhost_net]
[  185.066906]  [<f8023560>] handle_tx_kick+0x10/0x20 [vhost_net]
[  185.071080]  [<f801f8f9>] vhost_worker+0xb9/0x140 [vhost_net]
[  185.075374]  [<f801f840>] ? memory_access_ok.isra.9+0xa0/0xa0 [vhost_net]
[  185.079665]  [<c046e7c4>] kthread+0x94/0xa0
[  185.083875]  [<c09b0837>] ret_from_kernel_thread+0x1b/0x28
[  185.088291]  [<c046e730>] ? kthread_create_on_node+0xc0/0xc0
[  185.092654] Code: 8b 83 b0 00 00 00 8b 53 50 89 53 18 66 83 78 02 00 74 34 8b 8b a0 00 00 00 89 ce 2b b3 a8 00 00 00 f6 40 06 11 0f 84 f2 02 00 00 <0f> b6 49 0c c0 e9 04 0f b6 c9 8d 34 8e 0f b7 40 04 83 e8 01 0f
[  185.102014] EIP: [<c08b9c72>] dev_queue_xmit+0xb2/0x3c0 SS:ESP 0068:e9f05dbc
[  185.106624] CR2: 000000000000000c
crash> bt -a
PID: 1936   TASK: e7243fc0  CPU: 0   COMMAND: "vhost-1935"
 #0 [e9f05c70] crash_kexec at c04b3129
 #1 [e9f05cc8] update_sd_lb_stats at c0488858
 #2 [e9f05cf8] __do_page_fault at c09acd30
 #3 [e9f05d74] do_page_fault at c09aceb8
 #4 [e9f05d7c] error_code (via page_fault) at c09a9e75
    EAX: e7195f40  EBX: e71c5800  ECX: 00000000  EDX: 000016e2  EBP: e9f05de4 
    DS:  007b      ESI: 18e6c000  ES:  007b      EDI: e9dabe00  GS:  00e0
    CS:  0060      EIP: c08b9c72  ERR: ffffffff  EFLAGS: 00010202 
 #5 [e9f05db0] dev_queue_xmit at c08b9c72
 #6 [e9f05de8] macvlan_start_xmit at f7fcf2f4 [macvlan]
 #7 [e9f05e04] macvtap_get_user at f7fd8b12 [macvtap]
 #8 [e9f05e94] macvtap_sendmsg at f7fd8d55 [macvtap]
 #9 [e9f05eb0] handle_tx at f80231bc [vhost_net]
#10 [e9f05f3c] handle_tx_kick at f802355b [vhost_net]
#11 [e9f05f44] vhost_worker at f801f8f6 [vhost_net]
#12 [e9f05f70] kthread at c046e7c2
#13 [e9f05fb0] ret_from_kernel_thread at c09b0834

PID: 0      TASK: f70e8cc0  CPU: 1   COMMAND: "swapper/1"
 #0 [f70e7e70] crash_nmi_callback at c0431936
 #1 [f70e7e74] nmi_handle.isra.0 at c09aa7f6
 #2 [f70e7e94] do_nmi at c09aa903
 #3 [f70e7ec0] nmi at c09a9f32
    EAX: 00000020  EBX: 00000008  ECX: 00000001  EDX: 00000000  EBP: f70e7f14 
    DS:  007b      ESI: 00000004  ES:  007b      EDI: 00000020  GS:  00e0
    CS:  0060      EIP: c06f6fad  ERR: 00000020  EFLAGS: 00200046 
 #4 [f70e7ef4] intel_idle at c06f6fad
 #5 [f70e7f18] cpuidle_enter at c087ace1
 #6 [f70e7f28] cpuidle_wrap_enter at c087b4eb
 #7 [f70e7f44] cpuidle_enter_tk at c087b55d
 #8 [f70e7f50] cpuidle_enter_state at c087b1bb
 #9 [f70e7f5c] cpuidle_idle_call at c087b27d
#10 [f70e7f80] cpu_idle at c0419cb5

PID: 0      TASK: f70e9980  CPU: 2   COMMAND: "swapper/2"
 #0 [f70f1e70] crash_nmi_callback at c0431936
 #1 [f70f1e74] nmi_handle.isra.0 at c09aa7f6
 #2 [f70f1e94] do_nmi at c09aa903
 #3 [f70f1ec0] nmi at c09a9f32
    EAX: 00000020  EBX: 00000008  ECX: 00000001  EDX: 00000000  EBP: f70f1f14 
    DS:  007b      ESI: 00000004  ES:  007b      EDI: 00000020  GS:  00e0
    CS:  0060      EIP: c06f6fad  ERR: 00000020  EFLAGS: 00200046 
 #4 [f70f1ef4] intel_idle at c06f6fad
 #5 [f70f1f18] cpuidle_enter at c087ace1
 #6 [f70f1f28] cpuidle_wrap_enter at c087b4eb
 #7 [f70f1f44] cpuidle_enter_tk at c087b55d
 #8 [f70f1f50] cpuidle_enter_state at c087b1bb
 #9 [f70f1f5c] cpuidle_idle_call at c087b27d
#10 [f70f1f80] cpu_idle at c0419cb5

PID: 0      TASK: f70ea640  CPU: 3   COMMAND: "swapper/3"
 #0 [f70f3e70] crash_nmi_callback at c0431936
 #1 [f70f3e74] nmi_handle.isra.0 at c09aa7f6
 #2 [f70f3e94] do_nmi at c09aa903
 #3 [f70f3ec0] nmi at c09a9f32
    EAX: 00000020  EBX: 00000008  ECX: 00000001  EDX: 00000000  EBP: f70f3f14 
    DS:  007b      ESI: 00000004  ES:  007b      EDI: 00000020  GS:  00e0
    CS:  0060      EIP: c06f6fad  ERR: 00000020  EFLAGS: 00200046 
 #4 [f70f3ef4] intel_idle at c06f6fad
 #5 [f70f3f18] cpuidle_enter at c087ace1
 #6 [f70f3f28] cpuidle_wrap_enter at c087b4eb
 #7 [f70f3f44] cpuidle_enter_tk at c087b55d
 #8 [f70f3f50] cpuidle_enter_state at c087b1bb
 #9 [f70f3f5c] cpuidle_idle_call at c087b27d
#10 [f70f3f80] cpu_idle at c0419cb5
crash>exit

I do not see 9p being in the forefront, it seems to be in the macvtap stuff. I can dig more on the vmcore (provided you help me), or I can put it on a ftp server, after all, this happens when an ftp client tries to access my server!

regards
Laurent
Comment 4 Josh Boyer 2013-05-31 15:37:17 EDT
Looks like 954181
Comment 5 Laurent Jacquot 2013-05-31 17:23:00 EDT
I agree with Josh this is a very similar bactrace and similar weird crashes.

The vmcore I captured is availaible at dev.lutty.net through ftp with fedora as both usr and pwd
Comment 6 Andrew Jones 2013-06-03 04:43:57 EDT
Agreed. Dupping.

*** This bug has been marked as a duplicate of bug 954181 ***

Note You need to log in before you can comment on or make changes to this bug.