Description of problem: When booting the Fedora 26 kernel on top of the Fedora 26 qemu: [ 1.143648] general protection fault: 0000 [#1] SMP [ 1.143648] Modules linked in: virtio_pci(+) virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk virtio_ring virtio crc32_generic [ 1.143648] CPU: 0 PID: 1 Comm: init Not tainted 4.11.0-0.rc1.git0.1.fc26.x86_64+debug #1 [ 1.143648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.1-2.fc26 04/01/2014 [ 1.143648] task: ffff8f275ee1b100 task.stack: ffffac5a800c8000 [ 1.143648] RIP: 0010:vp_modern_find_vqs+0x39/0x70 [virtio_pci] [ 1.143648] RSP: 0018:ffffac5a800cb828 EFLAGS: 00000286 [ 1.143648] RAX: ffffac5a800fd000 RBX: 2d316f6974726976 RCX: 0000000000000000 [ 1.143648] RDX: 00000000000000fc RSI: ffffac5a800fd01c RDI: 0000000000000001 [ 1.143648] RBP: ffffac5a800cb848 R08: 0000000000000000 R09: 0000000000000000 [ 1.143648] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8f275bdf2d28 [ 1.143648] R13: 0000000000000000 R14: ffff8f275bdf2800 R15: 000000000000001f [ 1.143648] FS: 00007ffe35cb08e0(0000) GS:ffff8f275f000000(0000) knlGS:0000000000000000 [ 1.143648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.143648] CR2: 00007fb262ed6000 CR3: 000000001bd0b000 CR4: 00000000000006f0 [ 1.143648] Call Trace: [ 1.143648] init_vqs+0x1a9/0x2f0 [virtio_console] [ 1.143648] virtcons_probe+0xbf/0x450 [virtio_console] [ 1.143648] virtio_dev_probe+0x144/0x1e0 [virtio] [ 1.143648] driver_probe_device+0x106/0x450 [ 1.143648] __device_attach_driver+0x83/0x100 [ 1.143648] ? __driver_attach+0xf0/0xf0 [ 1.143648] bus_for_each_drv+0x6c/0xb0 [ 1.143648] __device_attach+0xe2/0x160 [ 1.143648] device_initial_probe+0x13/0x20 [ 1.143648] bus_probe_device+0xa3/0xb0 [ 1.143648] device_add+0x457/0x6a0 [ 1.143648] ? vp_modern_find_vqs+0x70/0x70 [virtio_pci] [ 1.143648] ? vp_modern_find_vqs+0x70/0x70 [virtio_pci] [ 1.143648] device_register+0x1a/0x20 [ 1.143648] register_virtio_device+0xdc/0x100 [virtio] [ 1.143648] virtio_pci_probe+0x94/0x110 [virtio_pci] [ 1.143648] local_pci_probe+0x42/0xa0 [ 1.143648] ? pci_match_device+0xe0/0x110 [ 1.143648] pci_device_probe+0xbe/0x150 [ 1.143648] ? _raw_spin_unlock+0x27/0x40 [ 1.143648] driver_probe_device+0x106/0x450 [ 1.143648] __driver_attach+0xa8/0xf0 [ 1.143648] ? driver_probe_device+0x450/0x450 [ 1.143648] bus_for_each_dev+0x75/0xc0 [ 1.143648] driver_attach+0x1e/0x20 [ 1.143648] bus_add_driver+0x1d3/0x270 [ 1.143648] ? 0xffffffffc029b000 [ 1.143648] driver_register+0x60/0xe0 [ 1.143648] ? 0xffffffffc029b000 [ 1.143648] __pci_register_driver+0x60/0x70 [ 1.143648] virtio_pci_driver_init+0x1e/0x1000 [virtio_pci] [ 1.143648] do_one_initcall+0x50/0x1a0 [ 1.143648] ? rcu_read_lock_sched_held+0x79/0x80 [ 1.143648] ? kmem_cache_alloc_trace+0x273/0x2e0 [ 1.143648] ? do_init_module+0x27/0x1e8 [ 1.143648] do_init_module+0x5f/0x1e8 [ 1.143648] load_module+0x2392/0x29e0 [ 1.143648] ? __symbol_put+0x70/0x70 [ 1.143648] ? show_coresize+0x30/0x30 [ 1.143648] SYSC_init_module+0x193/0x1d0 [ 1.143648] SyS_init_module+0xe/0x10 [ 1.143648] entry_SYSCALL_64_fastpath+0x1f/0xc2 [ 1.143648] RIP: 0033:0x400e03 [ 1.143648] RSP: 002b:00007ffe35cb07b8 EFLAGS: 00000217 ORIG_RAX: 00000000000000af [ 1.143648] RAX: ffffffffffffffda RBX: 000000000000800b RCX: 0000000000400e03 [ 1.143648] RDX: 0000000000403ba1 RSI: 000000000000800b RDI: 00007fb262ece010 [ 1.143648] RBP: 000000000000800b R08: 00000000ffffffff R09: 0000000000000000 [ 1.143648] R10: 0000000000400e03 R11: 0000000000000217 R12: 00007fb262ece010 [ 1.143648] R13: 0000000000000004 R14: 00007fb262ed8010 R15: 0000000000000000 [ 1.143648] Code: 54 53 49 89 fe e8 98 0d 00 00 85 c0 41 89 c5 75 44 49 8b 9e 28 05 00 00 4d 8d a6 28 05 00 00 4c 39 e3 74 31 49 8b 86 58 05 00 00 <0f> b7 7b 28 48 8d 70 16 e8 7a b7 23 f7 49 8b 86 58 05 00 00 bf [ 1.143648] RIP: vp_modern_find_vqs+0x39/0x70 [virtio_pci] RSP: ffffac5a800cb828 [ 1.143648] ---[ end trace 00e9dd75496f4561 ]--- [ 1.143648] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b Version-Release number of selected component (if applicable): kernel 4.11.0-0.rc1.git0.1.fc26 qemu 2:2.8.0-2.fc26 How reproducible: Unknown, at least once. Steps to Reproduce: 1. Boot kernel on qemu.
Created attachment 1261189 [details] build.log Full build log, contains qemu command line etc.
Created attachment 1261190 [details] root.og
FWIW the Rawhide kernel also fails in a similar place with a slightly different stack trace: supermin: internal insmod virtio_pci.ko [ 0.409906] virtio-pci 0000:00:02.0: PCI->APIC IRQ transform: INT A -> IRQ 10 [ 0.412600] scsi host2: Virtio SCSI HBA [ 0.413477] virtio-pci 0000:00:03.0: PCI->APIC IRQ transform: INT A -> IRQ 11 [ 0.420249] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5 [ 0.436727] scsi 2:0:1:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5 [ 0.438038] virtio-pci 0000:00:04.0: PCI->APIC IRQ transform: INT A -> IRQ 11 [ 0.440993] random: crng init done [ 0.458354] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 0.459004] IP: virtscsi_queuecommand_single+0x21/0x40 [virtio_scsi] [ 0.459004] PGD 1d701067 [ 0.459004] PUD 1d6fb067 [ 0.459004] PMD 0 [ 0.459004] [ 0.459004] Oops: 0002 [#1] SMP [ 0.459004] Modules linked in: virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk virtio_ring virtio crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul [ 0.459004] CPU: 0 PID: 78 Comm: kworker/u2:1 Not tainted 4.11.0-0.rc1.git0.1.fc27.x86_64 #1 [ 0.459004] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.459004] Workqueue: events_unbound async_run_entry_fn [ 0.459004] task: ffff91c41d642580 task.stack: ffff9e7d4027c000 [ 0.459004] RIP: 0010:virtscsi_queuecommand_single+0x21/0x40 [virtio_scsi] [ 0.459004] RSP: 0018:ffff9e7d4027fa40 EFLAGS: 00010293 [ 0.459004] RAX: ffff91c41d57c000 RBX: ffff91c41d118148 RCX: 0000000000000000 [ 0.459004] RDX: ffff91c41d118148 RSI: ffff91c41d118148 RDI: ffff91c41d57c000 [ 0.459004] RBP: ffff9e7d4027fa40 R08: 0000000000000020 R09: 0000000000000000 [ 0.459004] R10: 0000000000000000 R11: ffff91c41d020ec8 R12: ffff91c41d57c000 [ 0.459004] R13: ffff91c41d57c000 R14: ffff91c41d118148 R15: ffff91c41d151050 [ 0.459004] FS: 0000000000000000(0000) GS:ffff91c41f000000(0000) knlGS:0000000000000000 [ 0.459004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.459004] CR2: 0000000000000004 CR3: 000000001d6f9000 CR4: 00000000003406f0 [ 0.459004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 0.459004] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 0.459004] Call Trace: [ 0.459004] scsi_dispatch_cmd+0xe6/0x220 [ 0.459004] scsi_request_fn+0x4c4/0x690 [ 0.459004] ? blk_rq_map_kern+0x140/0x140 [ 0.459004] __blk_run_queue+0x33/0x40 [ 0.459004] blk_execute_rq_nowait+0xd9/0x140 [ 0.459004] blk_execute_rq+0x54/0x90 [ 0.459004] scsi_execute+0x128/0x270 [ 0.459004] scsi_probe_and_add_lun+0x244/0xeb0 [ 0.459004] ? __pm_runtime_resume+0x5b/0x80 [ 0.459004] __scsi_scan_target+0xfa/0x630 [ 0.459004] scsi_scan_channel+0x66/0xa0 [ 0.459004] scsi_scan_host_selected+0xe7/0x150 [ 0.459004] do_scsi_scan_host+0x8d/0xa0 [ 0.459004] do_scan_async+0x1c/0x190 [ 0.459004] async_run_entry_fn+0x39/0x170 [ 0.459004] process_one_work+0x19e/0x470 [ 0.459004] worker_thread+0x4e/0x4a0 [ 0.459004] ? process_one_work+0x470/0x470 [ 0.459004] kthread+0x11e/0x140 [ 0.459004] ? kthread_park+0x90/0x90 [ 0.459004] ret_from_fork+0x2c/0x40 [ 0.459004] Code: ab 01 00 00 eb 92 0f 1f 40 00 0f 1f 44 00 00 55 48 8b 4e 30 48 89 f8 48 89 f2 48 89 e5 48 8b 89 88 01 00 00 48 8b 89 08 03 00 00 <3e> ff 41 04 48 8d bf c0 07 00 00 48 8d b0 c8 09 00 00 e8 28 fd [ 0.459004] RIP: virtscsi_queuecommand_single+0x21/0x40 [virtio_scsi] RSP: ffff9e7d4027fa40 [ 0.459004] CR2: 0000000000000004 [ 0.459004] ---[ end trace 84e3e2fe9b7b0530 ]--- kernel-4.11.0-0.rc1.git0.1.fc27.x86_64 qemu-2.8.0-2.fc26.x86_64
Still crashing in 4.11.0-0.rc2.git2.2.fc26.x86_64 [ 0.423754] virtio-pci 0000:00:02.0: PCI->APIC IRQ transform: INT A -> IRQ 10 [ 0.426540] scsi host2: Virtio SCSI HBA [ 0.427441] virtio-pci 0000:00:03.0: PCI->APIC IRQ transform: INT A -> IRQ 11 [ 0.433912] scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 2.5+ PQ: 0 ANSI: 5 [ 0.437220] general protection fault: 0000 [#1] SMP [ 0.437897] Modules linked in: virtio_pci(+) virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk virtio_ring virtio crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul [ 0.440994] CPU: 0 PID: 1 Comm: init Not tainted 4.11.0-0.rc2.git2.2.fc26.x86_64 #1 [ 0.442018] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1.fc26 04/01/2014 [ 0.443191] task: ffff94755ee0a580 task.stack: ffffb8f1c00c8000 [ 0.443993] RIP: 0010:vp_modern_find_vqs+0x39/0x70 [virtio_pci] [ 0.444793] RSP: 0018:ffffb8f1c00cb830 EFLAGS: 00010282 [ 0.445500] RAX: ffffb8f1c00fd000 RBX: 2d316f6974726976 RCX: 0000000000000000 [ 0.446452] RDX: 00000000000000fc RSI: ffffb8f1c00fd01c RDI: 0000000000000001 [ 0.447414] RBP: ffffb8f1c00cb850 R08: 000000000001c960 R09: ffffffff9041f129 [ 0.448371] R10: fffffbc30075c640 R11: 0000000000000000 R12: ffff94755d767708 [ 0.449323] R13: 0000000000000000 R14: ffff94755d767400 R15: 000000000000001f [ 0.450301] FS: 00007fff55262a70(0000) GS:ffff94755f000000(0000) knlGS:0000000000000000 [ 0.451390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.452160] CR2: 00007f2e9af19000 CR3: 000000001d6fa000 CR4: 00000000003406f0 [ 0.453120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 0.454073] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 0.455025] Call Trace: [ 0.455368] init_vqs+0x1a0/0x2e0 [virtio_console] [ 0.456016] virtcons_probe+0xb9/0x360 [virtio_console] [ 0.456724] virtio_dev_probe+0x144/0x1e0 [virtio] [ 0.457377] driver_probe_device+0x106/0x450 [ 0.457955] __device_attach_driver+0x83/0x100 [ 0.458566] ? __driver_attach+0xe0/0xe0 [ 0.459098] bus_for_each_drv+0x69/0xb0 [ 0.459638] __device_attach+0xdd/0x160 [ 0.460155] device_initial_probe+0x13/0x20 [ 0.460727] bus_probe_device+0x92/0xa0 [ 0.461245] device_add+0x448/0x680 [ 0.461721] ? vp_modern_find_vqs+0x70/0x70 [virtio_pci] [ 0.462440] ? vp_modern_find_vqs+0x70/0x70 [virtio_pci] [ 0.463153] device_register+0x1a/0x20 [ 0.463665] register_virtio_device+0xcc/0xf0 [virtio] [ 0.464361] virtio_pci_probe+0x94/0x110 [virtio_pci] [ 0.465040] local_pci_probe+0x42/0xa0 [ 0.465556] pci_device_probe+0xbe/0x150 [ 0.466087] driver_probe_device+0x106/0x450 [ 0.466672] __driver_attach+0xa4/0xe0 [ 0.467180] ? driver_probe_device+0x450/0x450 [ 0.467785] bus_for_each_dev+0x6e/0xb0 [ 0.468306] driver_attach+0x1e/0x20 [ 0.468796] bus_add_driver+0x1d0/0x270 [ 0.469317] ? 0xffffffffc0336000 [ 0.469789] driver_register+0x60/0xe0 [ 0.470297] ? 0xffffffffc0336000 [ 0.470752] __pci_register_driver+0x4c/0x50 [ 0.471334] virtio_pci_driver_init+0x1e/0x1000 [virtio_pci] [ 0.472090] do_one_initcall+0x50/0x1a0 [ 0.472616] ? free_hot_cold_page+0x19a/0x300 [ 0.473201] ? kmem_cache_alloc_trace+0x15f/0x1c0 [ 0.473838] ? do_init_module+0x27/0x1e6 [ 0.474369] do_init_module+0x5f/0x1e6 [ 0.474881] load_module+0x22b7/0x2820 [ 0.475390] ? __symbol_put+0x60/0x60 [ 0.475887] SYSC_init_module+0x16f/0x1a0 [ 0.476431] SyS_init_module+0xe/0x10 [ 0.476927] entry_SYSCALL_64_fastpath+0x1a/0xa9 [ 0.477548] RIP: 0033:0x400e03 [ 0.477962] RSP: 002b:00007fff55262948 EFLAGS: 00000217 ORIG_RAX: 00000000000000af [ 0.478967] RAX: ffffffffffffffda RBX: 0000000000007f4b RCX: 0000000000400e03 [ 0.479988] RDX: 0000000000403ba1 RSI: 0000000000007f4b RDI: 00007f2e9af12010 [ 0.480942] RBP: 0000000000007f4b R08: 00000000ffffffff R09: 0000000000000000 [ 0.481888] R10: 0000000000400e03 R11: 0000000000000217 R12: 00007f2e9af12010 [ 0.482836] R13: 0000000000000004 R14: 00007f2e9af1b010 R15: 0000000000000000 [ 0.483783] Code: 54 53 49 89 fe e8 78 0d 00 00 85 c0 41 89 c5 75 44 49 8b 9e 08 03 00 00 4d 8d a6 08 03 00 00 4c 39 e3 74 31 49 8b 86 38 03 00 00 <0f> b7 7b 28 48 8d 70 16 e8 ba cd 10 d0 49 8b 86 38 03 00 00 bf [ 0.486288] RIP: vp_modern_find_vqs+0x39/0x70 [virtio_pci] RSP: ffffb8f1c00cb830 [ 0.487280] ---[ end trace 3e840e9062b919f1 ]---
FWIW I finally reproduced this with the upstream kernel from git (093b995e3b55a). It's rarer than I thought, at least in my test. Probably happens about 1 in 10 times. Also there are *other* crashes in this kernel that happen, so it's not very stable in general. The stack trace is essentially the same as comment 4 so I won't reproduce it here.
Created attachment 1265489 [details] bisect.sh I bisected this to the following commit: 07ec51480b5eb1233f8c1b0f5d7a7c8d1247c507 is the first bad commit commit 07ec51480b5eb1233f8c1b0f5d7a7c8d1247c507 Author: Christoph Hellwig <hch> Date: Sun Feb 5 18:15:19 2017 +0100 virtio_pci: use shared interrupts for virtqueues This lets IRQ layer handle dispatching IRQs to separate handlers for the case where we don't have per-VQ MSI-X vectors, and allows us to greatly simplify the code based on the assumption that we always have interrupt vector 0 (legacy INTx or config interrupt for MSI-X) available, and any other interrupt is request/freed throught the VQ, even if the actual interrupt line might be shared in some cases. This allows removing a great deal of variables keeping track of the interrupt state in struct virtio_pci_device, as we can now simply walk the list of VQs and deal with per-VQ interrupt handlers there, and only treat vector 0 special. Additionally clean up the VQ allocation code to properly unwind on error instead of having a single global cleanup label, which is error prone, and in this case also leads to more code. Signed-off-by: Christoph Hellwig <hch> Signed-off-by: Michael S. Tsirkin <mst> :040000 040000 79a8267ffb73f9d244267c5f68365305bddd4696 8832a160b978710bbd24ba6966f462b3faa27fcc M drivers This is the very same commit that the kernel developers already pointed to in https://bugzilla.kernel.org/show_bug.cgi?id=194911#c0 I'm also attaching the bisection script that I used, partly so I remember what I did if I come back to this in future, but also because it might be of interest to others.
Laura Abbott posted a scratch build which reverts a large number of upstream virtio IRQ changes (see full list at the end of this comment). https://koji.fedoraproject.org/koji/taskinfo?taskID=18529971 I tested this scratch build on my Fedora 26 machine using a loop which runs libguestfs-test-tool 200 times, checking for the vp_modern_find_vqs bug (this bug) and other general hangs which I was also seeing. The vp_modern_find_vqs problem did NOT occur in any of the 200 iterations. There are still a few hangs. I believe there is still another bug lurking in the ISA serial driver (or maybe qemu) which is causing hangs. --------------------- Full list of reverts in Laura's build: 07ec51480b5e ("virtio_pci: use shared interrupts for virtqueues") is linked to a bunch of issues. Unfortunately we can't just revert it by itself. Revert it and dependency patches as well. Revert "virtio: provide a method to get the IRQ affinity mask for a virtqueue" This reverts commit bbaba479563910aaa51e59bb9027a09e396d3a3c. Revert "virtio-console: avoid DMA from stack" This reverts commit c4baad50297d84bde1a7ad45e50c73adae4a2192. Revert "vhost: introduce O(1) vq metadata cache" This reverts commit f889491380582b4ba2981cf0b0d7d6a40fb30ab7. Conflicts: drivers/vhost/vhost.c Revert "virtio_scsi: use virtio IRQ affinity" This reverts commit 0d9f0a52c8b9f7a003fe1650b7d5fb8518efabe0. Revert "virtio_blk: use virtio IRQ affinity" This reverts commit ad71473d9c43725c917fc5a86d54ceb7001ee28c. Revert "blk-mq: provide a default queue mapping for virtio device" This reverts commit 73473427bb551686e4b68ecd99bfd27e6635286a. Revert "virtio: allow drivers to request IRQ affinity when creating VQs" This reverts commit fb5e31d970ce8b4941f03ed765d7dbefc39f22d9. Revert "virtio_pci: simplify MSI-X setup" This reverts commit 52a61516125fa9a21b3bdf4f90928308e2e5573f. Revert "virtio_pci: don't duplicate the msix_enable flag in struct pci_dev" This reverts commit 53a020c661741f3b87ad3ac6fa545088aaebac9b. Revert "virtio_pci: use shared interrupts for virtqueues" This reverts commit 07ec51480b5eb1233f8c1b0f5d7a7c8d1247c507.
There's a smaller patch posted which fixes this issue for me: https://lkml.org/lkml/2017/3/23/38
*** Bug 1433899 has been marked as a duplicate of this bug. ***
*** Bug 1434462 has been marked as a duplicate of this bug. ***
Discussed at 2017-03-23 Fedora 26 Alpha Go/No-Go meeting, acting as a blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-meeting-2/2017-03-23/f26-alpha-go-no-go-meeting-2nd.2017-03-23-17.00.log.txt . Note: the bug discussed was actually #1433899, but we considered at the time that these three bugs were probably all the same and considered 1433899 a proxy for all of them. We agreed at the meeting that, given the increased prevalence of VM testing since the criteria were originally written, the criteria should be adjusted such that major bugs in VM guest functionality like this can block Alpha, and that this bug should be covered by such a criteria modification. Thus this is accepted as an Alpha blocker. We will draft the criterion change on test@ soon.
kernel-4.11.0-0.rc3.git0.2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-64210363f1
Seems to work fine in my F26 VM.
I followed instructions from https://bugzilla.redhat.com/show_bug.cgi?id=1434462#c17 with the kernel from comment 12. It works (it made 15 iterations without fail).
*** Bug 1431330 has been marked as a duplicate of this bug. ***
pschindl: could you file karma on the update? Thanks!
kernel-4.11.0-0.rc3.git0.2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-64210363f1
kernel-4.11.0-0.rc3.git0.2.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
*** Bug 1431940 has been marked as a duplicate of this bug. ***