Bug 1419572 - Can't install Fedora 25 in ppc64le/pseries guest
Summary: Can't install Fedora 25 in ppc64le/pseries guest
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 25
Hardware: ppc64le
OS: Linux
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2017-02-06 14:29 UTC by Andrea Bolognani
Modified: 2017-08-01 16:40 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-08-01 16:40:31 UTC
Type: Bug

Attachments (Terms of Use)

Description Andrea Bolognani 2017-02-06 14:29:19 UTC
I believe this to be a kernel bug rather than an anaconda
bug, but I leave up to you to reassign it accordingly. Also
see the end of the report.

Long story short, I can't install Fedora 25 in a
ppc64le/pseries guest because the kernel oops and crashes
the whole thing.

I'm using virt-install as following:

  $ virt-install \
    --name fedora25 \
    --location https://mirrors.nic.cz/fedora-secondary/releases/25/Server/ppc64le/os/ \
    --arch ppc64le \
    --vcpus 8,sockets=1,cores=1,threads=8 \
    --ram 2048 \
    --accelerate \
    --os-variant fedora23 \
    --graphics none \
    --network network=default,model=virtio \
    --disk size=5,bus=scsi \
    --controller scsi,model=virtio-scsi

When the (text mode) installer starts, I perform the
following steps:

  Set timezone: Europe/Prague
  Set destination: QEMU harddisk / Use All Space / LVM
  Set root password

Then start the installation. After a while - usually after

  Performing post-installation setup tasks
  Running in chroot, ignoring request.

but at least once it was earlier than that - I get a stack
trace like

  [  323.497171] Oops: Exception in kernel mode, sig: 5 [#1]
  [  323.499592] SMP NR_CPUS=1024 NUMA pSeries
  [  323.499685] Modules linked in: xfs fcoe libfcoe libfc scsi_transport_fc zram virtio_balloon loop virtio_console virtio_net virtio_scsi ghash_generic vmx_crypto virtio_pci virtio_ring virtio sunrpc xts lrw gf128mul dm_crypt dm_round_robin linear raid10 raid456 async_raid6_recov async_memcpy libcrc32c crc32c_vpmsum async_pq async_xor xor async_tx raid6_pq raid1 raid0 scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua iscsi_tcp libiscsi_tcp libiscsi squashfs cramfs scsi_transport_iscsi dm_multipath
  [  323.501698] CPU: 4 PID: 1280 Comm: loop0 Not tainted 4.8.6-300.fc25.ppc64le #1
  [  323.501812] task: c000000003ad9180 task.stack: c000000003b40000
  [  323.501905] NIP: c0000000003078f4 LR: c000000000307d3c CTR: c00000000030a300
  [  323.502018] REGS: c000000003b43380 TRAP: 0700   Not tainted  (4.8.6-300.fc25.ppc64le)
  [  323.502130] MSR: 800000010282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]>  CR: 28224824  XER: 00000000
  [  323.502525] CFAR: c000000000307d38 SOFTE: 1
  GPR00: c000000000307d3c c000000003b43600 c000000001228400 c00000006c014680
  GPR04: c000000065aa32a0 0000000000000003 0000000002010000 d000000003d8e608
  GPR08: ffffffffffffffff 0000000000000001 c00000006c0146b8 d000000003cf4708
  GPR12: c00000000030a300 c00000000fdc2400 c000000003b43af0 0000000000000000
  GPR16: 0000000000000000 0000000000000000 0000000000005e61 000000000242000a
  GPR20: 00000000000d0000 0000000000000000 c000000003b43870 0000000000000000
  GPR24: 0000000000005e67 0000000000005e61 00011b2800000001 c000000065aa32c8
  GPR28: c00000006c017500 c000000065aa32a0 0000000000000003 c00000006c014680
  [  323.504209] NIP [c0000000003078f4] remove_zspage+0x54/0xc0
  [  323.504291] LR [c000000000307d3c] fix_fullness_group+0xdc/0xf0
  [  323.504410] Call Trace:
  [  323.504460] [c000000003b43600] [c000000003b43640] 0xc000000003b43640 (unreliable)
  [  323.504598] [c000000003b43640] [c000000000307d3c] fix_fullness_group+0xdc/0xf0
  [  323.504732] [c000000003b43690] [c00000000030a3c4] zs_free+0xc4/0x1a0
  [  323.504849] [c000000003b436f0] [d000000003cf1d80] zram_free_page+0x58/0x120 [zram]
  [  323.505020] [c000000003b43730] [d000000003cf1ec0] zram_slot_free_notify+0x78/0x140 [zram]
  [  323.505156] [c000000003b43780] [c0000000002bd254] swap_slot_free_notify+0xa4/0xd0
  [  323.505287] [c000000003b437b0] [c0000000002bde54] swap_readpage+0x214/0x330
  [  323.505402] [c000000003b437f0] [c0000000002befac] read_swap_cache_async+0x5c/0x80
  [  323.505535] [c000000003b43850] [c0000000002bf130] swapin_readahead+0x160/0x2f0
  [  323.505668] [c000000003b438f0] [c00000000027728c] shmem_swapin+0x7c/0xb0
  [  323.505781] [c000000003b439f0] [c000000000278014] shmem_getpage_gfp+0x734/0xda0
  [  323.505916] [c000000003b43ad0] [c000000000279160] shmem_file_read_iter+0x140/0x460
  [  323.506049] [c000000003b43b70] [c000000000314760] vfs_iter_read+0xb0/0x150
  [  323.506165] [c000000003b43be0] [d0000000024b3e94] loop_queue_work+0x3fc/0xb30 [loop]
  [  323.506316] [c000000003b43d20] [c0000000000fc710] kthread_worker_fn+0x90/0x250
  [  323.506447] [c000000003b43d80] [c0000000000fc9e0] kthread+0x110/0x130
  [  323.506562] [c000000003b43e30] [c000000000009968] ret_from_kernel_thread+0x5c/0x74
  [  323.506692] Instruction dump:
  [  323.506753] 7cbe2b78 7c7f1b78 7c9d2378 60000000 60000000 7bc926e4 39290008 7d5f482a
  [  323.506948] 7d3f4a14 7d295278 7d290074 7929d182 <0b090000> e93d0000 7929b762 7d290034
  [  323.507140] ---[ end trace 96b6c07f3ffb86ab ]---

and the installation is stuck. I can still switch between
tabs using Ctrl+b 1 etc, but the shell is unusable and the
logs don't seem to provide any interesting information.

One more hint that the issue is most likely in the kernel:
I have created, on the same hardware, a Fedora 25 guest
using virt-builder, and when I boot it with the 4.8.6-300
kernel (same one the installer is using AFAICT) it doesn't
boot (can't find the LVs), but both 4.8.8-300 and 4.9.4-201
work fine.

So I guess the issue was in the kernel and it has already
been fixed; however, that doesn't make life any easier for
people trying to install Fedora 25 on ppc64le, since the
kernel shipped with the installation media is affected by
the issue.

Is there any chance of respinning the installation media
with a fixed kernel? Or are people going to be forced to
install Fedora 24 and upgrade from it?

Comment 1 Tobias Hodges 2017-07-27 22:59:46 UTC
Indeed it seems to be a kernel bug, and if you are OK with it, not installing zram seemed to function as a workaround for us. 

Using QEMU Builder for Packer:

"boot_command": [
        "linux /ppc/ppc64/vmlinuz ro ",
        "ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/fedora-25/ks-ppc64le-openstack.cfg ",
        "inst.zram=0 <enter>",  ** <--- key addition **
        "initrd /ppc/ppc64/initrd.img<enter>",

We are now able to pass the point of failure from before and the installation continues successfully.

Screenshot of original exception: http://imgur.com/a/sduXy

Source of solution: https://docs.fedoraproject.org/en-US/Fedora//html/Installation_Guide/sect-boot-options-advanced.html

Information on zram: https://www.kernel.org/doc/Documentation/blockdev/zram.txt

Comment 2 Andrea Bolognani 2017-08-01 10:02:30 UTC
FWIW, the issue seem to have been resolved in Fedora 26.

Comment 3 Laura Abbott 2017-08-01 16:40:31 UTC
Thanks for letting us know. I'm going to close the bug, feel free to reopen if the problem shows up again,

Note You need to log in before you can comment on or make changes to this bug.