RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1523083 - [s390x] Qemu process terminated while booting with additional module enabled by mkinitrd
Summary: [s390x] Qemu process terminated while booting with additional module enabled ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-ma
Version: 7.5-Alt
Hardware: s390x
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Cornelia Huck
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-07 08:14 UTC by Zhengtong
Modified: 2017-12-08 08:34 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-08 08:34:02 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Zhengtong 2017-12-07 08:14:39 UTC 
Description of problem:
Reboot failed with new initramfs image file generated by mkinitrd enabling new kernel modules 

Version-Release number of selected component (if applicable):

qemu-kvm-ma-2.10.0-11.el7
host kernel:4.14.0-15.el7a.s390x
guest kernel:4.14.0-15.el7a.s390x

How reproducible:
3/3

Steps to Reproduce:
1.Boot up guest with the kernel 4.14.0-15.el7a.s390x, or lower version than -15

2.In guest, update kernel if the version is lower than -15.

3.Regenerate the initramfs file by:
#mkinitrd -f /boot/initramfs-4.14.0-15.el7a.s390x.img --with=virtio_blk 4.14.0-15.el7a.s390x
Aims to load the virtio_blk module while guest boot up.

4.Reboot the guest


Actual results:
Guest crash/qemu terminated
------------------------------------------------------------------------
Rebooting.
LOADPARM=[........]
Using virtio-scsi.
Using SCSI scheme.
.....
Uncompressing Linux... Ok, booting the kernel.
[    0.083018] Linux version 4.14.0-15.el7a.s390x (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-25) (GCC)) #1 SMP Tue Dec 5 16:22:12 EST 2017
[    0.083023] setup: Linux is running under KVM in 64-bit mode
[    0.083181] setup: The maximum memory size is 1024MB
[    0.083206] cpu: 1 configured CPUs, 0 standby CPUs
[    0.083290] Write protected kernel read-only data: 9476k
[    0.085778] Zone ranges:
[    0.085780]   DMA      [mem 0x0000000000000000-0x000000007fffffff]
[    0.085782]   Normal   empty
[    0.085783] Movable zone start for each node
[    0.085784] Early memory node ranges
[    0.085785]   node   0: [mem 0x0000000000000000-0x000000003fffffff]
[    0.085787] Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
[    0.086902] random: fast init done
[    0.086929] percpu: Embedded 23 pages/cpu @000000003ffbd000 s56320 r8192 d29696 u94208
[    0.086950] Built 1 zonelists, mobility grouping on.  Total pages: 258048
[    0.086952] Kernel command line: root=/dev/mapper/rhelaa-root console=tty0 console=ttyS0,115200 crashkernel=auto rd.lvm.lv=rhelaa/root rd.lvm.lv=rhelaa/swap biosdevname=0 net.ifnames=0 LANG=en_US.UTF-8
[    0.087826] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.087861] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.087878] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[    0.101439] Memory: 977024K/1048576K available (6184K kernel code, 879K rwdata, 3288K rodata, 608K init, 696K bss, 71552K reserved, 0K cma-reserved)
[    0.101474] SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.101476] ftrace: allocating 21425 entries in 84 pages
[    0.109289] Hierarchical RCU implementation.
[    0.109291] 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[    0.109292] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.110876] NR_IRQS: 3, nr_irqs: 3, preallocated irqs: 3
[    0.110898] clocksource: tod: mask: 0xffffffffffffffff max_cycles: 0x3b0a9be803b0a9, max_idle_ns: 1805497147909793 ns
[    0.110962] console [ttyS0] enabled
[    0.110986] console [ttyS1] enabled
[    0.111005] pid_max: default: 32768 minimum: 301
[    0.111020] Security Framework initialized
[    0.111023] SELinux:  Initializing.
[    0.111037] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.111041] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes)
[    0.111249] Hierarchical SRCU implementation.
[    0.111405] smp: Bringing up secondary CPUs ...
[    0.111406] smp: Brought up 1 node, 1 CPU
[    0.111542] devtmpfs: initialized
[    0.111652] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.111656] futex hash table entries: 512 (order: 5, 131072 bytes)
[    0.111743] NET: Registered protocol family 16
[    0.112214] HugeTLB registered 1.00 MiB page size, pre-allocated 0 pages
[    0.112362] SCSI subsystem initialized
[    0.112494] NetLabel: Initializing
[    0.112496] NetLabel:  domain hash size = 128
[    0.112497] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.112515] NetLabel:  unlabeled traffic allowed by default
[    0.125475] VFS: Disk quotas dquot_6.6.0
[    0.125484] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.125608] NET: Registered protocol family 2
[    0.125695] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[    0.125721] TCP bind hash table entries: 8192 (order: 5, 131072 bytes)
[    0.125763] TCP: Hash tables configured (established 8192 bind 8192)
[    0.125788] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.125795] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.125821] NET: Registered protocol family 1
[    0.125852] Unpacking initramfs...
[    0.629407] Initramfs unpacking failed: read error
[    0.634195] Freeing initrd memory: 41064K
[    0.634600] hypfs: The hardware system does not support hypfs
[    0.634616] hypfs: Initialization of hypfs failed with rc=-61
[    0.634751] audit: initializing netlink subsys (disabled)
[    0.634860] Initialise system trusted keyrings
[    0.634874] audit: type=2000 audit(1512632655.682:1): state=initialized audit_enabled=0 res=1
[    0.634884] workingset: timestamp_bits=46 max_order=18 bucket_order=0
[    0.686382] NET: Registered protocol family 38
[    0.686386] Key type asymmetric registered
[    0.686388] Asymmetric key parser 'x509' registered
[    0.686410] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.686427] io scheduler noop registered
[    0.686429] io scheduler deadline registered (default)
[    0.686453] io scheduler cfq registered
[    0.686454] io scheduler mq-deadline registered
[    0.686455] io scheduler kyber registered
[    0.686495] atomic64_test: passed
[    0.686526] hvc_iucv: The z/VM IUCV HVC device driver cannot be used without z/VM
[    0.686640] rdac: device handler registered
[    0.686652] hp_sw: device handler registered
[    0.686653] emc: device handler registered
[    0.686662] alua: device handler registered
[    0.686676] cio: Channel measurement facility initialized using format extended (mode autodetected)
[    0.686747] ap: The hardware system does not support AP instructions
[    0.686770] drop_monitor: Initializing network drop monitor service
[    0.686822] Initializing XFRM netlink socket
[    0.686863] NET: Registered protocol family 10
[    0.687339] Segment Routing with IPv6
[    0.687352] NET: Registered protocol family 17
[    0.687422] registered taskstats version 1
[    0.687424] Loading compiled-in X.509 certificates
[    0.687485] alg: No test for pkcs1pad(rsa,sha1) (pkcs1pad(rsa-generic,sha1))
[    0.688058] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: f8cf00e3232b2bc52bbfe321480ed4b18f1e19f4'
[    0.688238] Key type big_key registered
[    0.688242] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[    0.688286] md: Waiting for all devices to be available before autodetect
[    0.688287] md: If you don't use raid, use raid=noautodetect
[    0.688369] md: Autodetecting RAID arrays.
[    0.688371] md: autorun ...
[    0.688372] md: ... autorun DONE.
[    0.688384] List of all partitions:
[    0.688386] No filesystem could mount root, tried: 
[    0.688386] 
[    0.688389] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.688392] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.14.0-15.el7a.s390x #1
[    0.688393] Hardware name: IBM 2827 H43 400 (KVM/Linux)
[    0.688395] Call Trace:
[    0.688400] ([<00000000001138ba>] show_stack+0x62/0x78)
[    0.688405]  [<00000000006e90a6>] dump_stack+0x7e/0xb0 
[    0.688407]  [<00000000001404f4>] panic+0xfc/0x240 
[    0.688411]  [<0000000000b1e2e2>] mount_block_root+0x2d2/0x2e0 
[    0.688420]  [<0000000000b1e500>] prepare_namespace+0x180/0x1c0 
[    0.688422]  [<0000000000b1de3a>] kernel_init_freeable+0x26a/0x290 
[    0.688425]  [<00000000006fd0ca>] kernel_init+0x2a/0x138 
[    0.688428]  [<0000000000706d3a>] kernel_thread_starter+0x6/0xc 
[    0.688430]  [<0000000000706d34>] kernel_thread_starter+0x0/0xc 

Ncat: Broken pipe.
------------------------------------------------------------------------

Expected results:
Guest could be restarted and works well

Additional info:
Comment 2 Zhengtong 2017-12-07 08:17:56 UTC 
The issue happened on s390x platform with kernel:4.14.0-15.el7a.s390x


4.14.0-15.el7a.s390x  broken
4.14.0-14.el7a.s390x  good
4.14.0-12.el7a.s390x  good
4.14.0-8.el7a.s390x   good

So this bug is a regression.
Comment 4 Cornelia Huck 2017-12-07 08:33:08 UTC 
(In reply to Zhengtong from comment #2)
> The issue happened on s390x platform with kernel:4.14.0-15.el7a.s390x
> 
> 
> 4.14.0-15.el7a.s390x  broken
> 4.14.0-14.el7a.s390x  good
> 4.14.0-12.el7a.s390x  good
> 4.14.0-8.el7a.s390x   good
> 
> So this bug is a regression.

I assume this refers to the guest kernel?
Comment 5 Zhengtong 2017-12-07 08:51:24 UTC 
(In reply to Cornelia Huck from comment #4)

> I assume this refers to the guest kernel?

yes, It is. and the host kernel is always: 4.14.0-15.el7a.s390x
Comment 6 Zhengtong 2017-12-07 10:25:02 UTC 
oh...wait.  I think I can still hit this issue on -14 and -13 now.  I will do more test on the different versions. update them later.
Comment 7 Cornelia Huck 2017-12-07 10:43:43 UTC 
Just looked at your boot messages again. It seems you boot from a virtio-scsi disk, but add virtio-blk to your initrd. Does this reproduce if you add virtio-scsi as well?
Comment 8 Zhengtong 2017-12-07 11:17:16 UTC 
(In reply to Cornelia Huck from comment #7)
> Just looked at your boot messages again. It seems you boot from a
> virtio-scsi disk, but add virtio-blk to your initrd. Does this reproduce if
> you add virtio-scsi as well?

I tried with "virtio_net" and "virtio_scsi" module, Both  hit this issue, two.


In my latest test result. 

This issue happened on 4.14.0-9.el7a.s390x, 
            but not on 4.14.0-8.el7a.s390x.
Comment 9 Thomas Huth 2017-12-07 11:55:07 UTC 
Could you please provide the full console output of mkinitrd for the case where it is failing?
Comment 10 David Hildenbrand 2017-12-07 15:21:18 UTC 
Just to make sure:

a) you're booting into kernel < 4.14.0-15.el7a.s390x
b) you update the kernel to 4.14.0-15.el7a.s390x via rpm/yum
c) you generate an initrd manually
d) you reboot

Can you answer:

1. you need to generate the initrd manually shouldn't there get one generated automatically? Why is that necessary?

2. do you run "zipl" _after_ you generated the initrd? Otherwise this might be the problem.

Can you also provide your QEMU command line? (or libvirt xml)

Thanks!
Comment 11 Zhengtong 2017-12-08 02:50:51 UTC 
(In reply to Thomas Huth from comment #9)
> Could you please provide the full console output of mkinitrd for the case
> where it is failing?

mkinitrd application has no any std output while running .

And the console output of guest after ejecting "reboot" is in comment #0
Comment 12 Zhengtong 2017-12-08 03:20:11 UTC 
(In reply to David Hildenbrand from comment #10)
> Just to make sure:
> 
> a) you're booting into kernel < 4.14.0-15.el7a.s390x
> b) you update the kernel to 4.14.0-15.el7a.s390x via rpm/yum
> c) you generate an initrd manually
> d) you reboot
> 
> Can you answer:
> 
> 1. you need to generate the initrd manually shouldn't there get one
> generated automatically? Why is that necessary?
Actually, this test scenario comes from a autotest code. The new initrd should  install all the  virtio drivers, and prepare a easy ENV to do test.
Yes , there is already a initrd in directory "/boot/" before I making one.

> 
> 2. do you run "zipl" _after_ you generated the initrd? Otherwise this might
> be the problem.
Oh..no.  
I tried with "zipl" after re-making inird just now. the issue disappeared..
Thanks very much.
> 
> Can you also provide your QEMU command line? (or libvirt xml)
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine s390-ccw-virtio  \
    -nodefaults  \
    -vga none  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_ut7GP4/monitor-qmpmonitor1-20171115-020901-7Fni7fsG,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_ut7GP4/monitor-catch_monitor-20171115-020901-7Fni7fsG,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_ut7GP4/serial-serial0-20171115-020901-7Fni7fsG,server,nowait \
    -device sclpconsole,chardev=serial_id_serial0 \
    -device virtio-scsi-ccw,id=virtio_scsi_ccw0 \
    -drive file=/home/zhengtli/root_dir/ALT-Server-7.4-s390x-virtio-scsi.qcow2,id=drive0,if=none,format=qcow2,snapshot=on \
    -device scsi-hd,bus=virtio_scsi_ccw0.0,drive=drive0,id=hd1 \
    -netdev tap,id=hostnet0 \
    -device virtio-net-ccw,netdev=hostnet0,id=net0 \
    -device diag288,id=wdt0 \
    -m 1024  \
    -smp 1,maxcpus=1,cores=1,threads=1,sockets=1  \
    -cpu 'host'  \
    -vnc :1  \
    -rtc base=utc,clock=host,driftfix=slew \
    -boot strict=on \
    -monitor stdio \

> 
> Thanks!
Comment 13 Zhengtong 2017-12-08 03:23:19 UTC 
Based on comment #12.  

We need to to "zipl" to update the boot behaviors after re-making initrd.
Maybe we can close this bug now.

But I have a little doubt that why no need to do this(zipl) for kernels whose version is lower than 4.14.0-8.el7a.s390x
Comment 14 Thomas Huth 2017-12-08 06:48:56 UTC 
(In reply to Zhengtong from comment #13)
> Based on comment #12.  
> 
> We need to to "zipl" to update the boot behaviors after re-making initrd.
> Maybe we can close this bug now.

Likely yes.

> But I have a little doubt that why no need to do this(zipl) for kernels
> whose version is lower than 4.14.0-8.el7a.s390x

I think that was just by accident. If there was already a ramdisk before making a new one with mkinitrd, the new might overwrite the old one on the disk at the very same location (i.e. the very same sectors on the disk). So the zipl boot code then could load the new ramdisk from the old location. But as soon as one sector location changes, the ramdisk will be loaded corrupted when you did not run "zipl" before the reboot again.
Comment 15 Zhengtong 2017-12-08 07:17:44 UTC 
(In reply to Thomas Huth from comment #14)

> 
> > But I have a little doubt that why no need to do this(zipl) for kernels
> > whose version is lower than 4.14.0-8.el7a.s390x
> 
> I think that was just by accident. If there was already a ramdisk before
> making a new one with mkinitrd, the new might overwrite the old one on the
> disk at the very same location (i.e. the very same sectors on the disk). So
> the zipl boot code then could load the new ramdisk from the old location.
> But as soon as one sector location changes, the ramdisk will be loaded
> corrupted when you did not run "zipl" before the reboot again.

sounds reasonable, thanks
Comment 16 Cornelia Huck 2017-12-08 08:28:07 UTC 
OK, from the comments above, this is a test case bug (needs to include an invocation of zipl)?

If you agree, I'll close this.
Comment 17 Zhengtong 2017-12-08 08:31:46 UTC 
(In reply to Cornelia Huck from comment #16)
> OK, from the comments above, this is a test case bug (needs to include an
> invocation of zipl)?
> 
> If you agree, I'll close this.

Go ahead
Note You need to log in before you can comment on or make changes to this bug.