RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2167465 - Starting QEMU/KVM machine on RHEL 7 Kernel 3.10.0-1160.8X Causes Crash.
Summary: Starting QEMU/KVM machine on RHEL 7 Kernel 3.10.0-1160.8X Causes Crash.
Keywords:
Status: CLOSED DUPLICATE of bug 2143438
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.9
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Amnon Ilan
QA Contact: Chao Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-02-06 17:17 UTC by Bernie Hoefer
Modified: 2023-02-08 00:55 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-02-07 15:02:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-147660 0 None None None 2023-02-06 17:19:36 UTC

Description Bernie Hoefer 2023-02-06 17:17:47 UTC 
Description of problem:
My HP ProLiant DL380 G6 running RHEL 7 crashes when I start a QEMU/KVM virtual machine.


Version-Release number of selected component (if applicable):
The problem started with 3.10.0-1160.80.1.el7.x86_64.  Kernels 3.10.0-1160.81.1.el7.x86_64 and 3.10.0-1160.83.1.el7.x86_64 exhibit the exact same problem.
Kernel 3.10.0-1160.76.1.el7.x86_64 was the last kernel to work.


How reproducible:
Boot the server into a 3.10.0-1160.8X kernel.
Use laptop's Virtual Machine Manager to connect to the Libvirt daemon running on the server.  Right-click a virtual machine and choose "Run".


Actual results:
The server's kernel immediately crashes.


Expected results:
The server should continue running as normal.
Comment 3 Bernie Hoefer 2023-02-06 17:21:18 UTC 
This is the output when a QEMU/KVM machine is started under kernel-3.10.0-1160.83.1.el7.x86_64:


systemd[1]: Listening on Virtual machine log manager socket.
systemd[1]: Started Virtual machine log manager.
kernel: virbr1: port 2(vnet0) entered blocking state
kernel: virbr1: port 2(vnet0) entered disabled state
kernel: device vnet0 entered promiscuous mode
kernel: virbr1: port 2(vnet0) entered blocking state
kernel: virbr1: port 2(vnet0) entered listening state
NetworkManager[908]: <info>  [1675702616.6790] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/11)
NetworkManager[908]: <info>  [1675702616.6868] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.6912] keyfile: add connection /var/run/NetworkManager/system-connections/vnet0.nmconnection (43749f34-15fc-4989-9972-aa7c97d8e06c,"vnet0")
NetworkManager[908]: <info>  [1675702616.6922] device (vnet0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.6933] device (vnet0): Activation: starting connection 'vnet0' (43749f34-15fc-4989-9972-aa7c97d8e06c)
NetworkManager[908]: <info>  [1675702616.6934] device (vnet0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.6943] device (vnet0): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.6949] device (vnet0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.6952] device (virbr1): bridge port vnet0 was attached
NetworkManager[908]: <info>  [1675702616.6952] device (vnet0): Activation: connection 'vnet0' enslaved, continuing activation
NetworkManager[908]: <info>  [1675702616.6955] device (vnet0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.6966] device (vnet0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.6971] device (vnet0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
NetworkManager[908]: <info>  [1675702616.7210] device (vnet0): Activation: successful, device activated.
dbus[843]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
systemd[1]: Starting Network Manager Script Dispatcher Service...
dbus[843]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
systemd[1]: Started Network Manager Script Dispatcher Service.
nm-dispatcher[1813]: req:1 'up' [vnet0]: new request (4 scripts)
nm-dispatcher[1813]: req:1 'up' [vnet0]: start running ordered scripts...
systemd-machined[841]: New machine qemu-1-rhel86-002.
systemd[1]: Started Virtual Machine qemu-1-rhel86-002.
systemd[1]: Unit iscsi.service cannot be reloaded because it is inactive.
kvm[1838]: 1 guest now active
kernel: int3: 0000 [#1] SMP 
kernel: Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 bridge nf_defrag_ipv6 nf_nat_ipv6 stp ip6table_mangle llc ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support intel_powerclamp gpio_ich coretemp kvm_intel kvm irqbypass ipmi_ssif pcspkr lpc_ich hpilo hpwdt joydev sg i7core_edac ipmi_si pcc_cpufreq ipmi_devintf ipmi_msghandler acpi_power_meter ip_tables xfs libcrc32c radeon sd_mod crc_t10dif
kernel:  crct10dif_generic crct10dif_common i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel drm_panel_orientation_quirks hpsa serio_raw bnx2 scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod
kernel: CPU: 6 PID: 1841 Comm: qemu-kvm Tainted: G          I    ------------   3.10.0-1160.83.1.el7.x86_64 #1
kernel: Hardware name: HP ProLiant DL380 G6, BIOS P62 08/16/2015
kernel: task: ffff976d83ae4200 ti: ffff97678638c000 task.ti: ffff97678638c000
kernel: RIP: 0010:[<ffffffffc0813f55>]  [<ffffffffc0813f55>] setno+0x5/0x10 [kvm]
kernel: RSP: 0018:ffff97678638fc18  EFLAGS: 00000202
kernel: RAX: 0000000000000200 RBX: ffff976d786698c0 RCX: 000301001a242000
kernel: RDX: ffffffffc0813f54 RSI: 0000000000000000 RDI: ffff976d786698c0
kernel: RBP: ffff97678638fc48 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000001ae0 R11: ffff9767864f0008 R12: ffffffffc0834240
kernel: R13: 0000000000000006 R14: ffff976d78668000 R15: 0000000000000000
kernel: FS:  00007fc56b0f1700(0000) GS:ffff976787ac0000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000000 CR3: 0000000c06f48000 CR4: 00000000000027e0
kernel: Call Trace:
kernel:  [<ffffffffc081ff35>] ? x86_emulate_insn+0x635/0xd30 [kvm]
kernel:  [<ffffffffc07fe9d4>] x86_emulate_instruction+0x1e4/0x720 [kvm]
kernel:  [<ffffffffc088f689>] vmx_handle_exit+0x1f9/0xc00 [kvm_intel]
kernel:  [<ffffffffc087fc4d>] ? vmx_set_cr3+0xbd/0x190 [kvm_intel]
kernel:  [<ffffffffc08252a5>] ? kvm_apic_has_interrupt+0x45/0xa0 [kvm]
kernel:  [<ffffffffc07fa910>] vcpu_enter_guest+0x770/0x1470 [kvm]
kernel:  [<ffffffffc07fecb4>] ? x86_emulate_instruction+0x4c4/0x720 [kvm]
kernel:  [<ffffffffaaa4bb42>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
kernel:  [<ffffffffc0802a18>] kvm_arch_vcpu_ioctl_run+0x358/0x480 [kvm]
kernel:  [<ffffffffc07e3239>] kvm_vcpu_ioctl+0x2d9/0x700 [kvm]
kernel:  [<ffffffffaaa71458>] do_vfs_ioctl+0x3a8/0x5c0
kernel:  [<ffffffffaaa75838>] ? dput+0x68/0x1b0
kernel:  [<ffffffffaaa716f1>] SyS_ioctl+0x81/0xa0
kernel:  [<ffffffffaa9467e6>] ? __audit_syscall_exit+0x1f6/0x2b0
kernel:  [<ffffffffaafc539a>] system_call_fastpath+0x25/0x2a
kernel: Code: 00 00 48 85 ff 74 0a 55 48 89 e5 e8 46 78 0b ea 5d c3 cc cc cc cc 0f 90 c0 c3 cc cc cc cc cc cc cc cc cc cc cc cc 0f 91 c0 c3 cc <cc> cc cc cc cc cc cc cc cc cc cc 0f 92 c0 c3 cc cc cc cc cc cc 
kernel: RIP  [<ffffffffc0813f55>] setno+0x5/0x10 [kvm]
kernel:  RSP <ffff97678638fc18>
kernel: ---[ end trace fd1c470c579bcf99 ]---




This is the output when started under kernel-3.10.0-1160.76.1.el7.x86_64:


systemd[1]: Listening on Virtual machine log manager socket.
systemd[1]: Started Virtual machine log manager.
kernel: virbr1: port 2(vnet0) entered blocking state
kernel: virbr1: port 2(vnet0) entered disabled state
kernel: device vnet0 entered promiscuous mode
kernel: virbr1: port 2(vnet0) entered blocking state
kernel: virbr1: port 2(vnet0) entered listening state
NetworkManager[919]: <info>  [1675702183.5437] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/11)
NetworkManager[919]: <info>  [1675702183.5480] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5516] keyfile: add connection /var/run/NetworkManager/system-connections/vnet0.nmconnection (b2a12edf-29b8-40a6-a2b5-9f9d1e390fb4,"vnet0")
NetworkManager[919]: <info>  [1675702183.5524] device (vnet0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5535] device (vnet0): Activation: starting connection 'vnet0' (b2a12edf-29b8-40a6-a2b5-9f9d1e390fb4)
NetworkManager[919]: <info>  [1675702183.5537] device (vnet0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5544] device (vnet0): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5548] device (vnet0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5551] device (virbr1): bridge port vnet0 was attached
NetworkManager[919]: <info>  [1675702183.5551] device (vnet0): Activation: connection 'vnet0' enslaved, continuing activation
NetworkManager[919]: <info>  [1675702183.5554] device (vnet0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5564] device (vnet0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5567] device (vnet0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
NetworkManager[919]: <info>  [1675702183.5766] device (vnet0): Activation: successful, device activated.
dbus[834]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
systemd[1]: Starting Network Manager Script Dispatcher Service...
dbus[834]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.machine1.service'
dbus[834]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
systemd[1]: Started Network Manager Script Dispatcher Service.
nm-dispatcher[24582]: req:1 'up' [vnet0]: new request (4 scripts)
nm-dispatcher[24582]: req:1 'up' [vnet0]: start running ordered scripts...
systemd[1]: Starting Virtual Machine and Container Registration Service...
systemd[1]: Unit iscsi.service cannot be reloaded because it is inactive.
dbus[834]: [system] Successfully activated service 'org.freedesktop.machine1'
systemd[1]: Started Virtual Machine and Container Registration Service.
systemd-machined[24590]: New machine qemu-1-rhel86-002.
systemd[1]: Started Virtual Machine qemu-1-rhel86-002.
kvm[24608]: 1 guest now active
kernel: virbr1: port 2(vnet0) entered learning state
kernel: virbr1: port 2(vnet0) entered forwarding state
kernel: virbr1: topology change detected, propagating
kernel: IPv6: ADDRCONF(NETDEV_CHANGE): virbr1: link becomes ready
NetworkManager[919]: <info>  [1675702187.5848] device (virbr1): carrier: link connected
Comment 4 Bernie Hoefer 2023-02-06 17:26:44 UTC 
I do not currently have kdump enabled on this machine, but can set it up and provide a vmcore, if needed.  (I figured this might already be a known issue since it started happening when kernel-3.10.0-1160.80.1.el7 was released via RHSA-2022:7337 on 2022-11-02 -- but could not find a previously submitted Bugzilla ticket.)
Comment 5 John Ferlan 2023-02-07 12:51:35 UTC 
Is this a customer related issue or dependent product issue? Unclear on the expectation here.  The more information provided the better.

As noted by the bot, RHEL7 is old and not actively watched/developed/used/tested (pick your favorite). The only recent issue I'm aware of is bug 2143438, but that's a hang not a crash.

Assigning to Amnon as group director to allow him to sort out who to assign this to.
Comment 7 Bernie Hoefer 2023-02-07 15:02:13 UTC 
(In reply to John Ferlan from comment #5)
===
> Is this a customer related issue or dependent product issue?
===

No, this is just something I noticed on my own server.  Due to the holidays, other priorities and the fact that it has remained broken in all .8X versions of the 3.10.0-1160.8X kernel -- I just got around to reporting this, yesterday.


===
> The only recent issue I'm aware of is bug 2143438, but that's a hang not a crash.
===

Perhaps my terminology was wrong.  Even looking at:

  https://en.wikipedia.org/wiki/Hang_(computing)

  https://en.wikipedia.org/wiki/Crash_(computing)#Operating_system_crashes

...I'm still not 100% sure of the difference.  Either way, in my situation the server becomes totally unresponsive:  it becomes unresponsive over the network; the system console just displays the kernel trace and is unresponsive to keyboard input, including Ctrl+Alt+F# keys to switch to a different TTY.  The server is not configured for Alt+SysRq key presses, so I do not know if they would work or not.

Looking at BZ 2143438, it looks exactly like my problem.  Therefore, I am closing this Bugzilla ticket (BZ 2167465) as a duplicate of it.  Thank you!

*** This bug has been marked as a duplicate of bug 2143438 ***
Note You need to log in before you can comment on or make changes to this bug.