Moving this to qemu-kvm for now, but I am not sure if this is a qemu-kvm-rhev or kernel issue.

This is affecting (at least) RHEV-H so requesting 6.5.z

Hi,

The first thing that comes to mind when I see a soft lockup like this is that you are trying to run too many vCPUs given the number of physical cores you have in the hardware.  Could you share some more information about your configuration:

1. /proc/cpuinfo on the host
2. Details about the VM load on the host 
  a. how many vms
  b. how many vCPUs per vm
  c. how much memory assigned to each VM
  d. workload that is driving memory consumption inside the VMs
3. /var/log/vdsm/mom.log around the time of the crash so I can see the ksm settings that are being activated.

RHEV-H QE got the following conclusions after did three different scenarios. 

Test version:
rhev-hypervisor6-6.5-20140624.0.el6ev
ovirt-node-3.0.1-18.el6_5.11.noarch
vdsm-4.14.7-3.el6ev.x86_64
RHEVM av10


Test scenario 1 (Host: RHEV-H):
Run lots of VMs to full of memory for trigger KSM activation.

Test result:
ksm server can start automatic and RHEV-H will no crash.


Test scenario 2 (Host: RHEV-H):
1. Create a VM and made the memory more closely to the host(e.g. The memory of host= 48G, then set the Memory of VM to 48G)
2. Run eatmemory script on VM for run out of memory

Test result:
RHEV-H crashes.


Test scenario 3 (Host: RHEL):
Do the same steps with scenario 2 on RHEL host.

Test result:
1. The process of eatmemory will be killed automatic.
2. Host(RHEL) will not crash.

So the crash only occurs on RHEV-H but no RHEL.
You can find the script and crash.png in the attachment.

Thanks!

Created attachment 917023 [details]
eatmemory

Created attachment 917024 [details]
RHEV-H-crash.png

Hey Ying,

thanks for the excessive testing. Pelase provide some more details (see inline).

(In reply to shaochen from comment #6)
> Test scenario 2 (Host: RHEV-H):
> 1. Create a VM and made the memory more closely to the host(e.g. The memory
> of host= 48G, then set the Memory of VM to 48G)
> 2. Run eatmemory script on VM for run out of memory
> 
> Test result:
> RHEV-H crashes.

Please provide the output of `free -m` , or some details on how much swap was avilable.

> Test scenario 3 (Host: RHEL):
> Do the same steps with scenario 2 on RHEL host.
> 
> Test result:
> 1. The process of eatmemory will be killed automatic.
> 2. Host(RHEL) will not crash.

The same as above - please proved `free -m` or some details on the swap available.

> So the crash only occurs on RHEV-H but no RHEL.
> You can find the script and crash.png in the attachment.

IIUIC then it's quite normal that memory hogs get killed in cases that the kernel runs out of memory.

> Please provide the output of `free -m` , or some details on how much swap
> was avilable.
> 

=====================================================
[root@ibm-x3650m3-01 admin]# free -m swap
             total       used       free     shared    buffers     cached
Mem:         48259       4505      43754          0         30        195
-/+ buffers/cache:       4278      43981
Swap:        32323        868      31455
[root@ibm-x3650m3-01 admin]# free -m swap
             total       used       free     shared    buffers     cached
Mem:         48259       9817      38442          0         30        195
-/+ buffers/cache:       9591      38668
Swap:        32323        868      31455
[root@ibm-x3650m3-01 admin]# free -m swap
             total       used       free     shared    buffers     cached
Mem:         48259      22919      25340          0         30        196
-/+ buffers/cache:      22692      25567
Swap:        32323        867      31456
[root@ibm-x3650m3-01 admin]# free -m swap
             total       used       free     shared    buffers     cached
Mem:         48259      36447      11812          0         30        196
-/+ buffers/cache:      36220      12039
Swap:        32323        864      31459
[root@ibm-x3650m3-01 admin]# free -m swap
             total       used       free     shared    buffers     cached
Mem:         48259      46712       1547          0         30        196
-/+ buffers/cache:      46485       1774
Swap:        32323        864      31459
[root@ibm-x3650m3-01 admin]# free -m swap
             total       used       free     shared    buffers     cached
Mem:         48259      47990        269          0         22        148
-/+ buffers/cache:      47819        440
Swap:        32323       1137      31186

The scenario 2's error is different with original bug error.


> > Test scenario 3 (Host: RHEL):
> > Do the same steps with scenario 2 on RHEL host.
> > 
> > Test result:
> > 1. The process of eatmemory will be killed automatic.
> > 2. Host(RHEL) will not crash.
> 
> The same as above - please proved `free -m` or some details on the swap
> available.

====================================================
[root@dell-op740-03 ~]# free -m swap
             total       used       free     shared    buffers     cached
Mem:          7808       1428       6380          0         81        163
-/+ buffers/cache:       1183       6625
Swap:            0          0          0
[root@dell-op740-03 ~]# free -m swap
             total       used       free     shared    buffers     cached
Mem:          7808       3955       3853          0         81        163
-/+ buffers/cache:       3710       4098
Swap:            0          0          0
[root@dell-op740-03 ~]# free -m swap
             total       used       free     shared    buffers     cached
Mem:          7808       5995       1813          0         81        163
-/+ buffers/cache:       5750       2058
Swap:            0          0          0
[root@dell-op740-03 ~]# free -m swap
             total       used       free     shared    buffers     cached
Mem:          7808       7583        225          0         81        163
-/+ buffers/cache:       7338        470
Swap:            0          0          0


> 
> > So the crash only occurs on RHEV-H but no RHEL.
> > You can find the script and crash.png in the attachment.
> 
> IIUIC then it's quite normal that memory hogs get killed in cases that the
> kernel runs out of memory.

Chen, could you also please provide the informations Adam needs, see comment 5.

(In reply to Fabian Deutsch from comment #14)
> Chen, could you also please provide the informations Adam needs, see comment
> 5.

1. /proc/cpuinfo on the host
Please see attachment "cpuinfo.txt"
2. Details about the VM load on the host 
  a. how many vms
9 vms

  b. how many vCPUs per vm
1 or 2 vCPUs per vm

  c. how much memory assigned to each VM
The total Memory of the host is 48G, and 4800M memory assigned to each VM.

  d. workload that is driving memory consumption inside the VMs
99%
3. /var/log/vdsm/mom.log around the time of the crash so I can see the ksm settings that are being activated.
Please see attachment "mom.log"

Created attachment 918569 [details]
mom.log

Created attachment 918570 [details]
cpuinfo.txt

Hey Linqing,
   As comment 10 said, the scenario 2's error is different with original bug error. We suspect that it is another new bug, not the same issue as this bug. And we can not determine Shao Chen's test procedure is 100% steps to reproduce the customer bug. 
   Could you help to reproduce this bug in kernel side?

Thanks
Ying

Hey Alexander,
   See comment 5, could you reply and provide the information?

Thanks
Ying

Hey Adam, do the informations (thanks Chen) from comment 15 shed some more light on this?

Other two needinfo flags from got removed unintentionally.
Adding them back. Sorry for the confusion.

Attachment in comment #8 is a different bug than the one in comment #0 and comment #1.

For comment #8 please file another bugreport, I've fixed in my upstream aa git tree several issues with OOM handling related to ext4 I/O errors that even lead to remounting the fs readonly (found with trinity triggering floods of OOM).

For this bug (comment #0 and comment #1) it seems some sort of deadlock in smp_call_function_single/many. I would have expected the NMI watchdog to trigger too but checking the sos report it didn't. The softlockup shows the deadlock kept running for 67 seconds before full crash. The NMI watchdog should fire in 5 seconds much less. It's unclear if it's a lock inversion between all those smp_call_functions running simultanously or something else. One wouldn't expect bugs in the IPI delivery logic because it runs all the time.

It would help if you could run SYSRQ+L and SYSRQ+T while syslog is still able to log (i.e. within the first 67 seconds) and report it. A crash dump would also help as then we could see all CPU stacktraces. Just to make an example CPU 1 is not shown. It is possible the culprit is in one of those CPUs that don't show the softlockup.

I'll think more about the available stack trace next week.

And if this is only reproducible in a single NUMA system and not everywhere else, we could evaluate if there could be hardware issues in the NUMA IPI delivery. A lost IPI can explain this too: there is a CPU waiting in csd_lock_wait in generic_exec_single, that is just waiting the IPI to run. (again if the IPI doesn't run normally it means the irqs have been disabled for too long on such CPU, but then the NMI watchdog should have fired, or the IPI was lost by the hardware, or there is some other software bug in the IPI delivery).

"grep NMI /proc/interrupts" and "cat /proc/sys/kernel/nmi_watchdog" can also verify the NMI watchdog is running.

Shao Chen,
   As comment 25, could you help to submit a new bug for your comment #8? Thanks.

(In reply to Ying Cui from comment #28)
> Shao Chen,
>    As comment 25, could you help to submit a new bug for your comment #8?
> Thanks.

I can't reproduce this issue with rhev-hypervisor6-6.5-20140821.1.el6ev(kernel-2.6.32-431.29.2.el6.x86_64 + qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64).

The process of eatmemory will be killed automatic. so please ignore my comment.

Thanks!

./eatmemory 20000M
Eating 20971520000 bytes in chunks of 1024...
Killed

I'm building a patch after discussion with Andrea.

This request was evaluated by Red Hat Product Management for
inclusion in a Red Hat Enterprise Linux release.  Product
Management has requested further review of this request by
Red Hat Engineering, for potential inclusion in a Red Hat
Enterprise Linux release for currently deployed products.
This request is not yet committed for inclusion in a release.

*** Bug 1083448 has been marked as a duplicate of this bug. ***

Patch(es) available on kernel-2.6.32-527.el6

Test following scenarios with 

# uname -r
2.6.32-550.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.464.el6.x86_64

# rpm -qpi kernel-2.6.32-550.el6.x86_64.rpm --changelog |grep 1116398
- [x86] kvm: Avoid pagefault in kvm_lapic_sync_to_vapic (Paolo Bonzini) [1116398]


ENV:
host has 512G memory:
# free -g
             total       used       free     shared    buffers     cached
Mem:           504          3        501          0          0          0
-/+ buffers/cache:          2        501
Swap:            3          0          3


The cli of guests like this:
/usr/libexec/qemu-kvm -cpu Opteron_G1 -M rhel6.5.0 -enable-kvm -m 52G -smp 4,sockets=1,cores=4,threads=1 -name rhel6.4-64 -uuid 9a0e67ec-f286-d8e7-0548-0c1c9ec93009 -nodefconfig -nodefaults -monitor stdio -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/home/RHEL-Server-6.7-64-virtio-scsi.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,snapshot=on -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:51:12,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -vga qxl -spice port=5911,disable-ticketing,seamless-migration=on -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864


scenario1:
1.start 14 guests:
# ps aux |grep qemu -c
14

2.start stress in guest:
# stress -m 1 --vm-bytes 50000M --vm-keep

3.wait for the memory of host is full used to trigger KSM activation.
# free -m
             total       used       free     shared    buffers     cached
Mem:        516858     516337        521          0          1         46
-/+ buffers/cache:     516289        569
Swap:         4095       4040         55

# service ksm status
ksm is running

Result: wait for long time, host/guests work well, no crash/softlock occurs.

scenario2:
# service ksmtuned status
ksmtuned is stopped
# service ksm status
ksm is running

1.start a 50G memory guest

2.Start stress in guest:
# stress -m 1 --vm-bytes 50000M --vm-keep

3.try to execute the eatmemory program
# ./eatmemory 500000M
Eating 524288000000 bytes in chunks of 1024...


Killed

Result:guest/host work well.

So the tests pass, and this bug is fixed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1272.html