Hello,

this seems to crash in the page migration code of the host kernel (guest is irrelevant for this bug).

If KSM is into the equation and the page migration otherwise works fine (NUMA balancing and memory compaction are the two main users of page migration) this could mean something's wrong in the migration of KSM pages.

Unfortunately both stack traces are truncated so it's uncertain if it's really the migration of KSM pages that tripped on a bug, so above I'm guessing.

To proceed with proper debugging:

1) could you retrieve a full log? Before the Call Trace: and Stack: lines there's other useful information. Perhaps it's still in /var/log.

2) is a kernel crash dump available with kexec? could you reproduce with that enabled?

3) does it go away if you run the below commands shortly after boot before running the testcase?

    echo NO_NUMA > /sys/kernel/debug/sched_features
    echo never > /sys/kernel/mm/transparent_hugepage/defrag
    echo 1000000000 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs

Ignore if the first command fails if the host is not NUMA.

The point 3) really intends to disable page migration, but there's no global tweak to disable it, and no global tweak to disable compaction either. Even after point 3) is applied, drivers will still invoke compaction at will, so the crash may still happen even after point 3) sometime, but it should happen less frequently if page migration of KSM pages is the problem.

Andrea, This looks very ugly. I can't tell how often this happens. We don't want these messages filling up the logs. Do you know the cause? 

You said "crash" but this looks like just a backtrace. You said host, but Yang Meng said the BT was in the guest. Which is it?

Yang Meng, What is the qemu-kvm command line? Is there NUMA set up in the guests? Thanks!

I think I confused them for host stack traces because I wouldn't have expected KSM to be able to destabilize the guest like this and the main problem is still that they're truncated and we don't know if it's a crash or just a warning, could be a VM_BUG_ON a WARN_ON or an harmless guest softlockup induced by a host inefficiency.

They are definitely guest stacktraces (like the first comment correctly said), because do_async_page_fault is in one of them.

The header above that stack trace would tell it. We need the full logs.

Considering the other KSM bug I've been working on, I wonder if the host tried to migrate a KSM page and these are harmless guest softlockups that happened because KSM in the host computed for too long while trying to migrate a KSM page. I assume KSM has been running in the host only, if yes, I still think the bug is unlikely in the guest, but we need the full logs around the stacktraces to make progress on this one.

(In reply to Karen Noel from comment #3)
> Andrea, This looks very ugly. I can't tell how often this happens. We don't
> want these messages filling up the logs. Do you know the cause? 
> 
> You said "crash" but this looks like just a backtrace. You said host, but
> Yang Meng said the BT was in the guest. Which is it?
> 
> Yang Meng, What is the qemu-kvm command line? Is there NUMA set up in the
> guests? Thanks!

hi, numa didn't set up ,and the following is the commandline:


/usr/libexec/qemu-kvm \
    -S  \
    -name 'virt-tests-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20150831-181355-S3Y85kan,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/tmp/monitor-catch_monitor-20150831-181355-S3Y85kan,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idRGYTYM  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20150831-181355-S3Y85kan,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20150831-181355-S3Y85kan,path=/tmp/seabios-20150831-181355-S3Y85kan,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20150831-181355-S3Y85kan,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=03 \
    -drive id=drive_image1,if=none,cache=none,snapshot=on,aio=native,format=qcow2,file=/home/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.2-64.qcow2 \
    -device ide-hd,id=image1,drive=drive_image1,bus=ide.0,unit=0 \
    -device e1000,mac=9a:18:19:1a:1b:1c,id=idB5ehSJ,netdev=idvI6NWv,bus=pci.0,addr=04  \
    -netdev tap,id=idvI6NWv,fd=21  \
    -m 44958  \
    -smp 16,maxcpus=16,cores=8,threads=1,sockets=2  \
    -cpu 'Opteron_G4',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off   \
    -pidfile /tmp/virt-tests-vm1-20150831-181352-bHLc.pid \
    -enable-kvm

Hello Yang,

which other lines are in /var/log just before/around 19:51:37? and 19:29:09?

(In reply to Andrea Arcangeli from comment #6)
> Hello Yang,
> 
> which other lines are in /var/log just before/around 19:51:37? and 19:29:09?

Hi Andrea,
    the log is not got from /var/log ,it's just the log generated by autotest script, do you need the full log of autotest ?

If the guest images aren't wiped out automatically after the autotest script run, you could try to restsart the guest and try to grab those logs from `journalctl` or /var/log/. If those two fragments are the only thing that gets logged into the autotest log, there shall be more kernel messages before those Stack/Call trace lines into /var/log or journalctl. If there aren't other kernel messages in the autotest logs I probably don't need the full log of autotest, other kernel messages would have the exact same timestamp: 2015-08-31 19:29:09 and 2015-08-31 19:51:37. If there are other lines with those two timestamps I need those yes.

Thank you.