Bug 1473536

Asked for rjones's help, He thought this is a bug we have seen before: https://bugzilla.redhat.com/show_bug.cgi?id=1435432. It's caused by the *glib2* rebase (to glib2-2.50.3-3.el7.x86_64)

This bug is the same as bug1473605(https://bugzilla.redhat.com/show_bug.cgi?id=1473605), which provides more datailed info.

If we were to just fix this in qemu alone, I believe it would require
this patch:

commit ecbddbb106114f90008024b4e6c3ba1c38d7ca0e
Author: Richard W.M. Jones <rjones>
Date:   Fri Mar 31 21:51:33 2017 +0100

    main-loop: Acquire main_context lock around os_host_main_loop_wait.

However this is really a bug in glib2, not in qemu.  The above is
a work around as I remember it.

Also the patch would be needed for both qemu-kvm and qemu-kvm-rhev,
and is not present in either.

(In reply to Richard W.M. Jones from comment #5)
> Also the patch would be needed for both qemu-kvm and qemu-kvm-rhev,
> and is not present in either.

Oops, as danpb pointed out, this is present in qemu-kvm-rhev because
we rebased, but not in qemu-kvm.

*** Bug 1474405 has been marked as a duplicate of this bug. ***

It is instead of the glib2 fix.  Fixing the depends on.

Fix included in qemu-kvm-1.5.3-153.el7

QE tried to reproduce this bug with 'virt-rescue', the version QE can get is 1.36, with command:

    $ virt-rescue -e ^] --scratch
    ><rescue> while true; do ls -l /usr/bin; done

But failed.

And tried the other way, with command:

    $ while libguestfs-test-tool -t 60 >& /tmp/log ; do echo -n . ; done

When tested enough time (one day), still can't get hang.

As developer said, it is hard to reproduce this bug, so QE did sanity test and
patch review for this bz.

Versions:
libguestfs-1.36.10-4.el7.x86_64
kernel-3.10.0-830.el7.x86_64
qemu-kvm-1.5.3-153.el7.x86_64
glib2-2.54.2-2.el7.x86_64

Run the below command on a metal host for two days, 'libguestfs-test-tool' finished OK for 54000 times, no hang occured.

    $ while libguestfs-test-tool -t 60 >& /tmp/log ; do echo -n . ; done

# git checkout origin/rhel7/master-1.5.3
Note: checking out 'origin/rhel7/master-1.5.3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 2beece2... Update to qemu-kvm-1.5.3-153.el7

# git log --grep="Bugzilla: 1473536"
commit 6baaf82a7742a1de9160146b08ba0cc86b3d4e79
Author: Paolo Bonzini <pbonzini>
Date:   Wed Jan 10 17:02:21 2018 +0100

    main-loop: Acquire main_context lock around os_host_main_loop_wait.
    
    RH-Author: Paolo Bonzini <pbonzini>
    Message-id: <20180110170221.28975-1-pbonzini>
    Patchwork-id: 78541
    O-Subject: [RHEL7.5 qemu-kvm PATCH] main-loop: Acquire main_context lock around os_host_main_loop_wait.
    Bugzilla: 1473536
    RH-Acked-by: Jeffrey Cody <jcody>
    RH-Acked-by: John Snow <jsnow>
    RH-Acked-by: Miroslav Rezanina <mrezanin>
    
    Bugzilla: 1473536
    
    Brew build: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14912977
    
    When running virt-rescue the serial console hangs from time to time.
    Virt-rescue runs an ordinary Linux kernel "appliance", but there is
    only a single idle process running inside, so the qemu main loop is
    largely idle.  With virt-rescue >= 1.37 you may be able to observe the
    hang by doing:
    
      $ virt-rescue -e ^] --scratch
      ><rescue> while true; do ls -l /usr/bin; done
    
    The hang in virt-rescue can be resolved by pressing a key on the
    serial console.
    
    Possibly with the same root cause, we also observed hangs during very
    early boot of regular Linux VMs with a serial console.  Those hangs
    are extremely rare, but you may be able to observe them by running
    this command on baremetal for a sufficiently long time:
    
      $ while libguestfs-test-tool -t 60 >& /tmp/log ; do echo -n . ; done
    
    (Check in /tmp/log that the failure was caused by a hang during early
    boot, and not some other reason)
    
    During investigation of this bug, Paolo Bonzini wrote:
    
    > glib is expecting QEMU to use g_main_context_acquire around accesses to
    > GMainContext.  However QEMU is not doing that, instead it is taking its
    > own mutex.  So we should add g_main_context_acquire and
    > g_main_context_release in the two implementations of
    > os_host_main_loop_wait; these should undo the effect of Frediano's
    > glib patch.
    
    This patch exactly implements Paolo's suggestion in that paragraph.
    
    This fixes the serial console hang in my testing, across 3 different
    physical machines (AMD, Intel Core i7 and Intel Xeon), over many hours
    of automated testing.  I wasn't able to reproduce the early boot hangs
    (but as noted above, these are extremely rare in any case).
    
    Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1435432
    Reported-by: Richard W.M. Jones <rjones>
    Tested-by: Richard W.M. Jones <rjones>
    Signed-off-by: Richard W.M. Jones <rjones>
    Message-Id: <20170331205133.23906-1-rjones>
    [Paolo: this is actually a glib bug: recent glib versions are also
    expecting g_main_context_acquire around g_poll---but that is not
    documented and probably not even intended].
    Signed-off-by: Paolo Bonzini <pbonzini>
    (cherry picked from commit ecbddbb106114f90008024b4e6c3ba1c38d7ca0e)
    
    Signed-off-by: Miroslav Rezanina <mrezanin>

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0816