Bug 1759131

Summary: [OSP-13] Wait longer before issuing SIGKILL on instance destroy, to handle "libvirtError: Failed to terminate process <pid> with SIGKILL: Device or resource busy"
Product: Red Hat OpenStack Reporter: Kashyap Chamarthy <kchamart>
Component: openstack-novaAssignee: Kashyap Chamarthy <kchamart>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: dasmith, eglynn, ggrimaux, jhakimra, kchamart, lyarwood, mbooth, sbauza
Target Milestone: z11Keywords: Patch, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-nova-17.0.12-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-10 11:27:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kashyap Chamarthy 2019-10-07 13:05:55 UTC 
This bug was initially created as a copy of Bug #1759125

A proactive backport for RHOS-13


+++ This bug was initially created as a clone of Bug #1636190 +++

[...]


commit 3d38d05bc4157d32814683473a4919d84fb47ec7
Author: Kashyap Chamarthy <kchamart>
Date:   Mon Feb 25 13:26:24 2019 +0100

    libvirt: Rework 'EBUSY' (SIGKILL) error handling code path
    
    Change ID I128bf6b939 (libvirt: handle code=38 + sigkill (ebusy) in
    _destroy()) handled the case where a QEMU process "refuses to die" within
    a given timeout period set by libvirt.
    
    Originally, libvirt sent SIGTERM (allowing the process to clean-up
    resources), then waited 10 seconds, if the guest didn't go away.  Then
    it sent, the more lethal, SIGKILL and waited another 5 seconds for it to
    take effect.
    
    From libvirt v4.7.0 onwards, libvirt increased[1][2] the time it waits
    for a guest hard shutdown to complete.  It now waits for 30 seconds for
    SIGKILL to work (instead of 5).  Also, additional wait time is added if
    there are assigned PCI devices, as some of those tend to slow things
    down.
    
    In this change:
    
      - Increment the counter to retry the _destroy() call from 3 to 6, thus
        increasing the total time from 15 to 30 seconds, before SIGKILL
        takes effect.  And it matches the (more graceful) behaviour of
        libvirt v4.7.0.  This also gives breathing room for Nova instances
        running in environments with large compute nodes with high instance
        creation or delete churn, where the current timout may not be
        sufficient.
    
      - Retry the _destroy() API call _only_ if MIN_LIBVIRT_VERSION is lower
        than 4.7.0.
    
    [1] https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=9a4e4b9
        (process: wait longer 5->30s on hard shutdown)
    [2] https://libvirt.org/git/?p=libvirt.git;a=commit;h=be2ca04 ("process:
        wait longer on kill per assigned Hostdev")
    
    Conflicts:
        nova/virt/libvirt/driver.py
    
        (Trivial conflict: Rocky didn't have the QEMU-native TLS feature
        yet.)
    
    Related-bug: #1353939
    
    Change-Id: If2035cac931c42c440d61ba97ebc7e9e92141a28
    Signed-off-by: Kashyap Chamarthy <kchamart>
    (cherry picked from commit 10d50ca4e210039aeae84cb9bd5d18895948af54)
    (cherry picked from commit 75985e25bc147369efb90d4fa9f046631766c14c)
Comment 5 errata-xmlrpc 2020-03-10 11:27:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0759