Bug 2013573

Summary:	the network hook doesn't work well by "issuing shutdown" in split daemon mode
Product:	Red Hat Enterprise Linux 9	Reporter:	Jing Qi <jinqi>
Component:	libvirt	Assignee:	Michal Privoznik <mprivozn>
libvirt sub component:	General	QA Contact:	Jing Qi <jinqi>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	jdenemar, lmen, mprivozn, virt-maint, xuzhang
Version:	9.0	Keywords:	Regression, Triaged, Upstream
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-7.9.0-1.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-05-17 12:45:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:	7.9.0
Embargoed:

Description Jing Qi 2021-10-13 09:44:33 UTC

Description of problem:
In split daemon mode, the network hook doesn't work well when the vm with an interface was shutdown by issuing "shutdown" command in the guest.

Version-Release number of selected component (if applicable):
libvirt-7.7.0-3.el9.x86_64 
qemu-kvm-6.1.0-3.el9.x86_64

How reproducible:
Always

Steps to Reproduce:
1. In split daemon mode, config the network hook in /etc/libvirt/hooks/network.

#!/bin/bash
echo "$0" "$@" >> /tmp/network.log
exit 0

and change the file mode - "chmod +x network"

2. Start a domain with an interface and check /tmp/network.log
/etc/libvirt/hooks/network default port-created begin -

2. Login the console of the vm and run "shutdown 0". 

3. Check the log file and there is no expected log like
/etc/libvirt/hooks/network default port-deleted begin -


Actual results:
No expected log after step2 - 
/etc/libvirt/hooks/network default port-created begin -

Expected results:
Add below log to log file - 
/etc/libvirt/hooks/network default port-deleted begin -


Additional info:

1.In split daemon mode, shutdown the vm with "virsh destroy vm", the log can be found in the file.
2.In a monolithic mode, run "shutdown 0" in the guest, the log can be found.

Comment 1 Michal Privoznik 2021-10-25 12:56:48 UTC

Yes, this is a genuine bug. The problem is that virtqemud fails to connect to virtnetworkd thus the latter doesn't run hook scripts. It's just a coincidence that the TAP device disappears (because QEMU that had it open is gone). What I'm seeing is that in qemuProcessStop() the virGetConnectNetwork() fails and it does so because virIdentityGetCurrent() in virGetConnectGeneric() returned NULL. This is understandable because this whole code runs in a separate thread (per vm thread for processing events) and thus has no identity set (identity is stored in a thread local store). Maybe setting the identity would help, but it needs to be done carefully so that the identity is not leaked when the thread exits.

Comment 2 Michal Privoznik 2021-10-25 13:26:48 UTC

Oh, my bad, the thread I'm talking about is not per VM; so we have to do something more clever.

Comment 3 Michal Privoznik 2021-10-26 10:18:00 UTC

Patches proposed upstream:

https://listman.redhat.com/archives/libvir-list/2021-October/msg01020.html

Comment 4 Michal Privoznik 2021-10-27 15:14:10 UTC

Merged upstream:

efc1ccc75b qemu: Set "qemu-event" thread identity
0c8f1aeddf virthreadpool: Allow setting identity for workers

v7.9.0-rc1-5-gefc1ccc75b

Comment 5 Jing Qi 2021-10-29 01:19:26 UTC

Using upstream libvirt v7.9.0-rc1-3-g775de86975 with the above patch &qemu-6.1.0-9.fc36.x86_64

1. In split daemon mode, config the network hook in /etc/libvirt/hooks/network.

#!/bin/bash
echo "$0" "$@" >> /tmp/network.log
exit 0

and change the file mode - "chmod +x network"

2. Start a domain with an interface and check /tmp/network.log
/etc/libvirt/hooks/network default port-created begin -

3. Login the console of the vm and run "shutdown 0". 

4. Check the log file and the below expected log is recorded
/etc/libvirt/hooks/network default port-deleted begin -

Comment 8 Jing Qi 2021-11-12 02:23:32 UTC

Verified with libvirt-7.9.0-1.el9.x86_64 & qemu-kvm-6.1.0-6.el9.x86_64
Steps are the same as comments 5.

Comment 10 errata-xmlrpc 2022-05-17 12:45:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: libvirt), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2390