RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1889363 - virt-who's systemd unit is missing 'libvirtd-ro.socket' requirement
Summary: virt-who's systemd unit is missing 'libvirtd-ro.socket' requirement
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: virt-who
Version: 8.2
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.5
Assignee: William Poteat
QA Contact: Eko
URL:
Whiteboard:
Depends On:
Blocks: 1878724
TreeView+ depends on / blocked
 
Reported: 2020-10-19 13:29 UTC by Marcin Sobczyk
Modified: 2022-05-10 14:31 UTC (History)
10 users (show)

Fixed In Version: virt-who-1.30.9-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-10 14:07:06 UTC
Type: Bug
Target Upstream Version:
Embargoed:
hsun: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github candlepin virt-who pull 344 0 None open 1889363: Update dependency between libvirtd and virt-who 2021-10-01 11:43:24 UTC
Red Hat Knowledge Base (Solution) 5392541 0 None None None 2021-11-23 16:09:57 UTC
Red Hat Product Errata RHBA-2022:1879 0 None None None 2022-05-10 14:07:20 UTC

Description Marcin Sobczyk 2020-10-19 13:29:16 UTC
Description of problem:

virt-who uses libvirtd's read-only socket to connect to the daemon [1], but it doesn't require it at systemd unit level [2]. This causes scenarios where one needs to put down whole libvirtd daemon with all its sockets unpredictable.

How reproducible: Always


Steps to Reproduce:

 systemctl stop libvirt-who.service libvirtd-ro.socket libvirtd.service
 systemctl start libvirt-who.service

The service is active, but it reports errors in log:

 libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock-ro': Connection refused

and:

 systemctl status libvirtd-ro.socket

shows that the socket is indeed stopped.

The opposite is also true - when both virt-who and libvirtd are running, this:

 systemctl stop libvirtd.service libvirtd-ro.socket

should also stop virt-who, but it doesn't.


Expected results:

Starting virt-who.service should start libvirtd-ro.socket automatically and stopping libvirtd-ro.socket should stop virt-who.service.


[1] https://github.com/candlepin/virt-who/blob/4c7fdb032a66e2fe3324cc2d7579101c699e3b00/virtwho/virt/libvirtd/libvirtd.py#L282
[2] https://github.com/candlepin/virt-who/blob/master/virt-who.service

Comment 1 William Poteat 2021-04-30 17:59:04 UTC
virt-who does not always use libvirtd.service or libvirtd-ro.socket. That is part of the configuration of virt-who that the libvirtd-ro.socket start/stop is not going to recognize. We cannot implement actions that will stop the virt-who service for non-libvirt users based on the libvirt socket.

Secondly, there is no way around the condition you describe as to the libvirtd.service and libvritd-ro.socket start/stop. You can stop libvirtd-ro.socket while libvirtd.service is active, but it will not restart until you at least stop the libvirtd.service.

The only relief I can offer is for virt-who to programmatically start the libvirtd.service on the local virt-who machine. This would happen if libvirt is used by a configuration and if the service is not already active.


Please let me know if this will be of use in this situation.

Comment 2 William Poteat 2021-04-30 18:36:57 UTC
Addendum to the last comment:

Invoking virt-who from the command line or from the service will already activate libvirtd.service when run. I do not believe that there is much else that can be done here.

Comment 3 Marcin Sobczyk 2021-05-05 09:21:37 UTC
If the dependency on libvirtd-ro.socket is dynamic you should use the mechanisms for dynamic dependency injection systemd provides. Quote from [1]:

---
Along with a unit file foo.service, the directory foo.service.wants/ may exist. All unit files symlinked from such a directory are implicitly added as dependencies of type Wants= to the unit. Similar functionality exists for Requires= type dependencies as well, the directory suffix is .requires/ in this case. This functionality is useful to hook units into the start-up of other units, without having to modify their unit files. For details about the semantics of Wants=, see below. The preferred way to create symlinks in the .wants/ or .requires/ directory of a unit file is by embedding the dependency in [Install] section of the target unit, and creating the symlink in the file system with the enable or preset commands of systemctl(1).
---

That's the way for the system to behave well with socket activation in place.
Vdsm i.e. has the same problem - you can peek the implementation here [2].

[1] https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description
[2] https://github.com/oVirt/vdsm/blob/a7d59f0b52b4d9124a40c341777f453a0075976b/lib/vdsm/tool/configurators/libvirt.py#L137

Comment 4 William Poteat 2021-05-05 13:30:32 UTC
This only seems to correct one area of the issue: if libvirtd.service or ro.socket is shut down, then virt-who continues to run and logs an error that it cannot connect.
What if you are running virt-who with both a libvirt and esx setup? Is the idea that the thread running the libvirt process listen for this shutdown and stop the thread? There is no means to stop/restart individual configurations on the fly. They all start and stop together.

Please tell me if I am correct about libvirtd.service not allowing libvirt-ro.socket to start while libvirtd.service is running. It is what I observed while turning the socket off and then attempting to turn it back on without restarting the service. As virt-who will start the service and thus the socket when it invokes a libvirt configuration.


This may be a useful enhancement to virt-who. I am willing to hear what ideas you have and weigh them against the effort required to make a change.

Comment 5 Marcin Sobczyk 2021-05-06 09:28:54 UTC
(In reply to William Poteat from comment #4)
> This only seems to correct one area of the issue: if libvirtd.service or
> ro.socket is shut down, then virt-who continues to run and logs an error
> that it cannot connect.
> What if you are running virt-who with both a libvirt and esx setup? Is the
> idea that the thread running the libvirt process listen for this shutdown
> and stop the thread? There is no means to stop/restart individual
> configurations on the fly. They all start and stop together.

Let's start with stating the fact, that I'm not that familiar with virt-who
project yet, so forgive my noob questions.

AFAIU Looking at [1] and your comment it is possible to run virt-who with smth like:

 virt-who -i 5 --libvirt --esx

And that will make virt-who report guest ids from both?
 
> Please tell me if I am correct about libvirtd.service not allowing
> libvirt-ro.socket to start while libvirtd.service is running. It is what I
> observed while turning the socket off and then attempting to turn it back on
> without restarting the service.

Correct - the socket needs to be started before 'libvirtd.service' [2].

> As virt-who will start the service and thus
> the socket when it invokes a libvirt configuration.

Starting 'libvirtd.service' will also automatically start 'libvirtd-ro.socket'.

> This may be a useful enhancement to virt-who. I am willing to hear what
> ideas you have and weigh them against the effort required to make a change.

I filed this bug due to a problem we saw when deploying hosts in oVirt.
I'll try to describe the issue gradually.

Suppose there are no socket units and no socket activation - we simply have
'libvirtd.service' and 'virt-who.service' - and virt-who can be dynamically
configured to work with libvirtd.

If there is no dependency relation on systemd unit level between these two,
we can have libvirt down and virt-who up. In this scenario virt-who will simply
spit out errors about libvirt connection problems all the time. Also, if both libvirt
and virt-who are up, stopping 'libvirtd.service' won't affect 'virt-who.service'.

If we add the dependency relation (doesn't matter if dynamically or not) we get
a bit more well-behaved system - when you stop 'libvirtd.service', 'virt-who.service'
will be stopped as a dependency. Since virt-who is a daemon, this scenario seems more
reasonable to me than to simply "stop working". It comes with a price though - after
starting 'libvirtd.service' again we'd like 'virt-who.service' to also start. 
That's why we'd also need a reverse 'Wants' dependency on 'libvirtd.service'
towards 'virt-who.service' (also doable with dynamic dependencies).

Since libvirtd is kinda the "brain" on oVirt hosts it makes sense for processes like
host deployment not to be aware of other elements using libvirtd and to expect that running:

 systemctl stop libvirtd.service
 systemctl start libvirtd.service

will affect the other elements properly.

With socket activation the situation became a bit more complex. AFAIU our problem was that
in host deployment we did exactly as above - 'systemctl stop libvirtd.service', then done
some other processing, which took some time, and finally we 'systemctl start libvirtd.service'.
However 'libvirtd-ro.socket' was still active all the time. In the "other processing" period
virt-who used the active 'libvirtd-ro.socket' and brought up libvirtd. This made the deployment
process explode.

I think we simply fixed the problem by stopping the ro socket as well.

While the fix seems obvious, I hope I at least partially made my point regarding
the system "behaving well".

[1] https://linux.die.net/man/8/virt-who
[2] https://github.com/libvirt/libvirt/blob/7f71a641d0809924acbe899ddebe46babb3ee4e9/src/remote/libvirtd-ro.socket.in#L3

Comment 6 William Poteat 2021-05-06 13:20:54 UTC
The answer to the first question in comment 5 is yes. We use configuration files that can have the same virt-who instance run multiple threads on multiple fabrics at the same time.
Because of this, it does not make sense for the released virt-who to tie into the libvirt service for all users.

Instead of relying on a dynamic dependency relation, why not alter the virt-who.service file on your deployed system? You can create this dependency and it will not be an issue for others that are running non-libvirt installations.

Comment 7 Eko 2021-05-06 14:17:00 UTC
As QE's opinion, libvirt is only one supported hypervisor mode by virt-who, we still other more hypervisors modes: vcenter, hyperv, rhevm, kubevirt, xen.
It may not a good idea to let virt-who have a default dependence, the vcenter customers will feel confusion why I need to launch libvirt service for my vcenter connection?

Comment 8 Rehana 2021-05-10 14:37:27 UTC
Setting need info , please refer comment 6.

Comment 9 Rehana 2021-05-10 14:37:28 UTC
Setting need info , please refer comment 6.

Comment 10 Rehana 2021-06-11 13:11:44 UTC
Gentle Reminder on the need info request. 

Note: 
The above needinfo flag has been requested a month ago. Unfortunately , we will have to close the bug during the upcoming triage ( Tuesday)  with missing information.

Comment 11 Martin Perina 2021-06-13 16:47:39 UTC
Marcin si from RHV team, he just pointed out the issue which virt-who causes to RHV. But libvirt is maintained by platform virtualization team, please direct additional questions to them.

From RHV point of view this virt-who functionality broke RHV and should be fixed on virt-who side.

Comment 12 Daniel Berrangé 2021-07-29 15:33:04 UTC
> Please tell me if I am correct about libvirtd.service not allowing libvirt-ro.socket to start while libvirtd.service is running. It is what I observed while turning the socket off and then attempting to turn it back on without restarting the service. As virt-who will start the service and thus the socket when it invokes a libvirt configuration.

You shouldn't mess around with the socket units while the service is running. It won't break current connections but will prevent future connections until the service restarts. This is rarely desirable.

virt-who shouldn't need to directly start the service/socket units itself - just let systemd do it automatically.

Overall I'm not really seeing what the core problem here is.

If libvirt is not running when virt-who is started, then there are no libvirt guests whose existence needs reporting, so fact that libvirtd isn't started is a non-issue, aside from the log message. If the log message is getting spammed in the logs repeatedly, then throttle the logs, or only log it once per connected/disconnected transition.

If you really do want a depndancy in the unit files,  and also want virt-who to be usable without libvirtd being present, then it is likely sufficient to have  'Wants=libvirt-ro.socket'. This is a relatively weak dependancy that will let virt-who service start, even if the libvirt-ro.socket fails or is unavailable. This is possibly desirable if you want virt-who to "just work" no matter what set of hypervisors are configured.

Comment 13 William Poteat 2021-09-28 12:47:43 UTC
Please consult Daniel Berrangé when testing this issue. The change is what he mentioned in Comment #12. He should be able to tell you what to look for.

Comment 29 errata-xmlrpc 2022-05-10 14:07:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt-who bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1879


Note You need to log in before you can comment on or make changes to this bug.