Bug 1572671 - Large Number of VM shutdown/startup issue
Summary: Large Number of VM shutdown/startup issue
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-27 14:46 UTC by Prarit Bhargava
Modified: 2018-05-09 12:13 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-30 12:20:28 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Prarit Bhargava 2018-04-27 14:46:56 UTC
This bug doesn't belong in kernel/Virtualization but I'm not entirely sure where to file it.

I'm running RHEL7.4 and I have ~80 100GB guests running on the system.  When I shutdown the system I do

virsh list | grep running | awk ' { print $2 } ' | xargs virsh shutdown

and after rebooting the system I have to manually start each guest.

I cannot start all the guests at once as this brings the system almost to a halt.  

Is there a way to bring up the guests, one-by-one, in an automated fashion such that the system continues to make progress?

P.

Comment 3 Peter Krempa 2018-04-30 05:40:15 UTC
If you are shutting down and then restarting the guests you can use the libvirt-guests service which can do this in a serial fashion. The service shuths down any running guest and then starts back all of them that were running.

Note that while the service supports parallel shutdown, where you can configure the number of guests shutdown at once, you can't do the same for startup, since libvirt does not know when the guest finished booting.

Since there are so many possible options here your desired scenario might not be possible to achieve using libvirt-guests. If you require any complex policy for startup, the best solution will be to implement it programatically using e.g. the python bindings of libvirt.

Comment 4 Peter Krempa 2018-04-30 05:40:48 UTC
(In reply to Prarit Bhargava from comment #0)

[...]

> virsh list | grep running | awk ' { print $2 } ' | xargs virsh shutdown

Note that you can use 'virsh list --state-running --name' instead of grep and awk.

Comment 5 Jiri Denemark 2018-04-30 12:20:28 UTC
This looks like a usage question rather than a bug report. Please, target such
questions to libvirt-users@redhat.com.

Anyway, the question was already answered by Peter. Theoretically, we could
add some kind of a configurable delay between virsh start commands in
libvirt-guests, but it would be quite fragile and not generally usable anyway.
That using the custom script as suggested by Peter looks like the best option
since you can implement any kind of policy which fits your use case. Such as
starting several (but not all) domains at the same time, starting the domains
in a specific order so that the more important ones are started first, etc.
You can even wait until each domain is fully up and running before starting
another one. Such universality is really out of scope for libvirt-guests
script.

Comment 6 Prarit Bhargava 2018-05-09 12:13:50 UTC
(In reply to Jiri Denemark from comment #5)
> This looks like a usage question rather than a bug report. Please, target
> such
> questions to libvirt-users@redhat.com.

Heh :)  Thanks.  I was looking for a list to email this question to :)

> 
> Anyway, the question was already answered by Peter. Theoretically, we could
> add some kind of a configurable delay between virsh start commands in
> libvirt-guests, but it would be quite fragile and not generally usable
> anyway.
> That using the custom script as suggested by Peter looks like the best option
> since you can implement any kind of policy which fits your use case. Such as
> starting several (but not all) domains at the same time, starting the domains
> in a specific order so that the more important ones are started first, etc.
> You can even wait until each domain is fully up and running before starting
> another one. Such universality is really out of scope for libvirt-guests
> script.

I agree completely with everything you both have said.  I am somewhat surprised that no one has come up with a unified solution.  I do have a systemd unit file that starts 5 guests at a time, waits 3 minutes, and then starts the next five, etc.

P.


Note You need to log in before you can comment on or make changes to this bug.