Bug 2019822
| Summary: | [RFE] Support suspend/resume with vGPUs | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | chhu | |
| Component: | openstack-nova | Assignee: | OSP DFG:Compute <osp-dfg-compute> | |
| Status: | CLOSED UPSTREAM | QA Contact: | OSP DFG:Compute <osp-dfg-compute> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 17.0 (Wallaby) | CC: | alifshit, chhu, dasmith, eglynn, jhakimra, jjongsma, kchamart, sbauza, sgordon, vromanso, xuzhang, yafu, zhguo | |
| Target Milestone: | --- | Keywords: | FutureFeature, Triaged | |
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | libvirt_OSP_INT | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2022078 (view as bug list) | Environment: | ||
| Last Closed: | 2022-10-04 15:26:56 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2022078 | |||
|
Description
chhu
2021-11-03 12:16:16 UTC
Hi, Jonathon Do VM suspend, resume in libvirt env, only with "Suspended Paused,Resumed Unpaused" event. In OSP env, it include more operations as below, and after VM suspend, resume in OSP, the mdev device in VM is no longer exited. Do you have any comments here ? > - VM suspend > [root@overcloud-novacompute-0 /]# virsh event --all --loop > event 'device-removed' for domain 'instance-0000000e': hostdev0 > event 'lifecycle' for domain 'instance-0000000e': Suspended Paused > event 'migration-iteration' for domain 'instance-0000000e': iteration: '1' > event 'migration-iteration' for domain 'instance-0000000e': iteration: '2' > event 'migration-iteration' for domain 'instance-0000000e': iteration: '3' > event 'job-completed' for domain 'instance-0000000e': > operation: 2 > time_elapsed: 784 > downtime: 782 > setup_time: 2 > data_total: 4404879360 > data_processed: 546775060 > data_remaining: 0 > memory_total: 4404879360 > memory_processed: 546775060 > memory_remaining: 0 > memory_bps: 717050269 > memory_constant: 944251 > memory_normal: 131159 > memory_normal_bytes: 537227264 > memory_dirty_rate: 0 > memory_iteration: 3 > memory_postcopy_requests: 0 > memory_page_size: 4096 > event 'lifecycle' for domain 'instance-0000000e': Stopped Saved > > [root@overcloud-novacompute-0 /]# virsh nodedev-event --loop --event > lifecycle > event 'lifecycle' for node device net_tap98fd6ebe_bb_fe_16_3e_b0_e5_5e: > Deleted > event 'lifecycle' for node device net_tap0373cf59_40_d6_f7_f8_30_f3_6f: > Deleted > - Check VM xml: > [root@overcloud-novacompute-0 /]# virsh dumpxml instance-0000000e|grep hostdev -A 6 > <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'> > <source> > <address uuid='9639c6e1-601c-474d-ace2-8d76739147dc'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> > </hostdev> > > - VM Resume: > [root@overcloud-novacompute-0 /]# virsh event --all --loop > event 'lifecycle' for domain 'instance-0000000e': Defined Updated > event 'lifecycle' for domain 'instance-0000000e': Started Restored > event 'lifecycle' for domain 'instance-0000000e': Resumed Snapshot > > [root@overcloud-novacompute-0 /]# virsh nodedev-event --loop --event > lifecycle > event 'lifecycle' for node device net_tap98fd6ebe_bb_fe_16_3e_b0_e5_5e: > Created > event 'lifecycle' for node device net_tap0373cf59_40_f2_f8_c5_04_7c_22: > Created > > - Check VM xml > [root@overcloud-novacompute-0 /]# virsh dumpxml instance-0000000e|grep > hostdev -A 6 > ---No mdev device (In reply to chhu from comment #3) > Hi, Jonathon > > Do VM suspend, resume in libvirt env, only with "Suspended Paused,Resumed > Unpaused" event. > In OSP env, it include more operations as below, > and after VM suspend, resume in OSP, the mdev device in VM is no longer > exited. > Do you have any comments here ? I'm not entirely sure what OSP does when you ask it to suspend a vm, but as you can see below... > > - VM suspend > > [root@overcloud-novacompute-0 /]# virsh event --all --loop > > event 'device-removed' for domain 'instance-0000000e': hostdev0 Something seems to be removing the hostdev device here. If you can figure out why it's being removed, it might answer the question. Alternately, if you can provide a minimal reproducer using only libvirt commands, it would be much easier for me to investigate. I'd say this is the downstream manifestation of https://bugs.launchpad.net/nova/+bug/1948705 / https://review.opendev.org/c/openstack/nova/+/815373... (In reply to Artom Lifshitz from comment #5) > I'd say this is the downstream manifestation of > https://bugs.launchpad.net/nova/+bug/1948705 / > https://review.opendev.org/c/openstack/nova/+/815373... Giving we are talking about vGPU hot plug and unplug here, should we also have a doc point to https://docs.nvidia.com/grid/latest/grid-vgpu-release-notes-red-hat-el-kvm/index.html#vgpu-hot-plug-not-supported to let users know the limitation? During OSP suspend, it remove the hostdev device, during OSP resume, from the event, I can't see it add the hostdev device back, I guess this caused the mdev device lost in VM. FWIW, we document the case that suspend doesn't really work with both upstream Nova [1] and our OSP product [2]. As said in https://review.opendev.org/c/openstack/nova/+/815373 we could move on (now that libvirt no longer raises an exception when hot-unpluging) so I think this bug is valid. [1] upstream doc says : "Suspending a guest that has vGPUs doesn’t yet work because of a libvirt limitation (it can’t hot-unplug mediated devices from a guest). Workarounds using other instance actions (like snapshotting the instance or shelving it) are recommended until libvirt gains mdev hot-unplug support. If a user attempts to suspend the instance, the libvirt driver will raise an exception that will cause the instance to be set back to ACTIVE. The suspend action in the os-instance-actions API will have an Error state." https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats [2] eg. our 16.2 doc says : " Suspend operations on a vGPU-enabled instance is not supported due to a libvirt limitation. Instead, you can snapshot or shelve the instance." https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/configuring_the_compute_service_for_instance_creation/assembly_configuring-virtual-gpus-for-instances_vgpu#con_vgpu-supported-configurations-and-limitations_vgpu https://review.opendev.org/c/openstack/nova/+/815373/ Fixed in upstream yoga, will be in OSP 18.0. |