I would like to request a new document for RHV that describes the different ways of setting up vGPU / virtual technical workstation. The document needs to cover: - NVIDIA - Intel - AMD once supported and the following remote visualisation tools: - Mechdyne TGX - NICE DCV - HP RGS - spice-streaming once available The document should also cover how to install the drivers from NVIDIA/AMD as needed as well as how to set up and configure the guests.
Re-targeting to 4.2.x per previous request.
Martin, Ryan, I'm looking at Assigning GPU Devices [1] in the RHEL Virtualization Deployment and Administration Guide. I've been working on a draft based on this content (see comment 8), which Ryan shared with me. [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-device-GPU#sect-device-GPU-vGPU Questions based on Martin's original description: > The document needs to cover: > - Intel > - AMD once supported Are you talking about Intel and AMD onboard graphics? Or are you talking about AMD's discrete graphics cards? Or something else? Intel doesn't have a discrete graphics card right now. > The document should also cover how to install the drivers from NVIDIA/AMD as > needed I think that the existing links in the document pointing to the third party sites should do the job. We shouldn't be documenting third-party driver installation procedures. > as well as how to set up and configure the guests. What is missing from [1] in this regard?
(In reply to Steve Goodman from comment #9) > Martin, Ryan, > > I'm looking at Assigning GPU Devices [1] in the RHEL Virtualization > Deployment and Administration Guide. I've been working on a draft based on > this content (see comment 8), which Ryan shared with me. > > [1] > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ > html/virtualization_deployment_and_administration_guide/sect-device-GPU#sect- > device-GPU-vGPU > > Questions based on Martin's original description: > > > The document needs to cover: > > - Intel > > - AMD once supported > > Are you talking about Intel and AMD onboard graphics? Or are you talking > about AMD's discrete graphics cards? Or something else? Intel doesn't have a > discrete graphics card right now. Correct. Intel does have vGPU enablement on their onboard GPUs. This should work the same way NVIDIA does. I just don't think QE has tested it, but it is supported from RHEL 7.6 onwards. This said, it might be in 4.2.7 or in 4.3 (as we tried to reduce QE load as much as possible). So it should work, but I would put this in for RHV 4.3 earliest unless QE has already tested it. AMD is different, as they are using a SRIOV approach, and this is for their High End GPUs (the feature is named MxGPU) So same as for Intel: Ideally it should work as an SRIOV assignment already, but nobody did test it. I doubt this is a 4.3 issue even. That said, I would focus on NVIDIA and Intel for now as the overall process of assigning the vGPU (mdev) devices is identical. > > > The document should also cover how to install the drivers from NVIDIA/AMD as > > needed > > I think that the existing links in the document pointing to the third party > sites should do the job. We shouldn't be documenting third-party driver > installation procedures. +1 > > > as well as how to set up and configure the guests. > > What is missing from [1] in this regard? [1] only shows how to setup the host. The guest itself is missing. Probably we can again link to NVIDIA/Intel documentation for that. Cheers, Martin
Open related question from https://bugzilla.redhat.com/show_bug.cgi?id=1596289: Martin, If I understand correctly: - The original paragraph mentioned in the first comment in the bug is true and does not need to be edited. - We need to add the following to the list of vGPU requirements: - a 3rd party remote visualization solution. The following solutions have been tested: - Mechdyne TGX - HP RGS - Nice DCV NOTE: SPICE is not supported. Megan also asked about licenses, and I didn't see a response: > Finally I wanted to clarify what license/s are required from nVidia to make > this feature functional? The deep dive mentions requiring an enterprise account > to access the required drivers, is this accurate? It also mentions requiring an > nVidia GRID licensing server? Do you have any further information about this > requirement?
About Hardware Requirements: We discuss PCI device assignment (passing through a host GPU to a VM), and vGPU assignment. For host device passthrough, The Virtual Machine Management Guide specifies only three solutions, all NVIDIA, as being compatible with this usage model [1]: -------------------------------------------------------- Red Hat Enterprise Linux 7 supports PCI device assignment of the following PCIe-based GPU devices as non-VGA graphics devices: NVIDIA Quadro K-Series, M-Series, and P-Series (models 2000 series or higher) NVIDIA GRID K-Series NVIDIA Tesla K-Series and M-Series -------------------------------------------------------- Is this true? We only support NVIDIA solutions? What about AMD/Intel? [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-device-gpu#sect-device-GPU-asignment
(In reply to Steve Goodman from comment #11) > Open related question from > https://bugzilla.redhat.com/show_bug.cgi?id=1596289: > > Martin, > > If I understand correctly: > - The original paragraph mentioned in the first comment in the bug is true > and does not need to be edited. Correct. We can run 3D workloads (and even do with all our current vGPU enabled customers). > - We need to add the following to the list of vGPU requirements: > - a 3rd party remote visualization solution. The following solutions have > been tested: > - Mechdyne TGX > - HP RGS > - Nice DCV > NOTE: SPICE is not supported. > Correct. SPICE is not supported yet. It might be in the future once we figured out the licensing/patenting issues. > > Megan also asked about licenses, and I didn't see a response: > > > Finally I wanted to clarify what license/s are required from nVidia to make > > this feature functional? The deep dive mentions requiring an enterprise account > > to access the required drivers, is this accurate? It also mentions requiring an > > nVidia GRID licensing server? Do you have any further information about this > > requirement? Well you need the NVIDIA license server along with the Enterprise agreement (or something like this) from NVIDIA. Adding Tarun to further comment on this one. Maybe he can provide a Link to the appropriate NVIDIA page. @Tarun: Could you please update around the requirements from NVIDIA to be able to run vGPU(mdev) based workloads? (In reply to Steve Goodman from comment #12) > About Hardware Requirements: > > We discuss PCI device assignment (passing through a host GPU to a VM), and > vGPU assignment. > > For host device passthrough, The Virtual Machine Management Guide specifies > only three solutions, all NVIDIA, as being compatible with this usage model > [1]: > > -------------------------------------------------------- > Red Hat Enterprise Linux 7 supports PCI device assignment of the following > PCIe-based GPU devices as non-VGA graphics devices: > > NVIDIA Quadro K-Series, M-Series, and P-Series (models 2000 series or > higher) > NVIDIA GRID K-Series > NVIDIA Tesla K-Series and M-Series > > -------------------------------------------------------- > > Is this true? We only support NVIDIA solutions? What about AMD/Intel? > We support passthrough for at least AMD (which is the easiest one anyways) as well as for Intel (which I believe needs some quirks). From NVIDIA side the models seem to be correct, as consumer cards are not supported for passthrough. @Tarun: Can you doublecheck for NVIDIA, please? > > [1] > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ > html/virtualization_deployment_and_administration_guide/sect-device-gpu#sect- > device-GPU-asignment
As of now this is what I understand, where the last comment I see in the bug is from Alex, comment 17: – RHV will support Intel vGPUs (GVT-g) in RHV 4.3 but only as a tech preview – RHV does not support Intel device passthrough (GVT-d), not in RHV 4.2, and there are no plans to change this in RHV 4.3. I don't see acknowledgement from Martin yet. – RHV does not support AMD device passthrough nor does it support vGPU. Not in RHV 4.2, and not in RHV 4.3.
(In reply to Steve Goodman from comment #18) > As of now this is what I understand, where the last comment I see in the bug > is from Alex, comment 17: > > – RHV will support Intel vGPUs (GVT-g) in RHV 4.3 but only as a tech preview > – RHV does not support Intel device passthrough (GVT-d), not in RHV 4.2, and > there are no plans to change this in RHV 4.3. I don't see acknowledgement > from > Martin yet. > – RHV does not support AMD device passthrough nor does it support vGPU. Not > in > RHV 4.2, and not in RHV 4.3. Looks correct to me.
(In reply to Martin Tessun from comment #19) > (In reply to Steve Goodman from comment #18) > > As of now this is what I understand, where the last comment I see in the bug > > is from Alex, comment 17: > > > > – RHV will support Intel vGPUs (GVT-g) in RHV 4.3 but only as a tech preview > > – RHV does not support Intel device passthrough (GVT-d), not in RHV 4.2, and > > there are no plans to change this in RHV 4.3. I don't see acknowledgement > > from > > Martin yet. > > – RHV does not support AMD device passthrough nor does it support vGPU. Not > > in > > RHV 4.2, and not in RHV 4.3. > > Looks correct to me. I agree as well. On the NVIDIA side, I assume direct assignment support extends through the P and RTX series Quadros and likely includes the GV100 and PV100 Quadros as well (which breaks our 2000 series and higher statement). Likewise for Tesla, P, V, and T series cards should support both direct assignment and vGPU. I think we still need Tarun to confirm the NVIDIA support though.
Based on the comments and answers from the Google doc in comment in comment 5, and based on similar work being done for RHEL 8, I posted a new draft here: http://file.tlv.redhat.com/~sgoodman/BZ%231631224-Virtual_Machine_Management_Guide/ Here's what I know needs to be fixed: Many or all cross references to NVIDIA documentation have the same text, but link to different URLs. The link text should be more intuitive. The NVIDIA content targets 4.2.8. The Intel content will only appear in 4.3 Beta. Before sending this to QA, I need SME approval.
*** Bug 1485104 has been marked as a duplicate of this bug. ***
Looks mostly ok to me. You're right about the link text. I'm worried that most of 4.11.1.2.4. is misleading, though. First, referencing the vGPU software for a non-mediated device isn't really necessary (other than some older GRID cards). THe "normal" nVidia driver should be ok in these cases unless nVidia has changed this since last time I used it. Secondly, do we want to call the entire section "Managing vGPU devices"? It's probably misleading for the same reason as above (vGPU is specifically about mediated/SR-IOV devices rather than plain passthrough). It's probably more logical to simply name it something like "Configuring Virtual Machines with physical GPUs", though that's kind of a mouthful. At least it covers both cases (passthrough and mdev) by being more generic. I expect you'll find a snappier header
(In reply to Ryan Barry from comment #24) > Looks mostly ok to me. You're right about the link text. > > I'm worried that most of 4.11.1.2.4. is misleading, though. > > First, referencing the vGPU software for a non-mediated device isn't really > necessary (other than some older GRID cards). THe "normal" nVidia driver > should be ok in these cases unless nVidia has changed this since last time I > used it. I based this on the current NVIDIA documentation. In the section on passthrough, it says: "After configuring a Red Hat Enterprise Linux KVM for GPU pass through, install the NVIDIA graphics driver in the guest OS on the VM as explained in Installing the NVIDIA vGPU Software Graphics Driver." [1] And the section it refers to says: "The process for installing the NVIDIA vGPU software graphics driver depends on the OS that you are using. However, for any OS, the process for installing the driver is the same in a VM configured with vGPU, in a VM that is running pass-through GPU, or on a physical host in a bare-metal deployment. After you install the NVIDIA vGPU software graphics driver, you can license any NVIDIA vGPU software licensed products that you are using." [2] And then it lists the details for both Linux and Windows VMs. > > Secondly, do we want to call the entire section "Managing vGPU devices"? > It's probably misleading for the same reason as above (vGPU is specifically > about mediated/SR-IOV devices rather than plain passthrough). It's probably > more logical to simply name it something like "Configuring Virtual Machines > with physical GPUs", though that's kind of a mouthful. At least it covers > both cases (passthrough and mdev) by being more generic. > > I expect you'll find a snappier header How about: Setting Up a GPU and Using it on a Virtual Machine [1] https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#using-gpu-pass-through [2] https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#installing-grid-vgpu-display-drivers
(In reply to Steve Goodman from comment #25) > ... > And then it lists the details for both Linux and Windows VMs. > I don't have a quadro to test with, but I'm reasonably sure that it doesn't require any licensing for passthrough. Nisim, Martin? > > > > Secondly, do we want to call the entire section "Managing vGPU devices"? > > It's probably misleading for the same reason as above (vGPU is specifically > > about mediated/SR-IOV devices rather than plain passthrough). It's probably > > more logical to simply name it something like "Configuring Virtual Machines > > with physical GPUs", though that's kind of a mouthful. At least it covers > > both cases (passthrough and mdev) by being more generic. > > > > I expect you'll find a snappier header > > How about: Setting Up a GPU and Using it on a Virtual Machine > This sounds good to me.
I changed the text for the links so they are more intuitive. Of course the danger of this is that I'm not in control of the NVIDIA docs, so any change they make to their URLs breaks our links. If that does happen, there's enough guidance in the link text that our users should be able to find their way to the right location, albeit with a bit more effort. I also changed the section titles. I need final confirmation on the licensing requirements, but I just double-checked the NVIDIA docs, and they seem pretty clear. If the licensing model for passthrough and bare-metal is different, then their docs are buggy. What can confirm one or the other? Do we need someone with a relationship with NVIDIA to contact them to confirm?
> > The NVIDIA documentation for setting up GPU passthrough says that installing the guest drivers is the same for both passthrough and mdev installation. Take a look at Using GPU Passthrough. > > I notice that in the passthrough guide we prepared, we don't mention specifically installing the guest driver. We jumped from Attaching the GPU to a Virtual Machine to Updating and Enabling xorg. > > Was that a mistake on my part? I assume it is. It looks like a mistake. After attaching the GPU to a VM, Nvidia guest drivers needs to be installed and if installation is not asking for updating X configuration then we need to generate xorg.conf manually: 1. # X -configure 2. # cp /root/xorg.conf.new /etc/X11/xorg.conf 3. # init 5 as for https://bugzilla.redhat.com/show_bug.cgi?id=1631224#c26 I'm using Quadro K4200 and Quadro K2200 and it doesn't requires any licensing for passthrough.
(In reply to Nisim Simsolo from comment #28) > > as for https://bugzilla.redhat.com/show_bug.cgi?id=1631224#c26 > I'm using Quadro K4200 and Quadro K2200 and it doesn't requires any > licensing for passthrough. The official NVIDIA documentation says otherwise. Martin, don't we have a contact at NVIDIA that we can ask about this?
(In reply to Steve Goodman from comment #29) > (In reply to Nisim Simsolo from comment #28) > > > > as for https://bugzilla.redhat.com/show_bug.cgi?id=1631224#c26 > > I'm using Quadro K4200 and Quadro K2200 and it doesn't requires any > > licensing for passthrough. > > The official NVIDIA documentation says otherwise. Martin, don't we have a > contact at NVIDIA that we can ask about this? Tarun, can you help with this?
Restoring accidentally removed needinfos.
(In reply to Steve Goodman from comment #29) > (In reply to Nisim Simsolo from comment #28) > > > > as for https://bugzilla.redhat.com/show_bug.cgi?id=1631224#c26 > > I'm using Quadro K4200 and Quadro K2200 and it doesn't requires any > > licensing for passthrough. > > The official NVIDIA documentation says otherwise. Martin, don't we have a > contact at NVIDIA that we can ask about this? I also remember Tarun mentioning licensing was coming for direct assignment as well, but this was way back during the initial vGPU enablement, I don't know if it ever materialized. I expect older cards would be grandfathered in, so K-series Quadros are probably not representative of such a change. We'd probably need to look at M-series, P-series, or perhaps newer to see if NVIDIA ever implemented this.
UPDATE: I sent a request for clarification to the docs team at NVIDIA. (Their public docs have a link for doc feedback.) It's the holiday break for many people right now, so this will hopefully get addressed in January 2019.
Here's the answer I received from NVIDIA: > To answer your latest question: The requirement for an NVIDIA vGPU software license even for GPU pass. through deployments is not new and the documentation is correct on this point. > However, from a comment in your ongoing discussion on Bugzilla, I wonder if one source of confusion is that the official NVIDIA documentation to which you refer applies to software that is not supported on the GPUs that are being used: >>> >>> as for https://bugzilla.redhat.com/show_bug.cgi?id=1631224#c26 >>> I'm using Quadro K4200 and Quadro K2200 and it doesn't requires any >>> licensing for passthrough. >> The official NVIDIA documentation says otherwise. Martin, don't we have a contact at NVIDIA that we can ask about this? > Because NVIDIA vGPU software is not supported on Quadro K4200 and Quadro K2200 GPUs, any discussion in the NVIDIA vGPU software documentation about setup, driver installation, or licensing does not apply to these GPUs. > For the list of GPUs on which NVIDIA vGPU software is supported with Red Enterprise Linux with KVM and RHV, see Supported NVIDIA GPUs and Validated Server Platforms [1] in the Virtual GPU Software R410 for Red Hat Enterprise Linux with KVM Release Notes. > All NVIDIA vGPU software deployments (NVIDIA vGPU, pass through, bare metal, DDA, vDGA, RemoteFX vGPU, and vSGA) on these supported GPUs require a license. The type of license required (GRID Virtual Applications, GRID Virtual PC, or Quadro vDWS) and how licensing is enforced (through software or through the EULA) depend on the NVIDIA vGPU software deployment. For details, see How NVIDIA vGPU Software Licensing Is Enforced[2]. [1] https://docs.nvidia.com/grid/latest/grid-vgpu-release-notes-red-hat-el-kvm/index.html#hardware-configuration [2] https://docs.nvidia.com/grid/latest/grid-licensing-user-guide/index.html#how-grid-licensing-works
Latest correspondence with NVIDIA regarding my last comment (which I also asked them): > There’s a difference between what works on a GPU and what is supported. vGPU software is not supported on Quadro K4200 and Quadro K2200 GPUs. You may find that you can > get it to work, but you will not be in a position to request assistance for any bugs or issues you encounter while doing so. > Please refer to Supported NVIDIA GPUs and Validated Server Platforms for the list of vGPU software supported GPUs. So it appears that GPU passthrough has licensing requirements similar to vGPU instantiations. Which suggests that the document is correct as it is (that is, the install process for guest drivers that document is identical for both passthrough and vGPU. Please confirm and I'll move this forward.
Sounds good to me
Not blocking ovirt-4.3.0 on this. Moving to 4.3.1
Getting back to this. Martin, I see that we still a categorical statement regarding NVIDIA hardware support. We're now in the 4.3 beta stage. Here's what I see in the current 4.3 installation guide: Under Device Assignment Requirements [1]: >For GPU support, Red Hat Enterprise Linux 7 supports PCI device assignment of PCIe-based NVIDIA K-Series Quadro (model 2000 series or higher), GRID, and Tesla as non-VGA graphics devices. Currently up to two GPUs may be attached to a virtual machine in addition to one of the standard, emulated VGA interfaces. The emulated VGA is used for pre-boot and installation and the NVIDIA GPU takes over when the NVIDIA graphics drivers are loaded. Note that the NVIDIA Quadro 2000 is not supported, nor is the Quadro K420 card. But unless I misunderstand, comment 39 indicates that this is not correct. Can someone provide clear guidance? [1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3-beta/html-single/installation_guide/#Device_Assignment_Requirements_RHV_install
Also, Nisim, just confirming that the content we're preparing here will replace the following sections: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3-beta/html-single/installation_guide/#Preparing_a_Host_for_vGPU_Installation (the same content is in the VMM guide [1]) https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3-beta/html-single/administration_guide/index#Preparing_GPU_Passthrough [1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3-beta/html-single/virtual_machine_management_guide/#Installing_a_vGPU_on_a_Virtual_Machine
Hi Steve, (In reply to Steve Goodman from comment #49) > Getting back to this. > > Martin, I see that we still a categorical statement regarding NVIDIA > hardware support. We're now in the 4.3 beta stage. Here's what I see in the > current 4.3 installation guide: > > Under Device Assignment Requirements [1]: > > >For GPU support, Red Hat Enterprise Linux 7 supports PCI device assignment of PCIe-based NVIDIA K-Series Quadro (model 2000 series or higher), GRID, and Tesla as non-VGA graphics devices. Currently up to two GPUs may be attached to a virtual machine in addition to one of the standard, emulated VGA interfaces. The emulated VGA is used for pre-boot and installation and the NVIDIA GPU takes over when the NVIDIA graphics drivers are loaded. Note that the NVIDIA Quadro 2000 is not supported, nor is the Quadro K420 card. > > But unless I misunderstand, comment 39 indicates that this is not correct. > Can someone provide clear guidance? Well it depends. The above paragraph is for GPU passthrough only, no vGPU. vGPU itself is supported on NVIDIA Tesla GRID cards, starting with the Maxwell series (M6,10,60) to Pascal (P4,6,40,100) and Volta (T4, V100) Clear guidance on vGPU support can be found at NVIDIA's pages: https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#supported-gpus-grid-vgpu > > [1] > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3- > beta/html-single/installation_guide/ > #Device_Assignment_Requirements_RHV_install
Thanks, Martin. If NVIDIA is the source of truth, then is there any reason we shouldn't just refer to the NVIDIA site for supported hardware?
(In reply to Steve Goodman from comment #52) > Thanks, Martin. > > If NVIDIA is the source of truth, then is there any reason we shouldn't just > refer to the NVIDIA site for supported hardware? We can, but we should make sure that if we link to NVIDIA, we need to ensure that we don't point to a dead link from time to time. Cheers, Martin
(In reply to Martin Tessun from comment #53) > > ...we should make sure that if we link to NVIDIA, we need to ensure > that we don't point to a dead link from time to time. > > Cheers, > Martin You are 100% right, and this is an issue any time we link to an external document, and it is a challenge in lot of places in Red Hat's documentation set. However, we don't have a fool-proof solution to this problem. We have a link checker, but it only works when a human being initiates it. So it's possible that you check links on Jan 1 and it works fine, on Jan 2 the target page goes down, and you check the link again six months later, (which is a more-than-realistic frequency). In this scenario you'll have a broken link for six months before it's noticed. In my past workplace we dealt with this issue by cutting our losses: Instead of directly linking to this kind of target content, we mentioned the title of the target page and the top level domain, and left it to the user to search for it on the target site. Less user-friendly than providing a direct link, but more user-friendly than providing a broken link.
(In reply to Steve Goodman from comment #54) > (In reply to Martin Tessun from comment #53) > > > > ...we should make sure that if we link to NVIDIA, we need to ensure > > that we don't point to a dead link from time to time. > > > > Cheers, > > Martin > > You are 100% right, and this is an issue any time we link to an external > document, and it is a challenge in lot of places in Red Hat's documentation > set. However, we don't have a fool-proof solution to this problem. We have a > link checker, but it only works when a human being initiates it. So it's > possible that you check links on Jan 1 and it works fine, on Jan 2 the > target page goes down, and you check the link again six months later, (which > is a more-than-realistic frequency). In this scenario you'll have a broken > link for six months before it's noticed. Steve, the tests I set up in http://ccs-jenkins.gsslab.brq.redhat.com:8080/view/Tests%20-%20Red%20Hat%20Virtualization/ are actually automated and get updated every time a new commit is pushed to the repo. So we just need to figure out how often we want to check the test results.
Here's a WIP merge request for this: https://gitlab.cee.redhat.com/rhci-documentation/docs-Red_Hat_Enterprise_Virtualization/merge_requests/1181
*** Bug 1759460 has been marked as a duplicate of this bug. ***
The doc to review is here: https://pnt-devops-rad-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/ccs-mr-preview-prod/9033/artifact/assembly-Setting_up_a_gpu_on_a_vm/preview/index.html
This is now published to https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/setting_up_an_nvidia_gpu_for_a_virtual_machine_in_red_hat_virtualization/index