Bug 2155612
| Summary: | IOThreads are not bound to CPUs, nor there are enough CPUs allocated | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Nils Koenig <nkoenig> | ||||
| Component: | Virtualization | Assignee: | sgott | ||||
| Status: | NEW --- | QA Contact: | Kedar Bidarkar <kbidarka> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 4.10.9 | CC: | acardace, dholler, djdumas, fdeutsch, jhopper, jlejosne, nkoenig, vromanso | ||||
| Target Milestone: | --- | Flags: | nkoenig:
needinfo?
(vromanso) |
||||
| Target Release: | 4.15.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | Type: | Bug | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
It might be worthwhile to point out that the following settings were used:
isolateEmulatorThread: true
and then
dedicatedIOThread: true
for each disk.
In the KubeVirt docs https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#high-performance-features For the "IOThreads with Dedicated (pinned) CPUs" section, it appears the described behavior matches your description of what happened. Vladik, is there a way to achieve what Nils expected to happen? (In reply to sgott from comment #4) > In the KubeVirt docs > > https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#high- > performance-features > > For the "IOThreads with Dedicated (pinned) CPUs" section, it appears the > described behavior matches your description of what happened. > > Vladik, is there a way to achieve what Nils expected to happen? I think it's a bug that we are not treating all of the created IOthreads but the first one. We should pin all of these to the overallocated pCPU. With that said, I think there is no benefit of creating multiple threads when the emulator thread and IOThread are isolated on a separate pCPU. @vromanso
> I think it's a bug that we are not treating all of the created IOthreads but the first one. We should pin all of these to the overallocated pCPU.
> With that said, I think there is no benefit of creating multiple threads when the emulator thread and IOThread are isolated on a separate pCPU.
What about a usecase where you have two devices that sit on different NUMA nodes and hence would require different IOThreads?
We are pushing this Bug to 4.15 as per the severity and taking teams capacity into consideration. |
Created attachment 1933971 [details] libvirt domain xml Description of problem: Using IOThreads gives better performance for virtio based disks. Ideally the IOThread is bound to a specific CPU and stays on that CPU. In the current implementation there is one CPU specified automatically, but looking on which CPU the IOThreads are running on shows that this setting is not taken into account. Additionally when using multiple IOThreads, one would like to specify more then one CPU (one CPU per IOThread). Version-Release number of selected component (if applicable): $ oc version Client Version: 4.10.45 Server Version: 4.10.45 How reproducible: Using a VM with the attached configuration and looking at which CPUs the actual IOThreads are running on when the guest does it's IO. The domxml for the VM specifies: <emulatorpin cpuset='25'/> <iothreadpin iothread='1' cpuset='25'/> but looking where the IOThreads run, shows that it's only IOThread 1 that is using that pinning, the others can use almost all CPUs: # ps -T -p 832716 PID SPID TTY TIME CMD 832716 832716 ? 00:00:10 qemu-kvm 832716 832724 ? 00:00:00 IO iothread1 832716 832725 ? 00:00:01 IO iothread2 832716 832726 ? 00:01:18 IO iothread3 832716 832727 ? 00:02:33 IO iothread4 832716 832728 ? 00:00:05 IO iothread5 # taskset -cp 832724 pid 832724's current affinity list: 25 [root@worker-0 tools]# taskset -cp 832725 pid 832725's current affinity list: 1-25,28-111,113-136,140-223 [root@worker-0 tools]# taskset -cp 832726 pid 832726's current affinity list: 1-25,28-111,113-136,140-223 [root@worker-0 tools]# taskset -cp 832727 pid 832727's current affinity list: 1-25,28-111,113-136,140-223 [root@worker-0 tools]# taskset -cp 832728 pid 832728's current affinity list: 1-25,28-111,113-136,140-223 Expected results: The IOthreads should run on the CPU defined in the domain.xml. Additional info: Ideally, the CPUs for the emulator pin can be specified in the VM.yml. This is important since you would want those threads to be on the NUMA socket where also the storage controller is located.