RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2110203 - zerocopy capability can be enabled when set migrate capabilities with multifd and compress/xbzrle together
Summary: zerocopy capability can be enabled when set migrate capabilities with multifd...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Leonardo Bras
QA Contact: Li Xiaohui
URL:
Whiteboard:
: 2114852 (view as bug list)
Depends On: 2072049 2107466
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-24 14:21 UTC by Li Xiaohui
Modified: 2022-12-28 02:53 UTC (History)
16 users (show)

Fixed In Version: qemu-kvm-6.2.0-19.module+el8.7.0+16358+eef3c6a2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2107466
: 2117253 (view as bug list)
Environment:
Last Closed: 2022-11-08 09:20:55 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Gitlab redhat/rhel/src/qemu-kvm qemu-kvm merge_requests 201 0 None None None 2022-07-27 01:30:57 UTC
Red Hat Issue Tracker RHELPLAN-128795 0 None None None 2022-07-24 14:28:18 UTC
Red Hat Product Errata RHSA-2022:7472 0 None None None 2022-11-08 09:21:47 UTC

Description Li Xiaohui 2022-07-24 14:21:10 UTC
+++ This bug was initially created as a clone of Bug #2107466 +++

Description of problem:
For zerocopy, we don't support zerocopy enabled when compress is on. 
But when we set migrate capabilities multifd, zerocopy, compress together, they can succeed;
when set these capability separately, zerocopy can't be enabled under compress enabled, it's the expectation


Version-Release number of selected component (if applicable):
hosts info: hosts info: kernel-5.14.0-121.el9.x86_64 & qemu-kvm-7.0.0-8.el9.x86_64
guest info: kernel-5.14.0-125.el9.x86_64


How reproducible:
100%


Steps to Reproduce:
1.Boot a guest
2.Set multifd, zerocopy, compress capabilities on together:
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"xbzrle","state":false},{"capability":"auto-converge","state":false},{"capability":"rdma-pin-all","state":false},{"capability":"postcopy-ram","state":false},{"capability":"compress","state":true},{"capability":"pause-before-switchover","state":false},{"capability":"late-block-activate","state":false},{"capability":"multifd","state":true},{"capability":"dirty-bitmaps","state":false},{"capability":"return-path","state":false},{"capability":"zero-copy-send","state":true}]}}
{"return": {}}


Actual results:
Same as Steps, enable compress, multifd, zerocopy capabilities together succeed


Expected results:
Can't enable zerocopy and compress capabilities together

If we enable compress, multifd, zerocopy separately, zerocopy will fail to enable:
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"compress","state":true}]}}
{"return": {}}
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"multifd","state":true}]}}
{"return": {}}
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"zero-copy-send","state":true}]}}
{"error": {"class": "GenericError", "desc": "Zero copy only available for non-compressed non-TLS multifd migration"}}


Additional info:

--- Additional comment from Li Xiaohui on 2022-07-15 15:04:30 CST ---

As libvirt always set all migrate capabilities together through one qmp command "migrate-set-capabilities", we should fix this bug.

--- Additional comment from Li Xiaohui on 2022-07-15 15:23:44 CST ---

BTW, shall we also avoid to enable compress capability successfully if zerocopy enabled like:
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"multifd","state":true}]}}
{"return": {}}
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"zero-copy-send","state":true}]}}
{"return": {}}
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"compress","state":true}]}}
{"return": {}}

--- Additional comment from Leonardo Bras on 2022-07-16 04:41:12 CST ---

Thanks for reporting Li Xiaohui!

I think I have an idea on why this happens, and just wrote a probable fix.

I tested on a scratch build. Could you please give it a try?
https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=1295372

--- Additional comment from Li Xiaohui on 2022-07-24 17:13:06 CST ---

Thanks Leoardo to provide the scratch build in https://bugzilla.redhat.com/show_bug.cgi?id=1968509#c26.


I have tested the build (qemu-kvm-7.0.0-8.el9.leonardo202207190013.x86_64), the issues in Description, Comment 2 have been fixed, and we also don't support enable xbzrle with zerocopy enabled.


Only following question still exist, do you plan to fix it?
1.When try to migrate with tls + multifd + zerocopy (note tls certs has been set up on source and destination host)
a. if set tls creds on src and dst host firstly, then enable multifd, zerocopy, will get error prompt, it's the expectation:
{"execute": "migrate-set-parameters", "arguments": {"tls-creds": "tls0"}, "id": "hdTiagq5"}
...
{"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"capability": "zero-copy-send", "state": true}]}, "id": "wgr8gT3T"}
{"id": "wgr8gT3T", "error": {"class": "GenericError", "desc": "Zero copy only available for non-compressed non-TLS multifd migration"}}
b. but if enable multifd, zerocopy firstly, then set tls creds, all will succeed, but when start migration, migration will fail like:
{"execute": "query-migrate", "id": "qUqoTVuL"}
{"return": {"status": "failed", "error-desc": "Requested Zero Copy feature is not available: Invalid argument"}, "id": "qUqoTVuL"}

My question: for situation b, shall we avoid set tls creds successfully when zerocopy is enabled? or give accurate error prompt than above error-desc like zerocopy is enabled, but don't support tls migration under zerocopy.

Comment 1 John Ferlan 2022-07-25 20:29:14 UTC
Leo since you already own the cloned from bug, I'll assign directly to you.

Comment 2 Leonardo Bras 2022-07-26 01:35:27 UTC
(In reply to John Ferlan from comment #1)
> Leo since you already own the cloned from bug, I'll assign directly to you.

Sure, no problem.

It this one completely fixed by the brew I provided in the original bug, right?

Comment 3 Li Xiaohui 2022-07-26 07:33:54 UTC
(In reply to Leonardo Bras from comment #2)
> (In reply to John Ferlan from comment #1)
> > Leo since you already own the cloned from bug, I'll assign directly to you.
> 
> Sure, no problem.
> 
> It this one completely fixed by the brew I provided in the original bug,
> right?

This bug is for rhel8.7.0, and the brew you provided in https://bugzilla.redhat.com/show_bug.cgi?id=1968509#c26 have fixed the rhel9 issues.

So now you need to fix these issues for RHEL 8.7.0.

Comment 4 Leonardo Bras 2022-07-27 02:09:44 UTC
(In reply to Li Xiaohui from comment #3)
[...]
> So now you need to fix these issues for RHEL 8.7.0.

I have created a MR for those changes on top of rhel8.7: 
https://gitlab.com/redhat/rhel/src/qemu-kvm/qemu-kvm/-/merge_requests/201

It's still in 'draft' because there is a patch which is still under review upstream: the one that fixes zero-copy-send & tls-creds config order.[1]

I have created a brew which already includes the above build, if yoo think it's interesting to test:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=46761695


[1] http://patchwork.ozlabs.org/project/qemu-devel/patch/20220726010235.342927-1-leobras@redhat.com/)

Comment 5 John Ferlan 2022-08-01 18:22:08 UTC
(In reply to Leonardo Bras from comment #2)
> (In reply to John Ferlan from comment #1)
> > Leo since you already own the cloned from bug, I'll assign directly to you.
> 
> Sure, no problem.
> 
> It this one completely fixed by the brew I provided in the original bug,
> right?

Feel free to add a needinfo on me for questions... It'll get my attention quicker usually.

Essentially you have an upstream set of patches which you need to backport to both RHEL9 (bug 2107466) and RHEL8 (this one). In the long run the order isn't all that important, just that both get the patches. Now that you have this series, I'm going to add ITR=8.7.0 for this bug with a DTM=24 (Developer Target Milestone) which leaves us 2 weeks to get downstream reviews completed. QE then usually adds 2 to the DTM to set ITM (Internal Target Milestone). Our "goal" is to be "done" by ITM=26 and leave essentially critical things beyond that. In RHEL release teams that's when exceptions must be requested.

Furthermore, I assume this bug will eventually need a z-stream for 8.6.0 similar to bug 2072049, so I have added that as well with the dependency to that bug. Just trying to make sure everything we need is tracked.

Comment 7 Leonardo Bras 2022-08-10 21:54:16 UTC
A brew that is supposed to fix this bug is already available at MR[1]:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=47002821

[1] https://gitlab.com/redhat/rhel/src/qemu-kvm/qemu-kvm/-/merge_requests/201
The above MR is ready for merging, only waiting for 1 dev approval.

Comment 8 Li Xiaohui 2022-08-15 13:08:05 UTC
(In reply to Leonardo Bras from comment #7)
> A brew that is supposed to fix this bug is already available at MR[1]:
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=47002821

I have used this scratch build to test zerocopy, following issues has been solved on qemu-kvm-6.2.0-18.el8.root202208061423.x86_64:
1. zerocopy capability can not be enabled when set migrate capabilities with multifd and compress/xbzrle together;
2. compress/xbzrle can not be enabled after zerocopy and multifd enabled;
3. get proper prompt when set tls on tls client after zerocopy enabled:
{"execute": "migrate-set-parameters", "arguments": {"tls-creds": "tls0"}, "id": "PAwwXyyu"}
{"id": "PAwwXyyu", "error": {"class": "GenericError", "desc": "Zero copy only available for non-compressed non-TLS multifd migration"}}


So seems this build has fixed above issues. Thanks Leonardo.

> 
> [1] https://gitlab.com/redhat/rhel/src/qemu-kvm/qemu-kvm/-/merge_requests/201
> The above MR is ready for merging, only waiting for 1 dev approval.

Comment 10 Yanan Fu 2022-08-17 02:53:24 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 11 Li Xiaohui 2022-08-17 09:53:15 UTC
Verify this bug according to Comment 8 on qemu-kvm-6.2.0-19.module+el8.7.0+16358+eef3c6a2.x86_64, all issues have been fixed. 

Will mark this bug verified per test results after it's in on_qa status. 
Now remove 'SanityOnly' from 'Verified' since we have test steps to reproduce this bug

Comment 12 Li Xiaohui 2022-08-18 01:07:42 UTC
Hi Miroslav, 
can you help add this bug to errata system then it can be changed into on_qa so that QE can mark this bug verified per Comment 11?

Comment 16 Li Xiaohui 2022-09-01 06:34:17 UTC
*** Bug 2114852 has been marked as a duplicate of this bug. ***

Comment 18 errata-xmlrpc 2022-11-08 09:20:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7472


Note You need to log in before you can comment on or make changes to this bug.