Bug 632722 - [6.1 FEAT] QEMU static tracing framework
[6.1 FEAT] QEMU static tracing framework
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.1
other All
medium Severity medium
: beta
: 6.1
Assigned To: Jes Sorensen
Virtualization Bugs
: FutureFeature
Depends On:
Blocks: 538808 655920 580566 580954 LibvirtQEMUTrace 737763
  Show dependency treegraph
 
Reported: 2010-09-10 16:11 EDT by IBM Bug Proxy
Modified: 2013-01-09 18:06 EST (History)
15 users (show)

See Also:
Fixed In Version: qemu-kvm-0.12.1.2-2.131.el6
Doc Type: Enhancement
Doc Text:
Cause: Customer requesting support for dtrace style tracing in QEMU. Consequence: Without these changes, there was no/limited support for tracing of events within QEMU. Change: Applied patches provided by customer, which implements dtrace style tracing in QEMU Result: When used in conjunction with systemtap, it is now possible to trace internal QEMU events such as IO operations and memory allocations.
Story Points: ---
Clone Of:
: 737763 (view as bug list)
Environment:
Last Closed: 2011-05-19 07:28:55 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
tarball containing patches to backport qemu tracing for rhel 6.1 (58.88 KB, text/plain)
2010-11-20 07:44 EST, IBM Bug Proxy
no flags Details
Tarball containing set of patches ( 0001 to 0022 ) that add qemu-tracing capability. (28.27 KB, application/x-bzip2)
2010-11-22 06:01 EST, IBM Bug Proxy
no flags Details
Changes to qemu spec file to build qemu with tracing support. (1.43 KB, text/plain)
2010-11-22 06:01 EST, IBM Bug Proxy
no flags Details
Patch series for qemu tracing (1.17 KB, application/octet-stream)
2010-11-22 06:11 EST, IBM Bug Proxy
no flags Details
upstream commit patches pertaining to qemu tracing (30.08 KB, application/x-bzip2)
2010-12-17 02:43 EST, IBM Bug Proxy
no flags Details
SystemTap tracing backport onto qemu-kvm 0.12.1.2-2.121.el6 (19.95 KB, application/x-bzip)
2010-12-20 10:15 EST, Stefan Hajnoczi
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
IBM Linux Technology Center 67117 None None None Never

  None (edit)
Description IBM Bug Proxy 2010-09-10 16:11:34 EDT
1. Feature Overview:
Feature Id: [67117]
a. Name of Feature: [6.1 FEAT] QEMU static tracing framework
b. Feature Description
QEMU trace infrastructure is slated to go into upstream QEMU. We'd like that feature to be part of
QEMU shipped with RHEL6.1.

If RH moves to a new upstream QEMU version for 6.1, this feature may already be available there.
Else, we'll need a backport.


2. Feature Details:
Sponsor: LTC RAS
Architectures:  x86, x86_64, 

Arch Specificity: purely common code
Affects Kernel Modules: Field does not exist
Delivery Mechanism: Backport
Category: other
Request Type: Package - Update Version
d. Upstream Acceptance: Field does not exist
Sponsor Priority P3
f. Severity: normal
IBM Confidential: No
Code Contribution: IBM code
g. Component Version Target:---

3. Business Case
Tracing the happenings in QEMU is critical to zeroing on performance issues as well as bugs in the
layer in particular and the Linux Virtualization stack in general.

4. Primary contact at Red Hat:
John Jarvis, jjarvis@redhat.com

5. Primary contacts at Partner:
Project Management Contact:
Michael W. Wortman, wortman@us.ibm.com

Technical contact(s):
PRERNA SAXENA, saxena.prerna@in.ibm.com
Comment 2 IBM Bug Proxy 2010-10-04 12:05:25 EDT
------- Comment From rsisk@us.ibm.com 2010-10-04 11:32 EDT-------
Code Upstream Status: In Progress
Comment 5 Jes Sorensen 2010-11-10 12:23:15 EST
Is there a patch set available somewhere for the features you would like added?
Comment 8 IBM Bug Proxy 2010-11-20 07:44:46 EST
------- Comment From prerna@linux.vnet.ibm.com 2010-11-20 00:20 EDT-------
(In reply to comment #6)
> Created an attachment (id=57726) [details]
> tarball containing patches to backport qemu tracing for rhel 6.1
>
> This set of 23 patches ( 22 patches starting number 1694 to 1715 for qemu
> tracing; one more patch in addition, for patching the qemu-kvm.spec file ) adds
> qemu tracing capabilities for rhel 6.1
>
> This is used to enable qemu to be built with systemtap tracing backend. It
> requires systemtap >= 1.3 ( already specified in spec file  build dependencies
> )

This patch series includes the most recent version(v6) of systemtap backend ( Patch # 1714 and 1715 ).
Unfortunately an older version(v5) of this patchset was erroneously merged upstream ( see http://lists.gnu.org/archive/html/qemu-devel/2010-11/msg01789.html )
It would be good if upstream qemu also contained the most recent patchset.
Comment 9 IBM Bug Proxy 2010-11-20 07:44:52 EST
Created attachment 461716 [details]
tarball containing patches to backport qemu tracing for rhel 6.1


------- Comment on attachment From prerna@linux.vnet.ibm.com 2010-11-20 00:05 EDT-------


This set of 23 patches ( 22 patches starting number 1694 to 1715 for qemu tracing; one more patch in addition, for patching the qemu-kvm.spec file ) adds qemu tracing capabilities for rhel 6.1

This is used to enable qemu to be built with systemtap tracing backend. It requires systemtap >= 1.3 ( already specified in spec file  build dependencies )
Comment 10 IBM Bug Proxy 2010-11-22 06:01:17 EST
Created attachment 461967 [details]
Tarball containing set of patches ( 0001 to 0022 ) that add qemu-tracing capability.


------- Comment on attachment From prerna@linux.vnet.ibm.com 2010-11-22 05:55 EDT-------


This contains a set of patches that add qemu tracing support, along with systemtap trace backend which enables systemtap-userspace probing capability to trace qemu execution.

This depends on systemtap >=1.3
Comment 11 IBM Bug Proxy 2010-11-22 06:01:22 EST
Created attachment 461968 [details]
Changes to qemu spec file to build qemu with tracing support.


------- Comment on attachment From prerna@linux.vnet.ibm.com 2010-11-22 05:57 EDT-------


The qemu tracing framework is to be compiled with the 'dtrace' backend, which is specified with the '--trace-backend=dtrace' config option.

Also, it needs systemtap > 1.3 & systemtap-sdt-devel >= 1.3
Comment 12 IBM Bug Proxy 2010-11-22 06:11:29 EST
Created attachment 461973 [details]
Patch series for qemu tracing


------- Comment on attachment From prerna@linux.vnet.ibm.com 2010-11-22 06:02 EDT-------


This lists the patch series contained in attachment #57747 [details]
Comment 13 IBM Bug Proxy 2010-12-06 04:41:05 EST
------- Comment From prerna@linux.vnet.ibm.com 2010-12-06 04:35 EDT-------
Hi Redhat,
Would this patch series make it to RHEL 6.1 ?
Comment 14 Dor Laor 2010-12-06 07:15:39 EST
We'll try to get it into 6.1, it is a function of capacity.
Does all the code upstream by now, including the qmp commands?
Comment 15 Daniel Berrange 2010-12-06 07:31:44 EST
The final version 6 of the DTrace patches was committed upstream in

commit c276b17da65b7ff01627722a1abf2b7a684c8fd8
Author: Daniel P. Berrange <berrange@redhat.com>
Date:   Fri Nov 12 13:20:25 2010 +0000

    Add support for generating a systemtap tapset static probes

commit b3d08c029dd78ded5e35b74eaaa3d361821f83a7
Author: Daniel P. Berrange <berrange@redhat.com>
Date:   Fri Nov 12 13:20:24 2010 +0000

    Add a DTrace tracing backend targetted for SystemTAP compatability


There are no QMP commands required for this. So the backported patches posted by IBM on this bug should be sufficient.
Comment 16 Jes Sorensen 2010-12-15 11:33:05 EST
Prerna,

I have been trying to get this patches to apply to the current RHEL6.1
qemu-kvm, but not had any luck.

It wasn't clear to me at all which tree you had created the patches 
against, and the git commit IDs in the patches were missing, or against
a tree that I do not have. I also noticed that some of your patches were
a combination of multiple git commits from upstream, which makes it
really difficult to match.

I have tried to match all the commit messages to upstream git commits and
pull them in manually, but something isn't working. I spoke to Stefan
about this and he thinks something is missing.

Would it be possible for you to do a fresh respin of these patches against
the latest RHEL6.1 qemu-kvm package?

Thanks,
Jes
Comment 17 IBM Bug Proxy 2010-12-16 03:01:13 EST
------- Comment From mahesh@linux.vnet.ibm.com 2010-12-16 02:52 EDT-------
Reply from Ananth (mananth@in.ibm.com):

(In reply to comment #16)
> Prerna,
>
> I have been trying to get this patches to apply to the current RHEL6.1
> qemu-kvm, but not had any luck.
>
> It wasn't clear to me at all which tree you had created the patches
> against, and the git commit IDs in the patches were missing, or against
> a tree that I do not have. I also noticed that some of your patches were
> a combination of multiple git commits from upstream, which makes it
> really difficult to match.

Jes,

AFAIK, the patches were generated against the qemu-kvm sources shipped
with RHEL6.

>
> I have tried to match all the commit messages to upstream git commits and
> pull them in manually, but something isn't working. I spoke to Stefan
> about this and he thinks something is missing.
>
> Would it be possible for you to do a fresh respin of these patches against
> the latest RHEL6.1 qemu-kvm package?
>
> Thanks,
> Jes

We (IBM) don't have access to the latest RHEL6.1 source tree. Can you
provide us the latest source RPM to base the patches against?
Comment 18 Jes Sorensen 2010-12-16 09:00:53 EST
Hi Ananth,

I asked Anthony to pull a copy of the srpm so you can get it.

Cheers,
Jes
Comment 19 Jes Sorensen 2010-12-16 12:00:35 EST
Hi

Another question, when we get the patches sorted, which trace target
are we supposed to enable in the binary? dtrace or another one?

Thanks,
Jes
Comment 20 IBM Bug Proxy 2010-12-17 02:42:57 EST
------- Comment From prerna@linux.vnet.ibm.com 2010-12-17 02:41 EDT-------
Hi Jes,
(In reply to comment #16)
> Prerna,
>
> I have been trying to get this patches to apply to the current RHEL6.1
> qemu-kvm, but not had any luck.
> It wasn't clear to me at all which tree you had created the patches
> against, and the git commit IDs in the patches were missing, or against
> a tree that I do not have.

I had verified with Anthony around mid-november, and was asked to base my patches against RHEL 6 qemu-kvm sources. Considering a Dec 18 deadline for feature code requests, the information about a new rhel 6.1 codebase seems rather new!

> I also noticed that some of your patches were
> a combination of multiple git commits from upstream, which makes it
> really difficult to match.

Yes, quite a few upstream commits were related (some being minor fixes on the earlier features), so I'd coalesced them together into a single patch.

> I have tried to match all the commit messages to upstream git commits and
> pull them in manually, but something isn't working. I spoke to Stefan
> about this and he thinks something is missing.
>
> Would it be possible for you to do a fresh respin of these patches against
> the latest RHEL6.1 qemu-kvm package?
>

I still have no information of a rhel 6.1 srpm. I am slated to start my vacation from today, and would only be back in the first week of Jan. If this is urgent, would it be possible for you to rebase the upstream commits against the new code base you are working with? I'm attaching a tarball (attachment #58348 ) which contains all the upstream commit patches that I had backported for this feature. Each patch has the upstream commit details for your reference.

Pls note that this needs to be built with systemtap >=1.3. Also, ' trace-backend=dtrace ' needs to be provided at configuration time for the binary to be built with systemtap based tracing enabled. ( I had included a patch for the spec file for rhel 6 sources in the earlier set) Pls let me know if you need any other information.

Regards,
Prerna
Comment 21 IBM Bug Proxy 2010-12-17 02:43:04 EST
Created attachment 469304 [details]
upstream commit patches pertaining to qemu tracing


------- Comment on attachment From prerna@linux.vnet.ibm.com 2010-12-17 02:37 EDT-------


This is a 32-patch set that contains all upstream commits pertaining to qemu tracing ( Upstream Source : qemu tree @ git://git.qemu.org/qemu.git )
Comment 22 Jes Sorensen 2010-12-17 02:55:24 EST
Hi Prerna,

Sorry about the confusion of the RHEL6 and RHEL6.1 srpms. It is rare to
get such a large patchset like this one, but I think we need to try to
come up with a guide for how to do these kinds of backports. We have made
quite a few changes to the RHEL6.1 srpm which made it impossible for me
to apply your patchset. I hit so many conflicts that I started tracking
down the upstream patches, but it was made very difficult since you had
collapsed multiple commits into a single patch. I hit some problems with
the generated trace.h file, which indicated that something was completely
wrong with my backport. Stefan suggested I asked you for a respin.

If you don't have the srpm yet, please ping Anthony and Stefan. Otherwise
Stefan said that if you didn't have time, he would try to get it done
over the weekend.

Wrt systemtap-1.3 is that version already in RHEL6? If not that could
become a problem as we normally don't upgrade packages in between
releases. Is there an RFE for systemtap-1.3?

Regards,
Jes
Comment 23 IBM Bug Proxy 2010-12-17 06:11:26 EST
------- Comment From prerna@linux.vnet.ibm.com 2010-12-17 06:00 EDT-------
Hi Jes,
(In reply to comment #22)
> Hi Prerna,
>
> Sorry about the confusion of the RHEL6 and RHEL6.1 srpms. It is rare to
> get such a large patchset like this one, but I think we need to try to
> come up with a guide for how to do these kinds of backports. We have made
> quite a few changes to the RHEL6.1 srpm which made it impossible for me
> to apply your patchset. I hit so many conflicts that I started tracking
> down the upstream patches, but it was made very difficult since you had
> collapsed multiple commits into a single patch. I hit some problems with
> the generated trace.h file, which indicated that something was completely
> wrong with my backport. Stefan suggested I asked you for a respin.
>
> If you don't have the srpm yet, please ping Anthony and Stefan. Otherwise
> Stefan said that if you didn't have time, he would try to get it done
> over the weekend.
>

Totally understandable , the upstream sources have changed quite a bit from the qemu-kvm sources shipped with RHEL 6.1. Even I had to single out and port a lot of upstream commits to bring the base tree to a state where the tracing patches could be applied...
I have uploaded the complete set of trace framework upstream patches to be backported ( attachment #58348 ), but wont be able to take it up before Jan. Maybe Stefan or someone from Redhat can have a look ?

> Wrt systemtap-1.3 is that version already in RHEL6? If not that could
> become a problem as we normally don't upgrade packages in between
> releases. Is there an RFE for systemtap-1.3?

Rhel 6 was shipped with systemtap 1.2.
I'll defer the question of RFE for systemtap-1.3 to Dan Berrange and/or Frank Eigler.
From what I know, systemtap-1.3 came with a newer uprobes version that is used by the trace framework, and I dont think older versions of systemtap would be compatible with the qemu tracing framework. It would. however, be good to check with Dan Berrange / Rayson Ho, since they primarily worked on implementing this feature upstream.
Comment 24 Jes Sorensen 2010-12-17 06:24:25 EST
Dan,

Can you comment on the need for systemtap-1.3 or can we get around it with the systemtap-1.2 we have in RHEL6?

Thanks,
Jes
Comment 25 Stefan Hajnoczi 2010-12-17 12:53:54 EST
I'm surprised by the systemtap>=1.3 requirement because the box I tested QEMU --trace-backend=dtrace on only has 1.2 installed.  Will try out the latest patches again and report back.
Comment 26 Stefan Hajnoczi 2010-12-19 09:57:39 EST
I don't see an issue with SystemTap 1.2.  Will give backporting to RHEL6.1 a try.

Details:

Successfully built and tested qemu-kvm.git with --trace-backend=dtrace on RHEL6:

$ rpm -qf /usr/include/sys/sdt.h
systemtap-sdt-devel-1.2-9.el6.x86_64
$ rpm -qi systemtap
Name        : systemtap                    Relocations: (not relocatable)
Version     : 1.2                               Vendor: Red Hat, Inc.
Release     : 9.el6                         Build Date: Tue 29 Jun 2010 10:41:58 CDT
Install Date: Thu 12 Aug 2010 04:24:50 CDT      Build Host: x86-012.build.bos.redhat.com
Group       : Development/System            Source RPM: systemtap-1.2-9.el6.src.rpm
Size        : 7326854                          License: GPLv2+
Signature   : RSA/8, Wed 30 Jun 2010 07:15:18 CDT, Key ID 938a80caf21541eb
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://sourceware.org/systemtap/
Summary     : Instrumentation System
Description :
SystemTap is an instrumentation system for systems running Linux 2.6.
Developers can write instrumentation to collect data on the operation
of the system.

$ stap -L 'process("x86_64-softmmu/qemu-system-x86_64").mark("*")'
process("x86_64-softmmu/qemu-system-x86_64").mark("apic_deliver_irq") $arg1:uint8_t $arg2:uint8_t $arg3:uint8_t $arg4:uint8_t $arg5:uint8_t $arg6:uint8_t
process("x86_64-softmmu/qemu-system-x86_64").mark("apic_get_irq_delivered") $arg1:int
process("x86_64-softmmu/qemu-system-x86_64").mark("apic_local_deliver") $arg1:int $arg2:uint32_t
[...]

$ stap -x $(pgrep qemu) -I x86_64-softmmu -e 'probe qemu.system.x86_64.qemu_malloc { printf("malloc(%d) = %p\n", $arg1, $arg2); }'
malloc(40) = 0x00000000021035a0
malloc(8) = 0x00000000021035d0
malloc(5) = 0x0000000002103620
Comment 27 Jes Sorensen 2010-12-20 03:26:24 EST
Stefan,

The fact that it works fine with systemtap-1.2 is excellent news. It will
save us a good chunk of hassle I think.

As soon as you have some patches, let me know and I'll push them into
the official package.

Thanks,
Jes
Comment 28 Stefan Hajnoczi 2010-12-20 10:15:36 EST
Created attachment 469784 [details]
SystemTap tracing backport onto qemu-kvm 0.12.1.2-2.121.el6

Jes: Here's the backport.  It needs to modify the qemu-kvm.spec file, see the last commit.
Comment 29 Jes Sorensen 2010-12-27 11:07:11 EST
Stefan,

Thanks a lot for generating this updated patch set. With this set
I was able to build QEMU with the dtrace support enabled. Now I just
have to figure out how to test it :)

Cheers,
Jes
Comment 31 Daniel Berrange 2011-01-04 07:29:31 EST
I have only tested on SystemTAP 1.3, but I'm not aware of any reason why it wouldn't work with 1.2 also, so we should be fine in that respect.
Comment 32 Stefan Hajnoczi 2011-01-04 08:53:55 EST
Jes: In Comment 26 I used the following stap command-line to test the malloc probe:

$ stap -x $(pgrep qemu) -I x86_64-softmmu -e 'probe
qemu.system.x86_64.qemu_malloc { printf("malloc(%d) = %p\n", $arg1, $arg2); }'
malloc(40) = 0x00000000021035a0
malloc(8) = 0x00000000021035d0
malloc(5) = 0x0000000002103620

Perhaps that's a start to testing?
Comment 33 Jes Sorensen 2011-01-04 09:25:35 EST
Stefan,

Good point, that could be used as a simple basic functionality test.

Cheers,
Jes
Comment 35 John Jarvis 2011-01-13 13:23:15 EST
This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux minor release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux Update minor release.   
While it is a goal to include this enhancement in the next minor release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next minor release pending the next phase of actual 
code integration and successful Red Hat and partner testing.
Comment 42 Miya Chen 2011-03-09 00:15:13 EST
move to verified based on comment#40
Comment 43 IBM Bug Proxy 2011-03-29 05:01:49 EDT
------- Comment From ananth@in.ibm.com 2011-03-29 04:54 EDT-------
In RHEL6.1 beta 1, verified trace output using  command

stap -vve 'probe qemu.kvm.qemu_malloc {printf("qemu_malloc : size %d ptr %p\n", $size, $ptr)}'

Trace output is getting displayed in stdio. The feature is working.

qemu_malloc : size 49232 ptr 0x54265c0
qemu_malloc : size 49232 ptr 0x545bf40
qemu_malloc : size 49232 ptr 0x5467fa0
qemu_malloc : size 80 ptr 0x2c14fa0
qemu_malloc : size 24 ptr 0x2c19aa0
qemu_malloc : size 48 ptr 0x2c142f0
Comment 46 Jes Sorensen 2011-05-06 09:16:08 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:
Customer requesting support for dtrace style tracing
in QEMU.

Consequence:
Without these changes, there was no/limited support
for tracing of events within QEMU.

Change:
Applied patches provided by customer, which implements
dtrace style tracing in QEMU

Result:
When used in conjunction with systemtap, it is now
possible to trace internal QEMU events such as IO
operations and memory allocations.
Comment 47 errata-xmlrpc 2011-05-19 07:28:55 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html
Comment 48 errata-xmlrpc 2011-05-19 08:48:20 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html

Note You need to log in before you can comment on or make changes to this bug.