Bug 632722

------- Comment From rsisk.com 2010-10-04 11:32 EDT-------
Code Upstream Status: In Progress

Is there a patch set available somewhere for the features you would like added?

------- Comment From prerna.ibm.com 2010-11-20 00:20 EDT-------
(In reply to comment #6)
> Created an attachment (id=57726) [details]
> tarball containing patches to backport qemu tracing for rhel 6.1
>
> This set of 23 patches ( 22 patches starting number 1694 to 1715 for qemu
> tracing; one more patch in addition, for patching the qemu-kvm.spec file ) adds
> qemu tracing capabilities for rhel 6.1
>
> This is used to enable qemu to be built with systemtap tracing backend. It
> requires systemtap >= 1.3 ( already specified in spec file  build dependencies
> )

This patch series includes the most recent version(v6) of systemtap backend ( Patch # 1714 and 1715 ).
Unfortunately an older version(v5) of this patchset was erroneously merged upstream ( see http://lists.gnu.org/archive/html/qemu-devel/2010-11/msg01789.html )
It would be good if upstream qemu also contained the most recent patchset.

Created attachment 461716 [details]
tarball containing patches to backport qemu tracing for rhel 6.1


------- Comment on attachment From prerna.ibm.com 2010-11-20 00:05 EDT-------


This set of 23 patches ( 22 patches starting number 1694 to 1715 for qemu tracing; one more patch in addition, for patching the qemu-kvm.spec file ) adds qemu tracing capabilities for rhel 6.1

This is used to enable qemu to be built with systemtap tracing backend. It requires systemtap >= 1.3 ( already specified in spec file  build dependencies )

Created attachment 461967 [details]
Tarball containing set of patches ( 0001 to 0022 ) that add qemu-tracing capability.


------- Comment on attachment From prerna.ibm.com 2010-11-22 05:55 EDT-------


This contains a set of patches that add qemu tracing support, along with systemtap trace backend which enables systemtap-userspace probing capability to trace qemu execution.

This depends on systemtap >=1.3

Created attachment 461968 [details]
Changes to qemu spec file to build qemu with tracing support.


------- Comment on attachment From prerna.ibm.com 2010-11-22 05:57 EDT-------


The qemu tracing framework is to be compiled with the 'dtrace' backend, which is specified with the '--trace-backend=dtrace' config option.

Also, it needs systemtap > 1.3 & systemtap-sdt-devel >= 1.3

Created attachment 461973 [details]
Patch series for qemu tracing


------- Comment on attachment From prerna.ibm.com 2010-11-22 06:02 EDT-------


This lists the patch series contained in attachment #57747 [details]

------- Comment From prerna.ibm.com 2010-12-06 04:35 EDT-------
Hi Redhat,
Would this patch series make it to RHEL 6.1 ?

We'll try to get it into 6.1, it is a function of capacity.
Does all the code upstream by now, including the qmp commands?

The final version 6 of the DTrace patches was committed upstream in

commit c276b17da65b7ff01627722a1abf2b7a684c8fd8
Author: Daniel P. Berrange <berrange>
Date:   Fri Nov 12 13:20:25 2010 +0000

    Add support for generating a systemtap tapset static probes

commit b3d08c029dd78ded5e35b74eaaa3d361821f83a7
Author: Daniel P. Berrange <berrange>
Date:   Fri Nov 12 13:20:24 2010 +0000

    Add a DTrace tracing backend targetted for SystemTAP compatability


There are no QMP commands required for this. So the backported patches posted by IBM on this bug should be sufficient.

Prerna,

I have been trying to get this patches to apply to the current RHEL6.1
qemu-kvm, but not had any luck.

It wasn't clear to me at all which tree you had created the patches 
against, and the git commit IDs in the patches were missing, or against
a tree that I do not have. I also noticed that some of your patches were
a combination of multiple git commits from upstream, which makes it
really difficult to match.

I have tried to match all the commit messages to upstream git commits and
pull them in manually, but something isn't working. I spoke to Stefan
about this and he thinks something is missing.

Would it be possible for you to do a fresh respin of these patches against
the latest RHEL6.1 qemu-kvm package?

Thanks,
Jes

------- Comment From mahesh.ibm.com 2010-12-16 02:52 EDT-------
Reply from Ananth (mananth.com):

(In reply to comment #16)
> Prerna,
>
> I have been trying to get this patches to apply to the current RHEL6.1
> qemu-kvm, but not had any luck.
>
> It wasn't clear to me at all which tree you had created the patches
> against, and the git commit IDs in the patches were missing, or against
> a tree that I do not have. I also noticed that some of your patches were
> a combination of multiple git commits from upstream, which makes it
> really difficult to match.

Jes,

AFAIK, the patches were generated against the qemu-kvm sources shipped
with RHEL6.

>
> I have tried to match all the commit messages to upstream git commits and
> pull them in manually, but something isn't working. I spoke to Stefan
> about this and he thinks something is missing.
>
> Would it be possible for you to do a fresh respin of these patches against
> the latest RHEL6.1 qemu-kvm package?
>
> Thanks,
> Jes

We (IBM) don't have access to the latest RHEL6.1 source tree. Can you
provide us the latest source RPM to base the patches against?

Hi Ananth,

I asked Anthony to pull a copy of the srpm so you can get it.

Cheers,
Jes

Hi

Another question, when we get the patches sorted, which trace target
are we supposed to enable in the binary? dtrace or another one?

Thanks,
Jes

------- Comment From prerna.ibm.com 2010-12-17 02:41 EDT-------
Hi Jes,
(In reply to comment #16)
> Prerna,
>
> I have been trying to get this patches to apply to the current RHEL6.1
> qemu-kvm, but not had any luck.
> It wasn't clear to me at all which tree you had created the patches
> against, and the git commit IDs in the patches were missing, or against
> a tree that I do not have.

I had verified with Anthony around mid-november, and was asked to base my patches against RHEL 6 qemu-kvm sources. Considering a Dec 18 deadline for feature code requests, the information about a new rhel 6.1 codebase seems rather new!

> I also noticed that some of your patches were
> a combination of multiple git commits from upstream, which makes it
> really difficult to match.

Yes, quite a few upstream commits were related (some being minor fixes on the earlier features), so I'd coalesced them together into a single patch.

> I have tried to match all the commit messages to upstream git commits and
> pull them in manually, but something isn't working. I spoke to Stefan
> about this and he thinks something is missing.
>
> Would it be possible for you to do a fresh respin of these patches against
> the latest RHEL6.1 qemu-kvm package?
>

I still have no information of a rhel 6.1 srpm. I am slated to start my vacation from today, and would only be back in the first week of Jan. If this is urgent, would it be possible for you to rebase the upstream commits against the new code base you are working with? I'm attaching a tarball (attachment #58348 ) which contains all the upstream commit patches that I had backported for this feature. Each patch has the upstream commit details for your reference.

Pls note that this needs to be built with systemtap >=1.3. Also, ' trace-backend=dtrace ' needs to be provided at configuration time for the binary to be built with systemtap based tracing enabled. ( I had included a patch for the spec file for rhel 6 sources in the earlier set) Pls let me know if you need any other information.

Regards,
Prerna

Created attachment 469304 [details]
upstream commit patches pertaining to qemu tracing


------- Comment on attachment From prerna.ibm.com 2010-12-17 02:37 EDT-------


This is a 32-patch set that contains all upstream commits pertaining to qemu tracing ( Upstream Source : qemu tree @ git://git.qemu.org/qemu.git )

Hi Prerna,

Sorry about the confusion of the RHEL6 and RHEL6.1 srpms. It is rare to
get such a large patchset like this one, but I think we need to try to
come up with a guide for how to do these kinds of backports. We have made
quite a few changes to the RHEL6.1 srpm which made it impossible for me
to apply your patchset. I hit so many conflicts that I started tracking
down the upstream patches, but it was made very difficult since you had
collapsed multiple commits into a single patch. I hit some problems with
the generated trace.h file, which indicated that something was completely
wrong with my backport. Stefan suggested I asked you for a respin.

If you don't have the srpm yet, please ping Anthony and Stefan. Otherwise
Stefan said that if you didn't have time, he would try to get it done
over the weekend.

Wrt systemtap-1.3 is that version already in RHEL6? If not that could
become a problem as we normally don't upgrade packages in between
releases. Is there an RFE for systemtap-1.3?

Regards,
Jes

------- Comment From prerna.ibm.com 2010-12-17 06:00 EDT-------
Hi Jes,
(In reply to comment #22)
> Hi Prerna,
>
> Sorry about the confusion of the RHEL6 and RHEL6.1 srpms. It is rare to
> get such a large patchset like this one, but I think we need to try to
> come up with a guide for how to do these kinds of backports. We have made
> quite a few changes to the RHEL6.1 srpm which made it impossible for me
> to apply your patchset. I hit so many conflicts that I started tracking
> down the upstream patches, but it was made very difficult since you had
> collapsed multiple commits into a single patch. I hit some problems with
> the generated trace.h file, which indicated that something was completely
> wrong with my backport. Stefan suggested I asked you for a respin.
>
> If you don't have the srpm yet, please ping Anthony and Stefan. Otherwise
> Stefan said that if you didn't have time, he would try to get it done
> over the weekend.
>

Totally understandable , the upstream sources have changed quite a bit from the qemu-kvm sources shipped with RHEL 6.1. Even I had to single out and port a lot of upstream commits to bring the base tree to a state where the tracing patches could be applied...
I have uploaded the complete set of trace framework upstream patches to be backported ( attachment #58348 ), but wont be able to take it up before Jan. Maybe Stefan or someone from Redhat can have a look ?

> Wrt systemtap-1.3 is that version already in RHEL6? If not that could
> become a problem as we normally don't upgrade packages in between
> releases. Is there an RFE for systemtap-1.3?

Rhel 6 was shipped with systemtap 1.2.
I'll defer the question of RFE for systemtap-1.3 to Dan Berrange and/or Frank Eigler.
From what I know, systemtap-1.3 came with a newer uprobes version that is used by the trace framework, and I dont think older versions of systemtap would be compatible with the qemu tracing framework. It would. however, be good to check with Dan Berrange / Rayson Ho, since they primarily worked on implementing this feature upstream.

Dan,

Can you comment on the need for systemtap-1.3 or can we get around it with the systemtap-1.2 we have in RHEL6?

Thanks,
Jes

I'm surprised by the systemtap>=1.3 requirement because the box I tested QEMU --trace-backend=dtrace on only has 1.2 installed.  Will try out the latest patches again and report back.

I don't see an issue with SystemTap 1.2.  Will give backporting to RHEL6.1 a try.

Details:

Successfully built and tested qemu-kvm.git with --trace-backend=dtrace on RHEL6:

$ rpm -qf /usr/include/sys/sdt.h
systemtap-sdt-devel-1.2-9.el6.x86_64
$ rpm -qi systemtap
Name        : systemtap                    Relocations: (not relocatable)
Version     : 1.2                               Vendor: Red Hat, Inc.
Release     : 9.el6                         Build Date: Tue 29 Jun 2010 10:41:58 CDT
Install Date: Thu 12 Aug 2010 04:24:50 CDT      Build Host: x86-012.build.bos.redhat.com
Group       : Development/System            Source RPM: systemtap-1.2-9.el6.src.rpm
Size        : 7326854                          License: GPLv2+
Signature   : RSA/8, Wed 30 Jun 2010 07:15:18 CDT, Key ID 938a80caf21541eb
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://sourceware.org/systemtap/
Summary     : Instrumentation System
Description :
SystemTap is an instrumentation system for systems running Linux 2.6.
Developers can write instrumentation to collect data on the operation
of the system.

$ stap -L 'process("x86_64-softmmu/qemu-system-x86_64").mark("*")'
process("x86_64-softmmu/qemu-system-x86_64").mark("apic_deliver_irq") $arg1:uint8_t $arg2:uint8_t $arg3:uint8_t $arg4:uint8_t $arg5:uint8_t $arg6:uint8_t
process("x86_64-softmmu/qemu-system-x86_64").mark("apic_get_irq_delivered") $arg1:int
process("x86_64-softmmu/qemu-system-x86_64").mark("apic_local_deliver") $arg1:int $arg2:uint32_t
[...]

$ stap -x $(pgrep qemu) -I x86_64-softmmu -e 'probe qemu.system.x86_64.qemu_malloc { printf("malloc(%d) = %p\n", $arg1, $arg2); }'
malloc(40) = 0x00000000021035a0
malloc(8) = 0x00000000021035d0
malloc(5) = 0x0000000002103620

Stefan,

The fact that it works fine with systemtap-1.2 is excellent news. It will
save us a good chunk of hassle I think.

As soon as you have some patches, let me know and I'll push them into
the official package.

Thanks,
Jes

Created attachment 469784 [details]
SystemTap tracing backport onto qemu-kvm 0.12.1.2-2.121.el6

Jes: Here's the backport.  It needs to modify the qemu-kvm.spec file, see the last commit.

Stefan,

Thanks a lot for generating this updated patch set. With this set
I was able to build QEMU with the dtrace support enabled. Now I just
have to figure out how to test it :)

Cheers,
Jes

I have only tested on SystemTAP 1.3, but I'm not aware of any reason why it wouldn't work with 1.2 also, so we should be fine in that respect.

Jes: In Comment 26 I used the following stap command-line to test the malloc probe:

$ stap -x $(pgrep qemu) -I x86_64-softmmu -e 'probe
qemu.system.x86_64.qemu_malloc { printf("malloc(%d) = %p\n", $arg1, $arg2); }'
malloc(40) = 0x00000000021035a0
malloc(8) = 0x00000000021035d0
malloc(5) = 0x0000000002103620

Perhaps that's a start to testing?

Stefan,

Good point, that could be used as a simple basic functionality test.

Cheers,
Jes

This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux minor release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux Update minor release.   
While it is a goal to include this enhancement in the next minor release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next minor release pending the next phase of actual 
code integration and successful Red Hat and partner testing.

move to verified based on comment#40

------- Comment From ananth.com 2011-03-29 04:54 EDT-------
In RHEL6.1 beta 1, verified trace output using  command

stap -vve 'probe qemu.kvm.qemu_malloc {printf("qemu_malloc : size %d ptr %p\n", $size, $ptr)}'

Trace output is getting displayed in stdio. The feature is working.

qemu_malloc : size 49232 ptr 0x54265c0
qemu_malloc : size 49232 ptr 0x545bf40
qemu_malloc : size 49232 ptr 0x5467fa0
qemu_malloc : size 80 ptr 0x2c14fa0
qemu_malloc : size 24 ptr 0x2c19aa0
qemu_malloc : size 48 ptr 0x2c142f0

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause:
Customer requesting support for dtrace style tracing
in QEMU.

Consequence:
Without these changes, there was no/limited support
for tracing of events within QEMU.

Change:
Applied patches provided by customer, which implements
dtrace style tracing in QEMU

Result:
When used in conjunction with systemtap, it is now
possible to trace internal QEMU events such as IO
operations and memory allocations.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0534.html