Bug 913616 (nova_debug_dump_5.0) - [RFE][nova]: Provide for a live debug dump of system state
Summary: [RFE][nova]: Provide for a live debug dump of system state
Keywords:
Status: CLOSED UPSTREAM
Alias: nova_debug_dump_5.0
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Upstream M3
: 5.0 (RHEL 7)
Assignee: Solly Ross
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks: cinder_debug_dump_5.0 glance_debug_dump_5.0 keystone_debug_dump_5.0 neutron_debug_dump_5.0 swift_debug_dump_5.0 nova_debug_dump_4.0.z nova_debug_dump_3.0.z
TreeView+ depends on / blocked
 
Reported: 2013-02-21 15:55 UTC by Daniel Berrangé
Modified: 2019-09-09 14:32 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
: cinder_debug_dump_5.0 glance_debug_dump_5.0 keystone_debug_dump_5.0 neutron_debug_dump_5.0 swift_debug_dump_5.0 nova_debug_dump_4.0.z nova_debug_dump_3.0.z (view as bug list)
Environment:
Last Closed: 2014-04-08 21:28:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 32445 0 None None None Never
OpenStack gerrit 69058 0 None None None Never

Description Daniel Berrangé 2013-02-21 15:55:43 UTC
Description of problem:
Add the ability to obtain a debug dump of the system state for an OpenStack service based process.  eg upon sending SIGUSR1 trigger a dump of all native threads, green threads, live config and any other relevant info

See this upstream proposal

https://wiki.openstack.org/wiki/GuruMeditationReport

(Filed against Nova, but applicable to all)

Comment 3 Solly Ross 2013-06-06 20:30:37 UTC
@Daniel Berrange: were you thinking something like this: http://pastebin.test.redhat.com/146078 (`git apply` to nova)

(apply this: http://pastebin.test.redhat.com/146081 to python-novaclient to easily access without having to manually curl TENANT_ID/guru)

Comment 4 Daniel Berrangé 2013-06-07 09:50:18 UTC
No, this is not at all what I was considering for design. Integration into the RPC service, nova client, etc is absolutely not something that is intended. This needs to be a capability that is ultra reliable in the face of broken arbitrarily OpenStack infrastructure. As such it must not depends on things like the RPC system, database, or REST interfaces.

I wrote a proof of concept here

  https://github.com/berrange/nova/commits/gurumed2

however, I did not submit this upstream because I was intending to generalize it into a python package that is broadly independent of openstack code. Further alot of the same infrastructure is required for the related system tracing work in https://bugzilla.redhat.com/show_bug.cgi?id=913619

Comment 5 Solly Ross 2013-06-07 14:50:59 UTC
Oh, OK.  That wasn't exactly clear from the description and wiki page.  I'll take a look at the proof of concept and see if I can generalize it/integrate it into some of the framework that I wrote.

Comment 6 Solly Ross 2013-06-10 18:46:40 UTC
Patch submitted upstream to oslo-config to integrate oslo.report into Oslo

Comment 7 Solly Ross 2013-09-12 20:19:31 UTC
patch to Oslo merged upstream under 'openstack.common.report'.  Now to push functionality into nova...

Comment 8 Solly Ross 2014-01-24 23:08:51 UTC
Patch for integrating functionality into nova posted upstream.

Comment 9 Solly Ross 2014-04-08 21:28:38 UTC
Since this is targeting 5.0 and the change made it into Icehouse, I'm closing this "UPSTREAM"


Note You need to log in before you can comment on or make changes to this bug.