[This is an OKR 7 tracker RFE. For more information on ORK 7 as it applies to DFG:Compute, see https://docs.google.com/presentation/d/1xaYcX1S0K1kejcxHiAV0QhD7J0q7A1aj_h2UvusUMIs/edit] Title: QE need a comprehensive log collection tool + Ineffective bug reporting incurs a non-trivial cost. Details: QE would like a solution to fetch all the related logs that would help in easy debug/triage a CI failure + See https://mailman-int.corp.redhat.com/archives/rh-openstack-dev/2021-April/msg00131.html on how to write good but reports. Comment: Repetitive back and forth to get more information for debugging results in longer time to provide workarounds and resolutions for customer cases. AIs (from easiest/quickest to hardest/slowest): 1. Presentation that explains how to debug failures using request-ids, including understanding how to read tempest failures. 2. TripleO tooling to collect logs associated with specific request-ids. 3. Tempest improvements to better log request IDs. 4. Improve CI job logs collection and presentation.
Posting the mailing list content of "But why bother [writing good reports]?". Perhaps we should have an internal "blog post" with some surrounding context: But why bother? --------------- Writing a coherent report with a solid reproducer is hard, and takes time. Especially so, with a multi-layered product such as OSP. But it handsomely pays off to develop the habit of writing excellent bug reports. A reminder of reasons why: (1) It reduces needless round-trips of "needinfo" between various teams; thus saving money for Red Hat. An OSP bug may travel 4 layers deep into RHEL. A well-written report without any assumptions has more chances of your bug actually getting looked at. (2) When you're writing the context and reproducer details, you yourself might discover new clues that can help you solve the problem. (3) Developers who don't normally dwell in a given area can learn from a thoughtful bug report. It acts as "implicit training". (4) A test engineer trying to write an automation test (or a manual one) benefits from a well-written bug report. (5) Documentation writers will thank you for letting them understand why a bug-fix matters; and indirectly helping them with release notes. (6) PMs and Eng Managers, or any management folks who are not always in the trenches will be grateful for a report that helps them in a blocker meeting. (7) Our customers _and_ partners can often benefit when they look up some error and run into an excellent Red Hat bug report. Isn't it embarassing to imagine the shoddy impression we give to customers when our publicly accessible bugs are an incoherent mess? (8) Your fine bug report may spare some misery for frontline support, technical account managers (TAM), or solution architects. Especially if they're under customer duress. (9) Months later, the bug reporters _themselves_ will benefit from a robust report to help jog their memory when debugging a regression.
I'll also help with this given I have a very WIP request-id presentation I still want to deliver to CEE below: http://file.fab.redhat.com/lyarwood/request-id.html I'm also trying to find time to look into TripleO tooling around collecting logs that reference a specific request-id. Finally, I think this also needs to cover updating the openstack-nova bugzilla component template to include instance specific output like : $ openstack server show $instance $ openstack server event list --long $instance $ openstack server event show $instance $request-id
(In reply to Lee Yarwood from comment #2) > I'll also help with this given I have a very WIP request-id presentation I > still want to deliver to CEE below: > > http://file.fab.redhat.com/lyarwood/request-id.html > > I'm also trying to find time to look into TripleO tooling around collecting > logs that reference a specific request-id. > > Finally, I think this also needs to cover updating the openstack-nova > bugzilla component template to include instance specific output like : > > $ openstack server show $instance > > $ openstack server event list --long $instance > > $ openstack server event show $instance $request-id Thank you! Also the Tempest /CI items mentioned in the description (https://bugzilla.redhat.com/show_bug.cgi?id=1967671#c0) need collaboration from QE and CI
I assume sosreport which all RH products use is a no-go? (meaning, developing or extending a plugin for it)
I would add global ID issueing support for all tempest API calls. That (plus https://review.opendev.org/c/openstack/oslo.log/+/838190 would make the error tracing experience much better!)