Bug 849087 - [RFE] Improve journal operations scaling and resource consumption
Summary: [RFE] Improve journal operations scaling and resource consumption
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: beakerlib
Version: rawhide
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
Assignee: Ondrej Hudlicky
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 723944
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-08-17 09:23 UTC by Martin Kudlej
Modified: 2018-01-15 21:37 UTC (History)
3 users (show)

Fixed In Version: beakerlib-1.17-6
Clone Of: 723944
Environment:
Last Closed: 2018-01-15 21:37:00 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
inspiration for journalling.py (37.14 KB, text/x-python)
2014-06-03 13:50 UTC, Dalibor Pospíšil
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1320688 1 None None None 2021-01-20 06:05:38 UTC

Internal Links: 1320688

Description Martin Kudlej 2012-08-17 09:23:00 UTC
+++ This bug was initially created as a clone of Bug #723944 +++

Description of problem: It isn't effective to rewrite whole content of file for every log entry.
Every log of any log entry calls:
<code>
output = open(journal, 'wb')
output.write(newdoc.toxml().encode('utf-8'))
output.close()
</code>
Where newdoc is DOM of whole log which is parsed for every log of log entry.
So step for writing of every log entry are:
1. parse whole log file to DOM <- very resources consuming
2. add log entry
3. save DOM to file(practically save new file) <- very resource consuming

I recommend to add just XML elements at the end of the file for every log entry and at the end clean up the log by SAX or something similar because DOM parsing is very resources consuming.

Version-Release number of selected component (if applicable):
beakerlib-1.3-1.fc14.noarch

--- Additional comment from pmuller on 2011-07-22 08:51:48 EDT ---

Yeah. I've once measured the effect of this nastiness on performance against an prototype in-memory representation (a daemon keeping journal in memory, communication via sockets => no parsing/saving), and found out the current implementation starts to measurably differ from the daemon @ ~800-1000 records. Most of the tests are not so big, so we dropped the idea of rewriting the journal at that time.

Currently, there is an effort to rewrite the journal to be handled by a daemon, and XML just one output format of an object internal representation.

--- Additional comment from endoflife on 2012-08-16 12:51:49 EDT ---

This message is a notice that Fedora 14 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 14. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained.  At this time, all open bugs with a Fedora 'version'
of '14' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this 
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen 
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we were unable to fix it before Fedora 14 reached end of life. If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora, you are encouraged to click on 
"Clone This Bug" (top right of this page) and open it against that 
version of Fedora.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 1 Fedora End Of Life 2013-12-21 15:04:35 UTC
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 2 Dalibor Pospíšil 2014-06-03 13:50:32 UTC
Created attachment 901795 [details]
inspiration for journalling.py

Martin pointed me to the attached script which could be used as an inspiration for possibly faster journalling.py

Comment 3 Fedora Admin XMLRPC Client 2014-09-02 12:14:27 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.


Note You need to log in before you can comment on or make changes to this bug.