Bug 733379 - RFE: more efficient versioned-configuration serialization
Summary: RFE: more efficient versioned-configuration serialization
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: wallaby
Version: 2.0
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: 2.1
: ---
Assignee: Will Benton
QA Contact: Tomas Rusnak
URL:
Whiteboard:
Depends On:
Blocks: 743350
TreeView+ depends on / blocked
 
Reported: 2011-08-25 15:22 UTC by Will Benton
Modified: 2012-02-07 08:41 UTC (History)
4 users (show)

Fixed In Version: wallaby-0.10.6-1
Doc Type: Enhancement
Doc Text:
With this update, the Wallaby configuration service has been enhanced to use an alternate technique for storing versioned pool configuration. This technique significantly speeds up activation operations on most pools with complex configuration.
Clone Of:
Environment:
Last Closed: 2012-01-23 17:28:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2012:0045 normal SHIPPED_LIVE Red Hat Enterprise MRG Grid 2.1 bug fix and enhancement update 2012-01-23 22:22:58 UTC

Description Will Benton 2011-08-25 15:22:45 UTC
Description of problem:

Current versions of Wallaby store versioned configurations by serializing configurations for each node.  This is a bottleneck in Wallaby setups that have group configurations with many parameter-value mappings, since serializing large configurations to the database is expensive.

In almost all cases, it would be more efficient to store one configuration for each group, and to store for each node a set of memberships in (versioned) groups.

Version-Release number of selected component (if applicable):

wallaby-0.10.5-6

Comment 1 Will Benton 2011-08-25 15:23:26 UTC
This is fixed upstream.

Comment 2 Luigi Toscano 2011-09-08 16:38:41 UTC
How should this be reproduced? By defining few groups with many nodes each, and many parameters (key/value) for each group?

Comment 3 Will Benton 2011-09-08 17:07:55 UTC
Basically, that will do it.  The test I've used, which is based on a comically-overprovisioned execute node configuration, is in the Wallaby test suite; you can run it from the source tree by running the following command:

rake spec SPEC=spec/big_pool_torture_spec.rb

Depending on the hardware you run it on, the test might fail with the old config-serialization code; it probably won't with the new code.  (You can increase the TORTURE_TIMEOUT constant in the spec file to ensure that the test succeeds if you want to measure speedups.)

If you don't want to use the rspec torture tests, you can use their dataset, which is available in the big-pool.yaml file in the repository.  Load that up, change a param setting in the default group, and then activate.

http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=blob;f=spec/big_pool_torture_spec.rb
http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=blob;f=spec/big-pool.yaml

Comment 4 Will Benton 2011-10-04 18:47:35 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
The Wallaby configuration service now uses an alternate technique for storing versioned pool configurations.  This technique will result in significantly faster activation operations on most pools with complex configurations.

Comment 6 Tomas Rusnak 2011-10-18 15:26:33 UTC
Cannot be reproduced due to incompatibility between old wallaby and current rspec test:

wallaby-utils-0.10.5-3.el5

# ruby -Ispec $(which spec) wallaby/spec/big_pool_torture_spec.rb -b
/mnt/tests/distribution/MRG/Grid/grid_ptest_unit_wallaby/wallaby/spec/spec_helper.rb:23: undefined method `quiesce' for Mrg::Grid::Config::Store:Class (NoMethodError)
	from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
	from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
	from ./wallaby/spec/big_pool_torture_spec.rb:1
	from /usr/lib64/ruby/gems/1.8/gems/rspec-1.2.9/lib/spec/runner/example_group_runner.rb:15:in `load'
	from /usr/lib64/ruby/gems/1.8/gems/rspec-1.2.9/lib/spec/runner/example_group_runner.rb:15:in `load_files'
	from /usr/lib64/ruby/gems/1.8/gems/rspec-1.2.9/lib/spec/runner/example_group_runner.rb:14:in `each'
	from /usr/lib64/ruby/gems/1.8/gems/rspec-1.2.9/lib/spec/runner/example_group_runner.rb:14:in `load_files'
	from /usr/lib64/ruby/gems/1.8/gems/rspec-1.2.9/lib/spec/runner/options.rb:132:in `run_examples'
	from /usr/lib64/ruby/gems/1.8/gems/rspec-1.2.9/lib/spec/runner/command_line.rb:9:in `run'
	from /usr/lib64/ruby/gems/1.8/gems/rspec-1.2.9/bin/spec:5
	from /usr/bin/spec:19:in `load'
	from /usr/bin/spec:19

Retested on all supported platforms x86,x86_64/RHEL5,RHEL6 with:

wallaby-0.12.0-1.el6.noarch

RHEL6/64:
[11:29:39] Unit test big_pool_torture_spec.rb STARTED
.
Finished in 611.204436 seconds
1 example, 0 failures
[11:39:51] Unit test big_pool_torture_spec.rb PASSED

RHEL5/64:
[14:22:22] Unit test big_pool_torture_spec.rb STARTED
.
Finished in 457.512838 seconds
1 example, 0 failures
[14:30:00] Unit test big_pool_torture_spec.rb PASSED

RHEL6/32:
[07:44:51] Unit test big_pool_torture_spec.rb STARTED
.
Finished in 603.718119 seconds
1 example, 0 failures
[07:54:55] Unit test big_pool_torture_spec.rb PASSED

RHEL5/32:
[15:08:06] Unit test big_pool_torture_spec.rb STARTED
.
Finished in 520.715803 seconds
1 example, 0 failures
[15:16:47] Unit test big_pool_torture_spec.rb PASSED

>>> VERIFIED

Comment 7 Will Benton 2011-10-18 15:48:26 UTC
Yes, you'll need to check out the old spec_helper.rb to run the tests against the old wallaby.

http://git.fedorahosted.org/git/?p=grid/wallaby.git;a=tree;f=spec;h=d015057332884d01cf0dd891186bba593c3005bd;hb=e48d20644b62373560fa773778ebbf8974d2b29f

Comment 8 Tomas Capek 2011-11-16 15:39:47 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-The Wallaby configuration service now uses an alternate technique for storing versioned pool configurations.  This technique will result in significantly faster activation operations on most pools with complex configurations.+With this update, the Wallaby configuration service has been enhanced to use an alternate technique for storing versioned pool configuration. This technique significantly speeds up activation operations on most pools with complex configuration.

Comment 9 errata-xmlrpc 2012-01-23 17:28:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0045.html


Note You need to log in before you can comment on or make changes to this bug.