Bug 1094424

Summary: "Reindex Data Tables Nightly" setting can cause agents to time out during reindex
Product: [Other] RHQ Project Reporter: Elias Ross <genman>
Component: Configuration, PerformanceAssignee: Jay Shaughnessy <jshaughn>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: high    
Version: 4.9CC: hrupp, jshaughn
Target Milestone: GA   
Target Release: RHQ 4.12   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-12-15 11:35:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Elias Ross 2014-05-05 16:15:04 UTC
Description of problem:

The "Reindex Data Tables Nightly" option on large tables has caused daily trouble, until I discovered how to turn it off.

modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/system/SystemManagerBean.java

Runs:

ALTER INDEX {0} REBUILD UNRECOVERABLE

When being run on Oracle, this prevents inserts while the rebuild takes place, which may be several minutes. This has caused agents to time out and false alerts to be generated.

Unrecoverable is not a good idea as it locks the table during the calculation:
https://community.oracle.com/thread/1103546?tstart=0

The default should be false, as well UNRECOVERABLE is not a good idea either on Oracle because of the locking.

Comment 1 Elias Ross 2014-05-05 16:19:01 UTC
The primary trouble is with the traits table, although I suspect the availability table may also cause trouble during reindex.

Comment 2 Jay Shaughnessy 2014-07-01 15:50:56 UTC
Merged PR45

master commit 78541021d881c60dd97e3b21e0af1b0d59cc84ad
Merge: ba0c4d3 537c51f
Author: jshaughn <jshaughn>
Date:   Tue Jul 1 10:19:50 2014 -0400

    Merge pull request #45 from genman/BZ1094424

    BZ1094424 - do not reindex tables by default; don't use UNRECOVERABLE


Some more follow up work coming...

Comment 3 Jay Shaughnessy 2014-07-01 17:06:55 UTC
master commit a68c656eca8f0574dd551a6faf51a0df829f0f39
Author: Jay Shaughnessy <jshaughn>
Date:   Tue Jul 1 13:05:39 2014 -0400

    [1094424] "Reindex Data Tables Nightly" setting can cause agents to time
    This is follow-up work to PR45.  Add setting the nightly re-index option
    to false in db-upgrade.  Also, remove unused numeric data tables from
    being processed if in fact the option is set to true.

    This whole option should probably go away, see the discussion in the
    PR.

Comment 4 Heiko W. Rupp 2014-12-15 11:35:50 UTC
Bulk close of items fixed in RHQ 4.12

If you think this is not solved, then please open a *new* BZ and link to this one.