Bug 1416625 - Updating Remote execution status taking time at scale
Summary: Updating Remote execution status taking time at scale
Keywords:
Status: CLOSED DUPLICATE of bug 1417537
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Remote Execution
Version: 6.2.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact:
URL:
Whiteboard: scale_lab
: 1438656 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-26 05:32 UTC by Pradeep Kumar Surisetty
Modified: 2021-03-11 14:55 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-21 08:47:58 UTC
Target Upstream Version:


Attachments (Terms of Use)
package install (77.41 KB, image/png)
2017-01-26 05:33 UTC, Pradeep Kumar Surisetty
no flags Details
status page (11.48 KB, image/png)
2017-01-26 05:33 UTC, Pradeep Kumar Surisetty
no flags Details
yum install aide on 1290 (32.40 KB, image/png)
2017-01-26 05:42 UTC, Pradeep Kumar Surisetty
no flags Details
failure-success rate during ReX in progress (83.46 KB, image/png)
2017-01-26 06:11 UTC, Pradeep Kumar Surisetty
no flags Details

Description Pradeep Kumar Surisetty 2017-01-26 05:32:13 UTC
Description of problem:

Started ReX operation ( yum install -y aide) on 1024 nodes. 

Package is installed on node. But its taking longer time to update status for the host in Remote execution job UI

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. start Remote execution (install any package) at scale
2. monitor https://<satellite>/template_invocations/<anymachine> if package is installed
3. Check status on remote execution job UI
    https://<satellite>/job_invocations/<job_id> and 

Actual results:

update status UI after installing package takes longer  . 

Expected results:

update status UI after installing package. 

Additional info:

Comment 1 Pradeep Kumar Surisetty 2017-01-26 05:33:24 UTC
Created attachment 1244594 [details]
package install

Comment 2 Pradeep Kumar Surisetty 2017-01-26 05:33:51 UTC
Created attachment 1244595 [details]
status page

Comment 3 Pradeep Kumar Surisetty 2017-01-26 05:41:11 UTC
accounting of Succeeded is wrong. 

Package updated completed more than > 10 nodes. UI continue to list only

Comment 4 Pradeep Kumar Surisetty 2017-01-26 05:41:38 UTC
Package updated completed more than > 10 nodes. UI continue to list 2 only

Comment 5 Pradeep Kumar Surisetty 2017-01-26 05:42:09 UTC
Created attachment 1244597 [details]
yum install aide on 1290

Comment 6 Pradeep Kumar Surisetty 2017-01-26 06:11:03 UTC
package completion updated successfully on most of them (few in progress)
Dynamically failure rate graph is updating. But not success rate as shown in attachment

Comment 7 Pradeep Kumar Surisetty 2017-01-26 06:11:44 UTC
Created attachment 1244598 [details]
failure-success rate during ReX in progress

Comment 12 Shimon Shtein 2017-01-31 10:43:17 UTC
IMHO According to comment 24, It's related to https://bugzilla.redhat.com/show_bug.cgi?id=1416542#c24.

Should it be marked as duplicate?

Comment 13 Pradeep Kumar Surisetty 2017-02-01 07:42:25 UTC
Shimon, 

These two are different. 

For ex: 

Running a command via REX on 6100 hosts. Even 10% of them completed, (foreman task stopped with success,, but the main UI
https://<sat-server>/job_invocations/<job-id>  shows 100% Pending.

Comment 14 Ivan Necas 2017-04-04 09:04:30 UTC
This relates to the amount of messages the we need to process during the rex at scale. In order to enhance this, we need to have more info about the messages that are being processed:

http://projects.theforeman.org/issues/19153

And eliminate the amount of messages + suggest to run more workers accordingly.

Comment 15 Ivan Necas 2017-04-04 16:24:17 UTC
Here is a middleware that can help us collecting data about the thoughput of
the executor while at scale:

https://gist.github.com/iNecas/a60aaed4e76f239d4ddac459442b1b48

See the usage comments in the gist to know how to use that

Comment 16 Ivan Necas 2017-05-11 08:25:23 UTC
Also, this BZ is partially related https://bugzilla.redhat.com/show_bug.cgi?id=1417537

Comment 18 Ivan Necas 2017-08-21 08:47:58 UTC
I'm marking this as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1417537 as the description overlaps a lot. If you find different place/occasion that the data would not seem to update properly, please file new, more specific bug.

*** This bug has been marked as a duplicate of bug 1417537 ***

Comment 19 Pradeep Kumar Surisetty 2017-10-02 11:29:52 UTC
*** Bug 1438656 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.