Bug 1296237 - [sos] rhui plugin times out [NEEDINFO]
[sos] rhui plugin times out
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: Tools (Show other bugs)
All Linux
unspecified Severity low
: ---
: 3.0.0
Assigned To: RHUI Bug List
Irina Gulina
Depends On: 1358564
  Show dependency treegraph
Reported: 2016-01-06 11:38 EST by Jeff Stoner
Modified: 2016-11-01 19:49 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-11-01 12:04:25 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
igulina: needinfo? (bmr)

Attachments (Terms of Use)

  None (edit)
Description Jeff Stoner 2016-01-06 11:38:38 EST
Description of problem:
When running sosreport, the rhui plugin consistently times out, resulting in incomplete information submitted as part of sosreport.

Version-Release number of selected component (if applicable):
rh-rhui-tools-debug-script 2.1, sos 3.2

How reproducible:

Steps to Reproduce:
1. Run sosreport on a rhua or cds server that has several repositories synced.
2. Wait to see if you get  [plugin:rhui] command 'python /usr/share/rh-rhua/rhui-debug.py  --dir /tmp/sosreport/sos_commands/rhui' timed out after 300s
3. Profit!

Actual results:
sosreport may submit incomplete information to Red Hat Support.

Expected results:

Additional info:
I suspect these commands are the culprit:

CDS_COMMANDS = { "ls -lR /var/lib/pulp-cds": {"filename" : "ls_-lR_var.lib.pulp-cds", "access_path" : "/var/lib/pulp-cds"} }

PULP_COMMANDS = { "ls -lR /var/lib/pulp": {"filename" : "ls_-lR_var.lib.pulp", "access_path" : "/var/lib/pulp"} }

On a RHUA or CDS that has dozens of repos configured, these commands can take a LONG time to complete. The parent sosreport only allows 300 seconds for plugin scripts to execute.

Suggest making generating the file list an option of the plugin (default to NOT collecting the file list.)
Comment 3 Jeff Stoner 2016-01-06 16:51:21 EST
Diff that adds file list generation as a command line option. This is against Version : 2.1.37, Release : 2.el6

# diff -u /usr/share/rh-rhua/rhui-debug.py /home/jstoner/rhui-debug.py 
--- /usr/share/rh-rhua/rhui-debug.py    2014-10-20 11:04:31.000000000 -0400
+++ /home/jstoner/rhui-debug.py 2016-01-06 16:32:22.524121306 -0500
@@ -80,6 +80,12 @@
         help="If the script needs to be executed to collect CDS information instead of RHUA")
+    parser.add_option(
+        '--genfilelist',
+        dest='genfilelist',
+        action="store_true",
+        default=False,
+        help='Generate a file list of all repositories (warning: can be slow')
     return parser
 # Checks whether user has root access to run this script
@@ -160,11 +166,13 @@
         # Collect CDS specific debugging information
         copy_dirs(CDS_DIRS, base_dir)
         copy_files(CDS_FILES, base_dir)
-        run_commands(CDS_COMMANDS, base_dir)
+        if opt.genfilelist:
+            run_commands(CDS_COMMANDS, base_dir)
         # Collect RHUA specific debugging information
         copy_dirs(PULP_DIRS, base_dir)
         copy_files(PULP_FILES, base_dir)
-        run_commands(PULP_COMMANDS, base_dir)
+        if opt.genfilelist:
+            run_commands(PULP_COMMANDS, base_dir)
Comment 4 Bryn M. Reeves 2016-02-15 06:20:08 EST
This is not an sos problem - the paths mentioned:

CDS_COMMANDS = { "ls -lR /var/lib/pulp-cds": {"filename" : "ls_-lR_var.lib.pulp-cds", "access_path" : "/var/lib/pulp-cds"} }

PULP_COMMANDS = { "ls -lR /var/lib/pulp": {"filename" : "ls_-lR_var.lib.pulp", "access_path" : "/var/lib/pulp"} }

Are from the rhui-debug script (as is the patch in comment #3).

We could temporarily increase the timeout used for rhui-debug in RHEL however this would be a workaround (and not a great one) - if the script is blowing a 300s command timeout it's clear that that needs to be improved or made optional.
Comment 5 Bryn M. Reeves 2016-02-15 06:25:05 EST
I fixed the bug title to make it clear which plugin we're talking about but I think it would be better to move this to the relevant component for rhui-debug since that's where the problem is.
Comment 7 Irina Gulina 2016-07-26 09:30:13 EDT
I couldn't reproduce it on on RHEL7 iso 20160719

>> sosreport
 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Running 82/82: yum...                      
Creating compressed archive...

Your sosreport has been generated and saved in:

The checksum is: d83a15b9f458b411045288f8345b885f

Please send this file to your support representative

I can't check it for RHEL6 iso 20160719 because of BZ1358564
Comment 8 bizhang 2016-10-31 13:29:58 EDT
Looks like BZ1358564 is VERIFIED. 
Can you see if it is reproducible on RHEL6?
Comment 9 Irina Gulina 2016-11-01 11:45:11 EDT
I couldn't reproduce it on RHEL6 iso 201025. 

>> sosreport

 Setting up archive ...
 Setting up plugins ...
 Running plugins. Please wait ...

  Running 85/85: yum...                      
Creating compressed archive...

Your sosreport has been generated and saved in:

The checksum is: f971368c9c73a4672ac84595ee905d46

Please send this file to your support representative.
Comment 10 Bryn M. Reeves 2016-11-01 12:12:50 EDT
> I couldn't reproduce it on RHEL6 iso 201025. 

It's not so much the host, as the environment, that affects this: it's the quantity of data that foreman-debug needs to fetch and format that causes the very long runtimes.

A fresh install on a lightly-configured server isn't likely to hit this at all.
Comment 11 Jeff Stoner 2016-11-01 12:13:54 EDT
How many repositories did you configure in RHUI?

You need to configure many repositories and let them fully sync to properly reproduce the problem because the problem is triggered when you have 10,000+ files in the repo directories.
Comment 12 Irina Gulina 2016-11-01 19:47:39 EDT
Hello, Bryn and Jeff

I did it with 31 repos which includes e.g.
Beta RHEL RHUI Everything 7 Source Srpms (x86_64)
Beta RHEL RHUI Server 7 Debug (x86_64)
Beta RHEL RHUI Server 7 Optional OS (x86_64)
RHEL RHUI Server 6 Rhscl 1 Source Srpms (6Server-x86_64)
RHEL RHUI Server 7 Optional OS (7Server-x86_64)
Red Hat Enterprise Linux Server 6 (RPMs) (6Server-i386)
Red Hat Enterprise Linux Server 6 (SRPMS) (6Server-i386)

> [root@rhua ~]# cd /var/lib/pulp
> [root@rhua pulp]# find . -type f | wc -l

Please let me know if it's ok or I should do smth else.
Comment 13 Irina Gulina 2016-11-01 19:49:44 EDT
I forgot to mention, I tested it on RHUI3 iso, not RHUI2.

Note You need to log in before you can comment on or make changes to this bug.