This was removed a long time ago (sos-3.1), at the request of the Satellite team:

commit a6dc4a5d532a7213a1bf5149087d33b76c42583e
Author: Lukas Zapletal <lzap+git>
Date:   Thu Oct 10 14:09:53 2013 +0200

    Adding plugins foreman and katello
    
    Since CloudForms 1.0 has been renamed to Satellite 6.0, I am deleting this
    plugin and providing two other plugins. Both projects Katello and Foreman
    are upstream projects for Satellite 6.0 Red Hat products. Since both
    projects support deployment on non-redhat systems, they both deliver own
    reporting script that collects information from target system. Both scripts
    have options not to create tarball and to provide output to desired
    directory. This has been tested on RHEL 6.4.
    
    Signed-off-by: Lukas Zapletal <lzap+git>

As the commit message explains CloudForms was renamed, and as we understood it at the time there would be no further supported releases using this branding. If for some reason the plugin is not being enabled on systems where it should that's just a bug that should be fixed - the current manageiq plugin is enabled whenever the 'cfme' package is present, but if this is insufficient we can easily expand it to other packages.

It's not immediately clear from the information in comment #0 what products this request applies to - it would help to have an sosreport from an affected system (at least the installed-rpms file) to help diagnose this further.

(In reply to David Luong from comment #0)
> Description of problem:
> CloudForms plugin no longer exists in current version of sosreports, but it
> is available in upstream sos as manageiq plugin.  Please enable the plugin
> to be used in downstream RHEL as cfme or cloudforms.  The only differences I
> could foresee in upstream and downstream is just changing mentions of
> manageiq to cloudforms.

Change of _name_ of plugin does not affect when the plugin is enabled by default. manageiq is enabled whenever cfme package is installed. You can check that via running:

sosreport --list-plugins
..
The following plugins are currently enabled:

 acpid                ACPI daemon information
..

The following plugins are currently disabled:

 abrt                 inactive       Automatic Bug Reporting Tool
..

> 
> The reason for this request is that support currently has to ask for
> customers to download a script from a github repository in order to get logs
> from the cloudforms servers when the UI isn't able to collect them.  This
> change will bring it in parity with all other products that Red Hat offers
> when gathering logs from the command line thus leading to an improved and
> consistent customer experience in regards to CloudForms.

What script is required to download? Or in other words: in what situation (i.e. what package installed or what file or kernel module loaded) do you require from sosreport to automatically collect what data (exactly those from manageiq plugin?)?

And if you run "sosreport -e manageiq", does it collect what you want? (then we should just fix the trigger in the plugin - what package presence uniquely determines CloudForms is installed, then?)


> 
> Version-Release number of selected component (if applicable):
> 3.4

RHEL 7.6 ships sos 3.6 (currently sos-3.6-11.el7_6.noarch). Is the BZ applicable to this version as well? (I expect so as the plugin hasnt changed since 3.4, just worth to confirm)? Or does the 3.4 specific old version mean you will need some fix over 3.4 (on RHEL 7.4?). As RHEL 7.7 shall ship sos 3.7 per current plan.


In general: Please follow the BZ template to better understand specification of the issue.

(In reply to Pavel Moravec from comment #3)
> (In reply to David Luong from comment #0)
> > The reason for this request is that support currently has to ask for
> > customers to download a script from a github repository ..
> What script is required to download? 

Just FYI: we will strongly require enhancing sosreport to collect the same data the script does now, rather than calling the script directly. So if you specify the script location and what e.g. package presence shall automatically trigger collecting these data, it will be great.

Hello, this is the script that collects archive logs:
https://github.com/redhat-cfme-support/cloudforms_4.X_log_collection/blob/b6c861998361938cb260a2f906868550c068cefd/collect_CFME_archive_script.sh#L25

This is the script that collects current logs:
https://github.com/redhat-cfme-support/cloudforms_4.X_log_collection/blob/b6c861998361938cb260a2f906868550c068cefd/collect_CFME_current_script.sh#L25

It seems I was mistaken in the fact that manageiq plugin wasn't available, I must've overlooked it when looking for CloudForms since I was looking for our downstream branding when looking on RHEL.  However, it does seem a couple of files we normally (last_settings.txt comes to mind) collect are missing.  The links I provided links to the exact line where we collect and tar all the necessary files.

The cfme package should be sufficient enough trigger for the manageiq plugin, and it looks like it does.  So you can go ahead and close this RFE if you want and I can make a new one RFE to enhance the current plugin, or we can continue on with this bug remedying the couple of missing files not collected by the plugin.

Also, for the katello thing, it looks like the "CloudForms" plugin was removed right when Red Hat acquired ManageIQ, so what I think happened was that Red Hat renamed Katello in order for ManageIQ to adopt the CloudForms name, which I think would explain the discrepancy/confusion there, ie, the old plugin for "CloudForms" wasn't for the CloudForms we know today.

This info is accurate as of sos 3.6 and RHEL 7.6 as I just updated the server.  Thanks for the thorough and quick follow up!  Hope I didn't cause too much confusion here.

OK: that gives us a reasonably small set of changes to consider here. Let's continue to use this bug (I'll edit the title a bit in a moment), and keep all the conversation together in one place.

Thanks.  Is there also any way to prompt the user to choose archive or current logs?  The way we do it from those scripts is that we have 2 separate scripts for those.

So when 'cfme' package is installed / when manageiq plugin is enabled, sos should collect:

- from /var/www/miq/vmdb/ directory:
    BUILD
    GUID
    VERSION
    REGION
    log/*  (at least 25MB files, from newest to older ones - is this OK?)
    config/* (same limit)
    log/apache/* (same limit)
(can we restrict to e.g. *.* there? to prevent automatically collecting subdirectories)

- /var/log/* - what in particular is required? Other sos plugins collect various data already, so running sosreport by default collects lots of files from there.

- if $APPLIANCE_PG_DATA is set, then also:
    $APPLIANCE_PG_DATA/pg_log/*
    $APPLIANCE_PG_DATA/postgresql.conf

- we should exclude files specified in $(pwd)/exclude_files - as sosreport can be run from almost any directory, it is easy to miss this by the user; shall not be there some absolute path, rather?


(In reply to David Luong from comment #7)
> Thanks.  Is there also any way to prompt the user to choose archive or
> current logs?  The way we do it from those scripts is that we have 2
> separate scripts for those.

No interactive prompt, if possible. We can add a plugin option like:
- either "-k manageiq.logtypes=current" (default value, optionally "archive" value)
- or e.g. "-k manageiq.archivelogs=true" (or "yes"; default value = no/false, meaning current logs collected)

Why do you need so much to distinguish between collecting current (*.log) and archive (*) logfiles? In sosreport, the later will cause current logfiles (as newest generated) will be collected either way as the first one, then older one until default limit 25MB per file pattern is reached. To efficiently distinguish between current and archive logs, is there a file pattern specifying *just* archive logs?

Optionally, we can collect *.log by default, and * when sosreport is called with --all-logs option - that is the most typical use case in other plugins as well.


One further question: Aren't there different types of logfiles that you need to collect at least the newest instance every time, regardless if the logfiles are quite old? Assume use case of logs:

ls -ltrh /var/www/miq/vmdb/
..
10M  May 21  2018 install.log
30M  Jan 07  2019 access.log-rotated20190107
11M  Jan 20  2019 access.log

Here sosreport collects for /var/www/miq/vmdb/* filemask :
- first access.log
- then access.log-rotated20190107
- now cumulative size of collected data per filemask reached 25MB (default configurable limit), so no older files are collected

So if you need access.log, install.log and optionally some logrotated versions, we can't call just:

self.add_copy_spec("/var/www/miq/vmdb/*", log_size=optionally_changed)

but rather:

self.add_copy_spec("/var/www/miq/vmdb/access.log*", log_size=optionally_changed)
self.add_copy_spec("/var/www/miq/vmdb/install.log*", log_size=optionally_changed)


So my question is: are there different types / filemasks of logs you need to collect in either way?

Hello Pavel,
 
Yes, the --all-logs function seems sufficient to what I want for collecting archive and to collect *.log and *.txt by default.

For your further question, for current, we just need the current one, not the rotated ones.  The rotated ones would be the "archive" logs, for both cloudforms specific ones, and the logs provided by other programs like the access logs for apache.

So long story short, current would just need current unzipped/untar'd files, and archive logs collect everything, but I'm perfectly alright with the current collecting up to 25MB more past the current logs.

Thanks for feedback but I am still uncertain in some aspects of the specification.

- by default, the plugin will collect (apart of BUILD, GUID, VERSION and REGION):
  - log/*.log
  - config directory (worth to take complete in either case, I think)
  - log/apache/*.log

- with --all-logs, whole log and log/apache will be collected (up to 25MB limit per each dir, by default)

- nothing extra needed from /var/log (what other plugins expected to be enabled by default already collects) ?

- should be there some option to skip or collect current/archive logs, like the 2 current scripts distinguish? Or does --all-logs option cover this ?


Thanks in advance for this clarification.

Hello Pavel,

- the default will also need to collect log/*.txt

- 2nd point lgtm

- Nothing is needed from /var/log AFAIK with the exception of /var/log/tower.log.  IF /var/log/tower.log exists, it would be nice to have that though (this depends if embedded ansible is enabled for the application).

- I think all --all-logs cover this.  The way we currently have it set up is that current script just collects the non-tar'd/non zipped logs and archive collects everything.

Thanks for being so thorough Pavel!

When updating the manageiq plugin, I realized most of the specification requests are incorporated there already:

- plugin enabled by presence of cfme package (or few files it should collect)
- many configs and logfiles are collected from the specified directories
- --all-logs option used as agreed

I just added some reminders ("REGION", tower.log, postgres logs+config).

Not sure if the enumerated config files and log files cover all desired ones - please review that:
 - either by checking https://github.com/sosreport/sos/blob/be0cbb6d24e441615298fb1fc75055e621dbc2cc/sos/plugins/manageiq.py 
 - or testing the plugin itself:

mkdir bz1664464
cd bz1664464
wget https://github.com/sosreport/sos/archive/be0cbb6d24e441615298fb1fc75055e621dbc2cc.zip
unzip https://github.com/sosreport/sos/archive/be0cbb6d24e441615298fb1fc75055e621dbc2cc.zip
cd sos-be0cbb6d24e441615298fb1fc75055e621dbc2cc
./sosreport --batch --build

(In reply to Pavel Moravec from comment #12)
> mkdir bz1664464
> cd bz1664464
> wget
> https://github.com/sosreport/sos/archive/
> be0cbb6d24e441615298fb1fc75055e621dbc2cc.zip
> unzip
> https://github.com/sosreport/sos/archive/
> be0cbb6d24e441615298fb1fc75055e621dbc2cc.zip
> cd sos-be0cbb6d24e441615298fb1fc75055e621dbc2cc
> ./sosreport --batch --build

correction to this procedure (to be valid after force push changes in the PR):

mkdir bz1664464
cd bz1664464
wget https://github.com/sosreport/sos/archive/master.zip
unzip master.zip
cd sos-master
wget https://patch-diff.githubusercontent.com/raw/sosreport/sos/pull/1544.patch
cat 1544.patch | patch -p1

Hey Pavel, 

The changes look good, and the changes are correct.  However, I noticed that the for what files we need from those directories are outdated.  I can make a PR after you merge that one, or I can give you the list and you can rebase it, whichever one is easier for you.

Sincerely,
David Luong

(In reply to David Luong from comment #14)
> Hey Pavel, 
> 
> The changes look good, and the changes are correct.  However, I noticed that
> the for what files we need from those directories are outdated.  I can make
> a PR after you merge that one, or I can give you the list and you can rebase
> it, whichever one is easier for you.
> 
> Sincerely,
> David Luong

Easier is providing the changes now and I will update the PR (so there will be just one change/commit/PR in upstream and just 1 patch to backport), but the other option is also possible.

Hey Pavel, t

You can overwrite both lists with the lists below:

This is what we need to collect from config:

        'application.rb',
        'boot.rb',
        'brakeman.ignore',
        'brakeman.yml',
        'cable.yml.sample',
        'database.pg.yml',
        'default_replication_exclude_tables.yml',
        'dictionary_strings.rb',
        'environment.rb',
        'environments/development.rb',
        'environments/i18n.rb',
        'environments/patches/database_configuration.rb',
        'environments/production.yml',
        'environments/production.rb',
        'ha_admin.yml',
        'human_locale_names.yaml',
        'initializers/active_metrics.rb',
        'initializers/as_to_time.rb',
        'initializers/backtrace_silencers.rb',
        'initializers/create_documentation_for_containers_token.rb',
        'initializers/fast_gettext.rb',
        'initializers/instantiation_listener.rb',
        'initializers/marshal_autoloader.rb',
        'initializers/mime_types.rb',
        'initializers/nokogiri_as_xmlmini_backend.rb',
        'initializers/override_foreman_image_name.rb',
        'initializers/override_openshift_container_manager.rb',
        'initializers/permissions_repository.rb',
        'initializers/postgres_required_versions.rb',
        'initializers/rack_ruby_prof.rb',
        'initializers/secure_headers.rb',
        'initializers/session_memory_store.rb',
        'initializers/session_patch.rb',
        'initializers/session_store.rb',
        'initializers/wrap_parameters.rb',
        'initializers/yaml_autoloader.rb',
        'miq_expression.yml',
        'model_attributes.rb',
        'permissions.tmpl.yml',
        'permissions.yml',
        'preinitializer.rb',
        'puma.rb',
        'routes.rb',
        'secrets.yml.sample',
        'settings/development.yml',
        'settings/production.yml',
        'settings/test.yml',
        'settings.yml',
        'supported_locales.yml',
        'yaml_strings.rb',
        'cable.yml',
        'database.yml.old',
        'database.yml'

This is what we need to collect from log dir:

        'api.log',
        'appliance_console.log',
        'audit.log',
        'automation.log',
        'aws.log',
        'azure.log',
        'container_monitoring.log',
        'datawarehouse.log',
        'evm.log',
        'fog.log',
        'hourly_continuous_pg_maint_stdout.log',
        'kubernetes.log',
        'lenovo.log',
        'middleware.log',
        'nuage.log',
        'policy.log',
        'production.log',
        'rhevm.log',
        'scvmm.log',
        'top_output.log',
        'vcloud.log',
        'vim.log',
        'vmstat_output.log',
        'websocket.log',
        'apache/miq_apache.log',
        'apache/ssl_access.log',
        'apache/ssl_error.log',
        'apache/ssl_request.log',
        'apache/ssl_mirror_request.log',
        'apache/ssl_mirror_error.log',
        'apache/ssl_mirror_access_error.log',
        'gem_list.txt',
        'last_settings.txt',
        'last_startup.txt',
        'package_list_rpm.txt'

Instead of such big enumeration lists (error-prone, easy to overlook something, needs maintenance when adding new config,..), isn't it much concise and efficient to collect e.g.:

1) from config:
*.rb
*.yaml
*.yml
*.yml.sample
settings/*.yml
environments/*.rb
environments/*.yml
environments/patches/*.rb
initializers/*.rb
database.yml.old
brakeman.ignore


2) from logdir:
*.log
apache/*.log
*.txt

Two possible drawbacks:
- sos can collect something unwanted / confidential in an extra file (I dont expect so, though)
- if e.g. *.log files will have over 25MB in summary, just the newest files will be collected 
  - so for logs, we shall rather be conservative and collect all individual logfiles separately, plus *.txt ?

Let's do your suggestion and collect by *. The concerns you raise should be fine by us since our command line script does the * collection anyways already.

(In reply to David Luong from comment #18)
> Let's do your suggestion and collect by *. The concerns you raise should be
> fine by us since our command line script does the * collection anyways
> already.

Thanks, PR updated accordingly.

POSTed to upstream.

*** Bug 1690953 has been marked as a duplicate of this bug. ***

A coworker has informed me of a new conf directory, so I opened up a PR for it:
https://github.com/sosreport/sos/pull/1637

Let me know if I should open a new bug for this.

(In reply to David Luong from comment #26)
> A coworker has informed me of a new conf directory, so I opened up a PR for
> it:
> https://github.com/sosreport/sos/pull/1637
> 
> Let me know if I should open a new bug for this.

Out of scope of this BZ.

That fix will appear in RHEL8.2 for sure and quite probably in 7.8 if we will do rebase. If you want to ensure it is in 7.8, let raise BZ.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2295