Bug 1923835

What is the output of `systemctl list-units | grep -i insights`?

On the test server the output of that command:
 # systemctl list-units | grep -i insights
  insights-client-results.path                                                                loaded active waiting   Monitor /etc/insights-client/.lastupload for modifications
  insights-client.timer                                                                       loaded active waiting   Insights Client Timer Task

For the customer side please see comment #3 in the case.

Only the monitor and timer come up for both.
So no extra timers or the like.
You should be able to see this behavior in any normal running installation of insights.

I know that a log is made each time the client is used.
For example, during initial registration, there will be several log files for different tasks.
But that is a one time occurrence.

A new log file is created every time insights-client is run. The first log is generated by the nightly collection run of insights-client (triggered by insights-client.timer starting the insights-client.service unit). Towards the end of this process, it updates the file `/etc/insights-client/.lastupload`. The insights-client-results.path watches that file for changes and triggers the insights-client-results.service unit. _That_ unit runs `insights-client --check-results`. The second execution of insights-client with the `--check-results` flag then creates the second, smaller log file.

This is functioning "as designed", and is an unfortunate side-effect of a combination of factors based on the software's current architecture.

* Logs are rotated every time the process starts
* The process runs twice in an effort to retrieve the Advisor results from the archive that was most recently updated. This implementation can been seen in the `--check-results` and insights-client-results.{path,service} files. It was necessary to implement this feature in this way because insights-client does not have an event-triggering system, or a long-lived system daemon process that can run code at appropriate times. It can only operate when invoked (by a user, or by a process manager like systemd).

If you don't care about the "check-results" feature (where the JSON content of the most recent Advisor analysis is cached locally on the system), the simplest solution to this would be to disable and mask the path unit. This will stop the path unit and prevent it from starting. That, in turn, would prevent the second insights-client invocation from occurring and rotating the logs.

systemctl stop insights-client-results.path
systemctl mask insights-client-results.path

If you do use the check-results feature, then we would need to pursue a couple of possible solutions:

* Allow the number of logs kept to be modified by setting a value in the config file.
* Name the log file after the command string; then the log will rotate only on multiple invocations of the same command. So if I invoked "insights-client --check-results --verbose", a log file named insights-client_--check-results_--verbose.log would get created, and would only rotate if "insights-client --check-results --verbose" is invoked again. This has some obvious shortcomings that could leave log files around on the system indefinitely, so I'm not convinced this is a good approach yet.

Migrating this bug as part of the effort to close the cloud.redhat.com/Client Product/Component.

Link,

What application/process is using the feature of caching the Advisor JSON enabled?
Why is a feature to download content just to populate a cached value that potential be used enabled by default, for me this is not backwards compatbile.
If 'systemctl mask insights-client-results.path' is an approach that the user can be doing. Where is it documented?

Honestly the most simple approach is to change just the keep '4' to '16' logfiles. I do not see why the 4 logfiles is such a constraint.
Also implement logrotate with compression then space to keep 30 days of logs is also not an issue.


Peter

(In reply to Peter Vreman from comment #6)
> Link,
> 
> What application/process is using the feature of caching the Advisor JSON
> enabled?

This was added to enable an integration between Cockpit and insights-client. The Cockpit team wanted to enhance their Insights integration to show a summary of the number of Advisor "hits" a particular system had when navigating the Cockpit UI.

> Why is a feature to download content just to populate a cached value that
> potential be used enabled by default, for me this is not backwards
> compatbile.

This is a fair point. It was probably something more appropriately released with a new "Y release", rather than in a batch update. Since the time when this feature was released, we began putting in more effort to make insights-client releases that align with appropriate RHEL releases; this should make these sort of unexpected feature enhancements more palatable.

> If 'systemctl mask insights-client-results.path' is an approach that the
> user can be doing. Where is it documented?

We may not have explicit documentation describing how to disable the insights-client-results.path systemd unit using the systemd masking feature. I can ask around if it is documented any where. Generally, masking a unit is considered a reasonable way to permanently disable a systemd unit, so there may have been some assumptions made that a system administrator who wished to disable a unit would "just know" to mask it.

> Honestly the most simple approach is to change just the keep '4' to '16'
> logfiles. I do not see why the 4 logfiles is such a constraint.

It's pretty arbitrary, yes. At the time, 4 was probably selected because it seemed reasonable, along with the assumption that insights-client would only ever run once per 24 hours. I agree that exposing the number of log files to keep as a config option is the simplest approach. It's a reasonable feature to add, without any downside, so I'll add it to our list of feature enhancements.

> Also implement logrotate with compression then space to keep 30 days of logs
> is also not an issue.

I believe the current implementation uses a Python log rotation library. I don't know whether that uses compression or not, but that is a reasonable thing to add while we're making changes to the logging code.

> 
> Peter

Thanks for the https://access.redhat.com/solutions/5844811 that explains is nicely

For a real fix i recommand to look at the logfile topic as a whole. E.g. do you want to write a new log file for every action, or just append to an existing logfile.
E.g. also commands like '--version' and '--validate' are impacted.

I have raised RFE asking for logrotate in insights-client, sadly it is not picked up with the right mindset to improve a Insights, instead it was directly mentioned RHEL7 is end of life.

For me it is ok to have this BZ closed for the 'insights-client-result' fixed if the RFE to implement lograote is going to be implemented. If the logrotate is not going forward then I want to keep this BZ open because also simple actions like '--version' and '--validate' are createing new log files and are counted against the 4 logfiles history.

(In reply to Peter Vreman from comment #8)
> Thanks for the https://access.redhat.com/solutions/5844811 that explains is
> nicely
> 
> For a real fix i recommand to look at the logfile topic as a whole. E.g. do
> you want to write a new log file for every action, or just append to an
> existing logfile.
> E.g. also commands like '--version' and '--validate' are impacted.
> 
> I have raised RFE asking for logrotate in insights-client, sadly it is not
> picked up with the right mindset to improve a Insights, instead it was
> directly mentioned RHEL7 is end of life.

This was probably an automatic comment added as a side effect of us moving all the insights-client bugs under the Red Hat Enterprise Linux product family. Would you mind linking the bug number? We can take a look and see if it's really something that is RHEL 7 specific. If not, it can be moved the the Red Hat Enterprise Linux 8 product.

> 
> For me it is ok to have this BZ closed for the 'insights-client-result'
> fixed if the RFE to implement lograote is going to be implemented. If the
> logrotate is not going forward then I want to keep this BZ open because also
> simple actions like '--version' and '--validate' are createing new log files
> and are counted against the 4 logfiles history.

I'm more in favor of a general logrotate fix too, rather than a specific fix for this bug. It sounds like the workaround from the Solutions article is enough to mitigate this bug for you, correct?

Link,

The masking works.
To be one the safe side I implemented it with 2 masks

systemctl mask insights-client-results.service
systemctl mask insights-client-results.path


There is one downside of masking 'insights-client-results.path'. There is a systemd dependency info message is not that nice and should be maybe be mentioned in the KB that it can safely be ignored:
----
vrempet-admin@li-lc-1437 ~
$ sudo systemctl status insights-client-results.path
● insights-client-results.path
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead)

Mar 16 14:30:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 14:55:01 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 14:58:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 15:25:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 15:28:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 15:30:01 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 15:55:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 15:58:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 16:25:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.
Mar 16 16:28:00 li-lc-1437 systemd[1]: Cannot add dependency job for unit insights-client-results.path, ignoring: Unit is masked.

vrempet-admin@li-lc-1437 ~
$ sudo systemctl status insights-client-results.service
● insights-client-results.service
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead)

vrempet-admin@li-lc-1437 ~
$
----



The dependency is added by comes from:
----
/usr/lib/systemd/system/insights-client.timer:Also=insights-client-results.path
----
Is this really needed here? For most users insights-client-results.path is not needed at all and shall be managed by a 'cockpit-insight-client' integration and not for simple insights-client with the collected data sent to cloud.redhat.com

The work around found here:
   https://access.redhat.com/solutions/5844811
is satisfactory for the current logging system in the insights-client.

The KB has been updated with a note of the systemd errors.
I believe this case can be closed.

Any news to migrate the cockpit integration to a dedicated cockpit rpm and not in the base insights-client?

(In reply to Peter Vreman from comment #12)
> Any news to migrate the cockpit integration to a dedicated cockpit rpm and
> not in the base insights-client?

At this moment, there is no plan to do a dedicated cockpit rpm.

The KB seems to help solve this issue. 
There is a plan of future work in logrotate and improvements of logs in insights-client.
So it seems this case can be closed.

Peter, we dug up https://bugzilla.redhat.com/show_bug.cgi?id=1940267 and will be implementing a solution using logrotate. I think this provides a more flexible solution overall.

Link, thanks, yes using logrotate is the most flexible way. I also asked for logrotate support in the past already that i as a customer can control the logrotation to my own needs.