Created attachment 1217495 [details] grep of process 26692 containing several examples of problem Description of problem: In the following example, a worker process heartbeat is recorded as failing, but the log line type is not ERROR, but INFO. ===== [----] I, [2016-11-02T05:00:56.641451 #26692:e5798c] INFO -- : MIQ(MiqVimBrokerWorker::Runner) ID [50000001623777] PID [26692] GUID [f64863c8-a0de-11e6-b77e-005056827e39] Error heartbeating because PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "replication_set_table_pkey" DETAIL: Key (set_id, set_reloid)=(-299507980, 16492) already exists. ===== Version-Release number of selected component (if applicable):5.6.1.2 How reproducible: FALL-OUT from pglogical heartbeat issues. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: It is expected that a heartbeat failure for any reason would be reported on an ERROR log line and not an INFO log line. Not reporting this as an error makes this much harder to isolate.
Created attachment 1217537 [details] grep of entire two week logs of wk08 appliance showing scope of inconsistency of log line heartbeat error reporting
This issue is resolved by the hot fix provided in BZ 1392087
PR - https://github.com/ManageIQ/manageiq/pull/12513
The PR mentioned in comment 6 actually fixes the underlying issue which was causing the log message. It is possible in the future that other errors in the VimBrokerWorker get logged as info when they should be logged as error, and that is not yet addressed. Moving this out of POST and changing the priority and severity to low as this is just about the issue of log levels.
Nick, based on your comment above, I assume that you have disconnected all of the idiot lights on your dashboard, as they are just indicators of a problem and not the real problem. If I am incorrect in this assumption, allow me to suggest that these indicators of error condition are *not* low priority.
https://github.com/ManageIQ/manageiq/pull/12889
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/7596a5526d99bd4d4001d4bbba1e0975e051dfb8 commit 7596a5526d99bd4d4001d4bbba1e0975e051dfb8 Author: Nick Carboni <ncarboni> AuthorDate: Mon Nov 28 15:35:33 2016 -0500 Commit: Nick Carboni <ncarboni> CommitDate: Mon Nov 28 16:14:33 2016 -0500 Use the exit code when stopping the vim broker This will allow exits on error to log an error message instead of an info message. https://bugzilla.redhat.com/show_bug.cgi?id=1392087 app/models/miq_vim_broker_worker/runner.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Verified in 5.8.0.2