Bug 1422943

Summary: Reports are queued indefinitely
Product: Red Hat CloudForms Management Engine Reporter: Alex Mayberry <amayberr>
Component: ReportingAssignee: Gregg Tanzillo <gtanzill>
Status: CLOSED WONTFIX QA Contact: Dave Johnson <dajohnso>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.6.0CC: amayberr, dscott, itewksbu, jhardy, obarenbo, yrudman
Target Milestone: GA   
Target Release: cfme-future   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: report:distributed
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-01 18:59:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alex Mayberry 2017-02-16 15:37:57 UTC
Description of problem:
Initiating a report from a server that does not have the reporting role enabled will queue the report, but it will not be run until the server it was submitted from has the reporting role enabled, nor will it be picked up by another server.


Version-Release number of selected component (if applicable):
5.6.3.3

How reproducible:
100%

Steps to Reproduce:
1.  Disable reporting role for worker.
2.  Submit reporting job from worker.
3.  Monitor saved reports and note queued status.
4.  Verify no other workers pick up the reporting task.
5.  Enable reporting role, and verify the report runs.

Actual results:
Reporting job is indefinitely queued.

Expected results:
A:  Error message is generated and job submission is blocked if reporting is not enabled.
B:  Reporting jobs are accessible to other servers with the reporting role enabled. (preferred)

Additional info:

Comment 2 Dave Johnson 2017-03-01 21:27:25 UTC
Alex, can you reproduce this please.  Sounds like a distributed configuration problem.

Comment 3 Alex Mayberry 2017-03-01 21:46:05 UTC
Yes.
Are you seeing a different behavior in your test environment?

Today I deployed a fresh appliance, a single appliance.  Using this as a fresh test environment and disabling the reporting role, I see the report stay queued indefinitely.

Unlike the original scenario scenario, there are no other workers to pick it up, but it still allowed me to queue it.  Enabling the reporting role again, the task was picked up and executed.

I'm not sure what specific output to share with you to assist, but here are the tasks to show the stalled report, and then I re-enabled the reporting role, and the report runs.  finally, the queue clears.


vmdb_production=# select id,priority,method_name,state,queue_name,class_name,zone,role,msg_timeout from miq_queue;
       id       | priority |      method_name      | state | queue_name | class_name |  zone   | role | msg_timeout 
----------------+----------+-----------------------+-------+------------+------------+---------+------+-------------
 10000000003106 |      100 | _async_generate_table | ready | reporting  | MiqReport  | default |      |        3600
(1 row)

vmdb_production=# select id,priority,method_name,state,queue_name,class_name,zone,role,msg_timeout from miq_queue;
       id       | priority | method_name | state | queue_name |     class_name     |  zone   |    role    | msg_timeout 
----------------+----------+-------------+-------+------------+--------------------+---------+------------+-------------
 10000000003183 |       20 | dispatch    | ready | generic    | JobProxyDispatcher | default | smartstate |         600
(1 row)

vmdb_production=# select id,priority,method_name,state,queue_name,class_name,zone,role,msg_timeout from miq_queue;
 id | priority | method_name | state | queue_name | class_name | zone | role | msg_timeout 
----+----------+-------------+-------+------------+------------+------+------+-------------
(0 rows)

Comment 4 Alex Mayberry 2017-03-02 15:43:39 UTC
In the original environment, the software was updated from 4.1 to 4.2 (Version 5.7.1.3.20170221135006_818f133)

Within the UI zone (2 appliances), one appliance had the "reporting" role, and the other did not.  From the one that did not have the role, a report was submitted, and was picked up and processed by the one that had the role enabled.  (success!)

From the (DB zone), there is only the single DB appliance.  It has only the DB role, and the UI/Web roles enabled.  I submitted a job from there, and it remained queued indefinitely.

I enabled the "scheduler" role on the DB, and the queued job did not process.  I submitted a new reporting job, and it also remained queued indefinitely.

I enabled the "notifier" role on the DB, and the queued jobs did not process.  I submitted a new reporting job, and it also remained queued indefinitely.

I disabled "scheduler" and "notifier" roles and enabled "reporting", and all 3 jobs were processed.

So, it appears that the "reporting" role is all that's really needed (other than the UI role) to log in to the webUI and submit/run a report successfully.  Workers with the role enabled within that zone will pick it up and run it, even if the one you submit it from doesn't have the role.  But jobs submitted in zones where no workers have the role will queue indefinitely.

Comment 7 Yuri Rudman 2018-08-01 18:59:37 UTC
works as designed