Bug 1628658

Summary:	Power management via API falling into the wrong zone leading to permanently queued requests
Product:	Red Hat CloudForms Management Engine	Reporter:	Tuan <tuado>
Component:	API	Assignee:	Brandon Dunne <bdunne>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Parthvi Vala <pvala>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	5.9.3	CC:	bdunne, dmetzger, hkataria, lavenel, mfeifer, mpovolny, obarenbo, simaishi, smallamp
Target Milestone:	GA	Keywords:	TestOnly, ZStream
Target Release:	5.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	5.10.0.19	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1635764 (view as bug list)		Environment:
Last Closed:	2019-02-12 16:53:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	CFME Core	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1635764

Description Tuan 2018-09-13 16:23:32 UTC

Description of problem:
	
We have multi-regional setup (master region, vmware region, onsite region etc.). In this case we are talking about master <-> onsite. 

Onsite is used for vmware with native vmware vcenter connector.

On master we have 2 appliances in zone DefaultZone only.

On onsite we have 6 appliances, 2 in OnSite-Management zone and 4 in OnSite-01 zone.

We need to switch on and off VMs in OnSite-01 zone in Onsite region. If I try to use UI on master, everything works fine. But if I uses API on master my request is queued in Master region in OnSite-01 zone (which not exists in Master region).

This API call:
curl -X GET https://10.241.114.33/api/vms/10000000001382 -k -u username:password -d '{"action": "start"}'

Results to this task in queue in Master region:

#<MiqQueue:0x000000000a649240
 id: 99000002383731,
 target_id: nil,
 priority: 100,
 method_name: "stop",
 state: "ready",
 created_on: Wed, 12 Sep 2018 13:15:08 UTC +00:00,
 updated_on: Wed, 12 Sep 2018 13:15:08 UTC +00:00,
 lock_version: 0,
 task_id: nil,
 deliver_on: nil,
 queue_name: "generic",
 class_name: "ManageIQ::Providers::Vmware::InfraManager::Vm",
 instance_id: 10000000001382,
 args: [],
 miq_callback:
  {:class_name=>"MiqTask",
   :instance_id=>99000000035474,
   :method_name=>:queue_callback,
   :args=>["Finished"]},
 msg_data: nil,
 zone: "OnSite-01",
 role: "ems_operations",
 server_guid: nil,
 msg_timeout: 600,
 handler_id: nil,
 handler_type: nil,
 expires_on: nil,
 tracking_label: nil,
 user_id: nil,
 group_id: nil,
 tenant_id: nil>

We need to use API because we need to allow our customers to call power operations from service now.


Where are you experiencing the behavior?  What environment?
Production

When does the behavior occur? Frequently?  Repeatedly?   At certain times?
Every time we call API for every VM in different zone then DefaultZone. DefaultZone work because coincidence on zone name in region and master.

What information can you provide around timeframes and the business impact?

We are not able to provide service to our customers for all their VMs.


Version-Release number of selected component (if applicable):
5.9.3.4

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Tuan 2018-09-13 16:26:40 UTC

Created attachment 1483096 [details]
onsite

Comment 6 CFME Bot 2018-09-13 19:01:17 UTC

https://github.com/ManageIQ/manageiq/pull/17987

Comment 7 CFME Bot 2018-09-14 19:31:14 UTC

New commit detected on ManageIQ/manageiq/master:

https://github.com/ManageIQ/manageiq/commit/bf66236a6db960c8cab4225182a6bee80697bcf2
commit bf66236a6db960c8cab4225182a6bee80697bcf2
Author:     Brandon Dunne <brandondunne>
AuthorDate: Thu Sep 13 14:53:02 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Thu Sep 13 14:53:02 2018 -0400

    Prevent queueing things for a zone that doesn't exist in the region

    https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 app/models/miq_queue.rb | 2 +
 spec/lib/extensions/ar_nested_count_by_spec.rb | 45 +-
 spec/lib/tasks/evm_application_spec.rb | 19 +-
 spec/models/miq_action_spec.rb | 6 +-
 spec/models/miq_queue_spec.rb | 20 +-
 spec/models/miq_request_spec.rb | 5 +-
 spec/models/miq_task_spec.rb | 16 +-
 spec/models/mixins/authentication_mixin_spec.rb | 3 +-
 8 files changed, 70 insertions(+), 46 deletions(-)

Comment 8 CFME Bot 2018-09-19 22:28:37 UTC

https://github.com/ManageIQ/manageiq-api/pull/472

Comment 11 CFME Bot 2018-09-28 22:31:34 UTC

https://github.com/ManageIQ/manageiq/pull/18036

Comment 13 CFME Bot 2018-10-01 20:07:24 UTC

New commit detected on ManageIQ/manageiq/master:

https://github.com/ManageIQ/manageiq/commit/d5c75adf923ea1e57d18e26480ca09afa495ff71
commit d5c75adf923ea1e57d18e26480ca09afa495ff71
Author:     Brandon Dunne <brandondunne>
AuthorDate: Fri Sep 28 18:15:47 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Fri Sep 28 18:15:47 2018 -0400

    Currently on a successful ActionResult (start / stop) we return nil

    Returning the result of the api call would at least be a positive result
    https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 app/models/mixins/inter_region_api_method_relay.rb | 1 +
 spec/models/mixins/inter_region_api_method_relay_spec.rb | 8 +-
 2 files changed, 5 insertions(+), 4 deletions(-)

Comment 14 CFME Bot 2018-10-02 13:21:22 UTC

New commit detected on ManageIQ/manageiq/hammer:

https://github.com/ManageIQ/manageiq/commit/266b6c550dcb461206993916828ea9c5295b4e4a
commit 266b6c550dcb461206993916828ea9c5295b4e4a
Author:     Nick Carboni <ncarboni>
AuthorDate: Mon Oct  1 15:56:11 2018 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Mon Oct  1 15:56:11 2018 -0400

    Merge pull request #18036 from bdunne/action_result_shouldnt_be_nil

    Currently on a successful ActionResult (start / stop) we return nil

    (cherry picked from commit 2fb626056c1b7e53203ac7d727780693a1c619c1)

    https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 app/models/mixins/inter_region_api_method_relay.rb | 1 +
 spec/models/mixins/inter_region_api_method_relay_spec.rb | 8 +-
 2 files changed, 5 insertions(+), 4 deletions(-)

Comment 15 CFME Bot 2018-10-03 12:58:38 UTC

New commits detected on ManageIQ/manageiq-api/master:

https://github.com/ManageIQ/manageiq-api/commit/5d11ea815ee701ee826fbc207af174dbce71c7f2
commit 5d11ea815ee701ee826fbc207af174dbce71c7f2
Author:     Brandon Dunne <brandondunne>
AuthorDate: Wed Sep 19 14:46:12 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Wed Sep 19 14:46:12 2018 -0400

    Add CentralAdmin support to the API

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 app/controllers/api/mixins/central_admin.rb | 38 +
 1 file changed, 38 insertions(+)


https://github.com/ManageIQ/manageiq-api/commit/23b8a38980955f96c3147a6e5bf697824228cb70
commit 23b8a38980955f96c3147a6e5bf697824228cb70
Author:     Brandon Dunne <brandondunne>
AuthorDate: Wed Sep 19 16:04:41 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Wed Sep 19 16:04:41 2018 -0400

    Add CentralAdmin support to /api/vms for:

    - pause
    - reboot_guest
    - refresh
    - reset
    - scan
    - shelve
    - shelve_offload
    - shutdown_guest
    - start
    - stop
    - suspend

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 app/controllers/api/vms_controller.rb | 7 +
 spec/requests/central_admin/vms_spec.rb | 32 +
 2 files changed, 39 insertions(+)


https://github.com/ManageIQ/manageiq-api/commit/81bb2b755fb31b737838a03d8906dc2fd18b5da8
commit 81bb2b755fb31b737838a03d8906dc2fd18b5da8
Author:     Brandon Dunne <brandondunne>
AuthorDate: Wed Sep 19 18:19:15 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Wed Sep 19 18:19:15 2018 -0400

    Add CentralAdmin support to /api/instances for:

    - pause
    - reboot_guest
    - reset
    - shelve
    - start
    - stop
    - suspend
    - terminate

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 app/controllers/api/instances_controller.rb | 6 +
 spec/requests/central_admin/instances_spec.rb | 29 +
 2 files changed, 35 insertions(+)


https://github.com/ManageIQ/manageiq-api/commit/99221804f4d0cc8688dfb934f88be2d6b47f1b73
commit 99221804f4d0cc8688dfb934f88be2d6b47f1b73
Author:     Brandon Dunne <brandondunne>
AuthorDate: Wed Sep 19 23:20:51 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Wed Sep 19 23:20:51 2018 -0400

    Refactor into vms and instances shared_examples

    https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 spec/requests/central_admin/instances_spec.rb | 37 +-
 spec/requests/central_admin/vms_spec.rb | 43 +-
 spec/support/shared_examples/resource_power_operations.rb | 20 +
 3 files changed, 43 insertions(+), 57 deletions(-)


https://github.com/ManageIQ/manageiq-api/commit/f68519416f094a406042568a40b2a538ecdea2f7
commit f68519416f094a406042568a40b2a538ecdea2f7
Author:     Brandon Dunne <brandondunne>
AuthorDate: Wed Sep 19 23:25:07 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Wed Sep 19 23:25:07 2018 -0400

    Move central admin tests into existing spec files

    https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 spec/requests/central_admin/instances_spec.rb | 12 -
 spec/requests/central_admin/vms_spec.rb | 15 -
 spec/requests/instances_spec.rb | 10 +
 spec/requests/vms_spec.rb | 11 +
 4 files changed, 21 insertions(+), 27 deletions(-)


https://github.com/ManageIQ/manageiq-api/commit/504eda4bdfa1283c0dcf23cfc7817f9d74e6e240
commit 504eda4bdfa1283c0dcf23cfc7817f9d74e6e240
Author:     Brandon Dunne <brandondunne>
AuthorDate: Thu Sep 20 00:04:59 2018 -0400
Commit:     Brandon Dunne <brandondunne>
CommitDate: Thu Sep 20 00:04:59 2018 -0400

    Existing tests should be using the current region

    https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 spec/requests/instances_spec.rb | 2 +-
 spec/requests/vms_spec.rb | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Comment 17 CFME Bot 2018-10-11 14:30:01 UTC

New commit detected on ManageIQ/manageiq-api/hammer:

https://github.com/ManageIQ/manageiq-api/commit/60cb7c906f7ec9724e5ac56f76e2558bfae1fd0b
commit 60cb7c906f7ec9724e5ac56f76e2558bfae1fd0b
Author:     Gregg Tanzillo <gtanzill>
AuthorDate: Wed Oct  3 08:56:43 2018 -0400
Commit:     Gregg Tanzillo <gtanzill>
CommitDate: Wed Oct  3 08:56:43 2018 -0400

    Merge pull request #472 from bdunne/central_admin

    Add basic CentralAdmin support to the API

    (cherry picked from commit 4dbf6b0bbb3cc10be69d6259c069e385ee7bc10c)

    https://bugzilla.redhat.com/show_bug.cgi?id=1628658

 app/controllers/api/instances_controller.rb | 6 +
 app/controllers/api/mixins/central_admin.rb | 38 +
 app/controllers/api/vms_controller.rb | 7 +
 spec/requests/instances_spec.rb | 12 +-
 spec/requests/vms_spec.rb | 13 +-
 spec/support/shared_examples/resource_power_operations.rb | 20 +
 6 files changed, 94 insertions(+), 2 deletions(-)

Comment 18 Parthvi Vala 2018-11-01 08:34:17 UTC

FIXED. Verified on 5.10.0.21.