Bug 1620161 - issuing vm_reconfigure disk_remove via rest-api FAILS (WORKAROUND AVAILABLE)
Summary: issuing vm_reconfigure disk_remove via rest-api FAILS (WORKAROUND AVAILABLE)
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: API
Version: 5.8.4
Hardware: x86_64
OS: Linux
Target Milestone: GA
: 5.10.0
Assignee: William Fitzgerald
QA Contact: Parthvi Vala
Depends On: 1666593
Blocks: 1629096
TreeView+ depends on / blocked
Reported: 2018-08-22 14:47 UTC by Reartes Guillermo
Modified: 2019-10-23 18:30 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1629096 (view as bug list)
Last Closed: 2019-02-07 23:03:31 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:0212 None None None 2019-02-07 23:03:38 UTC

Description Reartes Guillermo 2018-08-22 14:47:01 UTC
Description of problem:

When issuing a reconfigure vm via rest-api to Cloudforms, specifically a disk removal, it fails.

I tried with two json objects (See below), one is using only "strings" and other is using only :symbols. No mater what, "requested" cannot read the disk name. 

WORKAROUND: Patch "requested" in one line, change :disk_name to "disk_name" and then it works. (details below)

Version-Release number of selected component (if applicable):
CF   : 4.5 (
RHV  : 4.1 (

The line i patched in "requested" is the same in CF 4.6, so it is most likely affected too.
How reproducible:

Steps to Reproduce:
1. add a new disk via CF reconfigure vm out of the box mechanism
2. get the disk name (an uuid, in case of RHV)
3. issue the "vm_reconfigure" "disk_remove" with either json_data_disk_remove_1 or json_data_disk_remove_2 objects.
4. it will faill in quota evaluation

Actual results:
unable to remove a disk to vm by issuing vm_reconfigure via rest-api

Expected results:
being able to remove the disk.

Additional Data #1 -> used JSON Objects for the rest-api:

json_data_disk_remove_1 = {
    "action"  =>"create",
    "options" => {
                  "src_ids" => [
                  "request_type" => "vm_reconfigure",
                  "disk_remove" => [
                                     "disk_name"      => "5aeff92b-bcb6-4a29-b219-57d842cbc879",
                                     "delete_backing" => false
    "auto_approve" => false
json_data_disk_remove_2 = {
    :action  => "create",
    :options => {
                 :src_ids => [
                 :request_type => "vm_reconfigure",
                 :disk_remove => [
                                   :disk_name      => "9984de0d-c52c-4185-9e63-0759da6c6913",
                                   :delete_backing => false
    :auto_approve => false

Comment 2 Reartes Guillermo 2018-08-22 14:51:30 UTC
Additional Info #2 (WORKAROUND DETAILS):

Faulted Method: / System / CommonMethods / QuotaMethods / requested

args_hash[:resource].options[:disk_remove].each do |disk|
   disk_num = disk[:disk_name].match(/_(\d).vmdk/)


$evm.log(:info, ""DEBUG -> '#{args_hash[:resource].options[:disk_remove]}'"")
args_hash[:resource].options[:disk_remove].each do |disk|
$evm.log(:info, ""DEBUG -> '#{disk}'"")
$evm.log(:info, ""DEBUG -> '#{disk['disk_name']}'"")
disk_num = disk['disk_name'].match(/_(\d).vmdk/)"

Note: :disk_name was changed to 'disk_name'

Comment 3 Reartes Guillermo 2018-08-22 14:51:58 UTC
Additional Info #3 (TRACEBACK):

<AEMethod requested> Request: VM Reconfigure for: vmx5b - Remove Disks: 1 id: 1000000000107 
AEMethod requested> DEBUG -> '[{""disk_name""=>""5aeff92b-bcb6-4a29-b219-57d842cbc879"", ""delete_backing""=>false}]'
AEMethod requested> DEBUG -> '{""disk_name""=>""5aeff92b-bcb6-4a29-b219-57d842cbc879"", ""delete_backing""=>false}'
AEMethod requested> DEBUG -> '5aeff92b-bcb6-4a29-b219-57d842cbc879'
<AEMethod requested> The following error occurred during method evaluation:
<AEMethod requested>   NoMethodError: undefined method `match' for nil:NilClass

<AEMethod requested>   /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:164:in `block in requested_storage'
/QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:161:in `each'
/QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:161:in `requested_storage'
/QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:103:in `vm_prov_option_value'
/QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:36:in `collect_template_totals'
/QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:25:in `get_total_requested'
/QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:19:in `calculate_requested'
Method STDERR: /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:164:in `block in requested_storage': undefined method `match' for nil:NilClass (NoMethodError)
Method STDERR: Did you mean?  catch
Method STDERR:         from /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:161:in `each'
Method STDERR:         from /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:161:in `requested_storage'
Method STDERR:         from /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:103:in `vm_prov_option_value'
Method STDERR:         from /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:36:in `collect_template_totals'
Method STDERR:         from /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:25:in `get_total_requested'
Method STDERR:         from /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:19:in `calculate_requested'
Method STDERR:         from /QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested:380:in `<main>'
<AEMethod [/QuotaVMReconfigure/System/CommonMethods/QuotaMethods/requested]> Ending
Aborting instantiation (unknown method return code) because [Method exited with rc=Unknown RC: [1]]
State=<requested> running  raised exception: <Method exited with rc=Unknown RC: [1]>
Error in State=[requested]"

Comment 4 dmetzger 2018-08-23 13:46:54 UTC
Please test on 5.9.4 and verify whether the issue persists.

Comment 5 Reartes Guillermo 2018-08-23 18:48:14 UTC
The affected line of code is the same.

Tested and the issue persists.

Comment 7 CFME Bot 2018-09-04 21:31:36 UTC
New commit detected on ManageIQ/manageiq-content/master:

commit 011643f7c38921d4800115fc6a70a8f9d256db3f
Author:     william fitzgerald <wfitzger@redhat.com>
AuthorDate: Mon Aug 27 17:39:58 2018 -0400
Commit:     william fitzgerald <wfitzger@redhat.com>
CommitDate: Mon Aug 27 17:39:58 2018 -0400

    Updated requested method to fix problem issuing vm_reconfigure via rest-api.

    Issuing a vm_reconfigure disk_remove via rest-api was failing.
    Very minor change (:disk_name was changed to 'disk_name')

    disk_num = disk['disk_name'].match(/_(\d).vmdk/)"


    Added a test for disk_remove

 content/automate/ManageIQ/System/CommonMethods/QuotaMethods.class/__methods__/requested.rb | 2 +-
 spec/automation/unit/method_validation/requested_spec.rb | 9 +
 2 files changed, 10 insertions(+), 1 deletion(-)

Comment 10 Parthvi Vala 2018-11-21 11:05:56 UTC
FIXED. Verified on with vmware(vCenter 6.7).

Request: POST /api/requests
Query: {
    "action"  :"create",
    "options" : {
                  "src_ids" : [
                  "request_type" : "vm_reconfigure",
                  "disk_remove" : [
                                     "disk_name"      : "[NFS_Datastore_1] pvala_vm_001/pvala_vm_001_3.vmdk",
                                     "delete_backing" : false
    "auto_approve" : false

Response: {
	"results": [
			"href": "https://<ip_address>/api/requests/5",
			"id": "5",
			"description": "VM Reconfigure for: pvala_vm_001 - Remove Disks: 1",
			"approval_state": "pending_approval",
			"type": "VmReconfigureRequest",
			"created_on": "2018-11-21T10:56:24Z",
			"updated_on": "2018-11-21T10:56:24Z",
			"fulfilled_on": null,
			"requester_id": "1",
			"requester_name": "Administrator",
			"request_type": "vm_reconfigure",
			"request_state": "pending",
			"message": "VM Reconfigure - Request Created",
			"status": "Ok",
			"options": {
				"src_ids": [
				"request_type": "vm_reconfigure",
				"disk_remove": [
						"disk_name": "[NFS_Datastore_1] pvala_vm_001/pvala_vm_001_3.vmdk",
						"delete_backing": false
			"userid": "admin",
			"source_id": null,
			"source_type": null,
			"destination_id": null,
			"destination_type": null,
			"tenant_id": "1",
			"service_order_id": null,
			"process": true,
			"cancelation_status": null

The disk is deleted successfully once the request is complete.

Comment 11 CFME Bot 2018-11-28 15:01:56 UTC
New commit detected on ManageIQ/manageiq-content/hammer:

commit f42263cbd3a992fd103b6eb422f20400dc5f1014
Author:     tina <tina.fitzgerald2@gmail.com>
AuthorDate: Tue Nov 27 16:29:28 2018 -0500
Commit:     tina <tina.fitzgerald2@gmail.com>
CommitDate: Tue Nov 27 16:29:28 2018 -0500

    Merge pull request #474 from billfitzgerald0120/quota_vm_reconfig_disk_remove_fix

    Changed disk_name to symbol in requested Quota method.

    (cherry picked from commit da43013051edc76032eb483238ade483ea54b787)

    Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1620161

 content/automate/ManageIQ/System/CommonMethods/QuotaMethods.class/__methods__/requested.rb | 5 +-
 spec/automation/unit/method_validation/requested_spec.rb | 4 +-
 2 files changed, 5 insertions(+), 4 deletions(-)

Comment 13 William Fitzgerald 2018-11-29 17:27:08 UTC

The json_data_disk_remove_2 is using a symbol (:disk_name) in the call instead of a string ("disk_name").  It is using a symbol for disk_name and delete_backing.

The new commit (PR 474) will work for Vmware and RHV.  The older code was trying to match on the file extension of .vmdk which is for Vmware only.  If you tested the old code on a RHV machine, it would not work.

You need to turn quota on and make sure the requested method runs.  Check the automaton log for results.  

Let me know if you have any more questions.


Comment 15 William Fitzgerald 2019-01-03 14:29:04 UTC

Do you have reproducer ?

Comment 17 William Fitzgerald 2019-01-03 19:09:49 UTC

This fix is working for me but something after quota is failing. 

I added a disk to your vm.  I turned quota on.
When I run the Requested method is running correctly but the process gets a 405 error after quota is finished.

This is what I am running from Postman:


Success In automation.log

[----] I, [2019-01-03T12:12:52.584526 #15110:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_request_8]) <AEMethod [/ManageIQ/System/CommonMethods/QuotaMethods/requested]> Starting 
[----] I, [2019-01-03T12:12:53.066826 #15110:7fc6870]  INFO -- : Q-task_id([r8_vm_reconfigure_request_8]) <AEMethod requested> Request: VM Reconfigure for: pvala_rhv41_vm_disk - Remove Disks: 1 id: 8 
[----] I, [2019-01-03T12:12:53.084838 #15110:7fc6870]  INFO -- : Q-task_id([r8_vm_reconfigure_request_8]) <AEMethod requested> Reconfigure Disk Removal: {:disk_name=>"9548a285-30fd-4f7e-a392-6612e339729d", :delete_backing=>false}
[----] I, [2019-01-03T12:12:53.107459 #15110:7fc6870]  INFO -- : Q-task_id([r8_vm_reconfigure_request_8]) <AEMethod requested> Reconfigure Disk Removal Disk: <9548a285-30fd-4f7e-a392-6612e339729d> Disk Size: <33 MB>
[----] I, [2019-01-03T12:12:53.108418 #15110:7fc6870]  INFO -- : Q-task_id([r8_vm_reconfigure_request_8]) <AEMethod requested> VM Reconfigure storage change: -34603008 Bytes
[----] I, [2019-01-03T12:12:53.141079 #15110:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_request_8]) <AEMethod [/ManageIQ/System/CommonMethods/QuotaMethods/requested]> Ending

Connection.rb failing in evm.log

[----] I, [2019-01-03T12:12:56.532085 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#execute) VM Reconfigure initiated
[----] I, [2019-01-03T12:12:56.557420 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#do_request) Config spec: ["numCoresPerSocket"](NilClass) = nil
[----] I, [2019-01-03T12:12:56.557525 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#do_request) Config spec: ["memoryMB"](NilClass) = nil
[----] I, [2019-01-03T12:12:56.557607 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#do_request) Config spec: ["numCPUs"](NilClass) = nil
[----] I, [2019-01-03T12:12:56.557865 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#do_request) Config spec: ["disksRemove"][0][:disk_name](String) = "9548a285-30fd-4f7e-a392-6612e339729d"
[----] I, [2019-01-03T12:12:56.557962 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#do_request) Config spec: ["disksRemove"][0][:delete_backing](FalseClass) = false
[----] I, [2019-01-03T12:12:56.558042 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#do_request) Config spec: ["disksAdd"](NilClass) = nil
[----] I, [2019-01-03T12:12:56.562549 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(ManageIQ::Providers::Redhat::InfraManager::Vm#run_command_via_parent) Invoking [vm_reconfigure] through EMS: [rhv41]
[----] I, [2019-01-03T12:12:56.562789 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(ManageIQ::Providers::Redhat::InfraManager::OvirtServices::Strategies::V4#vm_reconfigure) EMS: [rhv41] ManageIQ::Providers::Redhat::InfraManager::Vm: id [86], name [pvala_rhv41_vm_disk], ems_ref [/api/vms/6c5dd5ab-cb30-4583-9cca-ffb9f0f9fd43] Started...
[----] I, [2019-01-03T12:12:56.563080 #15118:49cf50]  INFO -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(ManageIQ::Providers::Redhat::InfraManager#with_provider_connection) Connecting through ManageIQ::Providers::Redhat::InfraManager: [rhv41]
[----] E, [2019-01-03T12:12:56.686765 #15118:49cf50] ERROR -- : Q-task_id([r8_vm_reconfigure_task_7]) MIQ(VmReconfigureTask#execute) [Error: HTTP response code is 405. The response body is empty.] encountered during VM Reconfigure
[----] E, [2019-01-03T12:12:56.686908 #15118:49cf50] ERROR -- : Q-task_id([r8_vm_reconfigure_task_7]) [OvirtSDK4::Error]: HTTP response code is 405. The response body is empty.  Method:[block (2 levels) in <class:LogProxy>]
[----] E, [2019-01-03T12:12:56.687096 #15118:49cf50] ERROR -- : Q-task_id([r8_vm_reconfigure_task_7]) /usr/share/gems/gems/ovirt-engine-sdk-4.2.4/lib/ovirtsdk4/connection.rb:385:in `raise_error'
/usr/share/gems/gems/ovirt-engine-sdk-4.2.4/lib/ovirtsdk4/service.rb:410:in `internal_read_body'
/usr/share/gems/gems/ovirt-engine-sdk-4.2.4/lib/ovirtsdk4/service.rb:99:in `check_fault'
/usr/share/gems/gems/ovirt-engine-sdk-4.2.4/lib/ovirtsdk4/service.rb:340:in `block in internal_remove'
/usr/share/gems/gems/ovirt-engine-sdk-4.2.4/lib/ovirtsdk4/service.rb:46:in `wait'
/usr/share/gems/gems/ovirt-engine-sdk-4.2.4/lib/ovirtsdk4/service.rb:342:in `internal_remove'
/usr/share/gems/gems/ovirt-engine-sdk-4.2.4/lib/ovirtsdk4/services.rb:6358:in `remove'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-ovirt-333da0e3931c/app/models/manageiq/providers/redhat/infra_manager/ovirt_services/strategies/v4.rb:824:in `block in remove_vm_disks'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-ovirt-333da0e3931c/app/models/manageiq/providers/redhat/infra_manager/ovirt_services/strategies/v4.rb:822:in `each'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-ovirt-333da0e3931c/app/models/manageiq/providers/redhat/infra_manager/ovirt_services/strategies/v4.rb:822:in `remove_vm_disks'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-ovirt-333da0e3931c/app/models/manageiq/providers/redhat/infra_manager/ovirt_services/strategies/v4.rb:299:in `block in vm_reconfigure'
/var/www/miq/vmdb/app/models/mixins/provider_object_mixin.rb:15:in `block in with_provider_object'
/var/www/miq/vmdb/app/models/ext_management_system.rb:413:in `with_provider_connection'
/var/www/miq/vmdb/app/models/mixins/provider_object_mixin.rb:12:in `with_provider_object'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-ovirt-333da0e3931c/app/models/manageiq/providers/redhat/infra_manager/ovirt_services/strategies/v4.rb:271:in `vm_reconfigure'
/opt/rh/cfme-gemset/bundler/gems/cfme-providers-ovirt-333da0e3931c/app/models/manageiq/providers/redhat/infra_manager.rb:155:in `vm_reconfigure'
/var/www/miq/vmdb/app/models/vm_or_template.rb:351:in `run_command_via_parent'
/var/www/miq/vmdb/app/models/vm_or_template/operations/configuration.rb:124:in `spec_reconfigure'
/var/www/miq/vmdb/app/models/vm_reconfigure_task.rb:50:in `do_request'
/var/www/miq/vmdb/app/models/miq_request_task.rb:201:in `execute'
/var/www/miq/vmdb/app/models/miq_queue.rb:455:in `block in dispatch_method'
/usr/share/ruby/timeout.rb:93:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:108:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:453:in `dispatch_method'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `block in deliver'
/var/www/miq/vmdb/app/models/user.rb:275:in `with_user_group'
/var/www/miq/vmdb/app/models/miq_queue.rb:430:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:104:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:137:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:155:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:149:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:329:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:326:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:127:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:402:in `block in start_runner_via_fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.4/lib/nakayoshi_fork.rb:23:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:400:in `start_runner_via_fork'

Comment 18 William Fitzgerald 2019-01-03 19:10:32 UTC
Can you test this again ?

Comment 19 Parthvi Vala 2019-01-04 07:14:37 UTC
Hey William,

I tested again, but received same errors in the log as you and same as I had received earlier. Quota enabled or not, I am still encountering the error. And removing disks from UI seems to be working well, but not with API. So it probably shouldn't have much to do with Quota. Or did I miss something while testing again? Let me know.


Comment 20 William Fitzgerald 2019-01-04 20:29:00 UTC

The requested method is working now. The errors you are getting is a different issue.


Comment 21 Parthvi Vala 2019-01-16 06:43:46 UTC
Hey Billy,

I have filed a BZ for the error that I'm getting, looks like this BZ will be blocked until the new BZ(1666593) gets fixed.


Comment 24 Sudhir Mallamprabhakara 2019-01-24 13:40:06 UTC
@Dennis - This is a 5.10 Blocker BZ and is not ready for QE.

Comment 25 Sudhir Mallamprabhakara 2019-01-24 13:42:12 UTC
moving the status back to Dev

Comment 26 Sudhir Mallamprabhakara 2019-01-24 13:43:21 UTC
@Loic - https://bugzilla.redhat.com/show_bug.cgi?id=1620161#c24

Comment 27 Parthvi Vala 2019-01-24 14:08:24 UTC
Marking this BZ as VERIFIED since the requested method is working fine, and the BZ (1666593) is not related to the fix.

Verified on

Comment 28 errata-xmlrpc 2019-02-07 23:03:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.