Bug 1379126

Summary: [RHEV] VM retire do not actually delete the VM on RHEV side
Product: Red Hat CloudForms Management Engine Reporter: Ilanit Stein <istein>
Component: ProvidersAssignee: Piotr Kliczewski <pkliczew>
Status: CLOSED NOTABUG QA Contact: Ilanit Stein <istein>
Severity: high Docs Contact:
Priority: high    
Version: 5.6.0CC: cpelland, istein, jfrey, jhardy, mgoldboi, obarenbo, oourfali
Target Milestone: GAKeywords: Regression
Target Release: 5.6.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: rhev
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-13 06:49:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
evm.log
none
rhevm.log
none
automation.log
none
policy.log
none
evm.log none

Description Ilanit Stein 2016-09-25 08:37:25 UTC
Description of problem:
VM retire operation from CFMe side seem successful:
On task windows display: "Task completed successfully"
VM is marked as "R",
but on RHEV side,
VM still exist.

Version-Release number of selected component (if applicable):
CFME-5.6.2.1.20160922130607_92d5b5e
RHV-4.0.4.2-0.1.

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Ilanit Stein 2016-09-25 08:38:05 UTC
Created attachment 1204512 [details]
evm.log

Comment 3 Ilanit Stein 2016-09-25 08:38:31 UTC
Created attachment 1204513 [details]
rhevm.log

Comment 4 Ilanit Stein 2016-09-25 08:39:01 UTC
Created attachment 1204514 [details]
automation.log

Comment 5 Piotr Kliczewski 2016-10-05 14:44:26 UTC
I do not see any issues in the logs.
Have you checked retirement with RHV 3.6?

Please provide policy.log.

Comment 6 Ilanit Stein 2016-10-10 09:07:23 UTC
I narrowed the problem.
VMs that were provisioned on RHV, from the CFME side,
were stopped & removed successfully, also on RHV side.

VMs that were provisioned on RHV, from the RHV side,
were stopped, but not removed on RHV side. These appear as "R" in CFME.

The above problem exist both on RHV-3.6 & RHV-4.0
(tested on RHV-3.6.8-0.1.el6 & RHV-4.0.4.2-0.1.el7ev)

Comment 7 Ilanit Stein 2016-10-10 09:12:56 UTC
Created attachment 1208773 [details]
policy.log

In this log, these cases can be seen:

1. VM that was created on RHV-3.6 side, and wasn't removed on RHV-side:

[----] I, [2016-10-10T04:33:32.654580 #13531:4bd994]  INFO -- : MIQ(policy-enforce_policy): Event: [vm_poweroff], To: [istein]

2. VM that was provisioned on RHV-4.0, from CFME side, and was stopped & removed successfully:
[----] I, [2016-10-10T03:48:36.499352 #13549:4bd994]  INFO -- : MIQ(policy-enforce_policy): Event: [vm_poweroff], To: [istein_5]

3. VM that was created on RHV-4.0 side, and wasn't removed on RHV-side:
[----] I, [2016-10-10T03:40:06.232987 #13549:4bd994]  INFO -- : MIQ(policy-enforce_policy): Event: [vm_poweroff], To: [istein_4]

Comment 8 Ilanit Stein 2016-10-10 09:19:03 UTC
* Audit log for VM that was provisioned and started on RHV-4.0, from CFME side, and was stopped & removed successfully (notice stop VM is done twice):

Oct 10, 2016 10:49:32 AM
VM istein_5 was successfully removed.

Oct 10, 2016 10:49:31 AM
VM istein_5 is already Down, Stop VM was skipped. User: admin@internal-authz.

Oct 10, 2016 10:48:12 AM
VM istein_5 is down. Exit message: Admin shut down from the engine

Oct 10, 2016 10:48:10 AM
VM istein_5 powered off by admin@internal-authz (Host: lilach-vdsa.tlv.redhat.com).

Oct 10, 2016 10:46:58 AM
VM istein_5 started on Host lilach-vdsa.tlv.redhat.com

Oct 10, 2016 10:46:21 AM
VM istein_5 was started by admin@internal-authz (Host: lilach-vdsa.tlv.redhat.com).

Oct 10, 2016 10:46:16 AM
Interface nic1 (VirtIO) was added to VM istein_5. (User: admin@internal-authz)

Oct 10, 2016 10:46:16 AM
Network Interface nic1 (VirtIO) was plugged to VM istein_5. (User: admin@internal-authz)

Oct 10, 2016 10:46:15 AM
VM istein_5 configuration was updated by admin@internal-authz.

Oct 10, 2016 10:46:15 AM
VM istein_5 configuration was updated by admin@internal-authz.

Oct 10, 2016 10:46:15 AM
VM istein_5 configuration was updated by admin@internal-authz.

Oct 10, 2016 10:46:15 AM
VM istein_5 configuration was updated by admin@internal-authz.

Oct 10, 2016 10:46:00 AM
VM istein_5 creation has been completed.

Oct 10, 2016 10:45:19 AM
VM istein_5 creation was initiated by admin@internal-authz.

* Audit log for VM that was created and started on RHV-4.0 side, and wasn't removed on RHV-side:

Oct 10, 2016 10:39:36 AM
VM istein_4 powered off by admin@internal-authz (Host: lilach-vdsa.tlv.redhat.com).

Oct 10, 2016 10:38:12 AM
VM istein_4 started on Host lilach-vdsa.tlv.redhat.com

Oct 10, 2016 10:37:55 AM
VM istein_4 was started by admin@internal-authz (Host: lilach-vdsa.tlv.redhat.com).

Oct 10, 2016 10:37:06 AM
VM istein_4 creation has been completed.

Oct 10, 2016 10:36:23 AM
VM istein_4 creation was initiated by admin@internal-authz.

Comment 9 Piotr Kliczewski 2016-10-10 09:20:30 UTC
Please provide evm log for both use cases when a vm is created in RHV and CFME and later removed.

Comment 10 Ilanit Stein 2016-10-11 06:55:16 UTC
Created attachment 1209024 [details]
evm.log

Comment 11 Ilanit Stein 2016-10-11 06:58:01 UTC
Comment on attachment 1209024 [details]
evm.log

- VM that was provisioned on RHV-4.0, from CFME side, and was stopped & removed successfully: [istein_5]

- VM that was created on RHV-4.0 side, and wasn't removed on RHV-side: [istein_4]

Comment 12 Ilanit Stein 2016-10-13 06:49:08 UTC
Greg McCullough:
"If the VM delete is being performed during retirement then the reported results would make sense.  Automate logic prevents VMs we did not provision from being deleted unless they have been tagged with "lifecycle/retire_full"."
Brad Ascar:

"That is the expected behavior. If you want to be able to delete a VM not provisioned by CF you need to add the tag as Greg described. "

For a VM created on RHV-4.0.4 side, & tagged with "LifeCycle: Fully retire VM and remove from Provider",
Indeed the VM was stopped & removed.

Based on the above closing this bug.