Bug 1753265 - Starting a VM after it has been stopped yields 'Forbidden Error'
Summary: Starting a VM after it has been stopped yields 'Forbidden Error'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Console Kubevirt Plugin
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.3.0
Assignee: Ido Rosenzwig
QA Contact: Guohua Ouyang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-18 13:49 UTC by Radim Hrazdil
Modified: 2020-01-23 11:06 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: when a VM was started after it has been stopped the old pod, which has been deleted, was used Consequence: showed an error: 'Forrbidden Error' Fix: use the newest pod related to the VM. Result: No Error when a VM was started after it has been stopped.
Clone Of:
Environment:
Last Closed: 2020-01-23 11:06:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Screenshot (13.03 KB, image/png)
2019-09-18 13:49 UTC, Radim Hrazdil
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift console pull 3115 0 'None' closed Bug 1753265: Add 'Stopping' status for Virtual Machines 2020-10-15 11:43:01 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:06:48 UTC

Description Radim Hrazdil 2019-09-18 13:49:52 UTC
Created attachment 1616230 [details]
Screenshot

Description of problem:
When a running VM is stopped (which can be done either by clicking Stop VM action button or by cloning running VM), an attempt to Start the VM againt leads to Forbidden error.

It might mean that the virt laucher pod is still being terminated at that time. If that is the case, we might want to consider adding a state 'Shutting down' or similar and do not allow user to perform start, stop and migrate actions.

Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-09-17-232025
HCO 2.1.0-47

How reproducible:
100%

Steps to Reproduce:
1. Stop running VM
2. Start the VM right after stopping it
3.

Actual results:
Forbidden

Expected results:


Additional info:

Comment 1 Tomas Jelinek 2019-09-18 14:02:47 UTC
@fabian, is there a good way we can detect that the VM is in a state that it is shutting down so we can not start it again yet? Currently we use only the "running" from vm spec to decide if the "start" should be enabled or not.

Comment 2 Fabian Deutsch 2019-09-19 07:27:19 UTC
Arik, thoughts?

Comment 3 Arik 2019-09-19 13:47:23 UTC
So the API server patches the VM, switching the "running" field to 'false' and therefore the front-end enables to start the VM.
But once the 'start' request arrives to the API server, the latter detects the VMI is still there (https://github.com/kubevirt/kubevirt/blob/master/pkg/virt-api/rest/subresource.go#L443).
IIUC, the problem is that the front-end looks at the desired state but the actual state is different at that period of time.
One way to solve this would be checking the VMI in the front-end as the back-end does, but that would likely mean adding a query, right?
But I think a better way is to look at the "ready" field in the status section of the VM and enable the 'start' button only when "running"==false and "ready"==false

Comment 4 Arik 2019-09-19 13:50:34 UTC
(In reply to Arik from comment #3)
> So the API server patches the VM, switching the "running" field to 'false'
> and therefore the front-end enables to start the VM.
> But once the 'start' request arrives to the API server, the latter detects
> the VMI is still there
> (https://github.com/kubevirt/kubevirt/blob/master/pkg/virt-api/rest/
> subresource.go#L443).
> IIUC, the problem is that the front-end looks at the desired state but the
> actual state is different at that period of time.

at that point in time*

> One way to solve this would be checking the VMI in the front-end as the
> back-end does, but that would likely mean adding a query, right?
> But I think a better way is to look at the "ready" field in the status

So I think*

> section of the VM and enable the 'start' button only when "running"==false
> and "ready"==false

Or when the status section is empty

Comment 5 Arik 2019-09-19 14:10:22 UTC
Actually, I may be wrong here - the pod may still be running with a different state then 'Ready', thus vm.status.ready field would be false and yet the back-end will reject 'start' requests.

So I guess there's a trade-off between the two approaches.
Tomas, do you query the VMI in that context?

Comment 6 Tomas Jelinek 2019-09-19 14:27:55 UTC
> So I guess there's a trade-off between the two approaches.
> Tomas, do you query the VMI in that context?

nope, we just check the "spec.running".

Comment 7 Arik 2019-09-19 15:07:16 UTC
ok, so as a temporary solution we can add a query for the VMI but I think it would be better to add a state to the VM similar to Pod's 'Terminating' state

Comment 8 Tomas Jelinek 2019-09-23 10:35:33 UTC
(In reply to Arik from comment #7)
> ok, so as a temporary solution we can add a query for the VMI 

I would like to avoid temporary solutions like this

> but I think it
> would be better to add a state to the VM similar to Pod's 'Terminating' state

this would be ideal. Moving to virt to decide if / when can this be implemented.

Comment 9 Fabian Deutsch 2019-09-23 14:03:36 UTC
Arik,

what could we do on the long rnu and what would be the owrkaround you mentioned?

Comment 11 Arik 2019-09-24 12:45:13 UTC
(In reply to Fabian Deutsch from comment #9)
> Arik,
> 
> what could we do on the long rnu and what would be the owrkaround you
> mentioned?

I think we could add an indication for the VM being terminated in its 'status' section.
I proposed, as a workaround, to have the front-end querying the VMI and applying the same logic that makes the back-end to reject the request.

Comment 12 sgott 2019-10-23 12:30:10 UTC
Per Comment #6, the UI is checking for spec.Running? This is incorrect. Since we've introduced RunStrategy, "Runnning" is only one way to mandate that a VM should be running. Running/RunStrategy are not indicators of current state. The correct approach is to check. status.Phase. virt-api considers the VM to be running is in a known non-final state (anything besides Failed, Succeeded, Unknown, vmPhaseUnset)

Comment 14 Guohua Ouyang 2019-11-11 10:16:16 UTC
verified on 4.3.0-0.nightly-2019-11-02-09233.

Comment 16 errata-xmlrpc 2020-01-23 11:06:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062


Note You need to log in before you can comment on or make changes to this bug.