Bug 1175313 - "Authentication Required" login screen that references RESTAPI
Summary: "Authentication Required" login screen that references RESTAPI
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.0
Hardware: All
OS: All
high
medium
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Alexander Wels
QA Contact: Jiri Belka
URL:
Whiteboard:
Depends On:
Blocks: 1206908
TreeView+ depends on / blocked
 
Reported: 2014-12-17 13:38 UTC by Bill Sanford
Modified: 2019-10-10 09:32 UTC (History)
16 users (show)

Fixed In Version: org.ovirt.engine-root-3.5.0-26
Doc Type: Bug Fix
Doc Text:
Previously, the REST API interface was not aware of a forced logout after 10 hours of being logged in, and would try to access the REST API after it. This caused an authentication popup to appear in the Administration Portal. Now, the REST API interface is aware of the forced logout and stops accessing the REST API after the logout happens, so the popup no longer appears.
Clone Of:
: 1206908 (view as bug list)
Environment:
Last Closed: 2016-03-09 20:53:04 UTC
oVirt Team: UX
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Screenshot of the login dialog box (175.17 KB, image/png)
2014-12-17 13:38 UTC, Bill Sanford
no flags Details
Screenshot of the login box instead of webadmin portal (936.70 KB, image/png)
2015-01-14 14:02 UTC, Bill Sanford
no flags Details
Screenshot of the Run once window (138.47 KB, image/png)
2015-02-17 15:01 UTC, Bill Sanford
no flags Details
cropped engine.log (15.87 MB, text/plain)
2015-03-04 00:28 UTC, Einav Cohen
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1168842 0 unspecified CLOSED user session timeout is not working 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1172726 0 urgent CLOSED Webadmin access not cleared on session expiration 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1192219 0 high CLOSED idle webadmin does not logout automatically if we are in the VMs main tab and there is a selected VM 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHEA-2016:0376 0 normal SHIPPED_LIVE Red Hat Enterprise Virtualization Manager 3.6.0 2016-03-10 01:20:52 UTC
oVirt gerrit 37837 0 None None None Never
oVirt gerrit 38453 0 master NEW webadmin: REST API login popup Never

Internal Links: 1168842 1172726 1192219

Description Bill Sanford 2014-12-17 13:38:23 UTC
Created attachment 970108 [details]
Screenshot of the login dialog box

Description of problem:
When 7.1 hosts are crashed and non-responsive in RHEV-M overnight, there is an "Authentication Required" login screen that appears, that says:

A username and password are being requested by https://rhevm35.spice.ml2.eng.bos.redhat.com. 

The site says: "RESTAPI"

This happens as https://bugzilla.redhat.com/show_bug.cgi?id=1175290 is happening.

Version-Release number of selected component (if applicable):
RHEV-M 3.5 (vt13.1/el6) on RHEL-6.6-20140926.0
RHEL-7.1-20141204.2 - Hosts

How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Einav Cohen 2014-12-19 14:43:23 UTC
this is not related to the hosts crash - this is some web-admin session vs rest-api-session out-of-sync issue that is probably happening once the engine session is expired (e.g. after leaving the web-admin idle for a long time as you did). 

sounds related to bug 1172726 and bug 1168842 (both 'infra' issues, but currently flagging as 'ux' based on the workflow) - please attempt to re-test in vt13.4. thanks. 

@Alexander - any additional thoughts/insights?

Comment 2 Alexander Wels 2014-12-19 18:05:38 UTC
Based on what is reported this is most likely related to bz1172726. The webadmin session is not expired but the engine session is. So the webadmin or a UI plugin makes a REST api call which has expired causing the login message to show up.

Under correct circumstances the webadmin http session would have expired as well, and the webadmin login screen would have been shown, but due to the above mentioned bug it didn't expire. So I think there is a high probability that the solution to the above bug will also fix this one.

Comment 3 Bill Sanford 2014-12-23 13:50:30 UTC
This still happens in 13.4.

My ISCSI storage went down, and I got this login screen the next am.

Comment 4 Einav Cohen 2015-01-06 19:28:24 UTC
(In reply to Bill Sanford from comment #3)
> This still happens in 13.4.
> 
> My ISCSI storage went down, and I got this login screen the next am.

right now (i.e. 13.4 and lager), you shouldn't be getting any pop-up as you do, unless your *engine server* was restarted/disconnected/etc. and/or your client was temporarily disconnected from the engine, for some reason, etc.  - are you sure that none of these have happened to you?

Comment 5 Bill Sanford 2015-01-06 19:50:31 UTC
I am using ISCSI storage that is sometimes under test and I lose the sendtargets at the host machines since the storage is not available. This is when I see the error.

Comment 6 Vojtech Szocs 2015-01-07 11:11:57 UTC
Attached screenshot shows "active" WebAdmin GUI, which would imply WebAdmin physical session (HttpSession) + Engine logical session both being alive; the "RESTAPI" auth dialog is probably due to REST physical session (HttpSession) dead.

Important question: is patch http://gerrit.ovirt.org/#/c/35248/ effective in tested RHEVM build? You can tell this by looking at user login events; if you see "user has logged in" *twice* then this patch is not effective.

If above mentioned patch is not effective, the root cause is timeout of separate Engine user session associated with REST physical session.

Comment 7 Einav Cohen 2015-01-07 14:07:43 UTC
@Vojtech: patch http://gerrit.ovirt.org/#/c/35248/ is included in vt13.4 which Bill tested as he mentioned in comment #3. 

The behavior that Bill is describing cannot happen, unless the engine dies / disconnects / ... - if the engine is up and running all the time, the heartbeat mechanism of the ui-plugins infrastructure is keeping both rest-api-http-session and engine-session alive indefinitely, even if the GUI is idle, which means that the browser-pop-up that requests credentials for the rest-api can/should never appear. 

(In reply to Bill Sanford from comment #5)
> I am using ISCSI storage that is sometimes under test and I lose the
> sendtargets at the host machines since the storage is not available. This is
> when I see the error.

Bill, the storage status is irrelevant to the credential-request-pop-up that you are seeing in your browser, unless you are using a hosted-engine environment in which your engine resides on that problematic iSCSI storage of yours. is this the case?

Comment 8 Vojtech Szocs 2015-01-07 14:21:33 UTC
Just a note - when Engine dies (goes down) and is started again, any previous server-side sessions are not valid anymore, so WebAdmin HTML has to be reloaded.

Comment 9 Bill Sanford 2015-01-07 15:36:53 UTC
Einav, I am using a VM on a stable machine and not the ISCSI storage that can be problematic.

Comment 10 Einav Cohen 2015-01-07 17:42:06 UTC
Bill's update: env was re-installed, issue is not reproduced. 

closing. 

[please reopen if reproduced]

Comment 11 Bill Sanford 2015-01-14 14:02:07 UTC
Created attachment 980024 [details]
Screenshot of the login box instead of webadmin portal

Comment 12 Bill Sanford 2015-01-14 21:57:02 UTC
I reopened since I have seen this the past two mornings.

Comment 13 Bill Sanford 2015-01-20 16:29:54 UTC
I am closing this since vt13.8/el6 does not show this behavior.

Comment 14 Bill Sanford 2015-01-22 19:53:29 UTC
When I had vt13.8/el6 installed and two Data Centers attached to one ISO, I didn't see this problem. When I detached and removed one Data Center, leaving one Data Center and one ISO "Storage Type," the error started appearing again, from overnight.

The problem I saw, it seems like the screen timeout to the Admin Portal is blocked from appearing with the "Authentication Required" login screen appearing first. With this RESTAPI login dialog box, I click either "Cancel" or "OK," twice, to clear the screen. When the RESTAPI dialog box is cleared, *then* the Admin Portal beginning screen will appear, like it does when the user times out.

Comment 15 Bill Sanford 2015-01-22 19:55:17 UTC
[root@rhevm35 ~]# rpm -qa | grep engine
rhevm-setup-plugin-ovirt-engine-common-3.5.0-0.30.el6ev.noarch
gtk2-engines-2.18.4-5.el6.x86_64
rhevm-setup-plugin-ovirt-engine-3.5.0-0.30.el6ev.noarch
[root@rhevm35 ~]#

Comment 16 Einav Cohen 2015-02-12 13:24:06 UTC
vt13.8 contains [1], [2] - reaching a situation in which the RESTAPI-credential-request pop-up appears should be impossible. user should properly be logged out of the web-admin once engine session timeout reaches (while the GUI is idle). 

the only possible scenario that comes to mind, which is most likely unrelated to the ui-plugins infrastructure, is that a java-script exception is thrown in the particular page that the GUI was left in (VMs main tab in the example), messing up the functionality of the page; 
i.e.:
- a java-script exception is thrown ->
- the main GUI code "freezes" -> 
- no requests from the main GUI to the engine take place at all ->
- engine-session dies, but GUI is remained seemingly "logged-in" (again, since main GUI code froze). 

in this situation: if, at some point, one of the ui-plugins (that its code is still executing somehow, since it is in a separate 'frame'/context) is attempting to perform a request to an already-dead engine-session/REST-API session, it will result in the RESTAPI-credential-request pop-up. 

@Alexander, can you please contact Bill and investigate on his setup assuming this is still reproducible? the only thing that we can try to do is to try and find that java-script exception (which is, again, most likely not related to the ui-plugins infrastructure IMO) and fix it, so that the behavior introduced in patches [1], [2] will work properly. 

Thanks. 

[1] http://gerrit.ovirt.org/#/c/36622/
[2] http://gerrit.ovirt.org/#/c/36737/

Comment 17 Einav Cohen 2015-02-12 13:25:56 UTC
BTW, patch [1] will introduce a global UncaughtExceptionHandler which should mitigate the unexpected behavior in case of a java-script exception. 

[1] http://gerrit.ovirt.org/#/c/25444/

Comment 19 Einav Cohen 2015-02-12 22:04:53 UTC
somewhat related: bug 1192219. 

bug 1192219 may explain why the web-admin hasn't automatically logged out (the periodical "refersh=true" query + the ui-plugins heartbeat mechanism are keeping the engine session alive forever - more details in bug 1192219, comment #0).

however, bug 1192219 still does not explain why the REST-API-credentials-request pop-up appeared: the periodical "refersh=true" query + the ui-plugins heartbeat mechanism are keeping both engine-session + REST API session alive -> there is no reason for REST-API-credentials to be requested -> there is no reason for the pop-up to appear (again: the only explanation that I can think of is some sort of a java-script exception that messes up the entire page functionality - as I mentioned in Comment #16).

Comment 28 Bill Sanford 2015-02-17 15:00:31 UTC
Alexander, all I do with the run once, is I move the CD to the top of the boot sequence, select to Attach CD, Pick the ISO I want and click ok. Will Attach screenshot.

Comment 29 Bill Sanford 2015-02-17 15:01:06 UTC
Created attachment 992732 [details]
Screenshot of the Run once window

Comment 35 Bill Sanford 2015-02-18 16:35:47 UTC
Alexander has already reproduced this.

Comment 36 Alexander Wels 2015-02-18 17:46:14 UTC
I have reproduced this by sitting on a screen that keeps refreshing the REST api due to a related bug. I am going to move his host into maintenance so it doesn't spam the log so much. Now that I can reproduce I will get some tools running so I can hopefully capture enough information to figure out the problem.

Comment 37 Eyal Edri 2015-02-25 08:42:12 UTC
3.5.1 is already full with bugs (over 80), and since none of these bugs were added as urgent for 3.5.1 release in the tracker bug, moving to 3.5.2

Comment 38 Einav Cohen 2015-03-04 00:28:21 UTC
Created attachment 997713 [details]
cropped engine.log

Comment 39 Einav Cohen 2015-03-04 01:01:48 UTC
I have managed to reproduce the REST API credentials request pop up on ~latest 'master'.

STEPS TO REPRODUCE:

0. Change engine session time-out to 3 minutes by invoking the following:
# engine-config -s UserSessionTimeOutInterval=3
and restart engine. 

1. Login to the web-admin

2. Navigate to the Storage main tab, select a Data storage domain and navigate to the VMs sub-tab. 

3. Leave the GUI idle for a time period >> 3 minutes.

4. Navigate away from the Storage main tab to e.g. the VMs main tab and select a VM. 

5. Wait for less than a minute. 

REST API credentials pop-up should appear. 
Attached engine.log (cropped) - see attachment 997713 [details]. 
Start time-stamp: 2015-03-03 16:34:33,737 (Login)
~End time-stamp:  2015-03-03 17:30:22,768 (This is when the Search "VMs:" query was invoked, i.e. we navigated away from the Storage main tab).

ADDITIONAL IMPORTANT INFORMATION: 

The GUI was left idle for a long time (step 3 above) but no automatic logout occurred ->
This implies that there was a repeating query in the background invoked with 'refresh=true', which kept the engine session alive -> 
you would expect that the ui-plugins' "keep alive" mechanism would invoke an "/api" request every ~1 minute or so.
However: When monitoring the networking traffic on the client, it seems that no "/api" requests were invoked *at all* during the time period in which the GUI was left idle (~1 hour in my case). 
[see http://i.imgur.com/FKlnQAf.png , http://i.imgur.com/n6MOclc.png]

This can imply one of two things:

(1) Even though there was no automatic logout, there weren't any requests that were invoked with 'refresh=true', which probably means that the engine kept the session alive even though it wasn't supposed to -> engine bug. 

(2) There was no automatic logout since there *were* requests that were invoked with 'refresh=true', however, for some reason, the UI Plugins "keep alive" mechanism missed them, and consequently haven't issued any "/api" requests -> possible bug in the UI plugins "keep alive" mechanism, or maybe an engine bug (see [1] for possible reasoning). 

NEXT STEPS:

Need to find out whether there were queries that were invoked with 'refresh=true' in this case or not; 
I didn't see any in the engine.log (attachment 997713 [details]), however maybe they are simply not logged - need to profile the engine more closely in order to find out. 

[1] maybe a query that was invoked in the GUI with 'refresh=false' contains, as part of its internal execution phase in the backend, call(s) to other query(ies) with 'refresh=true', which keep the engine session alive without the ui-plugins "keep alive" mechanism even knowing about it. 
In this case, it is probably an engine bug (since there is no reason for something that was invoked with 'refresh=false' to invoke internally something with 'refresh=true').

Comment 40 Alexander Wels 2015-03-06 19:57:44 UTC
There were 2 potential causes for this issue:

1. The hard limit of the engine session. The REST api pinging code did not take this hard limit into account and could potentially cause the popup when it tried to ping after the hard limit was reached and the engine session was terminated.
2. The UI was sending requests with refresh:false, but internally the engine was creating a sub query and set the refresh:true in that sub query causing the engine session to survive endlessly, but the REST api session would terminate after a while, causing the pop to appear. This scenario was the one Einav encountered in her testing.

The associated patches solve both issues.

Comment 42 Colin Coe 2015-04-27 01:15:57 UTC
Heyall

Is this patch going to be applied to the RHEV 3.5 stream?  If so, any ideas on the target release?  (/me hoping for v3.5.1)

Thanks

CC

Comment 43 Yaniv Lavi 2015-04-27 13:21:15 UTC
(In reply to Colin Coe from comment #42)
> Heyall
> 
> Is this patch going to be applied to the RHEV 3.5 stream?  If so, any ideas
> on the target release?  (/me hoping for v3.5.1)

Going to be in RHEV 3.5.3 the following release to RHEV 3.5.1.

> 
> Thanks
> 
> CC

Comment 44 Jiri Belka 2015-06-03 12:46:23 UTC
ok, same verification steps as in https://bugzilla.redhat.com/show_bug.cgi?id=1206908#8

ovirt-engine-backend-3.6.0-0.0.master.20150519172219.git9a2e2b3.el6.noarch

Comment 46 errata-xmlrpc 2016-03-09 20:53:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html


Note You need to log in before you can comment on or make changes to this bug.