Created attachment 970108 [details]
Screenshot of the login dialog box
Description of problem:
When 7.1 hosts are crashed and non-responsive in RHEV-M overnight, there is an "Authentication Required" login screen that appears, that says:
A username and password are being requested by https://rhevm35.spice.ml2.eng.bos.redhat.com.
The site says: "RESTAPI"
This happens as https://bugzilla.redhat.com/show_bug.cgi?id=1175290 is happening.
Version-Release number of selected component (if applicable):
RHEV-M 3.5 (vt13.1/el6) on RHEL-6.6-20140926.0
RHEL-7.1-20141204.2 - Hosts
Steps to Reproduce:
this is not related to the hosts crash - this is some web-admin session vs rest-api-session out-of-sync issue that is probably happening once the engine session is expired (e.g. after leaving the web-admin idle for a long time as you did).
sounds related to bug 1172726 and bug 1168842 (both 'infra' issues, but currently flagging as 'ux' based on the workflow) - please attempt to re-test in vt13.4. thanks.
@Alexander - any additional thoughts/insights?
Based on what is reported this is most likely related to bz1172726. The webadmin session is not expired but the engine session is. So the webadmin or a UI plugin makes a REST api call which has expired causing the login message to show up.
Under correct circumstances the webadmin http session would have expired as well, and the webadmin login screen would have been shown, but due to the above mentioned bug it didn't expire. So I think there is a high probability that the solution to the above bug will also fix this one.
This still happens in 13.4.
My ISCSI storage went down, and I got this login screen the next am.
(In reply to Bill Sanford from comment #3)
> This still happens in 13.4.
> My ISCSI storage went down, and I got this login screen the next am.
right now (i.e. 13.4 and lager), you shouldn't be getting any pop-up as you do, unless your *engine server* was restarted/disconnected/etc. and/or your client was temporarily disconnected from the engine, for some reason, etc. - are you sure that none of these have happened to you?
I am using ISCSI storage that is sometimes under test and I lose the sendtargets at the host machines since the storage is not available. This is when I see the error.
Attached screenshot shows "active" WebAdmin GUI, which would imply WebAdmin physical session (HttpSession) + Engine logical session both being alive; the "RESTAPI" auth dialog is probably due to REST physical session (HttpSession) dead.
Important question: is patch http://gerrit.ovirt.org/#/c/35248/ effective in tested RHEVM build? You can tell this by looking at user login events; if you see "user has logged in" *twice* then this patch is not effective.
If above mentioned patch is not effective, the root cause is timeout of separate Engine user session associated with REST physical session.
@Vojtech: patch http://gerrit.ovirt.org/#/c/35248/ is included in vt13.4 which Bill tested as he mentioned in comment #3.
The behavior that Bill is describing cannot happen, unless the engine dies / disconnects / ... - if the engine is up and running all the time, the heartbeat mechanism of the ui-plugins infrastructure is keeping both rest-api-http-session and engine-session alive indefinitely, even if the GUI is idle, which means that the browser-pop-up that requests credentials for the rest-api can/should never appear.
(In reply to Bill Sanford from comment #5)
> I am using ISCSI storage that is sometimes under test and I lose the
> sendtargets at the host machines since the storage is not available. This is
> when I see the error.
Bill, the storage status is irrelevant to the credential-request-pop-up that you are seeing in your browser, unless you are using a hosted-engine environment in which your engine resides on that problematic iSCSI storage of yours. is this the case?
Just a note - when Engine dies (goes down) and is started again, any previous server-side sessions are not valid anymore, so WebAdmin HTML has to be reloaded.
Einav, I am using a VM on a stable machine and not the ISCSI storage that can be problematic.
Bill's update: env was re-installed, issue is not reproduced.
[please reopen if reproduced]
Created attachment 980024 [details]
Screenshot of the login box instead of webadmin portal
I reopened since I have seen this the past two mornings.
I am closing this since vt13.8/el6 does not show this behavior.
When I had vt13.8/el6 installed and two Data Centers attached to one ISO, I didn't see this problem. When I detached and removed one Data Center, leaving one Data Center and one ISO "Storage Type," the error started appearing again, from overnight.
The problem I saw, it seems like the screen timeout to the Admin Portal is blocked from appearing with the "Authentication Required" login screen appearing first. With this RESTAPI login dialog box, I click either "Cancel" or "OK," twice, to clear the screen. When the RESTAPI dialog box is cleared, *then* the Admin Portal beginning screen will appear, like it does when the user times out.
[root@rhevm35 ~]# rpm -qa | grep engine
vt13.8 contains ,  - reaching a situation in which the RESTAPI-credential-request pop-up appears should be impossible. user should properly be logged out of the web-admin once engine session timeout reaches (while the GUI is idle).
the only possible scenario that comes to mind, which is most likely unrelated to the ui-plugins infrastructure, is that a java-script exception is thrown in the particular page that the GUI was left in (VMs main tab in the example), messing up the functionality of the page;
- a java-script exception is thrown ->
- the main GUI code "freezes" ->
- no requests from the main GUI to the engine take place at all ->
- engine-session dies, but GUI is remained seemingly "logged-in" (again, since main GUI code froze).
in this situation: if, at some point, one of the ui-plugins (that its code is still executing somehow, since it is in a separate 'frame'/context) is attempting to perform a request to an already-dead engine-session/REST-API session, it will result in the RESTAPI-credential-request pop-up.
@Alexander, can you please contact Bill and investigate on his setup assuming this is still reproducible? the only thing that we can try to do is to try and find that java-script exception (which is, again, most likely not related to the ui-plugins infrastructure IMO) and fix it, so that the behavior introduced in patches ,  will work properly.
BTW, patch  will introduce a global UncaughtExceptionHandler which should mitigate the unexpected behavior in case of a java-script exception.
somewhat related: bug 1192219.
bug 1192219 may explain why the web-admin hasn't automatically logged out (the periodical "refersh=true" query + the ui-plugins heartbeat mechanism are keeping the engine session alive forever - more details in bug 1192219, comment #0).
however, bug 1192219 still does not explain why the REST-API-credentials-request pop-up appeared: the periodical "refersh=true" query + the ui-plugins heartbeat mechanism are keeping both engine-session + REST API session alive -> there is no reason for REST-API-credentials to be requested -> there is no reason for the pop-up to appear (again: the only explanation that I can think of is some sort of a java-script exception that messes up the entire page functionality - as I mentioned in Comment #16).
Alexander, all I do with the run once, is I move the CD to the top of the boot sequence, select to Attach CD, Pick the ISO I want and click ok. Will Attach screenshot.
Created attachment 992732 [details]
Screenshot of the Run once window
Alexander has already reproduced this.
I have reproduced this by sitting on a screen that keeps refreshing the REST api due to a related bug. I am going to move his host into maintenance so it doesn't spam the log so much. Now that I can reproduce I will get some tools running so I can hopefully capture enough information to figure out the problem.
3.5.1 is already full with bugs (over 80), and since none of these bugs were added as urgent for 3.5.1 release in the tracker bug, moving to 3.5.2
Created attachment 997713 [details]
I have managed to reproduce the REST API credentials request pop up on ~latest 'master'.
STEPS TO REPRODUCE:
0. Change engine session time-out to 3 minutes by invoking the following:
# engine-config -s UserSessionTimeOutInterval=3
and restart engine.
1. Login to the web-admin
2. Navigate to the Storage main tab, select a Data storage domain and navigate to the VMs sub-tab.
3. Leave the GUI idle for a time period >> 3 minutes.
4. Navigate away from the Storage main tab to e.g. the VMs main tab and select a VM.
5. Wait for less than a minute.
REST API credentials pop-up should appear.
Attached engine.log (cropped) - see attachment 997713 [details].
Start time-stamp: 2015-03-03 16:34:33,737 (Login)
~End time-stamp: 2015-03-03 17:30:22,768 (This is when the Search "VMs:" query was invoked, i.e. we navigated away from the Storage main tab).
ADDITIONAL IMPORTANT INFORMATION:
The GUI was left idle for a long time (step 3 above) but no automatic logout occurred ->
This implies that there was a repeating query in the background invoked with 'refresh=true', which kept the engine session alive ->
you would expect that the ui-plugins' "keep alive" mechanism would invoke an "/api" request every ~1 minute or so.
However: When monitoring the networking traffic on the client, it seems that no "/api" requests were invoked *at all* during the time period in which the GUI was left idle (~1 hour in my case).
[see http://i.imgur.com/FKlnQAf.png , http://i.imgur.com/n6MOclc.png]
This can imply one of two things:
(1) Even though there was no automatic logout, there weren't any requests that were invoked with 'refresh=true', which probably means that the engine kept the session alive even though it wasn't supposed to -> engine bug.
(2) There was no automatic logout since there *were* requests that were invoked with 'refresh=true', however, for some reason, the UI Plugins "keep alive" mechanism missed them, and consequently haven't issued any "/api" requests -> possible bug in the UI plugins "keep alive" mechanism, or maybe an engine bug (see  for possible reasoning).
Need to find out whether there were queries that were invoked with 'refresh=true' in this case or not;
I didn't see any in the engine.log (attachment 997713 [details]), however maybe they are simply not logged - need to profile the engine more closely in order to find out.
 maybe a query that was invoked in the GUI with 'refresh=false' contains, as part of its internal execution phase in the backend, call(s) to other query(ies) with 'refresh=true', which keep the engine session alive without the ui-plugins "keep alive" mechanism even knowing about it.
In this case, it is probably an engine bug (since there is no reason for something that was invoked with 'refresh=false' to invoke internally something with 'refresh=true').
There were 2 potential causes for this issue:
1. The hard limit of the engine session. The REST api pinging code did not take this hard limit into account and could potentially cause the popup when it tried to ping after the hard limit was reached and the engine session was terminated.
2. The UI was sending requests with refresh:false, but internally the engine was creating a sub query and set the refresh:true in that sub query causing the engine session to survive endlessly, but the REST api session would terminate after a while, causing the pop to appear. This scenario was the one Einav encountered in her testing.
The associated patches solve both issues.
Is this patch going to be applied to the RHEV 3.5 stream? If so, any ideas on the target release? (/me hoping for v3.5.1)
(In reply to Colin Coe from comment #42)
> Is this patch going to be applied to the RHEV 3.5 stream? If so, any ideas
> on the target release? (/me hoping for v3.5.1)
Going to be in RHEV 3.5.3 the following release to RHEV 3.5.1.
ok, same verification steps as in https://bugzilla.redhat.com/show_bug.cgi?id=1206908#8
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.