Bug 1399347

I see several potential causes:

- If the client browser is IE, Edge or Safari, you probably need to update to Cockpit 0.122-3 (released in sync with Extras 7.3.1)
- If Cockpit is running as a service, systemd may need to reload and the web service restarted
- Are there any error messages in the javascript console of the browser?
- Are all cockpit subpackages upgraded to the same version?

If these items don't help, it would be good to know
- the installed package versions (cockpit, subpackages)
- to clarify if the user just stays on the login page after logging in or if something else happens.

Sure I can get a more detailed report posted but &Jericho has a sos report of my system that should have most of the info you are asking for. If it can get attached

Hi,
does the problem persist across reboots. (In case also systemd were updated, there is suggested  in erratum to reboot system too)
    Regards
    Honza

(In reply to Boris Kurktchiev from comment #3)
> Sure I can get a more detailed report posted but &Jericho has a sos report
> of my system that should have most of the info you are asking for. If it can
> get attached

Jan just looked at the installed versions of Cockpit, thanks.

Since cockpit-kubernetes has version 118, could you please try upgrading the other packages to the same version as well?

Upgraded to -118 on all components and am currently upgrading the system to latest RHEL 7.3.x release and then will bounce and see what shakes loose. Stay tuned.

After restart still the same effect. As far as looking at it with Dev tools:

GET https://cockpit.cloudapps.unc.edu/cockpit/login 401 (Authentication required)

However, if you look at the journal you can see that the system authenticated me just fine: 

Dec 06 09:20:24 osmaster0p.isis.unc.edu polkitd[818]: Registered Authentication Agent for unix-session:2 (system bus name :1.34 [cockpit-bridge], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Dec 06 09:20:24 osmaster0p.isis.unc.edu cockpit-ws[1170]: logged in user: boris
Dec 06 09:20:24 osmaster0p.isis.unc.edu cockpit-session[4102]: pam_unix(cockpit:session): session opened for user boris by (uid=0)

Dec 06 09:20:39 osmaster0p.isis.unc.edu cockpit-session[4102]: pam_unix(cockpit:session): session closed for user boris
Dec 06 09:20:39 osmaster0p.isis.unc.edu polkitd[818]: Unregistered Authentication Agent for unix-session:2 (system bus name :1.34, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Dec 06 09:20:39 osmaster0p.isis.unc.edu cockpit-ws[1170]: boris: timed out

After discussing this with Peter, we think that this could be one of two things:
- the cookies are not taking (if you have access, you could check those in the browser)
- the session is terminating before the login page has a chance to reload

It might also help to turn on debugging for Cockpit in the journal.

More info on enabling debug messages for Cockpit can be found on the upstream repository:

https://github.com/cockpit-project/cockpit/blob/master/HACKING.md#debug-logging-of-cockpit-processes

Cookies:
there is a cockpit cookie with what looks like a token: dj0yO2s9M2I3Zjk0MDU0N2E4MGQ0MDNkOGFiYzk0NjlhNDE0NGJhOGFjODUxZjNiY2JmNjg5OTY2YTU1ODBhZTIwOTBjZA

I will work on getting debugging setup and see what happens.

Here is the debug info

Dec 06 10:27:05 osmaster0p.isis.unc.edu atomic-openshift-node[3240]: I1206 10:27:05.039981    3240 node_auth.go:143] Node request attributes: namespace=, user=&user.DefaultInfo{Name:"system:serviceaccount:openshift-infra:heapster", UID:"cb45c36c-a82e-11e6-91bd-00505691bba9", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-infra", "system:authenticated", "system:authenticated"}, Extra:map[string][]string{}}, attrs=authorizer.DefaultAuthorizationAttributes{Verb:"create", APIVersion:"v1", APIGroup:"", Resource:"nodes/stats", ResourceName:"osmaster0p.isis.unc.edu", RequestAttributes:interface {}(nil), NonResourceURL:false, URL:"/stats/container/"}
Dec 06 10:27:05 osmaster0p.isis.unc.edu atomic-openshift-node[3240]: I1206 10:27:05.104376    3240 server.go:964] POST /stats/container/: (64.485704ms) 200 [[Go-http-client/1.1] 152.19.229.172:43948]
Dec 06 10:27:18 osmaster0p.isis.unc.edu cockpit-ws[34938]: received unknown/invalid credential cookie
Dec 06 10:27:18 osmaster0p.isis.unc.edu cockpit-ws[34938]: received unknown/invalid credential cookie
Dec 06 10:27:18 osmaster0p.isis.unc.edu cockpit-ws[34938]: /usr/libexec/cockpit-session auth pipe: setting up auth pipe 9 10
Dec 06 10:27:18 osmaster0p.isis.unc.edu cockpit-ws[34938]: spawning /usr/libexec/cockpit-session
Dec 06 10:27:18 osmaster0p.isis.unc.edu cockpit-ws[34938]: /usr/libexec/cockpit-session auth pipe: reporting message
Dec 06 10:27:18 osmaster0p.isis.unc.edu cockpit-ws[34938]: /usr/libexec/cockpit-session says: {  "error": "authentication-unavailable", "message": "Authentication service cannot retrieve authentication info"}
Dec 06 10:27:18 osmaster0p.isis.unc.edu cockpit-ws[34938]: negotiate auth is not available, disabling
Dec 06 10:27:20 osmaster0p.isis.unc.edu atomic-openshift-node[3240]: I1206 10:27:20.021128    3240 node_auth.go:143] Node request attributes: namespace=, user=&user.DefaultInfo{Name:"system:serviceaccount:openshift-infra:heapster", UID:"cb45c36c-a82e-11e6-91bd-00505691bba9", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-infra", "system:authenticated", "system:authenticated"}, Extra:map[string][]string{}}, attrs=authorizer.DefaultAuthorizationAttributes{Verb:"create", APIVersion:"v1", APIGroup:"", Resource:"nodes/stats", ResourceName:"osmaster0p.isis.unc.edu", RequestAttributes:interface {}(nil), NonResourceURL:false, URL:"/stats/container/"}
Dec 06 10:27:20 osmaster0p.isis.unc.edu atomic-openshift-node[3240]: I1206 10:27:20.075916    3240 server.go:964] POST /stats/container/: (54.925446ms) 200 [[Go-http-client/1.1] 152.19.229.172:43948]
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: received unknown/invalid credential cookie
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: /usr/libexec/cockpit-session auth pipe: setting up auth pipe 9 10
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: spawning /usr/libexec/cockpit-session
Dec 06 10:27:24 osmaster0p.isis.unc.edu unix_chkpwd[35164]: password check failed for user (boris)
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-session[35162]: pam_unix(cockpit:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=172.22.158.132  user=boris
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: cockpit-session: pam: dropping text message Warning: Your password will expire in 84 days on Tue Feb 28 18:38:18 2017
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-session[35162]: pam_krb5[35162]: error reading keytab 'FILE:/etc/krb5.keytab'
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-session[35162]: pam_krb5[35162]: TGT verified
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-session[35162]: pam_krb5[35162]: authentication succeeds for 'boris' (boris.EDU)
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-session[35162]: pam_ssh_add: Identity added: /websyshome/boris/.ssh/id_rsa (/websyshome/boris/.ssh/id_rsa)
Dec 06 10:27:24 osmaster0p.isis.unc.edu systemd-logind[819]: New session 12 of user boris.
Dec 06 10:27:24 osmaster0p.isis.unc.edu systemd[1]: Started Session 12 of user boris.
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-session[35162]: pam_unix(cockpit:session): session opened for user boris by (uid=0)
Dec 06 10:27:24 osmaster0p.isis.unc.edu systemd[1]: Starting Session 12 of user boris.
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: /usr/libexec/cockpit-session auth pipe: reporting message
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: /usr/libexec/cockpit-session auth pipe: Auth pipe closed
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: /usr/libexec/cockpit-session says: {  "user": "boris"}
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: user authenticated as boris
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: localhost: new session
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: boris: login is idle
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: sending 460f4dc4c7d654996fcd77fe38c847dd8d40390c153ab45926e2715d778b5940 credential id 'cockpit' for user 'boris'
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: logged in user: boris
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: launched dbus-daemon
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: session bus address: unix:abstract=/tmp/dbus-a74ACI2qtq,guid=9a96982cad65088a876048615846d8dc
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: package server port: 44138
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: base1: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: base1: added package at /usr/share/cockpit/base1
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: branding: no manifest found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: dashboard: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: dashboard: added package at /usr/share/cockpit/dashboard
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: docker: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: docker: added package at /usr/share/cockpit/docker
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: kubernetes: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: kubernetes: added package at /usr/share/cockpit/kubernetes
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: networkmanager: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: network: added package at /usr/share/cockpit/networkmanager
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: realmd: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: domain: added package at /usr/share/cockpit/realmd
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: selinux: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: selinux: added package at /usr/share/cockpit/selinux
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: shell: added package at /usr/share/cockpit/shell
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: sosreport: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: sosreport: added package at /usr/share/cockpit/sosreport
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: static: no manifest found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: subscriptions: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: subscriptions: added package at /usr/share/cockpit/subscriptions
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: systemd: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: system: added package at /usr/share/cockpit/systemd
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: tuned: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: performance: added package at /usr/share/cockpit/tuned
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: users: no override found
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: users: added package at /usr/share/cockpit/users
Dec 06 10:27:24 osmaster0p.isis.unc.edu polkitd[818]: Registered Authentication Agent for unix-session:12 (system bus name :1.61 [cockpit-bridge], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: registered polkit authentication agent for subject: unix-session:12
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-bridge[35175]: received init message
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: localhost: received init message
Dec 06 10:27:24 osmaster0p.isis.unc.edu cockpit-ws[34938]: received unknown/invalid credential cookie
Dec 06 10:27:25 osmaster0p.isis.unc.edu cockpit-ws[34938]: received unknown/invalid credential cookie

Looks to be cookie related.

What browser are you using? Or is it any browser?

Have you tried clearing cookies before logging in, to see if that makes a difference?

What we need to try to verify is that the contents of the "Set-Cookie" header you receive in response to the /cockpit/login request that succeeds is same as what get sent to the server in the subsequent reload.

Well this is interesting. I switched to Safari and got in just fine. Chrome (latest) in both standard and Incognito mode is unable to get me in.

Ok, progress so in chrome open developer tools.

Open the network tab. Go to the login page, then hit the delete button to clear the log. Then check the preserve log checkbox.

Try to login in.

The first network entry should be a call to /cockpit/login with a 200 response. The next entry should be a request to the page you were going to ei: / or /dashboard or something like that.

Can you send the full request / response headers for both of those calls?

yeah so looks like /login is sending a 401:

Also chrome really needs a dev tools log export option... Here is the crude copy/paste:

cockpit.cloudapps.unc.edu
200
OK
document	Other	33.6 KB
33.2 KB
56 ms
41 ms
branding.css
/cockpit/static
200
OK
stylesheet	(index):1009
Parser
(from disk cache)	5 ms
4 ms
OpenSans-Regular-webfont.woff
/cockpit/static/fonts
200
OK
font	onloadwff.js:1402
Script
(from memory cache)	0 ms
0 ms
OpenSans-Bold-webfont.woff
/cockpit/static/fonts
200
OK
font	onloadwff.js:1402
Script
(from memory cache)	0 ms
0 ms
bg-login.jpg
/cockpit/static
200
OK
jpg	(index):-Infinity
Parser
(from memory cache)	0 ms
0 ms
logo.png
/cockpit/static
200
OK
png	(index):-Infinity
Parser
(from memory cache)	0 ms
0 ms
login
/cockpit
401
Negotiate authentication not available
xhr	(index):197
Script
3.7 KB
3.4 KB
16 ms
15 ms
data:image/png;base…
200
OK
png	onloadwff.js:1255
Script
(from memory cache)	0 ms
0 ms
data:image/png;base…
200
OK
png	onloadwff.js:928
Script
(from memory cache)	0 ms
0 ms
data:image/png;base…
200
OK
png	onloadwff.js:1255
Script
(from memory cache)	0 ms
0 ms
data:image/png;base…
200
OK
png	onloadwff.js:1160
Script
(from memory cache)	0 ms
0 ms

the 401 error: Negotiate authentication not available

That's the first request, that's expected. There should be a second request to /cockpit/login after you submit a username/password. The two requests we care about are that second /cockpit/login request and the request that gets made directly afterwards.

Nope, after login attempt i get: 401 Authentication required

Also seeing: OpenSans-Bold-webfont.woff
/cockpit/static/fonts
(failed)
net::ERR_CONNECTION_RESET

But dont think the second one is relevant.

I thought the page was reloading and then taking you back to the login screen. Is that not the case? Are you getting an error message instead?

Be sure you are checking the "preserve log" checkbox. Otherwise chrome clears the list of http calls when the reload happens and you never see the 2nd call to /cockpit/login. Only the first one again because the login page is what reloads.

If this is easier debugged by chat, I'm around on #cockpit on freenode.

Yes I have it preserve the log and once I click the login button thats all the logs says. Spits me back out to the login page and the network tab says 401 Authentication required. There is no error message displayed on the login page. I am idling in #cockpit, ebalsumgo

Thanks for your help tracking this down Boris.

The bug was an instance of 
https://github.com/cockpit-project/cockpit/pull/5327/commits/9d8bd4cbd773d606a9867dd1c7b92951f20045ba 

Where we weren't parsing cookies correctly so a different cookie set by the load balancer was causing the cockpit auth cookie not to be found.

This was fixed in cockpit version 123.

Dominik Perpeet, will this fix be in the next release?

Thanks Boris and Peter for tracking this down. Unfortunately this patch didn't make it into RHEL Extras 7.3.1, but will make it into 7.3.2.

I will add this bug to the 7.3.2 release so you will get notified once it becomes available.

Apparently the bug wasn't properly attached, but the fix did go out with 7.3.2.

Sorry for the inconvenience.