Description of problem: After updating cockpit and all of its components from version 178-1 to version 193-1 login fails with message "Internal error in login process" Version-Release number of selected component (if applicable): 193-1.fc29 How reproducible: Do an update to 193-1 and try to login. Steps to Reproduce: 1. Updating system via dnf update or dnf update cockpit 2. Open browser, enter <ipadress>:9090 and try to login 3. Message shows up "Internal error in login process" Actual results: Cannot login Expected results: Successful login and cockpit dashboard shows up Additional info: I updated my systems both Fedora 29 Workstation and Server Edition last day and after that i cannot login in cockpit. Downgrade of cockpit and all of its components to version 178-3 is fixing this behavior.
Could you include any cockpit related log lines in the journal from after your upgrade? Cockpit always logs error messages to the journal for any such failure. The following command is a good way to find the log lines. Make sure to purge sensitive information out: sudo journalctl -b | grep cockpit
More specifically, can you please do this: sudo journalctl -fn0 then try to log in, and copy&paste the entire output? (Again, please watch out for private stuff like user names). Thanks!
sudo journalctl -b | grep cockpit Mai 16 17:36:45 your.domain.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=cockpit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Mai 16 17:36:45 your.domain.com cockpit-ws[31149]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert Mai 16 17:36:45 your.domain.com cockpit-ws[31149]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate Mai 16 17:36:45 your.domain.com cockpit-ws[31149]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate Mai 16 17:36:52 your.domain.com audit[31154]: USER_AUTH pid=31154 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:authentication grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:36:52 your.domain.com audit[31154]: USER_ACCT pid=31154 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:36:52 your.domain.com audit[31154]: CRED_ACQ pid=31154 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:36:52 your.domain.com audit[31154]: USER_ROLE_CHANGE pid=31154 uid=0 auid=1000 ses=384 subj=system_u:system_r:cockpit_session_t:s0 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:36:52 your.domain.com cockpit-session[31154]: pam_ssh_add: Failed adding some keys Mai 16 17:36:52 your.domain.com cockpit-session[31154]: pam_unix(cockpit:session): session opened for user username by (uid=0) Mai 16 17:36:52 your.domain.com audit[31154]: USER_START pid=31154 uid=0 auid=1000 ses=384 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_ssh_add,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:36:52 your.domain.com audit[31154]: CRED_REFR pid=31154 uid=0 auid=1000 ses=384 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:36:52 your.domain.com cockpit-ws[31149]: /usr/libexec/cockpit-session: incorrect protocol: received invalid length prefix Mai 16 17:36:52 your.domain.com polkitd[680]: Registered Authentication Agent for unix-session:384 (system bus name :1.2383 [cockpit-bridge], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale de_DE.UTF-8) Mai 16 17:36:52 your.domain.com audit[31154]: CRED_DISP pid=31154 uid=0 auid=1000 ses=384 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:36:52 your.domain.com cockpit-session[31154]: pam_unix(cockpit:session): session closed for user username Mai 16 17:36:52 your.domain.com audit[31154]: USER_END pid=31154 uid=0 auid=1000 ses=384 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_ssh_add,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' sudo journalctl -fn0 -- Logs begin at Tue 2019-05-14 22:23:33 CEST. -- Mai 16 17:27:14 your.domain.com audit[24905]: USER_AUTH pid=24905 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:authentication grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com audit[24905]: USER_ACCT pid=24905 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com audit[24905]: CRED_ACQ pid=24905 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com audit[24905]: USER_ROLE_CHANGE pid=24905 uid=0 auid=1000 ses=381 subj=system_u:system_r:cockpit_session_t:s0 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com cockpit-session[24905]: pam_ssh_add: Failed adding some keys Mai 16 17:27:14 your.domain.com systemd-logind[730]: New session 381 of user username. Mai 16 17:27:14 your.domain.com systemd[1]: Started Session 381 of user username. Mai 16 17:27:14 your.domain.com cockpit-session[24905]: pam_unix(cockpit:session): session opened for user username by (uid=0) Mai 16 17:27:14 your.domain.com audit[24905]: USER_START pid=24905 uid=0 auid=1000 ses=381 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_ssh_add,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com audit[24905]: CRED_REFR pid=24905 uid=0 auid=1000 ses=381 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com cockpit-ws[19427]: /usr/libexec/cockpit-session: incorrect protocol: received invalid length prefix Mai 16 17:27:14 your.domain.com polkitd[680]: Registered Authentication Agent for unix-session:381 (system bus name :1.2358 [cockpit-bridge], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale de_DE.UTF-8) Mai 16 17:27:14 your.domain.com polkitd[680]: Unregistered Authentication Agent for unix-session:381 (system bus name :1.2358, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale de_DE.UTF-8) Mai 16 17:27:14 your.domain.com audit[24905]: CRED_DISP pid=24905 uid=0 auid=1000 ses=381 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com cockpit-session[24905]: pam_unix(cockpit:session): session closed for user username Mai 16 17:27:14 your.domain.com audit[24905]: USER_END pid=24905 uid=0 auid=1000 ses=381 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_ssh_add,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="username" exe="/usr/libexec/cockpit-session" hostname=10.0.0.15 addr=10.0.0.15 terminal=? res=success' Mai 16 17:27:14 your.domain.com systemd[1]: session-381.scope: Consumed 194ms CPU time Mai 16 17:27:14 your.domain.com systemd-logind[730]: Session 381 logged out. Waiting for processes to exit. Mai 16 17:27:14 your.domain.com systemd-logind[730]: Removed session 381. sudo systemctl status cockpit Mai 16 17:36:45 your.domain.com systemd[1]: Starting Cockpit Web Service... Mai 16 17:36:45 your.domain.com systemd[1]: Started Cockpit Web Service. Mai 16 17:36:45 your.domain.com cockpit-ws[31149]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert Mai 16 17:36:45 your.domain.com cockpit-ws[31149]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate Mai 16 17:36:45 your.domain.com cockpit-ws[31149]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate Mai 16 17:36:52 your.domain.com cockpit-session[31154]: pam_ssh_add: Failed adding some keys Mai 16 17:36:52 your.domain.com cockpit-session[31154]: pam_unix(cockpit:session): session opened for user username by (uid=0) Mai 16 17:36:52 your.domain.com cockpit-ws[31149]: /usr/libexec/cockpit-session: incorrect protocol: received invalid length prefix Mai 16 17:36:52 your.domain.com cockpit-session[31154]: pam_unix(cockpit:session): session closed for user username Mai 16 17:38:16 your.domain.com systemd[1]: cockpit.service: Consumed 199ms CPU time
Just to inform you guys. I tried the same procedure with my Fedora Workstation 29 Edition and I got the same behavior. Afterwards I did downgrade cockpit-packages to 178-1 and wrote "systemctl status cockpit" in the console which did not showed up any error messages. After that i used "sudo journalctl -fn0" and logged in into Cockpit. No error messages at all. When i move through the menue I get the message: "cockpit-ws[16569]: couldn't read from connection: Peer sent fatal TLS alert: Unknown certificate" but everything's working anyway. My supicion is, that something's wrong with the certificate, isn't it?
This seems to be the root of the problem: /usr/libexec/cockpit-session: incorrect protocol: received invalid length prefix Which means that cockpit-ws and cockpit-session cannot properly talk to each other. However, it's unclear why that is. The "Unknown certificate" is normal for self-signed certificates. For me it looks like Mai 17 08:15:41 donald cockpit-ws[5251]: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert Mai 17 08:15:41 donald cockpit-ws[5251]: couldn't read from connection: Peer sent fatal TLS alert: CA is unknown as I'm using an sscg-generated certificate, but I also tried with self-signed and that works as well. So I think this is unrelated. Also, the "protocol" between cockpit-ws and cockpit-session has nothing to do with TLS. Also, the initial TLS connection seems to have worked -- you apparently saw the login page and entered your credentials? To be sure of this, can you please open a private browser window, and try to log into cockpit through http://localhost:9090, i. e. not use TLS?
I cannot try the Server Editon because it is a headless system. I tried login via localhost:9090 on my Workstation but it results in the same issue. I do not know how to test without using TLS...
Hi, I've been having a similar issue on Fedora 29 and then Fedora 30 currrently on: cockpit-194-1.fc30.x86_64 With an GUI error of "Internal error in login process" (it is correctly confirming user credentials - it detects password mistakes. I also get the log message error of: /usr/libexec/cockpit-session: incorrect protocol: received invalid length As requested I've added the following lines to /etc/cockpit/cockpit.conf and restarted the service. [WebService] AllowUnencrypted = true sudo systemctl status cockpit May 25 14:19:50 your.domain.com cockpit-ws[3690]: Using certificate: /etc/cockpit/ws-certs.d/~self-signed.cert May 25 14:19:55 your.domain.com cockpit-session[3701]: pam_ssh_add: Identity added: /home/username/.ssh/id_rsa (/home/username/.ssh/id_rsa) May 25 14:19:55 your.domain.com cockpit-session[3701]: pam_unix(cockpit:session): session opened for user username by (uid=0) May 25 14:19:56 your.domain.com cockpit-ws[3690]: /usr/libexec/cockpit-session: incorrect protocol: received invalid length prefix May 25 14:19:56 your.domain.com cockpit-session[3701]: pam_unix(cockpit:session): session closed for user username May 25 14:20:31 your.domain.com cockpit-session[4378]: pam_ssh_add: Identity added: /home/username/.ssh/id_rsa (/home/username/.ssh/id_rsa) May 25 14:20:31 your.domain.com cockpit-session[4378]: pam_unix(cockpit:session): session opened for user username by (uid=0) May 25 14:20:32 your.domain.com cockpit-ws[3690]: /usr/libexec/cockpit-session: incorrect protocol: received invalid length prefix May 25 14:20:32 your.domain.com cockpit-session[4378]: pam_unix(cockpit:session): session closed for user username May 25 14:21:20 your.domain.com systemd[1]: cockpit.service: Succeeded. sudo journalctl -fn0 when attempting to login to http://localhost:9090 on chromium May 25 14:20:31 your.domain.com audit[4378]: USER_AUTH pid=4378 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:authentication grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:31 your.domain.com audit[4378]: USER_ACCT pid=4378 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="username" exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:31 your.domain.com audit[4378]: CRED_ACQ pid=4378 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:31 your.domain.com audit[4378]: USER_ROLE_CHANGE pid=4378 uid=0 auid=1000 ses=4227 subj=system_u:system_r:cockpit_session_t:s0 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:31 your.domain.com cockpit-session[4378]: pam_ssh_add: Identity added: /home/username/.ssh/id_rsa (/home/username/.ssh/id_rsa) May 25 14:20:31 your.domain.com systemd-logind[1070]: New session 4227 of user username. May 25 14:20:31 your.domain.com systemd[1]: Started Session 4227 of user username. May 25 14:20:31 your.domain.com cockpit-session[4378]: pam_unix(cockpit:session): session opened for user username by (uid=0) May 25 14:20:31 your.domain.com audit[4378]: USER_START pid=4378 uid=0 auid=1000 ses=4227 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_ssh_add,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="username" exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:31 your.domain.com audit[4378]: CRED_REFR pid=4378 uid=0 auid=1000 ses=4227 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:32 your.domain.com cockpit-ws[3690]: /usr/libexec/cockpit-session: incorrect protocol: received invalid length prefix May 25 14:20:32 your.domain.com polkitd[1190]: Registered Authentication Agent for unix-session:4227 (system bus name :1.183930 [cockpit-bridge], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.utf8) May 25 14:20:32 your.domain.com polkitd[1190]: Unregistered Authentication Agent for unix-session:4227 (system bus name :1.183930, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.utf8) May 25 14:20:32 your.domain.com audit[4378]: CRED_DISP pid=4378 uid=0 auid=1000 ses=4227 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:setcred grantors=pam_unix acct="username" exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:32 your.domain.com cockpit-session[4378]: pam_unix(cockpit:session): session closed for user username May 25 14:20:32 your.domain.com audit[4378]: USER_END pid=4378 uid=0 auid=1000 ses=4227 subj=system_u:system_r:cockpit_session_t:s0 msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_ssh_add,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="username" exe="/usr/libexec/cockpit-session" hostname=::1 addr=::1 terminal=? res=success' May 25 14:20:32 your.domain.com systemd[1]: session-4227.scope: Succeeded. May 25 14:20:32 your.domain.com systemd-logind[1070]: Session 4227 logged out. Waiting for processes to exit. May 25 14:20:32 your.domain.com systemd-logind[1070]: Removed session 4227.
I just got an idea. I bet this broke between 191 and 192 due to https://github.com/cockpit-project/cockpit/pull/11575 . If your ~/.bashrc (or equivalent for your selected login shell) prints something, it would disturb the protocol. What's the output of /bin/bash -c 'echo hello' ? I. e. does that print anything else than "hello"?
/bin/bash -c 'echo hello' prints 'hello' on the screen. Nothing else.
Sorry, correction: /bin/bash -l -c 'echo hello' Background: This is most surely something from ~/.bashrc, but without -l that isn't run.
For me that's it: I was using autojump that prints the current folder. Running /bin/bash -l -c 'echo hello' /mnt/home hello Removing autojump I can login.
Running /bin/bash -l -c 'echo hello' prints just 'hello' again. I have no autojump installed.
Thanks Rick and Woti. Also from debugging with another affected user on IRC, I'm now reasonably sure that this is due to ~./bashrc or ~/.bash_profile printing *something* on login. This is annoyingly hard to fix, I'll think about it.
You are welcome. But that means updating of Fedora-Server 29 packages or upgrading to Fedora-Server 30 is not recommended until this issue is fixed? At the moment there's no workaround?
Upstream fix: https://github.com/cockpit-project/cockpit/pull/11978
FEDORA-2019-95ed893adc has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-95ed893adc
cockpit-194-2.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-95ed893adc
Cockpit-192-4.fc29 is working fine. Cockpit-191-1.fc30 is working fine. Cockpit-195-1.fc30 is not working fine.
@Woti: This bug is *not* fixed yet in Fedora 30, only in Fedora 29 in the above update.
Okay, thanks for the info. Best to wait with upgrading of my Fedora-Server Edition to 30. Where do I know when it will be fixed? Will it be reported here?
@Woti: It will be fixed in Fedora 30 next Wednesday with Cockpit 196.
cockpit-194-2.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
For Fedora 30 this is fixed in https://bodhi.fedoraproject.org/updates/FEDORA-2019-07ec9aac9f . Testing and feedback appreciated!
I can confirm that cockpit-196-1.fc30 for Fedora 30 is working fine.