Created attachment 1607580 [details] Screenshot of Cockpit Error So, I have an unusual one for you guys. I have cockpit stable installed on FC30 on my RasPi 3. Everything was going fine until I accidentally yanked out the power cable during an dnf update, and since then whenever I try to login to cockpit I get the following error being reported: "The cockpit package is not installed" (screenshot attached). Here is the gotcha, cockpit is installed. I tried reinstalling Cockpit and cockpit-ws thinking maybe they were corrupted but a dnf reinstall did jack. I'm planning on just reinstalling FC30 on it at some point, but before I do I figure I would keep it going long enough to see if this is a bug like I think it is or if it's a case of "Chance is a idiot." Version-Release number of selected component (if applicable): cockpit 200, FC30 latest. How reproducible: I'm not sure. Steps to Reproduce: 1. Run dnf update 2. Yank cable out of Raspberry Pi. Actual results: Cockpit Web UI reports that Cockpit is not installed on login. Expected results: Cockpit Web UI allows me to admin my system. Additional info: Screenshot of error attached. systemd logs are below. [administrator@localhost ~]$ systemctl status cockpit ● cockpit.service - Cockpit Web Service Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled) Active: inactive (dead) since Fri 2019-08-23 21:41:12 EDT; 5min ago Docs: man:cockpit-ws(8) Process: 5513 ExecStartPre=/usr/sbin/remotectl certificate --ensure --user=root --group=cockpit-ws --selinux-type=etc_t (code=exited, status=0/SUCCESS) Process: 5515 ExecStart=/usr/libexec/cockpit-tls (code=exited, status=0/SUCCESS) Main PID: 5515 (code=exited, status=0/SUCCESS) Aug 23 21:38:11 localhost.localdomain cockpit-tls[5515]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 23 21:38:11 localhost.localdomain cockpit-tls[5515]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 23 21:38:12 localhost.localdomain cockpit-tls[5515]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 23 21:38:18 localhost.localdomain cockpit-session[5520]: pam_unix(cockpit:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=administrator Aug 23 21:38:24 localhost.localdomain cockpit-tls[5515]: cockpit-tls: reading from client fd 7 TLS connection failed: The TLS connection was non-properly terminated. Aug 23 21:38:24 localhost.localdomain cockpit-session[5523]: pam_ssh_add: Failed to start ssh-agent Aug 23 21:38:24 localhost.localdomain cockpit-session[5523]: pam_unix(cockpit:session): session opened for user administrator by (uid=0) Aug 23 21:38:24 localhost.localdomain cockpit-tls[5515]: cockpit-session: can't exec /bin/bash: Permission denied Aug 23 21:38:24 localhost.localdomain cockpit-session[5523]: pam_unix(cockpit:session): session closed for user administrator Aug 23 21:41:12 localhost.localdomain systemd[1]: cockpit.service: Succeeded. [administrator@localhost ~]$ [administrator@localhost ~]$ systemctl status cockpit.socket ● cockpit.socket - Cockpit Web Service Socket Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; enabled; vendor preset: enabled) Active: active (listening) since Fri 2019-08-23 21:37:54 EDT; 9min ago Docs: man:cockpit-ws(8) Listen: [::]:9090 (Stream) Tasks: 0 (limit: 1030) Memory: 860.0K CGroup: /system.slice/cockpit.socket Aug 23 21:37:54 localhost.localdomain systemd[1]: Starting Cockpit Web Service Socket. Aug 23 21:37:54 localhost.localdomain systemd[1]: Listening on Cockpit Web Service Socket. [administrator@localhost ~]$
Just adding on now that I thought about it, I ran a dnf history to see if Cockpit was updated when the power went out, and it was not. The only transaction records are when I attempted to fix it by reinstalling. [administrator@localhost ~]$ sudo dnf history | grep cockpit [sudo] password for administrator: 9 | reinstall cockpit-ws | 2019-08-23 21:37 | R | 2 8 | reinstall cockpit | 2019-08-23 21:36 | R | 2 [administrator@localhost ~]$
Seems more likely that some PAM config or permission got damaged during that. Do you still get the login screen, and can enter your user/password? Or does it fail with that right away? Can you please check `ls -lZ /usr/libexec/cockpit-session`? Should be -rwsr-x---. 2 root cockpit-ws system_u:object_r:cockpit_session_exec_t:s0 65288 1. Jan 1970 /usr/libexec/cockpit-session Also, please run "sudo journalctl -f" and try to log in, then copy&paste the output. The "authentication failure" in your log snippet hints to some error there, but it doesn't have enough context.
Okay, I'm going to work backwards here if that is okay. journalctl logs: Aug 29 09:32:11 localhost.localdomain sudo[22469]: pam_unix(sudo:session): session opened for user root by administrator(uid=0) Aug 29 09:32:11 localhost.localdomain audit[22469]: USER_START pid=22469 uid=0 auid=1000 ses=5 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success' Aug 29 09:32:22 localhost.localdomain systemd[1]: Starting Cockpit Web Service... Aug 29 09:32:22 localhost.localdomain systemd[1]: Started Cockpit Web Service. Aug 29 09:32:22 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=cockpit comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Aug 29 09:32:23 localhost.localdomain cockpit-tls[22481]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 29 09:32:23 localhost.localdomain cockpit-tls[22481]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 29 09:32:23 localhost.localdomain cockpit-tls[22481]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 29 09:32:23 localhost.localdomain cockpit-tls[22481]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 29 09:32:23 localhost.localdomain cockpit-tls[22481]: cockpit-tls: TLS handshake failed: A TLS fatal alert has been received. Aug 29 09:32:24 localhost.localdomain cockpit-tls[22481]: cockpit-tls: reading from client fd 8 TLS connection failed: The TLS connection was non-properly terminated. Aug 29 09:32:24 localhost.localdomain cockpit-tls[22481]: cockpit-tls: reading from client fd 10 TLS connection failed: The TLS connection was non-properly terminated. Aug 29 09:32:24 localhost.localdomain cockpit-tls[22481]: cockpit-tls: reading from client fd 7 TLS connection failed: The TLS connection was non-properly terminated. Aug 29 09:32:30 localhost.localdomain audit[22486]: USER_AUTH pid=22486 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='op=PAM:authentication grantors=pam_succeed_if,pam_localuser,pam_unix acct="administrator" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain audit[22486]: USER_ACCT pid=22486 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="administrator" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain audit[22486]: CRED_ACQ pid=22486 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 msg='op=PAM:setcred grantors=pam_localuser,pam_unix acct="administrator" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain audit[22486]: USER_ROLE_CHANGE pid=22486 uid=0 auid=1000 ses=6 subj=system_u:system_r:unconfined_service_t:s0 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0 selected-context=unconfined_u:unconfined_r:unconfined_t:s0 exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain audit[22490]: AVC avc: denied { transition } for pid=22490 comm="cockpit-session" path="/usr/bin/bash" dev="dm-0" ino=8484081 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0 Aug 29 09:32:30 localhost.localdomain cockpit-session[22490]: pam_ssh_add: couldn't run /bin/sh: Permission denied Aug 29 09:32:30 localhost.localdomain cockpit-session[22486]: pam_ssh_add: Failed to start ssh-agent Aug 29 09:32:30 localhost.localdomain systemd-logind[840]: New session 6 of user administrator. Aug 29 09:32:30 localhost.localdomain systemd[1]: Started Session 6 of user administrator. Aug 29 09:32:30 localhost.localdomain cockpit-session[22486]: pam_unix(cockpit:session): session opened for user administrator by (uid=0) Aug 29 09:32:30 localhost.localdomain audit[22486]: USER_START pid=22486 uid=0 auid=1000 ses=6 subj=system_u:system_r:unconfined_service_t:s0 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_umask,pam_lastlog acct="administrator" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain audit[22486]: CRED_REFR pid=22486 uid=0 auid=1000 ses=6 subj=system_u:system_r:unconfined_service_t:s0 msg='op=PAM:setcred grantors=pam_localuser,pam_unix acct="administrator" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain audit[22491]: AVC avc: denied { transition } for pid=22491 comm="cockpit-session" path="/usr/bin/bash" dev="dm-0" ino=8484081 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0 Aug 29 09:32:30 localhost.localdomain cockpit-tls[22481]: cockpit-session: can't exec /bin/bash: Permission denied Aug 29 09:32:30 localhost.localdomain audit[22486]: CRED_DISP pid=22486 uid=0 auid=1000 ses=6 subj=system_u:system_r:unconfined_service_t:s0 msg='op=PAM:setcred grantors=pam_localuser,pam_unix acct="administrator" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain cockpit-session[22486]: pam_unix(cockpit:session): session closed for user administrator Aug 29 09:32:30 localhost.localdomain audit[22486]: USER_END pid=22486 uid=0 auid=1000 ses=6 subj=system_u:system_r:unconfined_service_t:s0 msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_umask,pam_lastlog acct="administrator" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success' Aug 29 09:32:30 localhost.localdomain systemd[1]: session-6.scope: Succeeded. Aug 29 09:32:30 localhost.localdomain systemd-logind[840]: Session 6 logged out. Waiting for processes to exit. Aug 29 09:32:30 localhost.localdomain systemd-logind[840]: Removed session 6. Aug 29 09:32:33 localhost.localdomain systemd[1]: Started dbus-:1.4-org.fedoraproject.Setroubleshootd. Aug 29 09:32:33 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=dbus-:1.4-org.fedoraproject.Setroubleshootd@1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' and the ls: -rwsr-x---. 1 root cockpit-ws system_u:object_r:cockpit_session_exec_t:s0 82592 Aug 7 16:55 /usr/libexec/cockpit-session After looking at the systemd logs, I see that cockpit is getting permission denied when starting the SSH agent and when trying to use bash. I'm pretty sure my install did actually get deep sixed by that outage. If you concur that's the issue, I'll go ahead and close out the ticket.
This does point out some SELinux problem: AVC avc: denied { transition } for pid=22491 comm="cockpit-session" path="/usr/bin/bash" dev="dm-0" ino=8484081 scontext=system_u:system_r:unconfined_service_t:s0 cockpit-tls[22481]: cockpit-session: can't exec /bin/bash: Permission denied Supposedly things work after "setenforce 0"? Could you try to uninstall and reinstall cockpit-ws?
I'm running into the same thing on a brand new F30 install on a helios4 armv7 device. type=AVC msg=audit(1570738364.843:347): avc: denied { transition } for pid=10096 comm="cockpit-session" path="/usr/bin/bash" dev="mmcblk0p3" ino=7809 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0 tclass=process permissive=0 I can confirm that setenforce 0 is a work around, but it's interesting this is manifesting in arm devices.
Please ignore my comments. I believe there is a bug in either the image or in one of the packages. dnf update -y pulled in cockpit-204-1.fc30.armv7hl and everything is fine after a reboot.
@Ben: Same issue with the SELinux policy for you -- cockpit-session is being denied to start the user session, in particular run bash. If this is reproducible with an official image? Maybe the last image you downloaded also updated selinux-policy? If it's fixed now, then all good, otherwise let's reopen and reassign to selinux-policy?