Bug 688949

Summary: Systemd says sshd is running, but it isn't
Product: [Fedora] Fedora Reporter: Horst H. von Brand <vonbrand>
Component: systemdAssignee: Lennart Poettering <lpoetter>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: unspecified    
Version: rawhideCC: johannbg, lpoetter, metherid, mschmidt, notting, plautrba
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-04-04 18:44:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Output of "systemctl status sshd.service" before restart
none
Output of "systemctl status sshd.service" after restart
none
Output of "systemctl status sshd.service" after restart, connection refused by sshd
none
Output of "systemctl status sshd.service" after restart, connection refused by sshd none

Description Horst H. von Brand 2011-03-18 15:52:28 UTC
Created attachment 486273 [details]
Output of "systemctl status sshd.service" before restart

Description of problem:
Today I could not connect to my remote machine. Looking around, I found that systemctl said sshd was running, but "pgrep ssh" gave nothing, and the processes reported by systemctl didn't exist. I had started screen(1) a while back via ssh, that one was still running. Restarting sshd made everything work again, I could connect to my screen(1) session.

Attached are the output from systemctl status sshd.service before and after restarting sshd.service.

Version-Release number of selected component (if applicable):
systemd-18-1.fc16.x86_64

How reproducible:
Happened once

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Horst H. von Brand 2011-03-18 15:53:01 UTC
Created attachment 486274 [details]
Output of "systemctl status sshd.service" after restart

Comment 2 Horst H. von Brand 2011-03-18 16:01:33 UTC
It is openssh-5.8p1-18.fc16.1.x86_64, installed on March 17 (but much earlier than the instant given for the start of the process in the "before" listing); I started screen(1) on March 7. systemd's install date is February 21.

screen-4.1.0-0.3.20101110git066b098.fc15.x86_64, installed February 10.

Comment 3 Michal Schmidt 2011-03-18 18:40:41 UTC
The user's processes are not supposed to be under sshd's cgroup. They should be put under systemd:/user by pam_systemd.so. Do you have "UsePAM yes" in sshd_config? Is pam_systemd.so run from /etc/pam.d/sshd or a file included from there (probably /etc/pam.d/password-auth)?

Comment 4 Horst H. von Brand 2011-03-18 19:11:38 UTC
(In reply to comment #3)
> The user's processes are not supposed to be under sshd's cgroup. They should be
> put under systemd:/user by pam_systemd.so.

Sounds reasonable.

>                                            Do you have "UsePAM yes" in
> sshd_config?

Yep.

>              Is pam_systemd.so run from /etc/pam.d/sshd

Not mentioned there

>                                                         or a file included from
> there (probably /etc/pam.d/password-auth)?

# grep -R pam_systemd /etc/pam.d
/etc/pam.d/smartcard-auth-ac:-session     optional      pam_systemd.so
/etc/pam.d/fingerprint-auth-ac:-session     optional      pam_systemd.so
/etc/pam.d/password-auth:-session     optional      pam_systemd.so
/etc/pam.d/password-auth-ac:-session     optional      pam_systemd.so
/etc/pam.d/smartcard-auth:-session     optional      pam_systemd.so
/etc/pam.d/system-auth:-session     optional      pam_systemd.so
/etc/pam.d/fingerprint-auth:-session     optional      pam_systemd.so
/etc/pam.d/system-auth-ac:-session     optional      pam_systemd.so

I now have a connection open to the machine, but it refuses further connections. Output from "systemctl status sshd.service" attached.

Comment 5 Horst H. von Brand 2011-03-18 19:12:33 UTC
Created attachment 486308 [details]
Output of "systemctl status sshd.service" after restart, connection refused by sshd

Comment 6 Horst H. von Brand 2011-03-18 19:18:07 UTC
Created attachment 486310 [details]
Output of "systemctl status sshd.service" after restart, connection refused by sshd

Sorry, mixed up ttys before.

Comment 7 Horst H. von Brand 2011-03-18 19:23:19 UTC
Restarting sshd fixed the "connection refused" from comment 6, now there is a sshd in the group.

Comment 8 Michal Schmidt 2011-03-18 19:31:54 UTC
Do you get anything about pam_systemd in /var/log/secure after logging in via ssh?

Comment 9 Horst H. von Brand 2011-03-21 14:00:46 UTC
(In reply to comment #8)
> Do you get anything about pam_systemd in /var/log/secure after logging in via
> ssh?

Mar 21 10:46:27 quelen sshd[10250]: Accepted publickey for vonbrand from 200.1.21.41 port 39742 ssh2
Mar 21 10:46:27 quelen sshd[10250]: pam_systemd(sshd:session): Moving new user session for vonbrand into control group /user/vonbrand/490.
Mar 21 10:46:27 quelen sshd[10250]: pam_unix(sshd:session): session opened for user vonbrand by (uid=0)

(sorry I didn't answer before, sshd had died on me again as in comment 6, and I was away from the machine...)

Comment 10 Horst H. von Brand 2011-03-22 17:15:24 UTC
Again, sshd died on me. I closed all the processes running under screen(1), restarted sshd (systemd-20-1.fc15.x86_64, openssh-5.8p1-18.fc16.1.x86_64, screen-4.1.0-0.3.20101110git066b098.fc15.x86_64). Now the cgroup looks sane:

# systemctl status sshd.service
sshd.service - LSB: Start up the OpenSSH server daemon
          Loaded: loaded (/etc/rc.d/init.d/sshd)
          Active: active (running) since Tue, 22 Mar 2011 11:53:41 -0300; 2h 22min ago
         Process: 24637 ExecStop=/etc/rc.d/init.d/sshd stop (code=exited, status=0/SUCCESS)
         Process: 24645 ExecStart=/etc/rc.d/init.d/sshd start (code=exited, status=0/SUCCESS)
        Main PID: 24652 (sshd)
          CGroup: name=systemd:/system/sshd.service
                  └ 24652 /usr/sbin/sshd

Will have to see if it exits again, but it looks fixed for now.

Comment 11 Lennart Poettering 2011-04-04 18:42:54 UTC
Hmm, is there anything left to fix in systemd here? Or may I close the bug?

Comment 12 Lennart Poettering 2011-04-04 18:44:49 UTC
Or, hmm, let's turn this around. I'll close this now, feel free to reopen if a problem remains!