Bug 688949 - Systemd says sshd is running, but it isn't
Summary: Systemd says sshd is running, but it isn't
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: Lennart Poettering
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-03-18 15:52 UTC by Horst H. von Brand
Modified: 2011-04-04 18:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-04-04 18:44:49 UTC
Type: ---


Attachments (Terms of Use)
Output of "systemctl status sshd.service" before restart (534 bytes, text/plain)
2011-03-18 15:52 UTC, Horst H. von Brand
no flags Details
Output of "systemctl status sshd.service" after restart (568 bytes, text/plain)
2011-03-18 15:53 UTC, Horst H. von Brand
no flags Details
Output of "systemctl status sshd.service" after restart, connection refused by sshd (299 bytes, text/plain)
2011-03-18 19:12 UTC, Horst H. von Brand
no flags Details
Output of "systemctl status sshd.service" after restart, connection refused by sshd (917 bytes, text/plain)
2011-03-18 19:18 UTC, Horst H. von Brand
no flags Details

Description Horst H. von Brand 2011-03-18 15:52:28 UTC
Created attachment 486273 [details]
Output of "systemctl status sshd.service" before restart

Description of problem:
Today I could not connect to my remote machine. Looking around, I found that systemctl said sshd was running, but "pgrep ssh" gave nothing, and the processes reported by systemctl didn't exist. I had started screen(1) a while back via ssh, that one was still running. Restarting sshd made everything work again, I could connect to my screen(1) session.

Attached are the output from systemctl status sshd.service before and after restarting sshd.service.

Version-Release number of selected component (if applicable):
systemd-18-1.fc16.x86_64

How reproducible:
Happened once

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Horst H. von Brand 2011-03-18 15:53:01 UTC
Created attachment 486274 [details]
Output of "systemctl status sshd.service" after restart

Comment 2 Horst H. von Brand 2011-03-18 16:01:33 UTC
It is openssh-5.8p1-18.fc16.1.x86_64, installed on March 17 (but much earlier than the instant given for the start of the process in the "before" listing); I started screen(1) on March 7. systemd's install date is February 21.

screen-4.1.0-0.3.20101110git066b098.fc15.x86_64, installed February 10.

Comment 3 Michal Schmidt 2011-03-18 18:40:41 UTC
The user's processes are not supposed to be under sshd's cgroup. They should be put under systemd:/user by pam_systemd.so. Do you have "UsePAM yes" in sshd_config? Is pam_systemd.so run from /etc/pam.d/sshd or a file included from there (probably /etc/pam.d/password-auth)?

Comment 4 Horst H. von Brand 2011-03-18 19:11:38 UTC
(In reply to comment #3)
> The user's processes are not supposed to be under sshd's cgroup. They should be
> put under systemd:/user by pam_systemd.so.

Sounds reasonable.

>                                            Do you have "UsePAM yes" in
> sshd_config?

Yep.

>              Is pam_systemd.so run from /etc/pam.d/sshd

Not mentioned there

>                                                         or a file included from
> there (probably /etc/pam.d/password-auth)?

# grep -R pam_systemd /etc/pam.d
/etc/pam.d/smartcard-auth-ac:-session     optional      pam_systemd.so
/etc/pam.d/fingerprint-auth-ac:-session     optional      pam_systemd.so
/etc/pam.d/password-auth:-session     optional      pam_systemd.so
/etc/pam.d/password-auth-ac:-session     optional      pam_systemd.so
/etc/pam.d/smartcard-auth:-session     optional      pam_systemd.so
/etc/pam.d/system-auth:-session     optional      pam_systemd.so
/etc/pam.d/fingerprint-auth:-session     optional      pam_systemd.so
/etc/pam.d/system-auth-ac:-session     optional      pam_systemd.so

I now have a connection open to the machine, but it refuses further connections. Output from "systemctl status sshd.service" attached.

Comment 5 Horst H. von Brand 2011-03-18 19:12:33 UTC
Created attachment 486308 [details]
Output of "systemctl status sshd.service" after restart, connection refused by sshd

Comment 6 Horst H. von Brand 2011-03-18 19:18:07 UTC
Created attachment 486310 [details]
Output of "systemctl status sshd.service" after restart, connection refused by sshd

Sorry, mixed up ttys before.

Comment 7 Horst H. von Brand 2011-03-18 19:23:19 UTC
Restarting sshd fixed the "connection refused" from comment 6, now there is a sshd in the group.

Comment 8 Michal Schmidt 2011-03-18 19:31:54 UTC
Do you get anything about pam_systemd in /var/log/secure after logging in via ssh?

Comment 9 Horst H. von Brand 2011-03-21 14:00:46 UTC
(In reply to comment #8)
> Do you get anything about pam_systemd in /var/log/secure after logging in via
> ssh?

Mar 21 10:46:27 quelen sshd[10250]: Accepted publickey for vonbrand from 200.1.21.41 port 39742 ssh2
Mar 21 10:46:27 quelen sshd[10250]: pam_systemd(sshd:session): Moving new user session for vonbrand into control group /user/vonbrand/490.
Mar 21 10:46:27 quelen sshd[10250]: pam_unix(sshd:session): session opened for user vonbrand by (uid=0)

(sorry I didn't answer before, sshd had died on me again as in comment 6, and I was away from the machine...)

Comment 10 Horst H. von Brand 2011-03-22 17:15:24 UTC
Again, sshd died on me. I closed all the processes running under screen(1), restarted sshd (systemd-20-1.fc15.x86_64, openssh-5.8p1-18.fc16.1.x86_64, screen-4.1.0-0.3.20101110git066b098.fc15.x86_64). Now the cgroup looks sane:

# systemctl status sshd.service
sshd.service - LSB: Start up the OpenSSH server daemon
          Loaded: loaded (/etc/rc.d/init.d/sshd)
          Active: active (running) since Tue, 22 Mar 2011 11:53:41 -0300; 2h 22min ago
         Process: 24637 ExecStop=/etc/rc.d/init.d/sshd stop (code=exited, status=0/SUCCESS)
         Process: 24645 ExecStart=/etc/rc.d/init.d/sshd start (code=exited, status=0/SUCCESS)
        Main PID: 24652 (sshd)
          CGroup: name=systemd:/system/sshd.service
                  └ 24652 /usr/sbin/sshd

Will have to see if it exits again, but it looks fixed for now.

Comment 11 Lennart Poettering 2011-04-04 18:42:54 UTC
Hmm, is there anything left to fix in systemd here? Or may I close the bug?

Comment 12 Lennart Poettering 2011-04-04 18:44:49 UTC
Or, hmm, let's turn this around. I'll close this now, feel free to reopen if a problem remains!


Note You need to log in before you can comment on or make changes to this bug.