+++ This bug was initially created as a clone of Bug #1755986 +++ sshd server misbehave without `setup` package openssh-server-8.0p1-3.el8.x86_64 How reproducible: always Steps to Reproduce: 1. yum remove -y setup After this package `sshd` service cannot be started/restared. Current running server refuses new connections with: > ssh -vv root.137.174 OpenSSH_7.9p1, LibreSSL 2.7.3 debug1: Reading configuration data /Users/andrei/.ssh/config debug1: /Users/andrei/.ssh/config line 71: Applying options for * debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug2: resolve_canonicalize: hostname 10.0.137.174 is address debug2: ssh_connect_direct debug1: Connecting to 10.0.137.174 [10.0.137.174] port 22. debug1: Connection established. debug1: identity file /Users/andrei/.ssh/id_rsa type 0 debug1: identity file /Users/andrei/.ssh/id_rsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.9 ssh_exchange_identification: read: Connection reset by peer Actual results: [root@ci-vm-10-0-137-174 ~]# journalctl -u sshd Sep 26 10:14:44 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: Server listening on 0.0.0.0 port 22. Sep 26 10:14:44 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug2: fd 6 setting O_NONBLOCK Sep 26 10:14:44 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug3: sock_set_v6only: set socket 6 IPV6_V6ONLY Sep 26 10:14:44 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug1: Bind to port 22 on ::. Sep 26 10:14:44 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: Server listening on :: port 22. Sep 26 10:14:44 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com systemd[1]: Started OpenSSH server daemon. Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug3: fd 7 is not O_NONBLOCK Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug1: Forked child 5275. Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug3: send_rexec_state: entering fd = 10 config len 752 Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug3: ssh_msg_send: type 0 Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[4824]: debug3: send_rexec_state: done Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[5275]: debug3: oom_adjust_restore Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[5275]: debug1: Set /proc/self/oom_score_adj to 0 Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[5275]: debug1: rexec start in 7 out 7 newsock 7 pipe 9 sock 10 Sep 26 10:15:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[5275]: fatal: Privilege separation user sshd does not exist --- Additional comment from David Cantrell on 2019-09-26 14:35:58 UTC --- Through some trial and error, we found that removing 'setup' obviously removes /etc/passwd and /etc/shadow which seem to cause havoc with sshd. My personal thought on this is that openssh should carry a Requires on 'setup' or on the specific files it needs in /etc at a bare minimum. It seems extraneous, but it's important from a testing standpoint and simulating what users do when they minimize VM images (for example). --- Additional comment from Tomas Mraz on 2019-09-26 15:09:03 UTC --- Does it really make sense? I mean removing /etc/passwd basically means no normal user resolution will work. Should we have dependency on setup for anything that uses user or group names? On the other hand in theory if nsswitch.conf does not contain files for passwd and group all the users could be stored and resolved by other mechanism and then depending on setup and/or the existence of /etc/passwd and /etc/shadow actually does not make sense at all. Really just arbitrarily adding dependency on setup into openssh-server does not make much sense to me. --- Additional comment from Jakub Jelen on 2019-09-26 15:32:20 UTC --- Can you provide more information what leads to you removing setup package? The report as is stated now does not sound like something that would make sense to fix. The user sshd is explicitly stated in the spec file to be created on the installation. If you basically break the system afterward by removing this user, it is nothing sshd can fix, except for adding the explicit dependency. But if you would like to go that way (system-user = dependency on setup package), I would suggest you to start with Fedora and packaging guidelines: https://docs.fedoraproject.org/en-US/packaging-guidelines/ --- Additional comment from Andrei Stepanov on 2019-09-26 16:05:26 UTC --- > Can you provide more information what leads to you removing setup package? It is very easy With current: cat /etc/yum.conf clean_requirements_on_remove=True dnf will remove `setup` package in 'Removing unused dependencies' section. About nsswitch.conf: I doubt that the sshd-system-user-account will be added to remove provider if its USERID is bellow some value. --- Additional comment from Andrei Stepanov on 2019-09-26 16:06:15 UTC --- remote* provider --- Additional comment from Tomas Mraz on 2019-09-27 06:23:48 UTC --- Then I would suggest to put the requires on /etc/passwd on the package that owns /usr/lib64/libnss_files.so.2 which is glibc to avoid the removal of the setup package this way. --- Additional comment from Andrei Stepanov on 2019-09-27 10:03:21 UTC --- I agree with Tomas (comment 6). There are also fails from other system components: 1. Sep 26 10:27:31 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com sshd[5283]: fatal: Privilege separation user sshd does not exist 2. Sep 26 10:19:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com systemd-tmpfiles[5277]: [/usr/lib/tmpfiles.d/var.conf:15] Unknown group 'utmp'. 3. Sep 26 10:19:52 ci-vm-10-0-137-174.hosted.upshift.rdu2.redhat.com systemd-tmpfiles[5277]: Failed to parse ACL "d:group:adm:r-x,d:group:wheel:r-x": Invalid argument. Ignoring Whole system functionality is under question. Re-assign to glibc. --- Additional comment from Carlos O'Donell on 2019-09-27 20:44:18 UTC --- The dependency is already there. -> openssh-server requires libc.so.6()(64bit) -> glibc requires basesystem e.g. Requires(pre): basesystem -> basesystem requires setup e.g. Requires(pre): setup filesystem So if you remove setup, which owns /etc/passwd, then you have to remove basesystem and glibc, and nothing works. The bug is that setup was removed. Let us ask the dnf maintainers what they think. Assigning to them. --- Additional comment from Florian Weimer on 2019-09-27 20:47:50 UTC --- Requires(pre): dependencies are only used for the scriptlet, they can be removed after installation. We probably need both. However, this change should be prototyped in Fedora. I don't feel like adding new dependency loops after a release. --- Additional comment from Carlos O'Donell on 2019-09-27 21:37:59 UTC --- (In reply to Florian Weimer from comment #9) > Requires(pre): dependencies are only used for the scriptlet, they can be > removed after installation. We probably need both. You are totally right. I had forgotten that Require(pre) could allow removal after install. > However, this change should be prototyped in Fedora. I don't feel like > adding new dependency loops after a release. Why does basesystem use Requires(pre) also? Why do we do to the trouble of making them all removable after install. It seems like there is some history here we don't quite understand. I'm looking to the dnf team to comment on the history here, because it seems like glibc and basesystem have both been using Requires(pre) in this way. Perhaps to avoid dependency loops? --- Additional comment from Jaroslav Mracek on 2019-09-30 14:09:05 UTC --- I believe that many packages uses Require(pre) incorrectly, but mostly there is no problem with it because dependency trees are redundant. I also think that reproducer is quite strange. Setup is mandatory package of Core group therefore it is top in dependency tree. Removal of core component is more that unexpected from user side. I also consider that a dependency between the setup package and other packages could be improved. When one package requires for its functionality other package then it must be required or could be conditionally required using richdeps. In case that relation is not clear, weakdeps could be used. But this a subject for discussions between developers and maintainers of related components. From DNF side this is not a bug.
DJ, Could you please have a look at a rawhide build with Requires(Pre): setup changed to Requires: setup. They key issue here is going to be testing an upgrade and downgrade on a box with that glibc. Then if that works move it into Rawhide and see what breaks. Thanks.
Those pre-dependencies are only good for breaking dependency loops, and that's probably the reason it's there for in the first place. The question is: will glibc work if setup happens to get installed after it, due to dependency loops being cut? There's nothing wrong with adding a second requires without the (pre) qualifier to fix the setup package being removable, that might well be the closest thing to truth considering what it is. Another possibility is using Requires(pre,post) which will also force it to stay after installation.
RPM 4.8 removed Prereq: and added Requires(pre) but they do different things. GLIBC 2.7.90-8 converted from Prereq to Requires(pre) but not quite correctly. The nscd package has its own pre/post requires; of the remaining subpackages, only glibc-headers has a %pre that looks at the filesystem, all other scriptlets (besides nscd) happen while glibc is "installed". So I suspect the following patch is correct, which sets a general dependency on basesystem, as well as a Requires(pre) in glibc-headers on it. diff --git a/glibc.spec b/glibc.spec index 11865a4..ab19929 100644 --- a/glibc.spec +++ b/glibc.spec @@ -174,7 +174,7 @@ Requires: glibc-common = %{version}-%{release} # Various components (regex, glob) have been imported from gnulib. Provides: bundled(gnulib) -Requires(pre): basesystem +Requires: basesystem # This is for building auxiliary programs like memusage, nscd # For initial glibc bootstraps it can be commented out @@ -356,6 +356,7 @@ which is highly discouraged. Summary: Header files for development using standard C libraries. Provides: %{name}-headers(%{_target_cpu}) Requires(pre): kernel-headers +Requires(pre): basesystem Requires: kernel-headers >= 2.2.1, %{name} = %{version}-%{release} BuildRequires: kernel-headers >= 3.2
> RPM 4.8 removed Prereq: and added Requires(pre) but they do different things. Rpm 4.8 did no such thing, Prereq was deprecated around and the fine-grained qualifiers added somehwere around rpm 4.0 - 4.1. 4.8 might've made the deprecation more verbose. Requires(pre) (and other similar "scriptlet dependencies") are only ever required to break dependency loops for ordering. When no loops are present, the order is unaffected. The catch with Requires(pre) is that it tells rpm that such a dependency can be removed once the install has completed, which is fine if it's a scriptlet-only dependency which is sometimes the case, but if the pre-dependency is to break a dependency loop then it's not. The glibc basesystem dependency is of the latter kind, there's a well-known and fragile dependency loop between the "early bootstrap" packages, I'd think twice before removing it. Like I noted in comment #2, I'd suggest either adding a second requires on basesystem without the "pre", or turn it into Requires(pre,postun) which prevents it from being removed after installation as well. For glibc-headers there's no point in depending on basesystem, but there is a missing Requires(pre) is coreutils because it actually uses 'rm' there.
I certainly don't mind putting in all the requires, as a belt-and-suspenders solution. I just didn't want to do it blindly ;-)
This is fixed in Fedora Rawhide with this commit: commit eb580e4bc609c9b07278527cd90a74db548a23e2 Author: DJ Delorie <dj> Date: Thu Oct 24 13:05:55 2019 -0400 Add Requires on basesystem for main package (#1757267) Add Requires on coreutils for glibc-headers (uses rm) The solution is for glibc to require basesystem.