Bug 2172956 - Update hostkey migration to support OSTree systems and dynamically detect hostkey location
Summary: Update hostkey migration to support OSTree systems and dynamically detect hos...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openssh
Version: 38
Hardware: All
OS: Linux
high
high
Target Milestone: ---
Assignee: Dmitry Belyavskiy
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedFreezeException AcceptedBlocker
Depends On:
Blocks: F38BetaFreezeException F38FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2023-02-23 15:29 UTC by Dusty Mabe
Modified: 2023-03-30 00:20 UTC (History)
13 users (show)

Fixed In Version: openssh-9.0p1-11.fc39.1 openssh-9.0p1-12.fc38.1 openssh-9.0p1-14.fc38.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-03-30 00:20:50 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FC-762 0 None None None 2023-02-23 15:31:50 UTC

Description Dusty Mabe 2023-02-23 15:29:15 UTC
Description of problem:

In https://src.fedoraproject.org/rpms/openssh/c/b615362fd0b4da657d624571441cb74983de6e3f?branch=rawhide we did a migration for host key permissions. We need to make this work for OSTree systems that don't run scriptlets on upgrade and also we need to handle not default hostkey locations that the user may have configured.

There is an open PR with a suggested change to fix this in https://src.fedoraproject.org/rpms/openssh/pull-request/39

I'm opening this bug so we can submit it as a Freeze Exception for Fedora 38 beta.

Comment 1 Fedora Blocker Bugs Application 2023-02-23 15:30:48 UTC
Proposed as a Freeze Exception for 38-beta by Fedora user dustymabe using the blocker tracking app because:

 Would be nice to get the updated migration script tested in beta.

Comment 2 Ben Cotton 2023-02-27 16:05:43 UTC
What's the impact of not granting an FE and letting this come in as a post-release update?

Comment 3 Fedora Update System 2023-03-03 16:36:52 UTC
FEDORA-2023-c884d35083 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-c884d35083

Comment 4 Fedora Update System 2023-03-03 16:37:39 UTC
FEDORA-2023-c884d35083 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 5 Fedora Update System 2023-03-03 16:48:51 UTC
FEDORA-2023-e2b6da44e4 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-e2b6da44e4

Comment 6 Dusty Mabe 2023-03-03 17:19:18 UTC
(In reply to Ben Cotton from comment #2)
> What's the impact of not granting an FE and letting this come in as a
> post-release update?

I'm not 100% sure. My goal here is to try to get as much testing as possible on this "migration" because the failure scenario is (possibly) a system the user can't remotely connect to because sshd doesn't come up. Depending on the server's environment this could be OK or it could be critical (i.e. no physical access and no console access).

This migration shouldn't matter much on the new system install workflow since the hostkeys should get created with correct permissions on first boot. The upgrade case is where we have concern.

Comment 7 Dusty Mabe 2023-03-03 17:20:06 UTC
This also requires https://src.fedoraproject.org/rpms/fedora-release/pull-request/252 to merge and be built for f38 and added to the update (I think).

Comment 8 Adam Williamson 2023-03-03 18:13:54 UTC
Yeah, a bug proposed as FE for F38 should not be closed by an F39 update, so re-opening. It'd be good to have some clarity on this (what's the problem, what does the solution involve, what are the risks?)

Comment 9 Dusty Mabe 2023-03-03 18:28:25 UTC
Noting here that https://src.fedoraproject.org/rpms/openssh/pull-request/40 is the ultimate PR that merged helping to improve the behavior and close this bug.

Comment 10 Dusty Mabe 2023-03-03 18:32:43 UTC
(In reply to Adam Williamson from comment #8)
> Yeah, a bug proposed as FE for F38 should not be closed by an F39 update, so
> re-opening. It'd be good to have some clarity on this (what's the problem,
> what does the solution involve, what are the risks?)

[The Problem]
In support of https://fedoraproject.org/wiki/Changes/SSHKeySignSuidBit the openssh RPM was modified to migrate host key permissions to be less permissive. The previous migration implementation had no consideration for OSTree systems or non-default host key locations, which leaves a system inaccessibly via SSH after upgrade. See https://github.com/coreos/fedora-coreos-tracker/issues/1394

[The Solution]
Deliver the migration code as a script that can be called via scriptlets or a new systemd unit. Augment the migration code to support detecting the location of host keys and operating on those rather than just operating on the default locations.

[The Risk]
None for newly installed systems. For upgrading systems it's possible the new code has a flaw that will need to be fixed, but this will go in to final anyway, so we might as well get some good testing on it for beta.

Comment 11 Fedora Update System 2023-03-04 02:35:01 UTC
FEDORA-2023-e2b6da44e4 has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-e2b6da44e4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Adam Williamson 2023-03-05 17:20:51 UTC
+3 in https://pagure.io/fedora-qa/blocker-review/issue/1050 , marking accepted FE.

Comment 13 Fedora Update System 2023-03-07 02:02:50 UTC
FEDORA-2023-e2b6da44e4 has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-e2b6da44e4

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 14 Fedora Update System 2023-03-07 08:54:24 UTC
FEDORA-2023-fdf3721184 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-fdf3721184

Comment 15 Adam Williamson 2023-03-07 16:00:58 UTC
What's going on with the edits to multiple updates here? We put fedora-release-38-0.30 and openssh-9.0p1-11.fc38.1 in Beta; is there a problem with those builds?

Comment 16 Dmitry Belyavskiy 2023-03-07 16:12:04 UTC
-12 (pushed today) contains some minor fixes so it would be better to respin.

Comment 17 Dusty Mabe 2023-03-07 16:25:46 UTC
Hey Adam,

openssh-9.0p1-12.fc38.1 has [1], which implemented a few late code review comments from [2]. They were mostly improvements and not 100% required so even if we don't get `-12` we should be OK, but if we do another RC let's try to pull it in.

[1] https://src.fedoraproject.org/rpms/openssh/pull-request/41 
[2] https://src.fedoraproject.org/rpms/openssh/pull-request/40

Comment 18 Fedora Update System 2023-03-08 02:24:40 UTC
FEDORA-2023-fdf3721184 has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-fdf3721184

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Fedora Update System 2023-03-09 22:53:35 UTC
FEDORA-2023-fdf3721184 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 Timothée Ravier 2023-03-14 13:45:03 UTC
This is not fixed for Silverblue/Kinoite (and likely IoT).

https://src.fedoraproject.org/rpms/fedora-release/pull-request/252#comment-133960

Comment 21 Fedora Blocker Bugs Application 2023-03-14 13:50:49 UTC
Proposed as a Blocker and Freeze Exception for 38-final by Fedora user siosm using the blocker tracking app because:

 This bugs breaks remote ssh access to systems. While only inconvenient for Silverblue/Kinoite, this is critical for IoT systems.

Comment 22 Colin Walters 2023-03-14 13:53:42 UTC
To enable the service, I'd suggest having the sshd.service have
Wants=openssh-key-migration.service

Comment 23 Timothée Ravier 2023-03-14 13:56:17 UTC
That should work too indeed.

Comment 24 Jonathan Lebon 2023-03-14 14:18:15 UTC
I think what we missed here was that for this to work, we needed to also add the systemd unit to this `%systemd_post` invocation: https://src.fedoraproject.org/rpms/openssh/blob/3a98e6f607c0b35eccb4fefb7a37ef98fe53b375/f/openssh.spec#_640, which is what actually applies the preset and creates the enablement symlink.

Anyway, using Wants= does indeed sound like a simpler and surer way to do this (and will make it easier to drop eventually).

Comment 25 Timothée Ravier 2023-03-14 16:08:07 UTC
First part in https://src.fedoraproject.org/rpms/fedora-release/pull-request/253

Comment 26 Timothée Ravier 2023-03-14 16:18:51 UTC
Second part in: https://src.fedoraproject.org/rpms/openssh/pull-request/45

Comment 27 Timothée Ravier 2023-03-14 16:19:36 UTC
Needs testing now. Will try to get to it.

Comment 28 Adam Williamson 2023-03-15 23:40:04 UTC
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1050 , marking accepted.

Comment 29 Adam Williamson 2023-03-23 17:46:38 UTC
Ping - did you get a chance to test this, Timothée? It's a Final blocker and final freeze is in a couple of weeks, so it'd be good to see some movement here. Thanks!

Comment 30 Timothée Ravier 2023-03-24 13:08:47 UTC
I've tested this change and posted my notes in https://src.fedoraproject.org/rpms/openssh/pull-request/45#comment-134454

I'm not a proven packager so I need an openssh package maintainer or a proven packager to merge this and push things further.

Comment 31 Dusty Mabe 2023-03-24 13:46:50 UTC
@dbelyavs - could you review the PR?

Comment 32 Adam Williamson 2023-03-24 15:22:09 UTC
Thanks a lot Timothée! I'm a provenpackager so could merge/build if necessary, but it'd be best for the real packager to do it if possible.

Don't we also need a fedora-release maintainer to merge/build the fedora-release part?

Comment 33 Dusty Mabe 2023-03-24 16:15:23 UTC
(In reply to Adam Williamson from comment #32)
> Thanks a lot Timothée! I'm a provenpackager so could merge/build if
> necessary, but it'd be best for the real packager to do it if possible.

Agree, hopefully they will respond soon.

> 
> Don't we also need a fedora-release maintainer to merge/build the
> fedora-release part?

Yes, but I think the changes need to ship together so we don't regress at least in Fedora CoreOS where presets *are* evaluated and added on upgrades.

Comment 34 Fedora Update System 2023-03-27 12:35:07 UTC
FEDORA-2023-bb12d0efe6 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-bb12d0efe6

Comment 35 Adam Williamson 2023-03-27 17:05:01 UTC
we now have an update with openssh but not fedora-release , do we need to merge the fedora-release change and add it to the update? I can try and get that taken care of.

Comment 36 Adam Williamson 2023-03-27 17:05:29 UTC
I disabled autopush on the update for now.

Comment 37 Dusty Mabe 2023-03-27 17:36:17 UTC
(In reply to Adam Williamson from comment #35)
> we now have an update with openssh but not fedora-release , do we need to
> merge the fedora-release change and add it to the update? I can try and get
> that taken care of.

That would be great! Thanks Adam.

Comment 38 Fedora Update System 2023-03-28 03:42:47 UTC
FEDORA-2023-bb12d0efe6 has been pushed to the Fedora 38 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-bb12d0efe6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 39 Fedora Update System 2023-03-30 00:20:50 UTC
FEDORA-2023-bb12d0efe6 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.