Hi Sabdro,

Unfortunately is not easy to fix , I think we need update containerd to 1.7.x

can you check if contained 1.7.0 from https://copr.fedorainfracloud.org/coprs/sergiomb/docker2/builds/ works for you ?

we have a lot of stuff to update ...


more info about the state here: 

https://src.fedoraproject.org/rpms/containerd/pull-request/15 
https://bodhi.fedoraproject.org/updates/FEDORA-2023-aaa2b3d20b

Thanks for your quick reply, unfortunately I get the same errors with containerd-1.7.0-5.fc39.x86_64

Actually correction, with containerd-1.7.0-5.fc39.x86_64 it works. If I can help in any way getting things updated in Fedora proper, let me know.

*** Bug 2239849 has been marked as a duplicate of this bug. ***

Discussed during the 2023-09-25 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug was made as we would like a clearer rationale for why it would be beneficial to grant an FE to this bug, given that containerd is not used by podman and probably isn't on any media aside from CoreOS. Our question: what does an FE achieve here that a 0-day update would not?

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2023-09-25/f39-blocker-review.2023-09-25-16.02.txt

Note: this is being considered as a Freeze Exception and *not* a Blocker Bug as stated above.

I'm not sure if this bit of info would or should play into the calculation, but for upgrading systems people theoretically upgrade from a working `docker` setup (f38) to one that won't work (f39).

but F38 (and F37) also have the same problems , anyone is going fix it ? , ATM I don't have time .

What needs to be done? Is it safe to just import all the builds from [1] to fedora? If so, I can help.


[1] https://copr.fedorainfracloud.org/coprs/sergiomb/docker2/builds/

(In reply to Sergio Basto from comment #8)
> but F38 (and F37) also have the same problems , anyone is going fix it ? ,
> ATM I don't have time .

The nuance here is that the updates for f37 and f38 never made it to `stable` repos. So users on those platforms are probably using `docker` without issue right now.

- https://bodhi.fedoraproject.org/updates/FEDORA-2023-5c4718e547
- https://bodhi.fedoraproject.org/updates/FEDORA-2023-aaa2b3d20b

When they switch to F39 is when they would have trouble.

> (In reply to Dusty Mabe from comment #10)
> (In reply to Sergio Basto from comment #8)
> > but F38 (and F37) also have the same problems , anyone is going fix it ? ,
> > ATM I don't have time .
> 
> The nuance here is that the updates for f37 and f38 never made it to
> `stable` repos. So users on those platforms are probably using `docker`
> without issue right now.
> 
> - https://bodhi.fedoraproject.org/updates/FEDORA-2023-5c4718e547
> - https://bodhi.fedoraproject.org/updates/FEDORA-2023-aaa2b3d20b
> 
> When they switch to F39 is when they would have trouble.

you need build containerd 1.6 with golang-github-containerd-ttrpc 1.1.0 to work,  can you try https://koji.fedoraproject.org/koji/buildinfo?buildID=2236784  ? it have 

DEBUG util.py:444:   golang-github-containerd-ttrpc-devel                              noarch  1:1.1.0-5.fc39                           build   38 k


@zebob.m should have a better status of containd 1.7 than me

(In reply to Sergio Basto from comment #11)
> > 
> > When they switch to F39 is when they would have trouble.
> 
> you need build containerd 1.6 with golang-github-containerd-ttrpc 1.1.0 to

For Fedora CoreOS we do have the ability to pin on older RPMs and we
pinned on 1.6.19-1.fc39 in https://github.com/coreos/fedora-coreos-config/pull/2624

We didn't do 1.6.19-2.fc39, which was the one done by releng for the
F39 mass rebuild because I didn't see it in the list of updates for
F39: https://bodhi.fedoraproject.org/updates/?search=&packages=containerd&releases=F39

I'll bump our pin to 1.6.19-2.fc39 now, but this doesn't help the
rest of Fedora that isn't Fedora CoreOS. In order to do that we'd
need to file a ticket with releng to untag the 1.6.23-1.fc39 build
so that 1.6.19-2.fc39 is the latest.

> @zebob.m should have a better status of containd 1.7 than me

I'll add needinfo to see if they can weigh in here.

"I'm not sure if this bit of info would or should play into the calculation, but for upgrading systems people theoretically upgrade from a working `docker` setup (f38) to one that won't work (f39)."

That does not really affect FE status, since being an FE does not guarantee a bug will be fixed. It simply *allows* for it. If there is a fix but we don't give it an FE, it would go out on day 0, and people upgrading on release day would still get the fixed version. If there wasn't a fix, the bug having an FE wouldn't make any difference.

There's a weak argument to give it an FE to avoid a regression for people upgrading *before* release day, but that's kinda the same for any bug, so...

For containerd we have the following review requests:

containerd:


github.com/container-orchestrated-devices/container-device-interface
github.com/containerd/cgroups/v3
github.com/containerd/nri

golang-github-container-orchestrated-devices-device-interface https://bugzilla.redhat.com/show_bug.cgi?id=2229406
golang-github-containerd-cgroups-3 https://bugzilla.redhat.com/show_bug.cgi?id=2229425




golang-github-containerd-nri:

github.com/containers/common
github.com/r3labs/diff/v3
github.com/sters/yaml-diff

golang-github-containers-common  https://bugzilla.redhat.com/show_bug.cgi?id=2229811
golang-github-r3labs-diff-3 https://bugzilla.redhat.com/show_bug.cgi?id=2229465
golang-github-sters-yaml-diff https://bugzilla.redhat.com/show_bug.cgi?id=2229466



golang-github-containers-common:


github.com/containers/storage
github.com/containers/image/v5
github.com/disiqueira/gotree/v3

golang-github-containers-storage https://bugzilla.redhat.com/show_bug.cgi?id=2229483
golang-github-containers-image-5 https://bugzilla.redhat.com/show_bug.cgi?id=2229518
golang-github-disiqueira-gotree-3 https://bugzilla.redhat.com/show_bug.cgi?id=2229481


golang-github-containers-storage:

github.com/google/go-intervals
github.com/mistifyio/go-zfs/v3

golang-github-google-intervals https://bugzilla.redhat.com/show_bug.cgi?id=2229478
golang-github-mistifyio-zfs-3 https://bugzilla.redhat.com/show_bug.cgi?id=2229479


golang-github-containers-image-5:

dario.cat/mergo
github.com/containers/libtrust
github.com/cyberphone/json-canonicalization
github.com/proglottis/gpgme
github.com/vbauerster/mpb/v8

golang-dario-mergo https://pagure.io/releng/fedora-scm-requests/issue/55275 B+
golang-github-containers-libtrust https://bugzilla.redhat.com/show_bug.cgi?id=2229469
golang-github-cyberphone-json-canonicalization https://bugzilla.redhat.com/show_bug.cgi?id=2229470
golang-github-proglottis-gpgme https://bugzilla.redhat.com/show_bug.cgi?id=2229471
golang-github-vbauerster-mpb-8 https://bugzilla.redhat.com/show_bug.cgi?id=2229477

I have reviewed  golang-github-container-orchestrated-devices-device-interface https://bugzilla.redhat.com/show_bug.cgi?id=2229406 
golang-github-sters-yaml-diff https://bugzilla.redhat.com/show_bug.cgi?id=2229466 

since containerd-1.7.0 seems work better I'm trying for now just bring containerd-1.7.0 to Fedora (containrd-1.7.4 needs even more new packages) 
I start build container 1.7.0 in a new copr repo: https://copr.fedorainfracloud.org/coprs/sergiomb/docker4/builds/

from my tests I did this report :  https://sergiomb.fedorapeople.org/docker_report3.txt

to build containerd we need  golang-opentelemetry-contrib [1] and we need update k8s.io/apimachinery and k8s.io/cri-api [2]


[1]
to unbootstrap golang-opentelemetry-contrib needs:
  golang-github-docker 	23.0.4-1
  new package  golang-github-moby-pubsub  1.0.0-1 (
  new package  golang-github-azure-ansiterm 0-0.1.20230420git306776e
  golang-github-docker-devel-23.0.4-1 built
  No matching package to install: 'golang(github.com/brunoscheufler/aws-ecs-metadata-go)'
  new package golang-github-brunoscheufler-aws-ecs-metadata-0-0.1.20230425git67e37ae


[2]
# update golang-github-googleapis-gnostic to 0.5.5-1 (not sure if it needed)
package rename from golang-github-googleapis-gnostic to golang-github-google-gnostic and update to 0.6.9-1
  new package golang-sigs-k8s-json-0-0.1.20230426gitbc3834c
  new package golang-github-flowstack-jsonschema-0.1.2-1

update golang-k8s-apimachinery to 1.26.4
update and bootstrap golang-k8s-apiserver to 1.26.4 (apiserver-kubernetes)
update golang-k8s-client to 1.26.4 with bootstrap %global __requires_exclude %{?__requires_exclude:%{__requires_exclude}|}^golang\\(.*\\)$
update golang-k8s-cri-api to 1.26.4
update golang-k8s-klog to 2.90.1-1.fc39

-4 FE in https://pagure.io/fedora-qa/blocker-review/issue/1345 , marking rejected. We feel there's a lot going on here and it'd be more suited to a 0-day update.

I also have this issue with F39 workstation beta.

Downgrading helps:

```
sudo dnf install ~/Downloads/containerd-1.6.19-2.fc39.x86_64.rpm
```

*** Bug 2246836 has been marked as a duplicate of this bug. ***

I just ran into this after upgrading from Fedora 38 to Fedora 39 and containerd-1.6.23-1.fc39.x86_64 was installed. Downgrading to containerd-1.6.19-2.fc39.x86_64 fixed it.

I then looked at my other working Fedora 39 and Kubernetes server where I had the docke-ce repo. With it I had containerd.io-1.6.24-3.1.fc39.x86_64, and then using that version on the original server I had an issue with works.

(In reply to Nathan G. Grennan from comment #19)
> I just ran into this after upgrading from Fedora 38 to Fedora 39 and
> containerd-1.6.23-1.fc39.x86_64 was installed. Downgrading to
> containerd-1.6.19-2.fc39.x86_64 fixed it.
> 
> I then looked at my other working Fedora 39 and Kubernetes server where I
> had the docke-ce repo. With it I had containerd.io-1.6.24-3.1.fc39.x86_64,
> and then using that version on the original server I had an issue with works.

  I ended up finding using containerd.io-1.6.24-3.1.fc39.x86_64 on this system only half worked, and ended up downgrading to containerd-1.6.19-2.fc39.x86_64. On the other system the actual runtime is docker, but on this system it is moby-engine. Mixing containerd.io from docker with moby-engine caused a different error with kubelet that prevented it from coming up properly. At least with Fedora 39 I can just use the kubernetes packages that came with Fedora instead of having to build my own.

Same error for a docker build command since F39 upgrade

*** Bug 2246041 has been marked as a duplicate of this bug. ***

(In reply to Nathan G. Grennan from comment #19)
> I just ran into this after upgrading from Fedora 38 to Fedora 39 and
> containerd-1.6.23-1.fc39.x86_64 was installed. Downgrading to
> containerd-1.6.19-2.fc39.x86_64 fixed it.

I just upgraded from Fedora 38 to Fedora 39 and can confirm, downgrading to containerd-1.6.19-2.fc39.x86_64 fixes the issue.

The actual package I used can be found here:
https://koji.fedoraproject.org/koji/packageinfo?packageID=25455

(In reply to bugzilla.redhat from comment #23)
> (In reply to Nathan G. Grennan from comment #19)
> > I just ran into this after upgrading from Fedora 38 to Fedora 39 and
> > containerd-1.6.23-1.fc39.x86_64 was installed. Downgrading to
> > containerd-1.6.19-2.fc39.x86_64 fixed it.
> 
> I just upgraded from Fedora 38 to Fedora 39 and can confirm, downgrading to
> containerd-1.6.19-2.fc39.x86_64 fixes the issue.
> 
> The actual package I used can be found here:
> https://koji.fedoraproject.org/koji/packageinfo?packageID=25455

Exactly the same for me. Downgrading also fixed the issue.

What is the plan? If per comment 15 upgrade of containerd to 1.7 needs some dependencies that are not in Fedora right now, can we just downgrade to the containerd-1.6.19-2 while bumping epoch, to provide people with the revert they are after?

I'm waiting for some help, I think you are making some confusion, we can't build  containerd 1.7 without dependencies , and rollback ttrpc can break other things I guess

Asking two questions about the plan after over a month after the last status comment causes confusion?

Is there an indication that the help you are waiting for is actually on its way? I wouldn't want to see us still waiting for help many months later.

Is the breakage caused by downgrade from 1.6.23 to 1.6.19 bigger than the current situation when the 1.6.23 setup is broken?

I added the https://copr.fedorainfracloud.org/coprs/sergiomb/docker2/ repo and installed the containerd package from there. Haven't seen adverse behavior.

Interesting to see that a package which is part of the distribution is supported like this --> that breakage is being accepted. If podman is the preferred way to go, then why not remove containerd and provide a comprehensive guide to migrate to podman?

I second the question as to what the plan is; is a downgrade to 1.6.19 not feasible because of issues with other packages?

I realize that the trend is going away from docker, but many of us have existing tools and scripts that rely on docker as it exists today, and having "docker run" work is basic and important in my circles, where the word of mouth atm is to hold off on f39.

(In reply to Stefan van der Eijk from comment #28)
> I added the https://copr.fedorainfracloud.org/coprs/sergiomb/docker2/ repo
> and installed the containerd package from there. Haven't seen adverse
> behavior.

I did that ( https://copr.fedorainfracloud.org/coprs/sergiomb/docker2/ ) with a very big effort and which cost me some family and work problems because I was very tired and even exhausted.

I know very little or nothing about the go language, go as it is packaged, has thousands of dependencies and some of them circular, after package reviews stop at the smallest details.

I want bring container 1.7.0 to Fedora for start , I think the problem of docker environment is get rid of cgroupsv2 and use cgroupsv3 , I think it was a security issue but I'm just speculating . Now it seems to me that updates are more slowly and peaceful, but we need to move all to container 1.7 and docker 24.0.0

I did a new repo  https://copr.fedorainfracloud.org/coprs/sergiomb/docker4/builds/ which is not finished with a refresh of the build that we need 


> Interesting to see that a package which is part of the distribution is
> supported like this --> that breakage is being accepted. If podman is the
> preferred way to go, then why not remove containerd and provide a
> comprehensive guide to migrate to podman?

I use docker on  my work and my employee don't want to change , so is not an option work with podman

Sergio: I think the idea is, since updating to 1.7.0 definitely is not trivial but people kinda need docker to work now, we should *first* ship a downgrade to 1.6.19 just to get things working again for folks, then we have time to get to 1.7.0 whenever we can get the pile of go deps packaged up.

Alternatively - maybe someone could figure out exactly what change between 1.6.19 and 1.6.23 broke things here, and we can see if it's feasible to just revert that? Unfortunately it seems like the patches in the package are a bit sensitive to version bumps, now I've started trying to do this...

(In reply to Adam Williamson from comment #31)
> Sergio: I think the idea is, since updating to 1.7.0 definitely is not
> trivial but people kinda need docker to work now, we should *first* ship a
> downgrade to 1.6.19 just to get things working again for folks, then we have
> time to get to 1.7.0 whenever we can get the pile of go deps packaged up.
> 
> Alternatively - maybe someone could figure out exactly what change between
> 1.6.19 and 1.6.23 broke things here, and we can see if it's feasible to just
> revert that? Unfortunately it seems like the patches in the package are a
> bit sensitive to version bumps, now I've started trying to do this...

ah for that, ( I already explained that several times but maybe I miss that in this bug report), we need rollback golang-github-containerd-ttrpc to 1.1.0 and rebuild containerd against it 

I did that already once https://src.fedoraproject.org/rpms/golang-github-containerd-ttrpc/c/e755e8bd10893cad974c506e4350ea7ccf2d34e6?branch=rawhide see https://src.fedoraproject.org/rpms/golang-github-containerd-ttrpc/commits/rawhide 

maybe we should try it again :) beacause only package that depends on it are cadvisor and containerd !

Also running into this issue, would greatly appreciate a fix 😇

For the folks that critically need to get things to work again I wanted to share this post that mentioned reinstalling docker that worked for me as an alternative to downgrading https://discussion.fedoraproject.org/t/docker-failed-to-create-shim-task-ttrpc/95902/3

There are official package (moby-engine). Install from docker repository can only be a workaround

Folks, this caused a regression on upgrading to Fedora 39! Right now Fedora 39 is non-functional with moby-engine and there is no way to downgrade.
Bug #2251293

How come there is no way to downgrade? Comment 23 points to where to take the downgrade rpms from in the Fedora build system. Specifically for containerd-1.6.19-2 the exact location would be https://koji.fedoraproject.org/koji/buildinfo?buildID=2236784.

(In reply to Jan Pazdziora from comment #37)
> How come there is no way to downgrade? Comment 23 points to where to take
> the downgrade rpms from in the Fedora build system. Specifically for
> containerd-1.6.19-2 the exact location would be
> https://koji.fedoraproject.org/koji/buildinfo?buildID=2236784.

Because this version is not available in the repository:

```
# dnf downgrade containerd --refresh
Copr repo for karellen-sysbox owned by karellen                                                                                                                                                             3.4 kB/s | 1.5 kB     00:00    
Copr repo for karellen-sysbox-ghar owned by karellen                                                                                                                                                        4.5 kB/s | 1.5 kB     00:00    
Copr repo for smartmontools-next owned by karellen                                                                                                                                                          7.8 kB/s | 1.5 kB     00:00    
Fedora 39 - x86_64                                                                                                                                                                                           65 kB/s |  24 kB     00:00    
Fedora 39 openh264 (From Cisco) - x86_64                                                                                                                                                                    4.3 kB/s | 989  B     00:00    
Fedora 39 - x86_64 - Updates                                                                                                                                                                                115 kB/s |  21 kB     00:00    
Package containerd of lowest version already installed, cannot downgrade it.
Dependencies resolved.
Nothing to do.
Complete!
```

From comment #1

can you check if contained 1.7.0 from https://copr.fedorainfracloud.org/coprs/sergiomb/docker2/builds/ works for you ?

Comment 3 Sandro Mani 2023-09-14 20:27:49 WEST

Actually correction, with containerd-1.7.0-5.fc39.x86_64 it works. If I can help in any way getting things updated in Fedora proper, let me know.



I prefer try go forward

(In reply to Arcadiy Ivanov from comment #36)
> Folks, this caused a regression on upgrading to Fedora 39! Right now Fedora
> 39 is non-functional with moby-engine and there is no way to downgrade.
> Bug #2251293

There is a way to downgrade, by installing the previous version of containerd directly from its Koji build:

    dnf install https://kojipkgs.fedoraproject.org//packages/containerd/1.6.19/2.fc39/x86_64/containerd-1.6.19-2.fc39.x86_64.rpm

I do that every time I launch `dnf upgrade` forgetting the option `--exclude containerd`.

Could containerd-1.6.19-2.fc39 be pushed in the Fedora repos and version 1.6.23-1.fc39 removed? That would at least avoid the growth of the number of victims. A few colleges of mines at work get bitten by that bug. I even hint not to upgrade to Fedora 39, for now.

(In reply to Laurent Rineau from comment #40)
> (In reply to Arcadiy Ivanov from comment #36)
> > Folks, this caused a regression on upgrading to Fedora 39! Right now Fedora
> > 39 is non-functional with moby-engine and there is no way to downgrade.
> > Bug #2251293
> 
> There is a way to downgrade, by installing the previous version of
> containerd directly from its Koji build:
> 
>     dnf install
> https://kojipkgs.fedoraproject.org//packages/containerd/1.6.19/2.fc39/x86_64/
> containerd-1.6.19-2.fc39.x86_64.rpm
> 
> I do that every time I launch `dnf upgrade` forgetting the option `--exclude
> containerd`.

```
dnf install 'dnf-command(versionlock)'
# dnf versionlock add containerd
Last metadata expiration check: 2:58:13 ago on Thu 30 Nov 2023 10:02:24 AM EST.
Adding versionlock on: containerd-0:1.6.19-2.fc39.*
```

But yeah, I can't do it in all machines that have automated upgrades.

(In reply to Laurent Rineau from comment #41)
> Could containerd-1.6.19-2.fc39 be pushed in the Fedora repos and version
> 1.6.23-1.fc39 removed? That would at least avoid the growth of the number of
> victims. A few colleges of mines at work get bitten by that bug. I even hint
> not to upgrade to Fedora 39, for now.

I second that. moby-engine needs to temporarily explicitly require containerd version below 1.6.23 and the .19 needs to be re-added to the repo.

"Could containerd-1.6.19-2.fc39 be pushed in the Fedora repos and version 1.6.23-1.fc39 removed?"

No, not really. By policy Fedora packages don't go backwards like that in stable releases. Sergio could bump the epoch and do a 1:1.6.19-3.fc39 build, but so far he has not chosen to do that.

(In reply to Adam Williamson from comment #44)
> "Could containerd-1.6.19-2.fc39 be pushed in the Fedora repos and version
> 1.6.23-1.fc39 removed?"
> 
> No, not really. By policy Fedora packages don't go backwards like that in
> stable releases. Sergio could bump the epoch and do a 1:1.6.19-3.fc39 build,
> but so far he has not chosen to do that.

Not to be mean, but is it also a Fedora policy to green-light OS releases that are known to result in non-functional containerization functionality post-upgrade? Moby-engine is not some obscure package that barely anybody uses and the fact that this issue is certain to break things on F39 upgrade has been, apparently, known since the end of September and the release went on anyway:

(In reply to Dusty Mabe from comment #10)
> The nuance here is that the updates for f37 and f38 never made it to
> `stable` repos. So users on those platforms are probably using `docker`
> without issue right now.
> 
> - https://bodhi.fedoraproject.org/updates/FEDORA-2023-5c4718e547
> - https://bodhi.fedoraproject.org/updates/FEDORA-2023-aaa2b3d20b
> 
> When they switch to F39 is when they would have trouble.

All policies have exceptions, an upgrade regression bug such as in this case likely warrants one.

The workarounds listed here are great and they work, except they are not (easily) compatible with automation.

"Not to be mean, but is it also a Fedora policy to green-light OS releases that are known to result in non-functional containerization functionality post-upgrade?"

Not to be blunt, but: yes. neither moby-engine nor docker is not release-blocking functionality in Fedora. The release validation process and criteria are entirely open and fully documented - see https://fedoraproject.org/wiki/Fedora_Release_Criteria , https://fedoraproject.org/wiki/QA:Release_validation_test_plan and https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process .

Podman is release-blocking for Fedora, you will find it covered in https://fedoraproject.org/wiki/Fedora_40_Beta_Release_Criteria#Podman_container_runtime (technically only for IoT, but in practice it's not really possible for it to be working in IoT but broken in other editions or vice versa).

Using the build from https://download.copr.fedorainfracloud.org/results/sergiomb/docker2/fedora-39-x86_64/05871259-containerd/containerd-1.7.0-5.fc39.x86_64.rpm I manage to get it working.
Just upgraded today to Fedora 39 and faced this issue, that definitely was not expecting, as I rely on docker for daily work.

(In reply to Adam Williamson from comment #46)
> Not to be blunt, but: yes. neither moby-engine nor docker is not
> release-blocking functionality in Fedora. The release validation process and
> criteria are entirely open and fully documented

Given the frequency of use among the Fedora users, should we perhaps consider adding moby-engine/docker alongside of podman among the release-blocking functionality?

FEDORA-2023-f3d6f65e2f has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-f3d6f65e2f

Do we have any particularly strong data on "frequency of use among the Fedora users"?

(In reply to Fedora Update System from comment #49)
> FEDORA-2023-f3d6f65e2f has been submitted as an update to Fedora 39.
> https://bodhi.fedoraproject.org/updates/FEDORA-2023-f3d6f65e2f

I rebuild containerd against golang-github-containerd-ttrpc-1.1.0 instead version 2.x.x , which should fix this particular bug , I don't know if we got more problems. 

I'm sorry about the delay. Please continue to report problems and help me to migrate docker from cgroups v2 to cgroups v3 . 

if are new in this bug , please read my comment #30

(In reply to Sergio Basto from comment #51)
> 
> I'm sorry about the delay. Please continue to report problems and help me to
> migrate docker from cgroups v2 to cgroups v3 .

I saw this and it sent me in the wrong direction for a moment. Just so we're clear you mean version 3 of the https://github.com/containerd/cgroups/releases/tag/v3.0.2 package and not linux cgroups [1] v3, which IIUC doesn't exist.

[1] https://en.wikipedia.org/wiki/Cgroups

containerd-1.6.23-2.fc39 works for me, thanks Sergio.

FEDORA-2023-f3d6f65e2f has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-f3d6f65e2f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-f3d6f65e2f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2023-f3d6f65e2f has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days