It appears as though all fedora kernel versions from f35 onward package perf without debuginfod client side support enabled. This may simply be because of the lack of BuildRequires: elfutils-debuginfod-client-devel in the .spec file. perf -vv should say HAVE_DEBUGINFOD_SUPPORT happy happy joy joy
https://src.fedoraproject.org/rpms/kernel-tools/pull-request/6
This bug appears to have been reported against 'rawhide' during the Fedora Linux 38 development cycle. Changing version to 38.