I think this might not need a fix at all, at least on Mutter side. I recently dropped support for the old DMA-BUFs (when modifiers are not announced). This was fixed with Chrome/Chromium M113 and will be backported to Firefox upstream soon. We probably just want to pick that change to our FF package instead. 

WebRTC fix: https://webrtc-review.googlesource.com/c/src/+/298700

This is going to be fixed with Firefox ESR 115. Let's leave this open just to make sure it gets tested.

Fix includes backporting following WebRTC upstream change: https://webrtc-review.googlesource.com/c/src/+/298700

(In reply to Jan Grulich from comment #7)
> This is going to be fixed with Firefox ESR 115. Let's leave this open just
> to make sure it gets tested.
> 
> Fix includes backporting following WebRTC upstream change:
> https://webrtc-review.googlesource.com/c/src/+/298700

Apparently it's not going to help.

I just patched Firefox to making sure DMABufs are not really advertised (there are not most likely advertised even in non-patched FF 115) and it still looks that Mutter enforces it and making screen sharing to fail.

I believe the best would be to completely ignore DMABufs on RHEL 8 for screen sharing as the version of Mutter doesn't implement them properly (according to recent specs) and we have only PipeWire 0.3.6 which doesn't have all the required features (e.g. renegotiation).

Jonas, can we do something about that? Or any idea whether we can workaround that somehow from FF side? Because this fix otherwise needs to be backported all the way to all RHEL 8 z-streams.

> Jonas, can we do something about that? Or any idea whether we can workaround that somehow from FF side? Because this fix otherwise needs to be backported all the way to all RHEL 8 z-streams.

We can disable it mutter side. The only potential consequence I can think of is OBS Studio getting worse performance, but I'm not sure it knows how to handle the legacy DMA buffer handling.

The other downside is gnome-remote-desktop getting slightly worse performance on i915.

(In reply to Jonas Ådahl from comment #11)
> > Jonas, can we do something about that? Or any idea whether we can workaround that somehow from FF side? Because this fix otherwise needs to be backported all the way to all RHEL 8 z-streams.
> 
> We can disable it mutter side. The only potential consequence I can think of
> is OBS Studio getting worse performance, but I'm not sure it knows how to
> handle the legacy DMA buffer handling.

OBS Studio does the same thing as WebRTC, they will announce SPA_DATA_DmaBuf support in case PipeWire 0.3.24 and newer is used or a modifier is passed (see #1). 

1) https://github.com/obsproject/obs-studio/blob/master/plugins/linux-pipewire/pipewire.c#L1043

We probably don't need to disable it, but I think the current version of Mutter (in RHEL 8) does something like:

>if (dma_buf_supported)
>  buffer_type = 1 << SPA_DATA_DmaBuf;
>else 
>  buffer_type = 1 << SPA_DATA_MemFD;

It's one or the other. If you announce both, it will be up to the client to decide whether to use SPA_DATA_DmaBuf based on their support.

Looking into the rest of OBS Studio code, it doesn't seem to support SPA_DATA_MemFd, only SPA_DATA_MemPtr and I guess with Mutter it's the other way around, am I right?

(In reply to Jan Grulich from comment #14)
> Looking into the rest of OBS Studio code, it doesn't seem to support
> SPA_DATA_MemFd, only SPA_DATA_MemPtr and I guess with Mutter it's the other
> way around, am I right?

Yes, looks I'm right looking into Mutter 3.32.2 we have in RHEL 8, but I might have found a bug in WebRTC. Old Mutter seems to skip this buffer type negotiation part and you just either push a MemFd or DmaBuf buffers, which is what decides what's going to be used. If this step is skipped, we in WebRTC don't even fallback to a DRM_FORMAT_MOD_INVALID used for modifier-less buffers and the modifier variable remains unset, resulting into importing DmaBufs with invalid modifier. 

I'm currently building a scratch build of Firefox to check whether this makes a difference.

(In reply to Jan Grulich from comment #14)
> Looking into the rest of OBS Studio code, it doesn't seem to support
> SPA_DATA_MemFd, only SPA_DATA_MemPtr and I guess with Mutter it's the other
> way around, am I right?

MemFd is for accessing shared memory, and MemPtr is for accessing memory mapped by PipeWire by passing PW_STREAM_FLAG_MAP_BUFFERS to pw_stream_connect() so that should not be an issue.

> in WebRTC don't even fallback to a DRM_FORMAT_MOD_INVALID used for modifier-less buffers and the modifier variable remains unset, resulting into importing DmaBufs with invalid modifier. 

Don't you need to handle using modifierless API for allocation either way when there are no explicit modifiers negotiated? There are IIRC drivers that simply do not support using the modifier-aware API variants.

(In reply to Jonas Ådahl from comment #16)
> > in WebRTC don't even fallback to a DRM_FORMAT_MOD_INVALID used for modifier-less buffers and the modifier variable remains unset, resulting into importing DmaBufs with invalid modifier. 
> 
> Don't you need to handle using modifierless API for allocation either way
> when there are no explicit modifiers negotiated? There are IIRC drivers that
> simply do not support using the modifier-aware API variants.

We handle that and looking into the logs it seems to fail on creating EGLImage (with EGL_BAD_PARAMETER). I was wrong with that assumption before and it ends up not having explicit modifier using DRM_FORMAT_MOD_INVALID to indicate that. I also tried OBS Studio and it doesn't work either. Could that be a VM limitation?

OBS Studio from Flathub I've been testing doesn't seem to work because of incompatibility of PipeWire library used in the runtime and the PipeWire running on the server (RHEL 8 - which is PW 0.3.6).

Cc:ing Wim: Seems to be some backward compatibility problems with newer libpipewire in Flatpaks working on RHEL8 hosts.

Created attachment 1985650 [details]
Patch allowing to fallback to MemFd buffers

I can make it work when I use the attached Mutter patch. I believe this is a correct thing to do. Not sure if my version is correct code-wise, but the logic is from modern Mutter releases. This allows clients to pick what they actually support.

Jonas, what do you think about the patch in comment 20? I have tested it in RHEL 8.4 and RHEL 8.6 and it makes Firefox to properly share a screen in a VM. You can think of it as a partial backport from recent Mutter. I can handle this bug and all the builds if you want and agree with this patch.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.