Proposed as a Blocker for 36-final by Fedora user augenauf using the blocker tracking app because:

 There are a number of selinux-policy issues that prevent flatpak from updating. Can't have a system where flatpaks don't update

Please collect SELinux denials that appeared on that machine:

# ausearch -m avc -m user_avc -m selinux_err -i -ts today

Thank you.

Without the list of SELinux denials, we cannot analyze the issue and provide a fix or a workaround.

Here is the output of `# ausearch -m avc -m user_avc -m selinux_err -i -ts today`;

https://paste.opensuse.org/43894250

I installed fresh F36 Workstation, I have selinux-policy-36.5-1.fc36 and flatpak-1.12.7-1.fc36, and I can't reproduce this issue. I installed gedit from fedora flatpak, and "flatpak update" (which does nothing, because there are no updates) worked fine. Then I enabled flathub filtered repo, installed bitwarden, downgraded it to a previous commit, and updated it using "flatpak update" - also worked fine.

I have an F36 VM where I don't experience the problem, I am only seeing this on a baremetal machine that I upgraded from F35. I agree, if I am the only person experiencing this, it's not a blocker bug...
How about I do 

    restorecon -vR /var/lib/flatpak

and we see if that changes anything?

... or restorecon on /, since apparently /etc/passwd is affected as well.

You are not the only person seeing this problem.  It seems to be a combination of flatpak, container-selinux and snappy.

Further observations:

 - restorecon -vR / did not fix the problem. 
 - running flatpak update with sudo works fine, "Updates complete".

Is there any other information or log that I can provide to help analyze this bug?

Discussed during the 2022-04-04 blocker review meeting: [1]

The decision to punt the classification of this bug was made:

"Per kparal’s comment in-bug and lruzicka’s comment during the meeting, neither were able to reproduce this. We would like to have more widespread testing before we vote one way or the other on this bug, so we will punt for now."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2022-04-04/f36-blocker-review.2022-04-04-16.00.log.html

*** Bug 2071939 has been marked as a duplicate of this bug. ***

*** Bug 2072695 has been marked as a duplicate of this bug. ***

Zdenek is there anything I can do in the container-selinux package to stop this from happening.  Could I remove the existing container.pp and then re-add it? Although this would cause 
containers to blow up.

I think the problem is selinux-policy is updated and rebuilt before custom modules are updated, and if the custom modules contain a reference to non-existing classes, failures are reported, the modules become inactive and types they provide do not exist.

Changing the order in specfile post/posttrans scriptlets could lead to other problems. I suggest removing the classes from F35 selinux-policy socket_class_set and rebuild all packages refering to socket_class_set.

FEDORA-2022-ee2ba7ba12 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-ee2ba7ba12

Could you add the classes back to f36 policy without adding them to the socket_class_set, and then remove them in f37.

I upgraded to the newly built package, still the same symptoms.

Here is another output of # ausearch -m avc -m user_avc -m selinux_err -i -ts today: https://paste.opensuse.org/63715694
Nothing changed.

For those who can't reproduce, I wanted to say that my install was a system-upgrade from f35.

Packages Altered:
    Upgrade  selinux-policy-36.6-1.fc36.noarch                           updates-testing
    Upgraded selinux-policy-36.5-1.fc36.noarch                           System
    Upgrade  selinux-policy-targeted-36.6-1.fc36.noarch                  updates-testing
    Upgraded selinux-policy-targeted-36.5-1.fc36.noarch                  System

    Upgrade  container-selinux-2:2.181.0-2.fc36.noarch commandline
    Upgraded container-selinux-2:2.181.0-1.fc36.noarch System

Similar situation here. I had osbuild-composer and friends installed in my system-upgrade; osbuild-selinux was included in that and seemed to be blocking the recompilation of policy. @dwalsh helped with my installation and can share more details here if needed.

FEDORA-2022-ee2ba7ba12 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-ee2ba7ba12`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-ee2ba7ba12

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Discussed during the 2022-04-11 blocker review meeting: [1]

The decision to classify this bug as an AcceptedBlocker was made:

"It violates the following criterion: “The installed system must be able to install software with the default console tool for the relevant software type. This should be interpreted to cover any other form of software distribution that we invent in future and include in an otherwise release-blocking deployment of Fedora..."."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2022-04-11/f36-blocker-review.2022-04-11-16.00.log.txt

FEDORA-2022-ee2ba7ba12 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

(In reply to Daniel Walsh from comment #15)
> Could you add the classes back to f36 policy without adding them to the
> socket_class_set, and then remove them in f37.

I actually did the opposite and removed them in F34 and F35. It requires container-selinux rebuilt with

selinux-policy-35.17-1.fc35 
https://bodhi.fedoraproject.org/updates/FEDORA-2022-c5bee6b70f

selinux-policy-34.27-1.fc34
https://bodhi.fedoraproject.org/updates/FEDORA-2022-eaef082697

Then how do we update them without causing the issue?

> Then how do we update them without causing the issue?

If F34/F35 container-selinux is built with the new policy, expanded socket_class_set does not contain references to the classes which will be removed in F36.

> Could you add the classes back to f36 policy without adding them to the
> socket_class_set, and then remove them in f37.

This would actually need to be there till f37 as updates over 2 releases are supported. I'd rather have it resolved earlier.

*** Bug 2070326 has been marked as a duplicate of this bug. ***

FEDORA-2022-c5bee6b70f has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-c5bee6b70f

FEDORA-2022-c5bee6b70f has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-c5bee6b70f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-c5bee6b70f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Zdenek, the update from comment 25 is for F35, so I assume that would avoid the problem when upgrading. But how do we fix the problem for people who already upgraded to F36? (Turns out my production system is affected by this, so I can easily test any proposed F36 update).

I had filed this bug report against f36, and I can confirm that I am not facing the issue anymore since update


End time       : Fri 15 Apr 2022 11:58:35 PM 

    Upgrade  flatpak-1.12.7-2.fc36.x86_64                @fedora
    Upgraded flatpak-1.12.7-1.fc36.x86_64                @@System
    Upgrade  flatpak-libs-1.12.7-2.fc36.x86_64           @fedora
    Upgraded flatpak-libs-1.12.7-1.fc36.x86_64           @@System
    Upgrade  flatpak-selinux-1.12.7-2.fc36.noarch        @fedora
    Upgraded flatpak-selinux-1.12.7-1.fc36.noarch        @@System
    Upgrade  flatpak-session-helper-1.12.7-2.fc36.x86_64 @fedora
    Upgraded flatpak-session-helper-1.12.7-1.fc36.x86_64 @@System

(In reply to Flo H. from comment #28)
>     Upgrade  flatpak-1.12.7-2.fc36.x86_64                @fedora

(In reply to Flo H. from comment #28)
>     Upgrade  flatpak-1.12.7-2.fc36.x86_64                @fedora

I still see the issue even with these package versions.

So as regards the blocker status of this bug: per the criteria, it's only clearly a blocker if the bug still occurs on a clean upgrade from fully-updated Fedora 35 to Fedora 36. Is this the case? If not, we should probably drop the AcceptedBlocker status. Remaining problems for people who upgraded while the bug was present are still an issue, but are less clearly release-blocking.

Adam, we don't have a reproducer, so we don't know how to verify this. It seems to be happening randomly just for certain systems during upgrade. All my test VMs were unaffected, my production system was affected.

Zdenek, do you have some advise how to verify if this was fixed at least for new upgrades? Also, what's going to happen to existing F36 users (see comment 27)? Thanks.

(In reply to Kamil Páral from comment #32)
> Adam, we don't have a reproducer, so we don't know how to verify this. It
> seems to be happening randomly just for certain systems during upgrade. All
> my test VMs were unaffected, my production system was affected.
> 
> Zdenek, do you have some advise how to verify if this was fixed at least for
> new upgrades? Also, what's going to happen to existing F36 users (see
> comment 27)? Thanks.

I tried the upgrade several times successfully, but it always was a clean system with just the named additional packages installed.

I didn't run into problems with existing F36 installations either, please add some details. It may be dependent on the packages version before and after the update.

The packages list contains: snapd-selinux container-selinux flatpak-selinux osbuild-selinux

> I tried the upgrade several times successfully, but it always was a clean system with just the named additional packages installed.

Yes, I tried that as well, had no problems. It seems we can't trigger the bug like this. There's something different about real-world systems. Or we just got lucky/unlucky.

> I didn't run into problems with existing F36 installations either, please add some details.

OK, I'll upload logs. Anytime I try `flatpak update`, I see:

Warning: Failed to get revokefs-fuse socket from system-helper: User flatpak does not exist in password file entry
Warning: Can't open system repo default: While opening repository /var/lib/flatpak/repo: opening repo: opendir(/var/lib/flatpak/repo): Permission denied
Warning: Failed to get revokefs-fuse socket from system-helper: User flatpak does not exist in password file entry
Warning: Can't open system repo default: While opening repository /var/lib/flatpak/repo: opening repo: opendir(/var/lib/flatpak/repo): Permission denied
Warning: Failed to get revokefs-fuse socket from system-helper: User flatpak does not exist in password file entry
...

Created attachment 1874009 [details]
dnf system-upgrade log

Created attachment 1874010 [details]
rpm -qa (current)

Created attachment 1874011 [details]
journal snippet after running flatpak update

(In reply to Kamil Páral from comment #34)
> > I didn't run into problems with existing F36 installations either, please add some details.
> 
> OK, I'll upload logs. Anytime I try `flatpak update`, I see:
> 
> Warning: Failed to get revokefs-fuse socket from system-helper: User flatpak
> does not exist in password file entry
This means the flatpak package was not installed successfully, reinstalling should resolve the problem.

Oh, great. I can confirm that `dnf reinstall flatpak\*` fixed the problem.

So is this related to bug 2056303 or a different problem?

I see a lot of selinux-related errors in my upgrade log, related to flatpak but not just flatpak, many other packages as well. Here's a flatpak-related snippet:

dub 14 11:02:00 hydra kernel: SELinux:  Context system_u:object_r:flatpak_helper_exec_t:s0 is not valid (left unmapped).
dub 14 11:02:00 hydra audit[6040]: AVC avc:  denied  { mac_admin } for  pid=6040 comm="restorecon" capability=33  scontext=system_u:system_r:setfiles_t:s0 tcontext=system_u:system_r:setfiles_t:s0 tclass=capability2 permissive=0
dub 14 11:02:00 hydra audit: SELINUX_ERR op=setxattr invalid_context="system_u:object_r:flatpak_helper_exec_t:s0"
dub 14 11:02:00 hydra audit[6040]: SYSCALL arch=c000003e syscall=189 success=no exit=-22 a0=7ffc9bf9d630 a1=7f84a1766251 a2=561fa35324c0 a3=2b items=0 ppid=5673 pid=6040 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="restorecon" exe="/usr/sbin/setfiles" subj=system_u:system_r:setfiles_t:s0 key=(null)
dub 14 11:02:00 hydra audit: PROCTITLE proctitle=2F7362696E2F726573746F7265636F6E002D65002F737973002D65002F70726F63002D65002F6D6E74002D65002F7661722F746D70002D65002F686F6D65002D65002F726F6F74002D65002F746D70002D69002D52002D66002D
dub 14 11:02:00 hydra kernel: kauditd_printk_skb: 9 callbacks suppressed
dub 14 11:02:00 hydra kernel: audit: type=1400 audit(1649926920.531:244): avc:  denied  { mac_admin } for  pid=6040 comm="restorecon" capability=33  scontext=system_u:system_r:setfiles_t:s0 tcontext=system_u:system_r:setfiles_t:s0 tclass=capability2 permissive=0
dub 14 11:02:00 hydra kernel: audit: type=1401 audit(1649926920.531:244): op=setxattr invalid_context="system_u:object_r:flatpak_helper_exec_t:s0"

Would you be able to compile a list of packages which are likely to be affected? We could issue an F36 'bump' update for those packages, so that all F36 users receive the fix automatically. Or is there a better solution?

(In reply to Kamil Páral from comment #39)
> Oh, great. I can confirm that `dnf reinstall flatpak\*` fixed the problem.
> 
> So is this related to bug 2056303 or a different problem?
> 
It rather is bz#2070350

> I see a lot of selinux-related errors in my upgrade log, related to flatpak
> but not just flatpak, many other packages as well. Here's a flatpak-related
> snippet:
> 
> dub 14 11:02:00 hydra kernel: SELinux:  Context
> system_u:object_r:flatpak_helper_exec_t:s0 is not valid (left unmapped).
> dub 14 11:02:00 hydra audit[6040]: AVC avc:  denied  { mac_admin } for 
> pid=6040 comm="restorecon" capability=33 
> scontext=system_u:system_r:setfiles_t:s0
> tcontext=system_u:system_r:setfiles_t:s0 tclass=capability2 permissive=0
> dub 14 11:02:00 hydra audit: SELINUX_ERR op=setxattr
> invalid_context="system_u:object_r:flatpak_helper_exec_t:s0"
> dub 14 11:02:00 hydra audit[6040]: SYSCALL arch=c000003e syscall=189
> success=no exit=-22 a0=7ffc9bf9d630 a1=7f84a1766251 a2=561fa35324c0 a3=2b
> items=0 ppid=5673 pid=6040 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0
> egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="restorecon"
> exe="/usr/sbin/setfiles" subj=system_u:system_r:setfiles_t:s0 key=(null)
> dub 14 11:02:00 hydra audit: PROCTITLE
> proctitle=2F7362696E2F726573746F7265636F6E002D65002F737973002D65002F70726F630
> 02D65002F6D6E74002D65002F7661722F746D70002D65002F686F6D65002D65002F726F6F7400
> 2D65002F746D70002D69002D52002D66002D
> dub 14 11:02:00 hydra kernel: kauditd_printk_skb: 9 callbacks suppressed
> dub 14 11:02:00 hydra kernel: audit: type=1400 audit(1649926920.531:244):
> avc:  denied  { mac_admin } for  pid=6040 comm="restorecon" capability=33 
> scontext=system_u:system_r:setfiles_t:s0
> tcontext=system_u:system_r:setfiles_t:s0 tclass=capability2 permissive=0
> dub 14 11:02:00 hydra kernel: audit: type=1401 audit(1649926920.531:244):
> op=setxattr invalid_context="system_u:object_r:flatpak_helper_exec_t:s0"
This is a result of previously failed updates.

> Would you be able to compile a list of packages which are likely to be
> affected? We could issue an F36 'bump' update for those packages, so that
> all F36 users receive the fix automatically. Or is there a better solution?
I don't have any information such a rebuild would help, reproducing steps are needed to assess a problem.

> It rather is bz#2070350

Hm. I upgraded to that flatpak version (flatpak-1.12.7-2.fc36) during my system upgrade, but I still ended up with a broken system. Why?

> This is a result of previously failed updates.

So what was the root cause in my case? Is that problem already solved in some selinux-policy update (F35 or F36)?

> I don't have any information such a rebuild would help, reproducing steps are needed to assess a problem.

What can we do for existing F36 systems at least related to this flatpak issue, then? A simple reinstall fixed it for me, and it's likely that there are many more machines like mine. Isn't rebuilding at least flatpak a good idea?

(In reply to Kamil Páral from comment #41)
> > It rather is bz#2070350
> 
> Hm. I upgraded to that flatpak version (flatpak-1.12.7-2.fc36) during my
> system upgrade, but I still ended up with a broken system. Why?
Please share some data, like dnf command output, package versions before and after, AVC denials, semodule command output. I am not aware of any problem based just on this description.

> > This is a result of previously failed updates.
> 
> So what was the root cause in my case? Is that problem already solved in
> some selinux-policy update (F35 or F36)?
There are not much data, so I don't really know. It might be that the new flatpak package needs rules brought by the flatpak-selinux package, so kind of ordering problem, but this is just a guess and needs to be verified.

> > I don't have any information such a rebuild would help, reproducing steps are needed to assess a problem.
> 
> What can we do for existing F36 systems at least related to this flatpak
> issue, then? A simple reinstall fixed it for me, and it's likely that there
> are many more machines like mine. Isn't rebuilding at least flatpak a good
> idea?
I am not convinced it would make any change.

(In reply to Zdenek Pytela from comment #42)
> Please share some data, like dnf command output, package versions before and
> after, AVC denials, semodule command output. I am not aware of any problem
> based just on this description.

I attached the whole upgrade log in comment 35. Doesn't it contain everything needed? It contains both the whole journal and the whole dnf transaction output. How can I gather additional data? (Note that I'm no longer affected after I reinstalled flatpak).

That's odd. Your system repor(In reply to Kamil Páral from comment #43)
> (In reply to Zdenek Pytela from comment #42)
> > Please share some data, like dnf command output, package versions before and
> > after, AVC denials, semodule command output. I am not aware of any problem
> > based just on this description.
> 
> I attached the whole upgrade log in comment 35. Doesn't it contain
> everything needed? It contains both the whole journal and the whole dnf
> transaction output. How can I gather additional data? (Note that I'm no
> longer affected after I reinstalled flatpak).

That's odd because your system reported "User flatpak does not exist in password file entry", while the script[1] to add user 'flatpak' (useradd -r -g flatpak```) was committed 3 years ago.

[1] https://src.fedoraproject.org/rpms/flatpak/c/b6456a51bb98ec1bd6b6dbdc5bcc46bad2f1930a

Chen, I assume that's because selinux didn't allow the process to read /etc/passwd or similar.

FEDORA-2022-c5bee6b70f has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.