Yeah, this is a pretty embarrassing bug.  In Fedora CoreOS we actually test the resulting ostree before shipping it to humans, because our build and test processes are integrated.  This isn't yet true of Silverblue or IoT.

I'm pretty sure Owen is looking at it.  In the meantime I'd guess you can `rpm-ostree override remove fedora-third-party`.  Or we could revert shipping it on rpm-ostree based systems until it's tested there.

I think more generally too, there is a fundamental clash here in that in the ostree model, we aim for two things:

- `/etc` is text files modified by humans or automation explicitly initiated by humans (e.g. ignition/ansible/puppet/etc.)
- `/usr` is owned by ostree, and conceptually all RPM/image stuff happens in `/usr`

You are trying to have stuff from the RPM/image side affect `/etc`.

Now of course, this *does* happen today with things like `ldconfig` which writes `/etc/ld.so.cache` - but that file is also not modified by humans.

So really, the more precise problem is you're trying to have RPM scripts change files that humans *do* edit.

Which introduces the possibility of data loss if a human is running `vi` on that file as the same time as an OS update.

(Just an aside related to this, in the ostree model, it is explicitly disallowed for RPM scripts to affect true user data in `/var`)

The ultimate fix here is to cleanly separate things so that we have e.g. /usr/lib/yum.repos.d and systemd-style masking like ln -s /dev/null /etc/yum.repos.d/fedora-thirdparty.repo to disable the repo.

And actually, while you *can* change files in `/etc` as part of a script, you can only change the *defaults*.  
IOW, as soon as someone edits one of the repo files (e.g. to change `baseurl`) any changes made by the fedora-third-party scripts will no longer take effect.

So ostree is preventing the "human edit vs OS update" concurrency issue here.

The hacky way to get out of this is to move it to a systemd unit as you're already doing, which doesn't fix any of these underlying issues - but it makes ostree not responsible =)

That said, systemd units are much more naturally under system administrator control, e.g. one can `systemctl disable` etc. that you can't with `%post` scripts.  So I'd consider switching to this model even for traditional-yum-rpm systems too.
The downside of that is it won't work in a non-systemd-container image though.

Which version of `fedora-third-party` is that?
It looks like the latest one may already have a fix for this: https://pagure.io/fedora-third-party/c/11803821f81b277e962b30a5015d51ebf5fd94f7?branch=main

https://src.fedoraproject.org/rpms/fedora-third-party > 0.5 hasn't been built yet

Hi, I have issues with fedora-third-party package and it's probably related to this issue (let me know if it's not):

Silverblue 35 cannot install anything, I haven't tried updating because I rebased from Silverblue 34 and it's already up-to-date.

It happens always.

Package versions:

fedora-third-party-0.5-1.fc35

rpm-ostree:
 Version: '2021.7'
 Git: 8f6533c34811d19be31a8bcd6fb986de4b5cee4b
 Features:
  - compose
  - rust
  - fedora-integration



Error:

error: Executing %transfiletriggerin for fedora-third-party: bwrap(/bin/sh): Child process killed by signal 1; run `journalctl -t 'rpm-ostree(fedora-third-party.transfiletriggerin)'` for more information

Traceback:

ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]: Traceback (most recent call last):
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/bin/fedora-third-party", line 33, in <module>
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     sys.exit(load_entry_point('fedora-third-party==0.5', 'console_scripts', 'fedora-third-party')())
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1137, in __call__
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     return self.main(*args, **kwargs)
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1062, in main
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     rv = self.invoke(ctx)
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1668, in invoke
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     return ctx.invoke(self.callback, **ctx.params)
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 763, in invoke
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     return __callback(*args, **kwargs)
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     return ctx.invoke(f, obj, *args, **kwargs)
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 763, in invoke
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     return __callback(*args, **kwargs)
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/fedora_third_party/cli.py", line 115, in refresh
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     cfg.update_seen_repositories()
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/fedora_third_party/config.py", line 53, in update_seen_repositories
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     self._write_state()
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:   File "/usr/lib/python3.10/site-packages/fedora_third_party/config.py", line 44, in _write_state
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]:     with open(self.state_path, 'w') as f:
ago 20 13:59:16 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[8237]: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/fedora-third-party/state'
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]: Traceback (most recent call last):
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/bin/fedora-third-party", line 33, in <module>
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     sys.exit(load_entry_point('fedora-third-party==0.5', 'console_scripts', 'fedora-third-party')())
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1137, in __call__
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     return self.main(*args, **kwargs)
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1062, in main
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     rv = self.invoke(ctx)
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1668, in invoke
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     return _process_result(sub_ctx.command.invoke(sub_ctx))
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     return ctx.invoke(self.callback, **ctx.params)
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 763, in invoke
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     return __callback(*args, **kwargs)
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     return ctx.invoke(f, obj, *args, **kwargs)
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/click/core.py", line 763, in invoke
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     return __callback(*args, **kwargs)
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/fedora_third_party/cli.py", line 115, in refresh
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     cfg.update_seen_repositories()
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/fedora_third_party/config.py", line 53, in update_seen_repositories
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     self._write_state()
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:   File "/usr/lib/python3.10/site-packages/fedora_third_party/config.py", line 44, in _write_state
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]:     with open(self.state_path, 'w') as f:
ago 20 16:17:13 fedora rpm-ostree(fedora-third-party.transfiletriggerin)[31103]: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/fedora-third-party/state'

I can confirm that the error only occurs when installing, not when updating. Also, sorry for the duplicate traceback above.

*** Bug 1997607 has been marked as a duplicate of this bug. ***

Just to add, trying to `rpm-ostree override remove fedora-third-party` will get you an error since the fedora workstation repositories depend on it.

Sorry that I didn't see this earlier.

(In reply to Óscar Fernández Díaz from comment #7)
> I can confirm that the error only occurs when installing, not when updating.
> Also, sorry for the duplicate traceback above.

It's actually not a duplicate traceback - it shows what happens in particular with the newer 0.5 version, which doesn't use dnf.

I'm unclear on the exact details of the environment where rpm-ostee is running triggers, but I should be able to update the code in a way that will be pretty robust against this - if the system looks "fresh" - there's no /var/lib/fedora-third-party state', then 'fedora-third-party refresh' should exit immediately.

The other approach would be to make the trigger detect that it's running in the rpm-ostree scripts environment explicity, but I'm not sure how to do that robustly, and the fedora-third-party change is more general.

(In reply to Colin Walters from comment #3)
> And actually, while you *can* change files in `/etc` as part of a script,
> you can only change the *defaults*.  
> IOW, as soon as someone edits one of the repo files (e.g. to change
> `baseurl`) any changes made by the fedora-third-party scripts will no longer
> take effect.
> 
> So ostree is preventing the "human edit vs OS update" concurrency issue here.

I'm not particularly worried about the race condition in this particular case - in the traditional model - the 'fedora-third-party refresh' script only affects newly installed repository files, so a user would need to be editing one of those before the RPM transaction finishes. In the rpm-ostree case, the trigger needs to do nothing, since it is never supposed to do anything on a freshly installed system.
 
> The hacky way to get out of this is to move it to a systemd unit as you're
> already doing, which doesn't fix any of these underlying issues - but it
> makes ostree not responsible =)

I don't really anticipate problems with users editing things at boot time, but I definitely agree that having the enablement status of repositories as editing a %config(noreplace) file in /etc is *not* the best way to do it, for a number of reasons. The biggest problem I see is that we can't update the repository files once they've been "edited" to turn them on, so if an URL changes or something, we'll have an issue.
 
> That said, systemd units are much more naturally under system administrator
> control, e.g. one can `systemctl disable` etc. that you can't with `%post`
> scripts.  So I'd consider switching to this model even for
> traditional-yum-rpm systems too.
> The downside of that is it won't work in a non-systemd-container image
> though.

Also, if people are updating the system live (not at boot time) the opt-in status won't be applied until the next reboot, which would definitely be confusing.

FEDORA-2021-b1b449aa6a has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-b1b449aa6a

Requesting a F35 Beta Freeze Exception, so that we can get F35 Silverblue working again as quickly as possible.

FEDORA-2021-b1b449aa6a has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-b1b449aa6a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-b1b449aa6a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

+3 in https://pagure.io/fedora-qa/blocker-review/issue/413 (counting bcotton as implicitly +1 BetaFE), so marking accepted FE.

FEDORA-2021-b1b449aa6a has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.