Bug 2170957
| Summary: | parsec service fails to start on Fedora IoT 37 upgraded from F36 IoT | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Micah Abbott <miabbott> |
| Component: | parsec | Assignee: | Peter Robinson <pbrobinson> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 40 | CC: | awilliam, bugzilla.somor, fedora, fzatlouk, jpazdziora, miabbott, pbrobinson, rgerardi, robatino, sdevlieg |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | AcceptedFreezeException AcceptedBlocker | ||
| Fixed In Version: | parsec-1.3.0-1.fc39 parsec-1.3.0-1.fc40 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-03-16 23:08:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2187793, 2187794 | ||
I can reproduce this with a fresh Fedora IoT 37 install using the installation ISO on x86_64. More interestingly, the initial deployment from the ISO (...) does start parsec correctly. After doing `rpm-ostree upgrade` and a reboot parsec fails to start with the same error. It works in commit 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0 and doesn't work in commit 8dde1800d7d3aeaf71e0737c9551e550114a20dd73046ba2e3b4d8aec3873f2e. In commit 8dde I have the same permissions on /var/lib/parsec/mappings as mentioned by Micah, in commit 2334 the permissions are as follows: ``` [root@localhost ~]# ls -laRZ /var/lib/parsec/ /var/lib/parsec/: total 12 drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 . drwxr-xr-x. 26 root root system_u:object_r:var_lib_t:s0 4096 Feb 17 21:18 .. drwx------. 2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 mappings /var/lib/parsec/mappings: total 8 drwx------. 2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 . drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 21:14 .. ``` It seems that betweens these two commits a relevant change might be that dnsmasq changed version from 2.87-1.fc37 to 2.88-1.fc37? I did a fresh install of F36 IoT and observed the following:
```
[root@fedora ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/x86_64/iot
Version: 36.20220618.0 (2022-06-18T10:37:36Z)
Commit: 329c785ac61ca4d6ccad4c7d65049a49fdbe682242dc3e49b0a764c10b89fed2
GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
[root@fedora ~]# rpm -q parsec
parsec-1.0.0-1.fc36.x86_64
[root@fedora ~]# ls -laRZ /var/lib/parsec/
/var/lib/parsec/:
total 12
drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 .
drwxr-xr-x. 25 root root system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 ..
drwxr-xr-x. 2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 mappings
/var/lib/parsec/mappings:
total 8
drwxr-xr-x. 2 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 .
drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Feb 17 20:16 ..
[root@fedora ~]# id parsec
uid=982(parsec) gid=982(parsec) groups=982(parsec),981(parsec-clients)
```
And looking at the UIDs on the F37 IoT system:
```
[root@localhost ~]# id parsec
uid=983(parsec) gid=983(parsec) groups=983(parsec),982(parsec-clients)
[root@localhost ~]# id dnsmasq
uid=982(dnsmasq) gid=981(dnsmasq) groups=981(dnsmasq)
```
So it looks like the UID mapping changed between Fedora releases?
And here's how the uids/gids changed: ``` # working (commit 2334737c8d840a9c57904ab1d2f786f05039702b1b0068608848c57f466e5dd0) [root@localhost ~]# id dnsmasq uid=984(dnsmasq) gid=984(dnsmasq) groups=984(dnsmasq) [root@localhost ~]# id parsec uid=982(parsec) gid=982(parsec) groups=982(parsec),981(parsec-clients) # non-working (commit 8dde1800d7d3aeaf71e0737c9551e550114a20dd73046ba2e3b4d8aec3873f2e) [root@localhost ~]# id dnsmasq uid=982(dnsmasq) gid=981(dnsmasq) groups=981(dnsmasq) [root@localhost ~]# id parsec uid=983(parsec) gid=983(parsec) groups=983(parsec),982(parsec-clients) ``` I suspect this may be due to how both packages handle user creation. The parsec package uses `useradd` to create users https://src.fedoraproject.org/rpms/parsec/blob/rawhide/f/parsec.spec#_75-81 The dnsmasq package uses systemd sysusers https://src.fedoraproject.org/rpms/dnsmasq/blob/rawhide/f/dnsmasq.spec#_177-179 I'm out of my depth here, but I wonder if `parsec` should adopt the sysusers approach. Regardless, I think it might benefit from some changes that can detect if the /var/lib/parsec/mappings folder is owned by a different UID and chown it to the current UID for the `parsec` user I had the same issue on a fresh install of Fedora IoT 37 aarch64 on Raspberry Pi 4. Issue persisted after updating to the latest version. Still present on latest Fedora 38 aarch64 on Pi 4 (fresh install with all updates) This message is a reminder that Fedora Linux 37 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '37'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 37 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed. This is still an issue in Fedora 39 I hit the same problem on fresh Fedora 39 IoT installation, rpm-ostree upgrade'd to
# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
Version: 39.20231231.0 (2023-12-31T16:28:21Z)
Commit: 9ccb803a38143ddb2803255c205d9cc73367f852d383dc517a6e6d9f8ffb70ea
GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
fedora-iot:fedora/stable/aarch64/iot
Version: 39.20231103.1 (2023-11-03T18:17:43Z)
Commit: cc8d419f72d84ac24d0a95e235c0cdf72844f73d9d6f42a41fcddf23dfb34f7d
GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
The ownership gets messed up to
# ls -laRZ /var/lib/parsec/
/var/lib/parsec/:
total 12
drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 .
drwxr-xr-x. 25 root root system_u:object_r:var_lib_t:s0 4096 Jan 4 20:46 ..
drwx------. 2 clevis parsec-clients system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 mappings
/var/lib/parsec/mappings:
total 8
drwx------. 2 clevis parsec-clients system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 .
drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Sep 27 00:00 ..
I've been digging into this a little. My first attempt was to add a
%attr(-,parsec,parsec) %ghost %config %{_localstatedir}/lib/parsec/mappings/
to the package spec. Unsure if that's the correct approach, but I was just going to test that out.
However, rebuilding parsec is blocked by a couple dependency updates that are breaking cargo/mockbuild. Some of those are fixable with more Cargofile patches, the remaining one that's still blocking me is https://bugzilla.redhat.com/show_bug.cgi?id=2229923 which I think has to be patched first to get the actual parsec build running again.
> %attr(-,parsec,parsec) %ghost %config %{_localstatedir}/lib/parsec/mappings/
>
> to the package spec. Unsure if that's the correct approach, but I was just
> going to test that out.
I think we need the following:
%attr(0750,parsec,parsec) %dir %{_localstatedir}/lib/parsec/mappings/
Will have an update to test shortly.
FEDORA-2024-a223050257 (parsec-1.3.0-1.fc40 and rust-cryptoki-0.6.2-1.fc40) has been submitted as an update to Fedora 40. https://bodhi.fedoraproject.org/updates/FEDORA-2024-a223050257 FEDORA-2024-6cf8aec019 (parsec-1.3.0-1.fc39, rust-cryptoki-0.6.2-1.fc39, and 6 more) has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-6cf8aec019 Proposed as a Blocker for 40-beta by Fedora user pbrobinson using the blocker tracking app because: Fedora IoT default service FEDORA-2024-6cf8aec019 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-6cf8aec019` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-6cf8aec019 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. Discussed at today's blocker review meeting: https://meetbot-raw.fedoraproject.org/blocker-review_matrix_fedoraproject-org/2024-03-11/f40-blocker-review.2024-03-11-16.00.html . This was rejected as a Beta blocker but accepted as a Final blocker, because it violates the "all system services must start" criterion after upgrade, which is a Final criterion. Also accepted as a Beta FE so we can fix upgrades ASAP. FEDORA-2024-a223050257 has been pushed to the Fedora 40 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-a223050257` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-a223050257 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2024-6cf8aec019 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-6cf8aec019` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-6cf8aec019 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. https://bodhi.fedoraproject.org/updates/FEDORA-2024-a223050257 fixes this issue. FEDORA-2024-6cf8aec019 (parsec-1.3.0-1.fc39, parsec-tool-0.4.0-6.fc39, and 8 more) has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report. Reopening as we need it for F-40 freeze exception and it was closed for F39 FEDORA-2024-a223050257 (parsec-1.3.0-1.fc40, parsec-tool-0.4.0-6.fc40, and 2 more) has been pushed to the Fedora 40 stable repository. If problem still persists, please make note of it in this bug report. |
The parsec service fails to start on Fedora IoT 37 (37.20230217.0): ``` # systemctl status parsec × parsec.service - Parsec Service Loaded: loaded (/usr/lib/systemd/system/parsec.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Fri 2023-02-17 19:37:57 UTC; 3min 45s ago Docs: https://parallaxsecond.github.io/parsec-book/parsec_service/install_parsec_linux.html Process: 744 ExecStart=/usr/libexec/parsec -c /etc/parsec/config.toml (code=exited, status=1/FAILURE) Main PID: 744 (code=exited, status=1/FAILURE) CPU: 3ms Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Scheduled restart job, restart counter is at 5. Feb 17 19:37:57 localhost.localdomain systemd[1]: Stopped parsec.service - Parsec Service. Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Start request repeated too quickly. Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Failed with result 'exit-code'. Feb 17 19:37:57 localhost.localdomain systemd[1]: Failed to start parsec.service - Parsec Service. ``` Version-Release number of selected component (if applicable): parsec-1.1.0-1.fc37.x86_64 Looking at the journal, there are a lot of `Error: Operation not permitted (os error 1)` ``` # journalctl -b -u parsec Feb 17 19:37:56 localhost systemd[1]: Starting parsec.service - Parsec Service... Feb 17 19:37:56 localhost parsec[706]: Error: Operation not permitted (os error 1) Feb 17 19:37:56 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE Feb 17 19:37:56 localhost systemd[1]: parsec.service: Failed with result 'exit-code'. Feb 17 19:37:56 localhost systemd[1]: Failed to start parsec.service - Parsec Service. Feb 17 19:37:56 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 1. Feb 17 19:37:56 localhost systemd[1]: Stopped parsec.service - Parsec Service. Feb 17 19:37:56 localhost systemd[1]: Starting parsec.service - Parsec Service... Feb 17 19:37:56 localhost parsec[730]: Error: Operation not permitted (os error 1) Feb 17 19:37:56 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE Feb 17 19:37:56 localhost systemd[1]: parsec.service: Failed with result 'exit-code'. Feb 17 19:37:56 localhost systemd[1]: Failed to start parsec.service - Parsec Service. Feb 17 19:37:57 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 2. Feb 17 19:37:57 localhost systemd[1]: Stopped parsec.service - Parsec Service. Feb 17 19:37:57 localhost systemd[1]: Starting parsec.service - Parsec Service... Feb 17 19:37:57 localhost parsec[731]: Error: Operation not permitted (os error 1) Feb 17 19:37:57 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE Feb 17 19:37:57 localhost systemd[1]: parsec.service: Failed with result 'exit-code'. Feb 17 19:37:57 localhost systemd[1]: Failed to start parsec.service - Parsec Service. Feb 17 19:37:57 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 3. Feb 17 19:37:57 localhost systemd[1]: Stopped parsec.service - Parsec Service. Feb 17 19:37:57 localhost systemd[1]: Starting parsec.service - Parsec Service... Feb 17 19:37:57 localhost parsec[732]: Error: Operation not permitted (os error 1) Feb 17 19:37:57 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE Feb 17 19:37:57 localhost systemd[1]: parsec.service: Failed with result 'exit-code'. Feb 17 19:37:57 localhost systemd[1]: Failed to start parsec.service - Parsec Service. Feb 17 19:37:57 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 4. Feb 17 19:37:57 localhost systemd[1]: Stopped parsec.service - Parsec Service. Feb 17 19:37:57 localhost systemd[1]: Starting parsec.service - Parsec Service... Feb 17 19:37:57 localhost parsec[744]: Error: Operation not permitted (os error 1) Feb 17 19:37:57 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE Feb 17 19:37:57 localhost systemd[1]: parsec.service: Failed with result 'exit-code'. Feb 17 19:37:57 localhost systemd[1]: Failed to start parsec.service - Parsec Service. Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Scheduled restart job, restart counter is at 5. Feb 17 19:37:57 localhost.localdomain systemd[1]: Stopped parsec.service - Parsec Service. Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Start request repeated too quickly. Feb 17 19:37:57 localhost.localdomain systemd[1]: parsec.service: Failed with result 'exit-code'. Feb 17 19:37:57 localhost.localdomain systemd[1]: Failed to start parsec.service - Parsec Service. ``` The config file ``` # cat /etc/parsec/config.toml # See a more complete example in /usr/shar/doc/parsec/config.toml [core_settings] [listener] listener_type = "DomainSocket" timeout = 200 # in milliseconds [authenticator] auth_type = "Direct" [[key_manager]] name = "on-disk-manager" manager_type = "OnDisk" [[provider]] provider_type = "MbedCrypto" key_info_manager = "on-disk-manager" # [[provider]] # provider_type = "Tpm" # key_info_manager = "on-disk-manager" # tcti = "device:/dev/tpmrm0" # owner_hierarchy_auth = "" ``` I installed `strace` on my system and ran the `parsec` binary through that; I think the failure is happening when the binary tries to `chmod` /var/lib/parsec/mappings ``` # sudo -u parsec strace /usr/libexec/parsec -c /etc/parsec/config.toml ... openat(AT_FDCWD, "/etc/parsec/config.toml", O_RDONLY|O_CLOEXEC) = 3 statx(0, NULL, AT_STATX_SYNC_AS_STAT, STATX_ALL, NULL) = -1 EFAULT (Bad address) statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=479, ...}) = 0 lseek(3, 0, SEEK_CUR) = 0 read(3, "# See a more complete example in"..., 479) = 479 read(3, "", 32) = 0 close(3) = 0 getuid() = 983 ioctl(2, TCGETS, {c_iflag=ICRNL|IUTF8, c_oflag=NL0|CR0|TAB0|BS0|VT0|FF0|OPOST|ONLCR, c_cflag=B38400|CS8|CREAD, c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|IEXTEN|ECHOCTL|ECHOKE, ...}) = 0 mkdir("/var/lib/parsec/mappings", 0777) = -1 EEXIST (File exists) statx(AT_FDCWD, "/var/lib/parsec/mappings", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 openat(AT_FDCWD, "/var/lib/parsec/mappings", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3 newfstatat(3, "", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_EMPTY_PATH) = 0 getdents64(3, 0x563f196fcf60 /* 2 entries */, 32768) = 48 getdents64(3, 0x563f196fcf60 /* 0 entries */, 32768) = 0 close(3) = 0 chmod("/var/lib/parsec/mappings", 0700) = -1 EPERM (Operation not permitted) write(2, "Error: ", 7Error: ) = 7 write(2, "Operation not permitted", 23Operation not permitted) = 23 write(2, " (os error ", 11 (os error ) = 11 write(2, "1", 11) = 1 write(2, ")", 1)) = 1 write(2, "\n", 1 ) = 1 sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0 munmap(0x7f0d27cd7000, 12288) = 0 exit_group(1) = ? +++ exited with 1 +++ ``` Looking at that directory, it looks like it is owned by the `dnsmasq` user and write access is only for that user: ``` # ls -laRZ /var/lib/parsec/ /var/lib/parsec/: total 12 drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 . drwxr-xr-x. 27 root root system_u:object_r:var_lib_t:s0 4096 Feb 15 23:26 .. drwxr-xr-x. 2 dnsmasq parsec-clients system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 mappings /var/lib/parsec/mappings: total 8 drwxr-xr-x. 2 dnsmasq parsec-clients system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 . drwxr-x---. 3 parsec parsec system_u:object_r:var_lib_t:s0 4096 Aug 24 15:40 .. ``` Since the directory is being created when the service/binary runs, it looks like in my case the directory was created as part of a previous deployment. This particular system started as an F36 IoT install, so perhaps the owner on that directory got mangled somehow? I'd need to test a fresh F36 IoT install.