Fedora Account System
Red Hat Associate
Red Hat Customer
Any update between May and now has changed either SELinux/kernel, Cockpit or the SELinux-policy in a way that breaks cockpit's virtual machine management when a SELinux confined user is used. By default, I assume it is an selinux-policy issue. I use F40 KDE, up to date with stable repos as of now. I have no 3rd party repos. My kernel is not tainted (cat /proc/sys/kernel/tainted = 0). My user account (in which I open the browser to use cockpit) is confined with sysadm_u with x boolean enabled. __default__ is set to user_u (which includes the user account I log into in the cockpit webinterface). This has properly worked before. I have not changed anything on my system since the last time when cockpit was working, except daily updates. Unfortunately, I have not used Cockpit since May or June, so it is unclear when it broke. The problem: When I log into cockpit and enter the virtual machine page of cockpit, I can start virtual machines, but once I try to open a remote screen, nothing happens. I tried it many times with different virtual machines. I then also checked the serial console which I can use instead of the remote screen within cockpit, and the serial console of the virtual machines outputs the following error in all cases: ``` error: failed to connect to the hypervisor error: Failed to connect socket to '/var/run/libvirt/virtqemud-sock': Permission denied disconnected ``` Then I checked and saw that several SELinux denials have occurred. I reproduced the error and saved a journal extract [1] and an ausearch [2]: The very case of the attached journal/ausearch documents the following actions (the very time of the respective action is at the end): open cockpit in browser, 162415 log into cockpit (direct to cockpit-machines page), 162435 start virtual machine, 162510 open page of the started virtual machine, 162525 try to open remote screen, attempt 1 162535 try to open remote screen, attempt 2 162550 go to serial console, 162605 go to desktop viewer, 162635 try to open remote screen, attempt 3, 162800 refresh cockpit page of very machine, 162825 try to open remote screen, attempt 4, 162855 All testing was done with 6.10.7-200.fc40.x86_64. [1] Output of `journalctl -r --boot=0 | grep "06 16:2"` (I removed all before 16:23:25): https://gitlab.com/py0xc3/tmp_e3835daaa878a97b4634/-/raw/main/sedenial-cockpit-machines_complete_journal [2] Output of `ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today` (the output of the very day is equal to the output of the very boot): https://gitlab.com/py0xc3/tmp_e3835daaa878a97b4634/-/raw/main/sedenial-cockpit-machines_complete_ausearch This is not urgent. Let me know if you need more information. Reproducible: Always Steps to Reproduce: See above: F40 KDE, up to date, with cockpit & cockpit-machines installed. User account has to be confined with sysadm_u (and x boolean has to be enabled), and __default__ has to be set to user_u. Then see steps above. Actual Results: Cockpit is not able to open remote screen or serial console. Likely due to access denial. Expected Results: Cockpit is able to open remote screen and serial console. No access denials.
I am not an selinux maintainer, but I do have some experience with selinux, VMs, and cockpit. For the record, could you post the version of these packages? $ rpm -q selinux-policy libvirt-daemon-driver-qemu cockpit systemd If you run "sudo setenforce permissive"[1], can you then connect to the remote VM? For the record, these are the cockpit-related AVCs that have permissive=0: $ fgrep permissive=0 sedenial-cockpit-machines_complete_ausearch-1.txt | fgrep cockpit type=AVC msg=audit(09/06/2024 16:24:38.016:644) : avc: denied { watch } for pid=49157 comm=cockpit-bridge path=/run/systemd dev="tmpfs" ino=2 scontext=user_u:user_r:user_t:s0 tcontext=system_u:object_r:init_var_run_t:s0 tclass=dir permissive=0 type=AVC msg=audit(09/06/2024 16:28:27.702:773) : avc: denied { watch } for pid=49157 comm=cockpit-bridge path=/run/systemd dev="tmpfs" ino=2 scontext=user_u:user_r:user_t:s0 tcontext=system_u:object_r:init_var_run_t:s0 tclass=dir permissive=0 This AVC appears to have been triggered by the virsh command. Do you recall what the specific virsh command was? It might be in your virsh history file ($HOME/.cache/libvirt/virsh/history): $ fgrep permissive=0 sedenial-cockpit-machines_complete_ausearch-1.txt | fgrep virt type=AVC msg=audit(09/06/2024 16:26:05.909:764) : avc: denied { write } for pid=50059 comm=virsh name=virtqemud-sock dev="tmpfs" ino=2396 scontext=user_u:user_r:user_t:s0 tcontext=system_u:object_r:virtqemud_var_run_t:s0 tclass=sock_file permissive=0 [1] Documentation: $ whatis -w \*enforce getenforce (8) - get the current mode of SELinux setenforce (8) - modify the mode SELinux is running in
Could you post the output from this sealert command in the journalctl output you linked? $ fgrep cockpit sedenial-cockpit-machines_complete_journal-1-reversed.txt | fgrep -m1 sealert Sep 06 16:24:42 fedora setroubleshoot[49359]: SELinux is preventing cockpit-bridge from watch access on the directory /run/systemd. For complete SELinux messages run: sealert -l adbbeaed-279e-48cc-aed5-15c2c0883a8c If you have "xclip" installed, you can run this command and paste directly into the bug report: $ sealert -l adbbeaed-279e-48cc-aed5-15c2c0883a8c | xclip
I no longer think this issue is related to confined users, although user confinement seems to impact the serial terminal. Also, SELinux does enforce even in permissive mode. ``` First, the above behavior occurred in `semanage login -l` Login Name SELinux User MLS/MCS Range Service __default__ user_u s0 * username sysadm_u s0 * root unconfined_u s0-s0:c0.c1023 * virtual user_u s0 * ``` where username is the account to login into KDE and open the Firefox browser, while virtual is the user whose credentials are used to log into the cockpit webinterface. When I do change user or __default__ to unconfined_u (semanage login -m -s unconfined_u <account>), the issue remains the same, but when I make virtual unconfined_u (semanage login -m -s unconfined_u virtual), at least the serial terminal changes its output to: ``` Connected to domain 'Fedora-40-KDE' Escape character is ^] (Ctrl + ]) ``` But the remote screen remains broken with SE denials in all cases. I verified each user account change with `semanage login -l`. The confinement condition of __default__ and user seem to NOT impact anything. Yet, as interesting as this is: the confinement has nothing to do with the issue of the major issue of the broken remote screen. When I unconfine ALL accounts to unconfined_u and verify with `semanage login -l`, the issue remains the same (except that now the serial console output is the one with "connected to domain ..."). Further, even when I do `setenforce permissive` (`setenforce permissive` and `setenforce Permissive` and `setenforce 0` all behave the same) and verify with `getenforce` (which indeed outputs "Permissive"), SELinux is still enforcing the denials: I did test while all accounts were unconfined_u (and I did reboot after I set all to unconfined_u), and after rebooting, I did again "setenforce permissive" and verified with getenforce. I did this BEFORE logging into any account (so I just started my notebook and directly went into the TTY root terminal): when I then log into my account and try again to use cockpit, the remote screen fails while SELinux denials are logged. After the last test, I again repeated `setenforce Permissive` and verified again with `getenforce` (Output: "Permissive"), I then returned to the cockpit page, refreshed the whole page with F5, and tried again. [1] is the extract of what happened at that time (journalctl -r | grep settroubleshoot), so only the last attempt that begins with refreshing the cockpit (all unconfined users). After that attempt, I just rechecked `getenforce` and still got "Permissive". Behavior is in all "Permissive" cases the same. I did all commands that manipulate SELinux in the root terminal of course (I don't use sudo). > This AVC appears to have been triggered by the virsh command. Do you recall what the specific virsh command was? No, I don't use virsh for years. Therefore, the ~/.cache/libvirt/virsh/history does not exist. I don't know if cockpit makes use of virsh, but I don't see what else can trigger it: I have `virt-manager` installed (which indeed makes use of virsh) and sometimes I use it for vm configs, but I did not open or use it during the logged attempts of my earlier post. > rpm -q selinux-policy libvirt-daemon-driver-qemu cockpit systemd selinux-policy-40.27-1.fc40.noarch libvirt-daemon-driver-qemu-10.1.0-4.fc40.x86_64 cockpit-323-1.fc40.x86_64 systemd-255.10-3.fc40.x86_64 > Could you post the output from this sealert command in the journalctl output you linked? > sealert -l adbbeaed-279e-48cc-aed5-15c2c0883a8c ``` SELinux is preventing cockpit-bridge from watch access on the directory /run/systemd. ***** Plugin catchall (100. confidence) suggests ************************** If you believe that cockpit-bridge should be allowed watch access on the systemd directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # ausearch -c 'cockpit-bridge' --raw | audit2allow -M my-cockpitbridge # semodule -X 300 -i my-cockpitbridge.pp Additional Information: Source Context user_u:user_r:user_t:s0 Target Context system_u:object_r:init_var_run_t:s0 Target Objects /run/systemd [ dir ] Source cockpit-bridge Source Path cockpit-bridge Port <Unknown> Host fedora Source RPM Packages Target RPM Packages SELinux Policy RPM selinux-policy-targeted-40.27-1.fc40.noarch Local Policy RPM selinux-policy-targeted-40.27-1.fc40.noarch Selinux Enabled True Policy Type targeted Enforcing Mode Enforcing Host Name fedora Platform Linux fedora 6.10.7-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 30 00:08:59 UTC 2024 x86_64 Alert Count 44 First Seen 2023-12-22 17:10:05 CET Last Seen 2024-09-08 12:31:29 CEST Local ID adbbeaed-279e-48cc-aed5-15c2c0883a8c Raw Audit Messages type=AVC msg=audit(1725791489.845:352): avc: denied { watch } for pid=5967 comm="cockpit-bridge" path="/run/systemd" dev="tmpfs" ino=2 scontext=user_u:user_r:user_t:s0 tcontext=system_u:object_r:init_var_run_t:s0 tclass=dir permissive=0 Hash: cockpit-bridge,user_t,init_var_run_t,dir,watch ``` -> Obviously, changing the source context with unconfined_u has not solved the remote screen issue :) A general question: Does cockpit work for you at the moment? In the current update state of Fedora? Also, if you have time, can you verify if it works with user confinement (the user that opens cockpit webinterface needs at least sysadm_u)? I just tried it again with kernel 6.10.8, but the outcome is the same. I somehow hope that I have an error in reasoning or so somewhere :O [1]: ``` Sep 08 13:16:17 fedora systemd[1]: setroubleshootd.service: Consumed 2.216s CPU time. Sep 08 13:16:17 fedora audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=setroubleshootd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 08 13:16:17 fedora systemd[1]: setroubleshootd.service: Deactivated successfully. Sep 08 13:16:07 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd-backe from map access on the file /memfd:memory-backend-memfd (deleted). Sep 08 13:16:07 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd-backe from map access on the file /memfd:memory-backend-memfd (deleted). For complete SELinux messages run: sealert -l dbdcdb89-23fc-4812-9ac3-875638163ad7 Sep 08 13:16:07 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd-backe from write access on the file /memfd:memory-backend-memfd (deleted). Sep 08 13:16:07 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd-backe from write access on the file /memfd:memory-backend-memfd (deleted). For complete SELinux messages run: sealert -l 8a9572f7-14f2-4ba6-991f-7b2fcf5e5fb4 Sep 08 13:16:03 fedora setroubleshoot[9069]: SELinux is preventing rpc-virtqemud from open access on the chr_file /dev/pts/2. Sep 08 13:16:03 fedora setroubleshoot[9069]: SELinux is preventing rpc-virtqemud from open access on the chr_file /dev/pts/2. For complete SELinux messages run: sealert -l 30c10120-0bde-4bc4-bdbe-8386b9af3a4a Sep 08 13:16:03 fedora setroubleshoot[9069]: failed to retrieve rpm info for path '/dev/pts/2': Sep 08 13:15:59 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd from mount access on the filesystem /proc. Sep 08 13:15:59 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd from mount access on the filesystem /proc. For complete SELinux messages run: sealert -l 85aa0c01-849e-4ba1-9dfb-9ee1535fabfe Sep 08 13:15:59 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd from using the setpcap capability. Sep 08 13:15:59 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd from using the setpcap capability. For complete SELinux messages run: sealert -l fedff7a3-d238-4345-89a0-f0938d181481 Sep 08 13:15:59 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd from using the sys_resource capability. Sep 08 13:15:59 fedora setroubleshoot[9069]: SELinux is preventing virtiofsd from using the sys_resource capability. For complete SELinux messages run: sealert -l 2f1e9e7a-6ec3-48da-b7a8-8a911638b635 Sep 08 13:15:56 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=setroubleshootd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 08 13:15:56 fedora systemd[1]: Started setroubleshootd.service - SETroubleshoot daemon for processing new SELinux denial logs. Sep 08 13:15:56 fedora systemd[1]: Starting setroubleshootd.service - SETroubleshoot daemon for processing new SELinux denial logs... Sep 08 13:12:08 fedora systemd[1]: setroubleshootd.service: Consumed 2.040s CPU time. Sep 08 13:12:08 fedora audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=setroubleshootd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 08 13:12:08 fedora systemd[1]: setroubleshootd.service: Deactivated successfully. ```
Chris: Thanks for your followup report. > with `getenforce` (which indeed outputs "Permissive"), SELinux is still enforcing the denials: Selinux AVCs will *still be logged* in that configuration, but selinux should allow all accesses: "The permissive option enables the SELinux code, but causes it to operate in a mode where accesses that would be denied by policy are permitted but audited." (selinux(8)) Could you revert all your selinux customizations, so we can debug a default configuration? Here is what I have on my F40 Xfce system: $ sudo semanage login -l Login Name SELinux User MLS/MCS Range Service __default__ unconfined_u s0-s0:c0.c1023 * root unconfined_u s0-s0:c0.c1023 * Also, thanks for confirming that you do not explicitly run virsh.
After disabling all your selinux customizations, could you run the following test? Reboot. $ sudo setenforce permissive # run as root, if you prefer. Run your cockpit test case. Post the output from this command: $ sudo ausearch -i -ts boot -m avc This is not part of the test -- restore enforcing mode or reboot: $ sudo setenforce enforcing
The above test was done in a default setting. I work with the maintainer to improve SELinux policies to fit confined user accounts. So I report to them if issues occur with a confined user and test their subsequent policy adjustments from bodhi/koji before they end up in stable. I do not customize Fedora's policies manually. So my SELinux and its policies are all what has been installed by default at the era of F37 plus all official subsequent updates (now at F40). Compared to the default with which Fedora is shipped, I only set my user account separated to sysadm_u (now also the virtual user for cockpit) and __default__ to user_u. However, since this issue seems so far to be not related to confined users, I have already set all accounts to unconfined_u in the above test. Therefore, the recent tests impose the default setting of confinement -> all accounts are set unconfined_u. MLS/MCS is not in use. Keep in mind that it is equal if an account is separately set to unconfined_u or if it gets unconfined_u through __default__.
(In reply to Christopher Klooz from comment #6) > The above test was done in a default setting. Thanks for your clarification. Then all that is needed is the report from "ausearch".
> could you run the following test? My previous test already implements your test, although it does not capture the denials of logging in but only attempts to use the VM management and to open a remote screen (so beginning with having the already logged in cockpit-webinterface refreshed, as elaborated above). Here is the ausearch output of the very time of the above extract (the maintainer prefers to always also have some more details including `ausearch -i -m avc,user_avc,selinux_err,user_selinux_err`): ``` ---- type=AVC msg=audit(09/08/2024 13:15:54.158:474) : avc: denied { sys_resource } for pid=8979 comm=virtiofsd capability=sys_resource scontext=system_u:system_r:virtqemud_t:s0 tcontext=system_u:system_r:virtqemud_t:s0 tclass=capability permissive=1 ---- type=AVC msg=audit(09/08/2024 13:15:54.160:475) : avc: denied { setpcap } for pid=8979 comm=virtiofsd capability=setpcap scontext=system_u:system_r:virtqemud_t:s0 tcontext=system_u:system_r:virtqemud_t:s0 tclass=capability permissive=1 ---- type=AVC msg=audit(09/08/2024 13:15:54.160:476) : avc: denied { mount } for pid=8982 comm=virtiofsd name=/ dev="proc" ino=1 scontext=system_u:system_r:virtqemud_t:s0 tcontext=system_u:object_r:proc_t:s0 tclass=filesystem permissive=1 ---- type=AVC msg=audit(09/08/2024 13:16:01.009:506) : avc: denied { open } for pid=2062 comm=rpc-virtqemud path=/dev/pts/2 dev="devpts" ino=5 scontext=system_u:system_r:virtqemud_t:s0 tcontext=system_u:object_r:svirt_devpts_t:s0 tclass=chr_file permissive=1 ---- type=AVC msg=audit(09/08/2024 13:16:05.603:510) : avc: denied { write } for pid=8982 comm=virtiofsd-backe path=/memfd:memory-backend-memfd (deleted) dev="tmpfs" ino=113 scontext=system_u:system_r:virtqemud_t:s0 tcontext=system_u:object_r:svirt_tmpfs_t:s0 tclass=file permissive=1 ---- type=AVC msg=audit(09/08/2024 13:16:05.604:511) : avc: denied { map } for pid=8982 comm=virtiofsd-backe path=/memfd:memory-backend-memfd (deleted) dev="tmpfs" ino=113 scontext=system_u:system_r:virtqemud_t:s0 tcontext=system_u:object_r:svirt_tmpfs_t:s0 tclass=file permissive=1 ---- ``` Sorry, I forgot to add the ausearch outputs in the last test :) I try to find some time next week to add a full test from logging into the user account through logging into cockpit and then trying to use VMs/remote screen etc., but everything with all accounts as unconfined_u and with doing everything in permissive mode (and check if it is permissive at the beginning and at the end). So just like my initial test with the exact times when I did what, but in permissive and with unconfined_u (so that should also implement your test case). I'll let you know. I don't wonder about such issues in confinement or issues between policies and cockpit (we had that before - especially after major updates in cockpit), but the behavior in permissive makes me really wondering.
I just reviewed and realized the denials logged permissive=1 (as far as I remember the permissive=1 entries fit the times when I had set it to permissive). Is it possible that SELinux was actually permissive but the log entries of SELinux are a little unclear in permissive? So that they still log terms like "denied" and formulations like "SELinux is preventing * from write access" even if it is only logged but not enforced? That would explain the situation and indicate that it is actually no SELinux issue at all.
> Is it possible that SELinux was actually permissive but the log entries of SELinux are a little unclear in permissive? Very possible. :-) When "permissive=1" is in an AVC record, the word "denied" is indeed misleading. I only look for "permissive=0" or "permissive=1". Further, AVC records are frequently missing essential details, so Zdenek has to ask for full auditing to be enabled: https://fedoraproject.org/wiki/SELinux/Debugging#Enable_full_auditing > That would explain the situation and indicate that it is actually no SELinux issue at all. That is what I was thinking too, although your ausearch report in Comment 8 still shows a possible issue with "comm=virtiofsd". Presumably, that command is from this package: $ rpm -qf /usr/libexec/virtiofsd virtiofsd-1.10.1-1.fc40.x86_64 If you want to investigate further, you could try enabling full auditing, as mentioned above. Documentation: auditctl (8) - a utility to assist controlling the kernel's audit system auditd (8) - The Linux Audit daemon
> That would explain the situation and indicate that it is actually no SELinux issue at all. > That is what I was thinking too With this in mind, I just checked for cockpit updates in bodhi and saw there is a major update that also contains bug fixes: https://bodhi.fedoraproject.org/updates/FEDORA-2024-35b9a68bf3 I have just updated to that cockpit version ( cockpit-324-1.fc40 ) and repeated the full test after a reboot. I did a full test with all unconfined_u and SELinux in permissive mode (I set all to unconfined_u before the boot of the test). I was setting SELinux to permissive once the system booted (before logging into my account to open Firefox and then cockpit within) and verified SELinux's permissive mode in the second test before and after the test: in remained permissive during the test, and the log entries contain all permissive=1, although the logs are formulated in an "enforcing" way. The issue remains the same in the new cockpit version. However, in unconfined_u modes, I can confirm that `virt-manager` works fine. So it is not related to the KVM/qemu/libvirt. The new tests took already place on kernel 6.10.8-200.fc40.x86_64. Here the tests that are logged in the ausearch output of [1]: 2) unconfined_u and permissive: BOOT -1 b7fd7a7376bc4851b4301c08a79c1640 Sun 2024-09-08 18:22:01 CEST Sun 2024-09-08 18:26:12 CEST 182403 logged into cockpit (directly to a vm) 182425 start vm 182450 click "Launch remote viewer" 182505 click "Launch remote viewer" again 182515 switch to Serial Console, which outputs: ``` Connected to domain 'Fedora-40-KDE' Escape character is ^] (Ctrl + ]) ``` [1] `ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today` (extract is limited to the two boots) -> https://gitlab.com/py0xc3/tmp_e3835daaa878a97b4634/-/raw/main/ausearch_b7 @zpytela Since there are several denials even in unconfined_u, I leave the ticket at selinux-policy for your consideration. But with the assumption that SELinux was really in permissive mode despite the formulation in the logs, I assume this is an issue with cockpit. I suggest to put this to cockpit. Does that make sense? Otherwise, let me know if and what you need from me. @y9t7sypezp Thanks for your support ;)
Chris: Thanks for your careful testing and detailed reporting. Comment 3: "Does cockpit work for you at the moment?" With Cockpit, I can run and shutdown remote VMs on the server[1]. With a remote F40 Server VM[2], I can get a serial console. With a remote F40 Xfce VM, I can get a serial console after booting if I append "console=ttyS0" to the kernel command line from the GRUB menu (in the Cockpit serial console window). In the Cockpit serial console window, can you see the GRUB menu at all on the remote VM? (It is severely garbled, but still functional.) I cannot get a desktop viewer with the F40 Xfce VM, although that could be due to misconfiguration. [1] The server is an Intel NUC11 on my LAN. The server is running F40 Xfce in headless mode booting to multi-user.target. [2] https://fedoraproject.org/server/download
(In reply to Steve from comment #12) > I cannot get a desktop viewer with the F40 Xfce VM, although that could be due to misconfiguration. Here is one configuration change I had to make. In virt-manager, for the remote VM, "Display Spice" must have this configuration: <graphics type="spice" port="5900" autoport="yes" listen="0.0.0.0"> <listen type="address" address="0.0.0.0"/> </graphics> And when the VM is running, port 5900 should be open and the listen address should be 0.0.0.0: $ sudo ss -np state listening sport inet::5900 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp 0 4096 0.0.0.0:5900 0.0.0.0:* users:(("qemu-system-x86",pid=8721,fd=11)) Documentation (under "Using virt-manager"): https://www.spice-space.org/spice-user-manual.html
(In reply to Steve from comment #12) > With a remote F40 Xfce VM, I can get a serial console after booting > if I append "console=ttyS0" to the kernel command line from the GRUB menu (in the Cockpit serial console window). That isn't necessary if the service is explicitly enabled: $ sudo systemctl enable --now serial-getty $ systemctl -q list-units -a serial-getty serial-getty loaded active running Serial Getty on ttyS0
What you have described initially sounded like you hit the same bug as I: I can also start, stop VMs and such. The issue starts with opening the remote screen. A major point is that I have not changed any related configurations, neither on host nor in the VM, whereas I have used the remote screen regularly in the past (actually, I did most things with the remote screen). Confinement seems to have an impact on the serial console, but on nothing else. At the same time, virt-manager has no problems with opening the same machines in its remote screen (we talk here about testing without confinement). > Here is one configuration change I had to make. In virt-manager, for the remote VM, "Display Spice" must have this configuration: Does that mean with the change in the "graphics type" configuration you made the remote screen working again in a VM in which it did not work before that change? Then the issue is maybe a change in cockpit that is no longer compatible to some older configurations of VM. I might try your config later. The major tests above were done using a VM that has this config: ``` <graphics type="spice" autoport="yes"> <listen type="address"/> <image compression="off"/> </graphics> ``` Yet, it remains working for virt-manager and used to work in cockpit (my other VMs I used with remote screen all have the same configuration). And I wonder that this type of change causes such an effect, as I do not even get the link of the remote screen file from cockpit at the moment (effectively it looks like the button is broken). That's why I ask if that change on itself fixed the issue from not-working remote screen to a working one :) Thanks for contributing btw ;)
(In reply to Christopher Klooz from comment #15) > > Here is one configuration change I had to make. In virt-manager, for the remote VM, "Display Spice" must have this configuration: > > Does that mean with the change in the "graphics type" configuration you made the remote screen working again in a VM in which it did not work before that change? I apologize for not being clear. That "Display Spice" configuration did not get remote VM graphics working with Cockpit. I have reverted it. > The major tests above were done using a VM that has this config: > ``` > <graphics type="spice" autoport="yes"> > <listen type="address"/> > <image compression="off"/> > </graphics> > ``` Thanks for posting that. My current "Display Spice" configuration (before booting the VM) is: <graphics type="spice" autoport="yes"> <listen type="address"/> </graphics> Where did your <image compression="off"/> option come from?
> Where did your <image compression="off"/> option come from? It is the default when `virt-manager` creates a VM with Display. I cannot verify it at the moment, but I think Cockpit uses it as well when it creates VMs (I can see that this "image compression" property is in all my VMs with Display, and I think some of them have been created with cockpit, but I am not 100% sure). By the way, I have posted the issue to bodhi [1] with karma so that the maintainers of cockpit know about it (I added a link to the bug ticket here). So far, I think this is simply a bug in cockpit: The "test case" for cockpit in bodhi does not contain a test of virtual machines. So that bug could have remain hidden in testing, and thus it only appears if someone uses cockpit with this specific function (so remote screen) and then also reports about it. Afaik, cockpit machines is not installed by default with cockpit. So it is not clear if the use of this function is widespread. Let's see if the maintainers have some idea and if they can verify it. I would wait for Zdenek to check if this ticket is relevant for him, otherwise I will move it to cockpit component. [1] https://bodhi.fedoraproject.org/updates/FEDORA-2024-35b9a68bf3
(In reply to Christopher Klooz from comment #17) > > Where did your <image compression="off"/> option come from? > > It is the default when `virt-manager` creates a VM with Display. I cannot > verify it at the moment, but I think Cockpit uses it as well when it creates > VMs (I can see that this "image compression" property is in all my VMs with > Display, and I think some of them have been created with cockpit, but I am > not 100% sure). Thanks. I must be doing something different. I recreated my remote VM with virt-manager using the same disk image and did not get the <image compression="off"/> option under "Display Spice". When creating the VM, I chose "Fedora Linux 40" for the "Operating System", and configured vCPU=2, Memory=4096/8192. The connection protocol is: qemu+ssh://[remote user]@[remote host].lan/system Is there anything else I should be considering?
This test case without cockpit works as expected. SSH into the remote system and start a VM with a graphical desktop: $ virsh -c qemu:///system virsh # start fedora-xfce-4 Domain 'fedora-xfce-4' started virsh # list Id Name State ------------------------------- 5 fedora-xfce-4 running On the local system: $ virt-viewer --connect qemu+ssh://[remote user]@[remote system].lan/system fedora-xfce-4 NB: SSH on the remote system has been configured with public key authentication, so there are no password prompts.
(In reply to Steve from comment #18) > I recreated my remote VM with virt-manager using the same disk image and did not get the <image compression="off"/> option under "Display Spice". Several of my local VMs do indeed have the <image compression="off"/> option. As a test, I connected with VNC to my remote system and created a test VM. That VM also had the <image compression="off"/> option. So the difference seems to depend on whether libvirt "thinks" the VM is being run locally or remotely. That seems like a good idea, but such differences can make it tricky to reproduce problems.
(In reply to Christopher Klooz from comment #8) > type=AVC msg=audit(09/08/2024 13:15:54.158:474) : avc: denied { sys_resource } for pid=8979 comm=virtiofsd capability=sys_resource scontext=system_u:system_r:virtqemud_t:s0 tcontext=system_u:system_r:virtqemud_t:s0 tclass=capability permissive=1 That AVC appears to be the same as the one reported here: Bug 2303663 - SELinux is preventing virtiofsd from using the 'sys_resource' capabilities. If you can reproduce it, it would be better to report it in that bug. See, in particular, Bug 2303663, Comment 4.
> So the difference seems to depend on whether libvirt "thinks" the VM is being run locally or remotely. Agreed :) I was thinking the same when I read this: > The connection protocol is: qemu+ssh://[remote user]@[remote host].lan/system ... because I think this is the major difference between your VMs and mine: my virt-manager/cockpit connect to localhost. ------------ > That AVC appears to be the same as the one reported here: Indeed, I added a comment there. I guess it makes sense to emphasize the denial stuff in the other topic and focus here on the cockpit issue, which I think is (strongly) indicated to be no SELinux issue. I thus move this ticket to cockpit.
(In reply to Christopher Klooz from comment #22) Thanks for confirming that you are testing with VMs on localhost. That greatly simplifies things! :-) Now that you have changed the component for this bug report to "cockpit", the "needinfo" flag for Zdenek is probably not needed.
We might be able to get more details by enabling cockpit debugging. The relevant section is about three-quarters of the way down the page: Debug logging of Cockpit processes https://cockpit-project.org/external/source/HACKING
(In reply to Steve from comment #24) > We might be able to get more details by enabling cockpit debugging. Maybe not. I got a flurry of these: $ sudo ausearch -i -ts boot -m avc ---- type=AVC msg=audit(09/10/2024 11:16:26.950:408) : avc: denied { dac_override } for pid=17664 comm=cockpit-tls capability=dac_override scontext=system_u:system_r:cockpit_ws_t:s0 tcontext=system_u:system_r:cockpit_ws_t:s0 tclass=capability permissive=0 ---- ... $ cat /etc/systemd/system/cockpit.service.d/debug.conf [Service] Environment=G_MESSAGES_DEBUG=cockpit-ws,cockpit-bridge User=root Group=
The issue from #c1 can be addressed in selinux-policy: https://github.com/fedora-selinux/selinux-policy/pull/2350 Denials for virt drivers are still being worked on in rawhide. I can't see here anything else for resolving in selinux-policy, please let me know if you do.
> We might be able to get more details by enabling cockpit debugging. Auto-pushing to stable for the next cockpit update has been disabled for now anyway, although I am not sure if that helps as earlier versions are affected too as far as I experience it. However, I suggest to wait for the maintainer and their thoughts. > the "needinfo" flag for Zdenek is probably not needed. Yeah, I too seldomly work here in bugzilla. I am not used that tagging a user always leads to a permanent needinfo flag in the ticket ;) Sorry Zdenek :) > Denials for virt drivers are still being worked on in rawhide. Thanks!
Now testing cockpit with F42 (rawhide) Workstation in a VM: $ journalctl --no-hostname -b -p warning | egrep -i 'cockpit|virtqemud' Sep 11 06:37:13 cockpit-session[4362]: pam_ssh_add: Failed adding some keys Sep 11 06:38:15 virtqemud[6826]: Guest agent is not responding: QEMU guest agent is not connected Tested with: $ rpm -q cockpit-ws cockpit-machines qemu-guest-agent libvirt-daemon-driver-qemu selinux-policy systemd cockpit-ws-324-1.fc42.x86_64 cockpit-machines-319-1.fc42.noarch qemu-guest-agent-9.1.0-1.fc42.x86_64 libvirt-daemon-driver-qemu-10.7.0-1.fc42.x86_64 selinux-policy-41.16-1.fc42.noarch systemd-256.5-1.fc42.x86_64 $ uname -r 6.11.0-0.rc7.56.fc42.x86_64
(In reply to Steve from comment #28) > Now testing cockpit with F42 (rawhide) Workstation in a VM: After starting the test VM with Cockpit, when I click on "Launch remote viewer", Cockpit opens a Firefox* downloads window listing files like this: $ cat ./Downloads/CXa3_I6l [virt-viewer] type=spice host=127.0.0.1 port=5900 delete-this-file=1 fullscreen=0 [...............................GraphicsConsole] After clicking on the file name in the Firefox downloads window, the test VM's graphical desktop is displayed. IOW, with F42 Workstation, Cockpit is working as expected. Here are some of the configuration changes that were needed: $ sudo systemctl enable --now cockpit.socket $ sudo dnf install virt-viewer Earlier, I had used virt-manager to install and configure the the test VM (under the qemu:///system connection). I had also added my test user to the "libvirt" group. * Started from the command line: $ firefox --safe-mode
>$ sudo systemctl enable --now cockpit.socket >$ sudo dnf install virt-viewer I'm wondering that you were using cockpit without these two in the past? Without the latter, rmeote viewer should have never worked at all. The first is necessary so that cockpit starts on boot. In any case, I have already both. So that's not the origin of the problem. It used to work in the configuration I have, and at some update (of any package), it broke. However, I saw in a virtual machine in which I installed cockpit today (with its current defaults as of today) that in this machine/configuration, cockpit is able to create the file to open the VM in the remote viewer. It is F40 KDE with stable repos, just like my host. > After starting the test VM with Cockpit, when I click on "Launch remote viewer", Cockpit opens a Firefox* downloads window listing files like this: Yes, that's intended. These files can be set to be directly opened with the virt-viewer. The issue in this ticket is effectively that these files are not created. So the buttons are "dead". I am wondering if some settings have changed and are no longer compatible to earlier settings. I will try tomorrow to renew my config files of cockpit and see if it then works. > I had also added my test user to the "libvirt" group. I have that already, so definitely not the origin. I think that could not explain the issue anyway. The firefox in the working VM and on the host with the broken buttons is the same as well. It has to be something around the cockpit configuration/settings that has changed (or changed its compatibility), or something with which it interfaces. The issue occurs within a Firefox tab (= a button on a website does not contain a link to a file that needs to be downloaded/opened), so I exclude that it has something to do with KDE/Qt/Plasma-related stuff. Let's see what resetting configs will do.
(In reply to Christopher Klooz from comment #30) > The issue in this ticket is effectively that these [vv] files are not created. So the buttons are "dead". I enabled the Firefox inspector (ctrl-shift-i), and the Console showed this when I clicked on "Launch remote viewer": Content-Security-Policy: The page’s settings blocked the loading of a resource (frame-src) at data:application/x-virt-viewer,%5Bvirt-v… because it violates the following directive: “default-src 'self'” Subsequent clicks increase the counter in the blue ovoid at the right end of that line. Tested with: $ firefox --safemode $ rpm -q firefox firefox-130.0-3.fc40.x86_64
(In reply to Steve from comment #31) > Content-Security-Policy: The page’s settings blocked the loading of a resource (frame-src) at data:application/x-virt-viewer,%5Bvirt-v… because it violates the following directive: “default-src 'self'” Apparently, "CSP" is a whole subject: Content Security Policy (CSP) https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP > $ firefox --safemode Firefox doesn't complain about that, but it should be: $ firefox --safe-mode ^
A blog post from 2018 describes Cockpit's Content Security Policy in the section titled: Security Policy within the browser https://cockpit-project.org/blog/is-cockpit-secure.html ``` In Cockpit’s case we send a strict Content-Security-Policy header that only allows code installed in Cockpit packages on the logged into system to be run. ``` ``` The default security policy looks like this: Content-Security-Policy: default-src 'self' connect-src 'self' ws: wss: ```
The Machines CSP issue has been fixed in cockpit-machines 319. https://github.com/cockpit-project/cockpit-machines/commit/7b1ef1a9e3441a9c11d278fb3129c59e675caeda
Thanks Jelle! I have just updated to cockpit-machines-319-1.fc40 [1] and this indeed solves the issue. These commands solved the issue: ``` dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-9ea187ad82 systemctl restart cockpit.socket systemctl restart cockpit.service ``` Maybe it is also necessary to clear cache & cookies in the browser (mine does it automatically). After that, everything worked fine :) Steve, I assume this also solves your affected hosts. But if not, feel free to let me know here so that I can reopen the ticket for you. Otherwise, I assume the three above commands solve the issue for everybody, and in a few hours/days, only a `dnf update` will be necessary. Greetings & thanks to the cockpit team :) I adjusted Karma in `cockpit-324-1.fc40` to +1 as it seems that package is not related. [1] https://bodhi.fedoraproject.org/updates/FEDORA-2024-9ea187ad82
Updating to cockpit-machines-0:319-1.fc40.noarch does indeed stop the CSP errors reported in Comment 31.* However, "Launch remote viewer" is still not working as expected. The Firefox inspector (ctrl-shift-i) console shows these errors: 08:52:56.778 The resource from “http://localhost:9090/cockpit/@localhost/*/po.manifest.js” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). localhost:9090 08:52:56.778 The resource from “http://localhost:9090/cockpit/@localhost/*/po.js” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). localhost:9090 08:52:56.904 The resource from “http://localhost:9090/cockpit/@localhost/*/po.manifest.js” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). localhost:9090 08:52:56.905 Loading failed for the <script> with source “http://localhost:9090/cockpit/@localhost/*/po.manifest.js”. localhost:9090:14:39 * Actually, I updated all the cockpit packages and rebooted: $ dnf -C repoquery --installed cockpit\* cockpit-0:324-1.fc40.x86_64 cockpit-bridge-0:324-1.fc40.x86_64 cockpit-machines-0:319-1.fc40.noarch cockpit-networkmanager-0:324-1.fc40.noarch cockpit-packagekit-0:324-1.fc40.noarch cockpit-storaged-0:324-1.fc40.noarch cockpit-system-0:324-1.fc40.noarch cockpit-ws-0:324-1.fc40.x86_64 Testing is with: $ firefox --safe-mode $ rpm -q firefox firefox-130.0-3.fc40.x86_64
There is already much consolidated in this ticket, and there is thus already a long mailing list for each post. Because the very issue of this ticket is solved, I think it makes sense to open a new ticket for the Firefox inspector entries that are caused by 319-1, doesn't it?