Bug 1917024

Summary: [qemu-guest-agent] guest-ssh-* commands not working in 5.2.0 package
Product: Red Hat Enterprise Linux 9 Reporter: Nikita Kalyanov <nikitakalyanov>
Component: selinux-policyAssignee: Nikola Knazekova <nknazeko>
Status: CLOSED ERRATA QA Contact: Milos Malik <mmalik>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.1CC: berrange, cfergeau, crobinso, dholler, dwalsh, grepl.miroslav, itamar, lvrabec, marcandre.lureau, mmalik, omosnace, ondrejj, pbonzini, philmd, pkoncity, rjones, sgott, virt-maint, vmojzis, zpytela
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: selinux-policy-38.1.6-1.el9 Doc Type: Bug Fix
Doc Text:
Cause: Missing permissions prevents virt_qemu_ga_t to read ssh home directory Consequence: QEMU Guest Agent new feature for managing guest ssh keys is not working Fix: Boolean to allow qemu-ga read ssh home directory Result: QEMU Guest Agent new feature for managing guest ssh keys is working
 Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-09 08:16:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
terminal log of a rhel9 guest
none
terminal log of rhel 9.1 none

Description Nikita Kalyanov 2021-01-16 17:55:26 UTC 
Description of problem:
QEMU Guest Agent 5.2.0 added some new features including commands for managing guest ssh keys (guest-ssh-get-authorized-keys, guest-ssh-add-authorized-keys, guest-ssh-remove-authorized-keys). These are not working in 2:5.2.0-4.fc34 rawhide package due to permissions errors.


Version-Release number of selected component (if applicable):
qemu-guest-agent.x86_64                       2:5.2.0-4.fc34

How reproducible: always


Steps to Reproduce:
1. Launch a VM with qemu-kvm + libvirt using Fedora 33 image (https://alt.fedoraproject.org/cloud/)
2. Install qemu-guest-agent package from rawhide inside the guest:
sudo dnf install fedora-repos-rawhide
sudo dnf --disablerepo=* --enablerepo=rawhide --releasever=34 install qemu-guest-agent
sudo systemctl start qemu-guest-agent
3. Try to use ssh keys-related commands from host with virsh:
sudo virsh qemu-agent-command instance-0000000a '{"execute":"guest-ssh-get-authorized-keys", "arguments": {"username": "fedora"}}'

Actual results:
error: internal error: unable to execute QEMU agent command 'guest-ssh-get-authorized-keys': failed to read '/home/fedora/.ssh/authorized_keys': Failed to open file “/home/fedora/.ssh/authorized_keys”: Permission denied

There may also be other errors for different users/commands:
sudo virsh qemu-agent-command instance-0000000a '{"execute":"guest-ssh-add-authorized-keys", "arguments": {"username": "fedora", "keys": ["test"]}}'
error: internal error: unable to execute QEMU agent command 'guest-ssh-add-authorized-keys': failed to create directory '/home/fedora/.ssh': Permission denied
sudo virsh qemu-agent-command instance-0000000a '{"execute":"guest-ssh-add-authorized-keys", "arguments": {"username": "root", "keys": ["test"], "reset": false}}'
error: internal error: unable to execute QEMU agent command 'guest-ssh-add-authorized-keys': failed to create directory '/root/.ssh': File exists
sudo virsh qemu-agent-command instance-0000000a '{"execute":"guest-ssh-get-authorized-keys", "arguments": {"username": "root"}}'
error: internal error: unable to execute QEMU agent command 'guest-ssh-get-authorized-keys': failed to read '/root/.ssh/authorized_keys': Failed to open file “/root/.ssh/authorized_keys”: Permission denied

The remove command returns successfully:
sudo virsh qemu-agent-command instance-0000000a '{"execute":"guest-ssh-remove-authorized-keys", "arguments": {"username": "root", "keys": ["test"]}}' | jq
{
  "return": {}
}
but it does not actually removes any keys inside the guest.





Expected results:
The commands work without errors


Additional info:
It seems like a bug/misconfiguration of the qemu-guest-agent systemd service. When I run the guest agent "manually" (without systemd):
sudo systemctl stop qemu-guest-agent
FSFREEZE_HOOK_PATHNAME=/etc/qemu-ga/fsfreeze-hook sudo /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=${BLACKLIST_RPC} -F${FSFREEZE_HOOK_PATHNAME}
the commands are working as expected.

Some more versions:
guest OS:
uname -sr
Linux 5.8.15-301.fc33.x86_64
host virsh:
virsh --version
3.9.0
Comment 1 Cole Robinson 2021-01-19 14:37:24 UTC 
Thanks for the report. Can you try `setenforce 0` in the VM and then retry? That can help narrow down the permission issues

cc marc-andre
Comment 2 Nikita Kalyanov 2021-01-19 15:08:03 UTC 
(In reply to Cole Robinson from comment #1)
> Thanks for the report. Can you try `setenforce 0` in the VM and then retry?
> That can help narrow down the permission issues
> 
> cc marc-andre

After `setenforce 0` the commands work fine.
Comment 3 Ben Cotton 2021-02-09 15:40:46 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.
Comment 4 Cole Robinson 2022-02-23 16:00:01 UTC 
Still relevant with f35. selinux-policy needs to be extended here to allow qemu_ga to read + write .ssh/authorized_keys

These qemu-guest-agent guest-ssh-* management APIs were requested by kubevirt devs here: https://bugzilla.redhat.com/show_bug.cgi?id=1885332

The goal is to provide a targeted operation rather than the wide open guest-open-file or guest-exec commands which selinux locks down (and RHEL explicitly disable those commands).

@zpytela I think this covers the info you were asking about in bug 2028762
Comment 5 Zdenek Pytela 2022-03-04 10:19:19 UTC 
We are discussing internally about possible approach towards resolving the issues.
Comment 6 Dominik Holler 2022-11-09 16:45:47 UTC 
Created attachment 1923396 [details]
terminal log of a rhel9 guest

please note that rhel9 seems to be affected, too.
Comment 7 Ben Cotton 2022-11-29 16:51:20 UTC 
This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.
Comment 8 Dominik Holler 2022-12-06 14:44:39 UTC 
Changing to RHEL to prevent the bug from being closed.

(In reply to Ben Cotton from comment #7)
> This message is a reminder that Fedora Linux 35 is nearing its end of life.
> Fedora will stop maintaining and issuing updates for Fedora Linux 35 on
> 2022-12-13.
> It is Fedora's policy to close all bug reports from releases that are no
> longer
> maintained. At that time this bug will be closed as EOL if it remains open
> with a
> 'version' of '35'.
> 
> Package Maintainer: If you wish for this bug to remain open because you
> plan to fix it in a currently maintained version, change the 'version' 
> to a later Fedora Linux version.
> 
> Thank you for reporting this issue and we are sorry that we were not 
> able to fix it before Fedora Linux 35 is end of life. If you would still
> like 
> to see this bug fixed and are able to reproduce it against a later version 
> of Fedora Linux, you are encouraged to change the 'version' to a later
> version
> prior to this bug being closed.
Comment 10 Milos Malik 2022-12-06 16:05:59 UTC 
Please gather SELinux denials which appear during your use case:

# ausearch -m avc -m user_avc -m selinux_err -i -ts today

and attach them to this BZ.

The attachment in comment#6 is not sufficient.

Thank you.
Comment 11 Dominik Holler 2022-12-07 14:44:18 UTC 
Created attachment 1930827 [details]
terminal log of rhel 9.1

(In reply to Milos Malik from comment #10)
> Please gather SELinux denials which appear during your use case:
> 
> # ausearch -m avc -m user_avc -m selinux_err -i -ts today
> 
> and attach them to this BZ.
> 
> The attachment in comment#6 is not sufficient.
> 
> Thank you.

Thanks for having a look and for the precise request!

[root@gacheck ~]# ausearch -m avc -m user_avc -m selinux_err -i -ts today
----
type=PROCTITLE msg=audit(12/07/22 15:40:24.129:54) : proctitle=/usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist=guest-file-open,guest-file-c 
type=SYSCALL msg=audit(12/07/22 15:40:24.129:54) : arch=x86_64 syscall=openat success=no exit=EACCES(Permission denied) a0=AT_FDCWD a1=0x562d0c88a5f0 a2=O_RDONLY a3=0x0 items=0 ppid=1 pid=691 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=qemu-ga exe=/usr/bin/qemu-ga subj=system_u:system_r:virt_qemu_ga_t:s0 key=(null) 
type=AVC msg=audit(12/07/22 15:40:24.129:54) : avc:  denied  { search } for  pid=691 comm=qemu-ga name=.ssh dev="vda4" ino=12164 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:ssh_home_t:s0 tclass=dir permissive=0
Comment 12 Dominik Holler 2022-12-07 14:47:41 UTC 
[root@gacheck ~]# sealert -l e686141c-9ab0-4e31-9a92-53f3ce2c8b73
SELinux is preventing /usr/bin/qemu-ga from search access on the directory .ssh.

*****  Plugin catchall_boolean (89.3 confidence) suggests   ******************

If you want to allow virt to qemu ga read nonsecurity files
Then you must tell SELinux about this by enabling the 'virt_qemu_ga_read_nonsecurity_files' boolean.

Do
setsebool -P virt_qemu_ga_read_nonsecurity_files 1

*****  Plugin catchall (11.6 confidence) suggests   **************************

If you believe that qemu-ga should be allowed search access on the .ssh directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'qemu-ga' --raw | audit2allow -M my-qemuga
# semodule -X 300 -i my-qemuga.pp


Additional Information:
Source Context                system_u:system_r:virt_qemu_ga_t:s0
Target Context                system_u:object_r:ssh_home_t:s0
Target Objects                .ssh [ dir ]
Source                        qemu-ga
Source Path                   /usr/bin/qemu-ga
Port                          <Unknown>
Host                          gacheck
Source RPM Packages           qemu-guest-agent-7.0.0-13.el9.x86_64
Target RPM Packages           
SELinux Policy RPM            selinux-policy-targeted-34.1.43-1.el9.noarch
Local Policy RPM              selinux-policy-targeted-34.1.43-1.el9.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     gacheck
Platform                      Linux gacheck 5.14.0-162.6.1.el9_1.x86_64 #1 SMP
                              PREEMPT_DYNAMIC Fri Sep 30 07:36:03 EDT 2022
                              x86_64 x86_64
Alert Count                   1
First Seen                    2022-12-07 15:40:24 CET
Last Seen                     2022-12-07 15:40:24 CET
Local ID                      e686141c-9ab0-4e31-9a92-53f3ce2c8b73

Raw Audit Messages
type=AVC msg=audit(1670424024.129:54): avc:  denied  { search } for  pid=691 comm="qemu-ga" name=".ssh" dev="vda4" ino=12164 scontext=system_u:system_r:virt_qemu_ga_t:s0 tcontext=system_u:object_r:ssh_home_t:s0 tclass=dir permissive=0


type=SYSCALL msg=audit(1670424024.129:54): arch=x86_64 syscall=openat success=no exit=EACCES a0=ffffff9c a1=562d0c88a5f0 a2=0 a3=0 items=0 ppid=1 pid=691 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm=qemu-ga exe=/usr/bin/qemu-ga subj=system_u:system_r:virt_qemu_ga_t:s0 key=(null)

Hash: qemu-ga,virt_qemu_ga_t,ssh_home_t,dir,search
Comment 22 errata-xmlrpc 2023-05-09 08:16:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (selinux-policy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2483