RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1396597 - Libvirt should update device CGroups on hotplug
Summary: Libvirt should update device CGroups on hotplug
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: yisun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-18 17:32 UTC by Michal Privoznik
Modified: 2017-08-01 23:59 UTC (History)
7 users (show)

Fixed In Version: libvirt-2.5.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 17:19:14 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description Michal Privoznik 2016-11-18 17:32:23 UTC
Description of problem:
While working on a different issue, I've came across device hotplug code in QEMU driver. Trouble is, that for some devices we don't whitelist them in device CGroup corresponding to the domain. I've identified RNG and chardev.

Version-Release number of selected component (if applicable):
Any

How reproducible:
100%

Steps to Reproduce:
1. Consider the following RNG:

    <rng model='virtio'>
      <rate bytes='1234' period='2000'/>
      <backend model='random'>/dev/sdb</backend>
    </rng>

Yes, /dev/sdb is really crappy random source, but it will serve its purpose here.

2. Try to hotplug it to a domain.
3. Observe qemu denied access.

Actual results:
Device can't be hotplugged.


Expected results:
Device is hotplugged.


Additional info:

Comment 1 Michal Privoznik 2016-11-18 17:33:25 UTC
Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2016-November/msg00854.html

Comment 2 Michal Privoznik 2016-11-23 15:50:05 UTC
Moving to POST:

commit 5d9c2c708111835ee8e625a9c8a98353687c3fa5
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Nov 18 11:45:44 2016 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Wed Nov 23 16:38:02 2016 +0100

    qemu: Update cgroup on chardev hotplug
    
    Just like in the previous commit, we are not updating CGroups on
    chardev hot(un-)plug and thus leaving qemu unable to access any
    non-default device users are trying to hotplug.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 085692c8bb9154f6f5e66b2e3fa150daef03fa19
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Nov 18 11:17:51 2016 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Wed Nov 23 16:37:57 2016 +0100

    qemu: Update cgroup on RNG hotplug
    
    If users try to hotplug RNG device with a backend different to
    /dev/random or /dev/urandom the whole operation fails as qemu is
    unable to access the device. The problem is we don't update
    device CGroups during the operation.
    
    Signed-off-by: Michal Privoznik <mprivozn>


v2.4.0-196-g5d9c2c7

Comment 4 yisun 2017-05-31 08:50:57 UTC
(In reply to Michal Privoznik from comment #2)
> Moving to POST:
> 
> commit 5d9c2c708111835ee8e625a9c8a98353687c3fa5
> Author:     Michal Privoznik <mprivozn>
> AuthorDate: Fri Nov 18 11:45:44 2016 +0100
> Commit:     Michal Privoznik <mprivozn>
> CommitDate: Wed Nov 23 16:38:02 2016 +0100
> 
>     qemu: Update cgroup on chardev hotplug
>     
>     Just like in the previous commit, we are not updating CGroups on
>     chardev hot(un-)plug and thus leaving qemu unable to access any
>     non-default device users are trying to hotplug.
>     
>     Signed-off-by: Michal Privoznik <mprivozn>
> 
> commit 085692c8bb9154f6f5e66b2e3fa150daef03fa19
> Author:     Michal Privoznik <mprivozn>
> AuthorDate: Fri Nov 18 11:17:51 2016 +0100
> Commit:     Michal Privoznik <mprivozn>
> CommitDate: Wed Nov 23 16:37:57 2016 +0100
> 
>     qemu: Update cgroup on RNG hotplug
>     
>     If users try to hotplug RNG device with a backend different to
>     /dev/random or /dev/urandom the whole operation fails as qemu is
>     unable to access the device. The problem is we don't update
>     device CGroups during the operation.
>     
>     Signed-off-by: Michal Privoznik <mprivozn>
> 
> 
> v2.4.0-196-g5d9c2c7
Hi Michal, 
I saw you update chardev related code, but during my try. I just cannot hotplug some typical chardev to a running vm.

## cat g
<graphics type='spice' autoport='yes'>
    <listen type='address'/>
    <image compression='off'/>
</graphics>

## virsh attach-device r g
error: Failed to attach device from g
error: Operation not supported: live attach of device 'graphics' is not supported

## cat c
<console type='pty'>
   <target type='serial' port='0'/>
</console>

## virsh attach-device r c
error: Failed to attach device from c
error: Operation not supported: attaching serial console is not supported

So what kind of chardev you'll suggest to be tested? and do you think we'll support such kind of hotplug in future(maybe a rfe could be open)? thx

Comment 5 Michal Privoznik 2017-05-31 10:19:10 UTC
(In reply to yisun from comment #4)
> (In reply to Michal Privoznik from comment #2)
> > Moving to POST:
> > 
> > commit 5d9c2c708111835ee8e625a9c8a98353687c3fa5
> > Author:     Michal Privoznik <mprivozn>
> > AuthorDate: Fri Nov 18 11:45:44 2016 +0100
> > Commit:     Michal Privoznik <mprivozn>
> > CommitDate: Wed Nov 23 16:38:02 2016 +0100
> > 
> >     qemu: Update cgroup on chardev hotplug
> >     
> >     Just like in the previous commit, we are not updating CGroups on
> >     chardev hot(un-)plug and thus leaving qemu unable to access any
> >     non-default device users are trying to hotplug.
> >     
> >     Signed-off-by: Michal Privoznik <mprivozn>
> > 
> > commit 085692c8bb9154f6f5e66b2e3fa150daef03fa19
> > Author:     Michal Privoznik <mprivozn>
> > AuthorDate: Fri Nov 18 11:17:51 2016 +0100
> > Commit:     Michal Privoznik <mprivozn>
> > CommitDate: Wed Nov 23 16:37:57 2016 +0100
> > 
> >     qemu: Update cgroup on RNG hotplug
> >     
> >     If users try to hotplug RNG device with a backend different to
> >     /dev/random or /dev/urandom the whole operation fails as qemu is
> >     unable to access the device. The problem is we don't update
> >     device CGroups during the operation.
> >     
> >     Signed-off-by: Michal Privoznik <mprivozn>
> > 
> > 
> > v2.4.0-196-g5d9c2c7
> Hi Michal, 
> I saw you update chardev related code, but during my try. I just cannot
> hotplug some typical chardev to a running vm.
> 
> ## cat g
> <graphics type='spice' autoport='yes'>
>     <listen type='address'/>
>     <image compression='off'/>
> </graphics>

This is not a chardev nor RNG. Live hotplug of graphics is not implemented as qemu doesn't support it yet.

> 
> ## virsh attach-device r g
> error: Failed to attach device from g
> error: Operation not supported: live attach of device 'graphics' is not
> supported
> 
> ## cat c
> <console type='pty'>
>    <target type='serial' port='0'/>
> </console>

Hotplug for chardevs is supported only for virtio.

Comment 6 yisun 2017-05-31 11:47:06 UTC
(In reply to Michal Privoznik from comment #5)
> (In reply to yisun from comment #4)
> > (In reply to Michal Privoznik from comment #2)
> > > Moving to POST:
> > > 
> > > commit 5d9c2c708111835ee8e625a9c8a98353687c3fa5
> > > Author:     Michal Privoznik <mprivozn>
> > > AuthorDate: Fri Nov 18 11:45:44 2016 +0100
> > > Commit:     Michal Privoznik <mprivozn>
> > > CommitDate: Wed Nov 23 16:38:02 2016 +0100
> > > 
> > >     qemu: Update cgroup on chardev hotplug
> > >     
> > >     Just like in the previous commit, we are not updating CGroups on
> > >     chardev hot(un-)plug and thus leaving qemu unable to access any
> > >     non-default device users are trying to hotplug.
> > >     
> > >     Signed-off-by: Michal Privoznik <mprivozn>
> > > 
> > > commit 085692c8bb9154f6f5e66b2e3fa150daef03fa19
> > > Author:     Michal Privoznik <mprivozn>
> > > AuthorDate: Fri Nov 18 11:17:51 2016 +0100
> > > Commit:     Michal Privoznik <mprivozn>
> > > CommitDate: Wed Nov 23 16:37:57 2016 +0100
> > > 
> > >     qemu: Update cgroup on RNG hotplug
> > >     
> > >     If users try to hotplug RNG device with a backend different to
> > >     /dev/random or /dev/urandom the whole operation fails as qemu is
> > >     unable to access the device. The problem is we don't update
> > >     device CGroups during the operation.
> > >     
> > >     Signed-off-by: Michal Privoznik <mprivozn>
> > > 
> > > 
> > > v2.4.0-196-g5d9c2c7
> > Hi Michal, 
> > I saw you update chardev related code, but during my try. I just cannot
> > hotplug some typical chardev to a running vm.
> > 
> > ## cat g
> > <graphics type='spice' autoport='yes'>
> >     <listen type='address'/>
> >     <image compression='off'/>
> > </graphics>
> 
> This is not a chardev nor RNG. Live hotplug of graphics is not implemented
> as qemu doesn't support it yet.
> 
> > 
> > ## virsh attach-device r g
> > error: Failed to attach device from g
> > error: Operation not supported: live attach of device 'graphics' is not
> > supported
> > 
> > ## cat c
> > <console type='pty'>
> >    <target type='serial' port='0'/>
> > </console>
> 
> Hotplug for chardevs is supported only for virtio.

thx for the info.
I tried with a virtio console, and seems it cannot be connected if hotplugged, but can be connected with cold plug, pls have a check.

1. make sure no console setup for vm
## virsh console vm1
Connected to domain vm1
Escape character is ^]
error: internal error: cannot find character device <null>

2. prepare a console xml
## cat vcon
<console type='pty'>
      <target type='virtio' port='0'/>
</console>

3. do the hot plug
## virsh attach-device vm1 vcon
Device attached successfully


4. try to connect to the console
## virsh console vm1
Connected to domain vm1
Escape character is ^]
<=== just hanging there ~~


well, if I cold plug the console with "virsh edit"
1. edit vm's xml before starting it
## virsh dumpxml vm1 --inactive | grep console -a2
...
    <console type='pty'>
      <target type='virtio' port='0'/>
    </console>

## virsh start vm1
Domain vm1 started

2. try to connect to the console
## virsh console vm1
Connected to domain vm1
Escape character is ^]

Red Hat Enterprise Linux Server 7.3 Beta (Maipo)
Kernel 3.10.0-461.el7.x86_64 on an x86_64

localhost login: 
<=== can connect to guest console

Comment 7 Michal Privoznik 2017-05-31 14:52:18 UTC
(In reply to yisun from comment #6)
> 
> thx for the info.
> I tried with a virtio console, and seems it cannot be connected if
> hotplugged, but can be connected with cold plug, pls have a check.
> 
> 1. make sure no console setup for vm
> ## virsh console vm1
> Connected to domain vm1
> Escape character is ^]
> error: internal error: cannot find character device <null>
> 
> 2. prepare a console xml
> ## cat vcon
> <console type='pty'>
>       <target type='virtio' port='0'/>
> </console>
> 
> 3. do the hot plug
> ## virsh attach-device vm1 vcon
> Device attached successfully
> 
> 
> 4. try to connect to the console
> ## virsh console vm1
> Connected to domain vm1
> Escape character is ^]
> <=== just hanging there ~~

This is because guest OS hasn't spawned any getty process on the console. You can either try running it by hand (from say SSH session), or just verify there's new /dev/vportXpY after you've hotplugged the console.

Comment 8 yisun 2017-06-01 07:35:06 UTC
seems needs to needinfo you again~~
I tested rng hotplug on:
    libvirt-3.2.0-7.el7.x86_64
    qemu-kvm-rhev-2.9.0-7.el7.x86_64
    kernel-3.10.0-660.el7.x86_64

I tried to attach a rng device with backing device = /dev/sdd and it's failed.
Steps:
## ll /dev/sdd
brw-rw----. 1 root disk 8, 48 Jun  1 15:28 /dev/sdd

## cat rng
    <rng model='virtio'>
      <rate bytes='1234' period='2000'/>
      <backend model='random'>/dev/sdd</backend>
    </rng>

## virsh attach-device vm1 rng
error: Failed to attach device from rng
error: internal error: unable to execute QEMU command 'object-add': Could not open '/dev/sdd': Permission denied
<==== here still a permission denied error

well, if I manually chmod the /dev/sdd to a+rw, it can be attached but seems not what you wanted to implement.

## chmod a+rw /dev/sdd
## ll /dev/sdd
brw-rw-rw-. 1 root disk 8, 48 Jun  1 15:28 /dev/sdd
## virsh attach-device vm1 rng
Device attached successfully

Comment 9 Michal Privoznik 2017-06-01 12:25:41 UTC
(In reply to yisun from comment #8)
> seems needs to needinfo you again~~
> I tested rng hotplug on:
>     libvirt-3.2.0-7.el7.x86_64
>     qemu-kvm-rhev-2.9.0-7.el7.x86_64
>     kernel-3.10.0-660.el7.x86_64
> 
> I tried to attach a rng device with backing device = /dev/sdd and it's
> failed.
> Steps:
> ## ll /dev/sdd
> brw-rw----. 1 root disk 8, 48 Jun  1 15:28 /dev/sdd
> 
> ## cat rng
>     <rng model='virtio'>
>       <rate bytes='1234' period='2000'/>
>       <backend model='random'>/dev/sdd</backend>
>     </rng>
> 
> ## virsh attach-device vm1 rng
> error: Failed to attach device from rng
> error: internal error: unable to execute QEMU command 'object-add': Could
> not open '/dev/sdd': Permission denied
> <==== here still a permission denied error
> 
> well, if I manually chmod the /dev/sdd to a+rw, it can be attached but seems
> not what you wanted to implement.

In fact, this is what we really want to have. RNGs are special type of devices. Usually, you don't have /dev/sdd as a RNG source but /dev/random or /dev/hwrng. Both these are meant to be shared with other applications on the system. Therefore if libvirt would label them, other apps might lose access. So we don't.

Comment 10 yisun 2017-06-02 12:03:01 UTC
verified on:
libvirt-3.2.0-7.el7.x86_64

1. chardev
in vm: 
# ll /dev/vport0p*
crw-------. 1 root root 246, 0 Jun  2 19:58 /dev/vport0p0
crw-------. 1 root root 246, 1 Jun  2 19:58 /dev/vport0p1
crw-------. 1 root root 246, 5 Jun  2 19:58 /dev/vport0p5

in host:
## cat vport
    <console type='pty'>
      <target type='virtio' port='0'/>
    </console>

## virsh attach-device vm1 vport
Device attached successfully

in vm:
# ll /dev/vport0p*
crw-------. 1 root root 246, 0 Jun  2 19:58 /dev/vport0p0
crw-------. 1 root root 246, 1 Jun  2 19:58 /dev/vport0p1
crw-------. 1 root root 246, 2 Jun  2 19:59 /dev/vport0p2 ** newly added
crw-------. 1 root root 246, 5 Jun  2 19:58 /dev/vport0p5


2. rng dev
in vm:
# ll /dev | grep -i rng
<== nothing

in host:
## ll /dev/sdd
brw-rw-rw-. 1 root disk 8, 48 Jun  1 15:28 /dev/sdd

## cat rng
    <rng model='virtio'>
      <rate bytes='1234' period='2000'/>
      <backend model='random'>/dev/sdd</backend>
    </rng>

## virsh attach-device vm1 rng
Device attached successfully

in guest:
# ll /dev | grep -i rng
crw-------. 1 root root     10, 183 Jun  2 20:02 hwrng  ** newly added

Comment 11 errata-xmlrpc 2017-08-01 17:19:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 12 errata-xmlrpc 2017-08-01 23:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846


Note You need to log in before you can comment on or make changes to this bug.