Bug 1356937

Without any changes, using a build of the same libvirtd referenced by this bz, I'm not seeing the same results using your example steps. 

Here's my example using your steps, but slightly expanded (there's just one domain that will be running, hence use of machine-qemu* and iothread*):

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
cat: /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us: No such file or directory

# virsh start dom
Domain  started

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
cat: /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us: No such file or directory

# virsh schedinfo dom
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : -1
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1

# virsh dumpxml dom | grep iothr
# virsh iothreadadd dom 1
# virsh dumpxml dom | grep iothr
  <iothreads>1</iothreads>
  <iothreadids>
    <iothread id='1'/>
  </iothreadids>

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
# virsh schedinfo dom --set vcpu_quota=1000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# virsh schedinfo dom --set global_quota=2000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2000

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1

...

So as you can see by this point I can set global quota and you'll also note that there is a -1 in my *iothread*/cpu.cfs_quota_us file. Since yours shows 4000, I'm wondering if by chance you "altered" the *iothread1/cpu.cfs_quota_us file to 4000 *before* you tried modifying the global...

It seems so, if I modify the file "by hand" to a value *lower* than what I'm about to set to the global value, then I get the same failure you note when trying to set to 2000 (since the IOThread is at 4000). If instead of 2000 I use 5000, then the global setting works. That is:

# echo "4000" > /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
# virsh schedinfo dom --set vcpu_quota=1000
...works
# virsh schedinfo dom --set global_quota=2000
Scheduler      : posix
error: Invalid value '2000' for 'cpu.cfs_quota_us': Invalid argument
# virsh schedinfo f18 --set global_quota=5000
...works


So in a way, your example is slightly flawed. Since global setting was only added in libvirt 1.3.3 (e.g. after RHEL 7.2), there isn't any sort of regression here.  So I think this bz may need to move to rhel-7.4. Although I will try for 7.3 - may not happen since there's a change to a header file required (libvirt-domain.h).


Regardless of where it falls, I have put together some patches that would allow one to modify the the "iothread_quota" and "iothread_period" much in the same way it is done for vcpu, emulator, and global values.  This at least then prints out whatever value that IOThreads are using and allows setting for IOThreads specifically.

Used a different domain with 4 IOThreads already defined... The number doesn't matter, just that they're defined. Here's the results:

# virsh start iothrdom
# virsh schedinfo iothrdom
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : -1
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
-1
-1
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1
-1
-1
# virsh schedinfo iothrdom --set vcpu_quota=1000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1
-1
-1
# virsh schedinfo iothrdom --set global_quota=2000 
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2000
iothread_period: 100000
iothread_quota : -1

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1
-1
-1

# virsh schedinfo iothrdom --set global_quota=2500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : -1

# virsh schedinfo iothrdom --set iothread_quota=1500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : 1500

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
1500
1500
1500
1500

The patches are posted upstream as:

http://www.redhat.com/archives/libvir-list/2016-July/msg01024.html

Upstream change made:

commit e4e4d17c9c2828b3a7d2909295e724c57e60a334
Author: John Ferlan <jferlan>
Date:   Mon Jul 25 07:07:43 2016 -0400

    qemu: Add support to get/set IOThread period and quota cgroup values
    
...
    
    Add support for IOThread quota/bandwidth and period parameters for non
    session mode. If in session mode, then error out.  Uses all the same
    places where {vcpu|emulator|global}_{period|quota} are adjusted and
    adds the iothread values.

Document check with libvirt-docs-2.0.0-8.el7.x86_64:

     </dd><dt><code>iothread_period</code></dt><dd>
        The optional <code>iothread_period</code> element specifies the
        enforcement interval(unit: microseconds) for IOThreads. Within
        <code>iothread_period</code>, each IOThread of the domain will
        not be allowed to consume more than <code>iothread_quota</code>
        worth of runtime. The value should be in range [1000, 1000000].
        An iothread_period with value 0 means no value.
        <span class="since">Only QEMU driver support since 2.1.0</span>
      </dd><dt><code>iothread_quota</code></dt><dd>
        The optional <code>iothread_quota</code> element specifies the maximum
        allowed bandwidth(unit: microseconds) for IOThreads. A domain with
        <code>iothread_quota</code> as any negative value indicates that the
        domain IOThreads have infinite bandwidth, which means that it is
        not bandwidth controlled. The value should be in range
        [1000, 18446744073709551] or less than 0. An <code>iothread_quota</code>
        with value 0 means no value. You can use this feature to ensure that
        all IOThreads run at the same speed.
        <span class="since">Only QEMU driver support since 2.1.0</span>

Scenario 1: iothread_quota test

# virsh schedinfo virtlab_test_copy
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1


# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
4000
4000
4000
4000



# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
cat: /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us: No such file or directory


# virsh iothreadadd virtlab_test_copy 1
# virsh iothreadadd virtlab_test_copy 2


# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1

# virsh schedinfo virtlab_test_copy --set vcpu_quota=1000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1


# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
1000
1000

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1


# virsh schedinfo virtlab_test_copy --set global_quota=2000 
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2000
iothread_period: 100000
iothread_quota : -1

#  cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/vcpu*/cpu.cfs_quota_us
1000
1000
1000
1000

# cat  /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu*/iothread*/cpu.cfs_quota_us
-1
-1


# virsh schedinfo virtlab_test_copy --set global_quota=500
Scheduler      : posix
error: invalid argument: value of 'global_quota' is out of range [1000, 18446744073709551]


# virsh schedinfo virtlab_test_copy --set global_quota=2500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : -1


# virsh schedinfo virtlab_test_copy --set iothread_quota=1500
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : 1500


# virsh schedinfo virtlab_test_copy --set iothread_quota=1300
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 100000
iothread_quota : 1300


# virsh schedinfo virtlab_test_copy --set iothread_quota=2600
Scheduler      : posix
error: Invalid value '2600' for 'cpu.cfs_quota_us': Invalid argument
<==== should not larger than 2500, as expected

# virsh schedinfo virtlab_test_copy --set global_quota=1200
Scheduler      : posix
error: Invalid value '1200' for 'cpu.cfs_quota_us': Invalid argument
<==== should not less than 1300, as expected

# virsh dumpxml virtlab_test_copy | grep thread_quota -i
    <iothread_quota>1300</iothread_quota>

Hi John, 
for iothread_period, I don't quite get the logic
Is it working with global_period?

In my test, I have two iothreads, 4 vcpus and global_period is 100000, as follow:
# virsh dumpxml virtlab_test_copy | egrep "iothread|vcpu"
  <vcpu placement='static'>4</vcpu>
  <iothreads>2</iothreads>
  <iothreadids>
    <iothread id='2'/>
    <iothread id='1'/>
  </iothreadids>
    <iothread_period>51999</iothread_period>
    <iothread_quota>1300</iothread_quota>

And I can set iothread_period to 51999, but not 51998, what's the logic?

[root@localhost ~]# virsh schedinfo virtlab_test_copy --set iothread_period=51999
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : 2500
iothread_period: 51999
iothread_quota : 1300

[root@localhost ~]# virsh schedinfo virtlab_test_copy --set iothread_period=51998
Scheduler      : posix
error: Invalid value '51998' for 'cpu.cfs_period_us': Invalid argument

Not quite sure. The error comes from *deep* in the bowels of the cgroup logic - when a virFileWriteStr() fails. That would mean the "EINVAL" (Invalid argument) was returned from either an open(), write(), or close() command. Neither makes sense since the previous adjustment worked. 

In the long run the iothread_period logic is essentially a copy of vcpu_period and emulator_period logic, so do those fail similarly? Did you try another update after that (e.g. back to 51999 or 60000 or 1000)?  How much more debugging could be done to see where that error comes from? It'd probably be useful to get some sort of syscall tracking to know which was called last (open, write, close).

Of course attempting a similar sequence on my upstream libvirt doesn't have the same issue.

Verified on: 
libvirt-2.0.0-9.el7.x86_64
kernel-3.10.0-508.el7.x86_64


# virsh start virtlab_test_copy
Domain virtlab_test_copy started

# virsh iothreadadd virtlab_test_copy 1

#  virsh schedinfo virtlab_test_copy
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 100000
iothread_quota : -1

# virsh schedinfo virtlab_test_copy --set iothread_period=2000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 2000
iothread_quota : -1

# virsh schedinfo virtlab_test_copy --set iothread_quota=2000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 4000
emulator_period: 100000
emulator_quota : -1
global_period  : 100000
global_quota   : -1
iothread_period: 2000
iothread_quota : 2000

# virsh dumpxml virtlab_test_copy | grep "iothread"
  <iothreads>1</iothreads>
  <iothreadids>
    <iothread id='1'/>
  </iothreadids>
    <iothread_period>2000</iothread_period>
    <iothread_quota>2000</iothread_quota>

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d4\\x2dvirtlabtestcopy.scope/iothread1/cpu.cfs_quota_us 
2000

# cat /sys/fs/cgroup/cpu\,cpuacct/machine.slice/machine-qemu\\x2d4\\x2dvirtlabtestcopy.scope/iothread1/cpu.cfs_period_us 
2000


=======
the question mentioned in comment 9 is now doced in a new cgroup bug 1377574.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html