Bug 1476830

The sosreport does not include the Nova logs in DEBUG. In order to investigate we need to see what is the domain XML generated by Nova and passed to libvirt.

in /etc/nova/nova.conf please turn the debug to True. Restart the compute services and start an instance with the flavor related. Instead of a sosreport you could just copy/paste the domain XML generated from the nova-compute.log.

effectively we should see a 'iotune' element under the 'disk' element. Can you test to set into the flavor extra-specs integers instead-of strings?

 os-flavor-access:is_public | True                                                                                                                     |
| properties                 | quota:disk_io_limit='100', quota:disk_read_iops_sec='100', quota:disk_total_iops_sec='100',                              |
|                            | quota:disk_write_iops_sec='100', quota:vif_inbound_average='1024', quota:vif_outbound_average='1024'                     |
| ram                        | 2048    


That should look to something:

quota:disk_write_iops_sec=100, quota:vif_inbound_average=1024...

Please feel free to reopen if necessary.

I'm going to take a look into this. To start, it's worth point out that 'quota:disk_io_limit' is an invalid option for the libvirt drive, so you can drop this. As for the rest of this, could I ask you to provide the following information. I know you've submitted some of it before, but I'd like to sanity check it.

1. The commands you're using to configure the flavor
2. The output of 'openstack flavor show FLAVOR-ID' for the flavor, once configured
3. The commands you're using to boot the instance
4. The output of 'openstack server show SERVER-ID' for the instance, once booted
5. The output of 'virsh dumpxml DOMAIN' for the instance, once booted
6. The nova.conf file you're using
7. A complete sosreport with logging at DEBUG level

At present, I really think this is a silly misconfiguration issue, but I'm willing to be proven wrong :)

Dropping triaged keyword as we're no longer sure if this is a nova issue or a libvirt issue

Would it be possible to get the XML for this flavor too?

(In reply to Stephen Finucane from comment #24)
> Would it be possible to get the XML for this flavor too?

I've never been asked to provide this before. Can you please tell us how to get that from the system?

Ah, sorry. It's step 5 in comment 10.

  5. The output of 'virsh dumpxml DOMAIN' for the instance, once booted

(In reply to Stephen Finucane from comment #26)
> Ah, sorry. It's step 5 in comment 10.
> 
>   5. The output of 'virsh dumpxml DOMAIN' for the instance, once booted

Yes, along with that, also please get the complete QEMU command-line (and attach it as a text file to this bug) for that guest from comment#25.

It is located in (on the Compute node where the instance is running):

   $ /var/log/libvirt/qemu/$instance-XXXXXXXX.log

(In reply to Siggy Sigwald from comment #22)

[...]

> The test was previously sent with the flavor with the configuration
> indicated:
> 
> [root@limit-io fio-2.0.14]# fio --name=randwrite --ioengine=libaio
> --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=512M --numjobs=8
> --runtime=240 --group_reporting
> randwrite: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio,
> iodepth=1

[...]

Talking to an QEMU I/O Layer expert, who's also versed with `fio` Stefan 
Hajnoczi (thank you).  Some comments.

Your goal seems to be to throttle disk read and write IOPS.

And your existing `fio` benchmark in comment#22 is using "--direct=0", 
it uses page cache, the I/O may never reach the disk!  So your existing
benchmark is stressing the VFS and page cache, _not_ the disk.

Also, with "--direct=0", there is no way to predict how QEMU I/O
throttling will perform since many requests don't even hit the disk.

Therefore, we suggest to try with: "--direct=1".

IOW, if you are trying to benchmark the disk, make sure requests are
being sent directly to the disk and not the software's page cache.

Couple of related recommendations from Stefan:

  - If you want to benchmark the disk it also helps to have a dedicated
    block device (e.g. /dev/vdb) and use fio filename=/dev/vdb to avoid
    going through a file system.

  - When using a whole block device with filename=/dev/vdb you can drop
    size=512M and let the random writes hit any part of the device.  
    This is a little fairer because most randomly chosen writes will be
    far away from each other.

    What this is hinting at is that: if you set size=8M or something too
    small then the benchmark may just be writing to the disk's write
    cache.)

  - You might also want to use 'ramp_time=30' with `fio` so that to let
    the workload run for a while before measurement begins (so that the
    system is in a steady state.

I have closed this bug as it has been waiting for more info for at least 2 weeks. We only do this to ensure that we don't accumulate stale bugs which can't be addressed. If you are able to provide the requested information, please feel free to re-open this bug.

Meta-comment:

I spent most of this week spending many hours trying to set up a 
reproducer environment for this.  I tried local and remote test
machines, I couldn't get a RHOS-9 running on RHEL 7.4. 

The methods I tried:

  - Upstream Mitaka (EOL) on Fedora-25 -- no luck, fails in many ways,
    requirements conflicts, etc

  - Upstream Mitaka on CentOS 7.4 -- no luck, fails miserably (even
    upstream CI job is not tended to) with conflicting requirements,
    causing you to do whack-a-mole. 

  - RHOS-9 with PackStack 'all-in-one' with essential services: also 
    fails in different ways.

    * * *

TripleO / RHOS Director-- I don't have requisite hardware to set this 
up.  So I'm in the process of getting a temporary environment from our 
fine CI folks (thanks: Migi).

Created attachment 1380486 [details]
RHOS-9 reproducer (confirmation)

Seems like I managed to make my own reproducer where I don't even see the 'iotune' element in the guest XML (refer attachment).

Now to examine what's on the upstream Git/master.  And then work from there.

I'm going to close this. While we were able to reproduce the issue, this is against OSP 9, which is rapidly approaching EOL, and there haven't been any updates in over a year.