Bug 1481858

Since 2017-08-09, openQA has been plagued by failures caused by typing errors - cases where key presses sent by the openQA test driver (os-autoinst) to the virtual machine running the actual test were missed. We've always had such failures, but they're usually quite rare, especially when typing at a console (we have a few tricks to mitigate the problem when typing in graphical desktops); the failure rate would be something like 1-2% of tests. Since 2017-08-10, something like 50% or more of all tests that do any significant amount of typing are failing.

After some investigation, it looks like the boxes where the tests run were updated on 2017-08-09 (as part of a mass infra update/reboot task), and went from qemu-2.9.0-1.fc26.1 to  qemu-2.9.0-3.fc26 . The most recent test batch where most tests passed as usual was run with qemu-2.9.0-1.fc26.1 ; the earliest test batch where a large number of tests failed due to typing errors was run with qemu-2.9.0-3.fc26 . I have now forcibly downgraded qemu on the openQA worker host boxes and re-run one set of tests, and for the first time since 2017-08-09, they mostly passed:

https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=25&build=Update-FEDORA-2017-bd0324f3e9&groupid=2

(the failure is not a typing failure but a screenshot that needs updating). So we're fairly confident the qemu update is the cause of the problem.

It seems highly likely the culprit is this commit:

https://github.com/qemu/qemu/commit/fa18f36a461984eae50ab957e47ec78dae3c14fc

which is intended to fix CVE-2017-8379 . Cole also notes that a SUSE developer, Alex Graf, has landed three commits since that one which seem very relevant to the specific case of openQA:

https://github.com/qemu/qemu/commit/77b0359bf414ad666d1714dc9888f1017c08e283
https://github.com/qemu/qemu/commit/51dbea77a29ea46173373a6dad4ebd95d4661f42
https://github.com/qemu/qemu/commit/d3b0db6dfea6b3a9ee0d96aceb796bdcafa84314

so we're going to check whether applying those commits on top of the 'limit kbd queue depth' commit makes things behave better. I'll do a scratch build and test this soon.