Created attachment 1452843 [details]
base.py hacked to reproduce using a hotplug "sdb" device (see "HACK TO REPRODUCE ISSUE")
Description of problem:
The plugins/base.py code iterates through devices to perform commands:
def _execute_all_device_commands(self, instance, devices):
for command in filter(lambda command: command["per_device"], self._commands.values()):
new_value = self._variables.expand(instance.options.get(command["name"], None))
if new_value is None:
----> for device in devices:
self._execute_device_command(instance, command, device, new_value)
If for some reason the devices set is modified while being iterated, a "RuntimeError: Set changed size during iteration" is thrown and tuned dies. Typical use case is when a SCSI device is being discovered at boot.
Version-Release number of selected component (if applicable):
tuned-2.9.0-1.el7.noarch (RHEL7.5) but also seen on tuned-2.7.1-3.el7.noarch (RHEL7.3 EUS)
Always, using the manual reproducer in attachment
Steps to Reproduce:
1. Set up a VM with 1 disk ("sda")
2. Install the hack as /usr/lib/python2.7/site-packages/tuned/plugins/base.py
3. Restart tuned and monitor the log
# tail -f /var/log/tuned.log &
# systemctl restart tuned
4. Attach a new disk ("sdb") to the VM
# tail -f /var/log/tuned/tuned.log
2018-06-19 10:11:19,207 INFO tuned.daemon.daemon: starting tuning
2018-06-19 10:11:19,211 INFO tuned.plugins.base: instance cpu: assigning devices cpu0
2018-06-19 10:11:19,216 WARNING tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias
2018-06-19 10:11:19,216 INFO tuned.plugins.base: instance disk: assigning devices sda, dm-0, dm-1
2018-06-19 10:11:19,217 INFO tuned.plugins.base: Applying command 'readahead' to device 'sda'
2018-06-19 10:11:19,218 INFO tuned.plugins.base: Waiting for hotplug device 'sdb' to show up
2018-06-19 10:11:42,078 INFO tuned.plugins.hotplug: device 'sdb' added
2018-06-19 10:11:42,078 INFO tuned.plugins.hotplug: instance disk: adding new device sdb
2018-06-19 10:11:42,249 INFO tuned.plugins.base: Device 'sdb' is now there, continuing loop
# journalctl -u tuned
systemd: Starting Dynamic System Tuning Daemon...
tuned: Exception in thread Thread-1:
tuned: Traceback (most recent call last):
tuned: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
tuned: File "/usr/lib64/python2.7/threading.py", line 765, in run
tuned: self.__target(*self.__args, **self.__kwargs)
tuned: File "/usr/lib/python2.7/site-packages/tuned/daemon/daemon.py", line 134, in _thread_code
tuned: File "/usr/lib/python2.7/site-packages/tuned/units/manager.py", line 90, in start_tuning
tuned: File "/usr/lib/python2.7/site-packages/tuned/plugins/instance/instance.py", line 73, in apply_tuning
tuned: File "/usr/lib/python2.7/site-packages/tuned/plugins/base.py", line 250, in instance_apply_tuning
tuned: File "/usr/lib/python2.7/site-packages/tuned/plugins/base.py", line 298, in _instance_apply_static
tuned: self._execute_all_device_commands(instance, instance.devices)
tuned: File "/usr/lib/python2.7/site-packages/tuned/plugins/base.py", line 418, in _execute_all_device_commands
tuned: for device in devices:
tuned: RuntimeError: Set changed size during iteration
systemd: tuned.service start operation timed out. Terminating.
Yeah, Tuned doesn't really do a whole lot of locking when accessing shared resources. Thanks for the report.
Upstream pull request:
Fixed by upstream PR:
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.