Created attachment 1452843 [details]
base.py hacked to reproduce using a hotplug "sdb" device (see "HACK TO REPRODUCE ISSUE")

Description of problem:

The plugins/base.py code iterates through devices to perform commands:

        def _execute_all_device_commands(self, instance, devices):
                for command in filter(lambda command: command["per_device"], self._commands.values()):
                        new_value = self._variables.expand(instance.options.get(command["name"], None))
                        if new_value is None:
                                continue
---->                   for device in devices:
                                self._execute_device_command(instance, command, device, new_value)

If for some reason the devices set is modified while being iterated, a "RuntimeError: Set changed size during iteration" is thrown and tuned dies. Typical use case is when a SCSI device is being discovered at boot.


Version-Release number of selected component (if applicable):

tuned-2.9.0-1.el7.noarch (RHEL7.5) but also seen on tuned-2.7.1-3.el7.noarch (RHEL7.3 EUS)


How reproducible:

Always, using the manual reproducer in attachment


Steps to Reproduce:
1. Set up a VM with 1 disk ("sda")
2. Install the hack as /usr/lib/python2.7/site-packages/tuned/plugins/base.py
3. Restart tuned and monitor the log

  # tail -f /var/log/tuned.log &
  # systemctl restart tuned

4. Attach a new disk ("sdb") to the VM


Actual results:

# tail -f /var/log/tuned/tuned.log 
2018-06-19 10:11:19,207 INFO     tuned.daemon.daemon: starting tuning
2018-06-19 10:11:19,211 INFO     tuned.plugins.base: instance cpu: assigning devices cpu0
2018-06-19 10:11:19,216 WARNING  tuned.plugins.plugin_cpu: your CPU doesn't support MSR_IA32_ENERGY_PERF_BIAS, ignoring CPU energy performance bias
2018-06-19 10:11:19,216 INFO     tuned.plugins.base: instance disk: assigning devices sda, dm-0, dm-1
2018-06-19 10:11:19,217 INFO     tuned.plugins.base: Applying command 'readahead' to device 'sda'
2018-06-19 10:11:19,218 INFO     tuned.plugins.base: Waiting for hotplug device 'sdb' to show up
2018-06-19 10:11:42,078 INFO     tuned.plugins.hotplug: device 'sdb' added
2018-06-19 10:11:42,078 INFO     tuned.plugins.hotplug: instance disk: adding new device sdb
2018-06-19 10:11:42,249 INFO     tuned.plugins.base: Device 'sdb' is now there, continuing loop

# journalctl -u tuned

systemd[1]: Starting Dynamic System Tuning Daemon...
tuned[1959]: Exception in thread Thread-1:
tuned[1959]: Traceback (most recent call last):
tuned[1959]: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
tuned[1959]: self.run()
tuned[1959]: File "/usr/lib64/python2.7/threading.py", line 765, in run
tuned[1959]: self.__target(*self.__args, **self.__kwargs)
tuned[1959]: File "/usr/lib/python2.7/site-packages/tuned/daemon/daemon.py", line 134, in _thread_code
tuned[1959]: self._unit_manager.start_tuning()
tuned[1959]: File "/usr/lib/python2.7/site-packages/tuned/units/manager.py", line 90, in start_tuning
tuned[1959]: instance.apply_tuning()
tuned[1959]: File "/usr/lib/python2.7/site-packages/tuned/plugins/instance/instance.py", line 73, in apply_tuning
tuned[1959]: self._plugin.instance_apply_tuning(self)
tuned[1959]: File "/usr/lib/python2.7/site-packages/tuned/plugins/base.py", line 250, in instance_apply_tuning
tuned[1959]: self._instance_apply_static(instance)
tuned[1959]: File "/usr/lib/python2.7/site-packages/tuned/plugins/base.py", line 298, in _instance_apply_static
tuned[1959]: self._execute_all_device_commands(instance, instance.devices)
tuned[1959]: File "/usr/lib/python2.7/site-packages/tuned/plugins/base.py", line 418, in _execute_all_device_commands
tuned[1959]: for device in devices:
tuned[1959]: RuntimeError: Set changed size during iteration
systemd[1]: tuned.service start operation timed out. Terminating.