Bug 1666680

Summary: tuned fails due to "RuntimeError: Set changed size during iteration"
Product: Red Hat Enterprise Linux 8 Reporter: Ondřej Lysoněk <olysonek>
Component: tunedAssignee: Jaroslav Škarvada <jskarvad>
Status: CLOSED DUPLICATE QA Contact: qe-baseos-daemons
Severity: medium Docs Contact:
Priority: medium    
Version: 8.0CC: jeder, jskarvad, lcapitulino, olysonek
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1592743 Environment:
Last Closed: 2019-01-24 21:12:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1592743    
Bug Blocks: 1640832, 1665995    

Comment 1 Luiz Capitulino 2019-01-24 18:07:21 UTC
We seem to be triggering this issue with tuned-2.10.0-14.el8
when applying the realtime-virtual-host profile:

2019-01-14 09:01:12,156 ERROR    tuned.units.manager: BUG: Unhandled exception in start_tuning: dictionary changed size during iteration

It seems that the realtime-virtual-host profile is not fully
applied because of this bug.

Do you think it's the same issue? If yes, do we have a plan to
get it fixed?

Comment 2 Ondřej Lysoněk 2019-01-24 19:17:35 UTC
Oh wow, I misread the error message. I thought it's the same as in this bug, sorry! Could you try to reproduce the issue with the following patch applied to /usr/lib/python3.6/site-packages/tuned/units/manager.py? Alternatively, you can pass the '-D' option to tuned in /usr/lib/systemd/system/tuned.service. Please provide a new log after doing that. I'll provide a test build, if this is any trouble.

diff --git a/tuned/units/manager.py b/tuned/units/manager.py
index 62ed0a8..2e9928e 100644
--- a/tuned/units/manager.py
+++ b/tuned/units/manager.py
@@ -77,7 +77,7 @@ class Manager(object):
 			trace = traceback.format_exc()
 			log.error("BUG: Unhandled exception in %s: %s"
 					% (caller, str(e)))
-			log.debug(trace)
+			log.error(trace)
 			return exc_ret
 
 	def destroy_all(self):
--

Anyway, I've been working on the 'Set changed size' problem. Here's a WIP branch. I haven't tested the patch yet, so I'm sorry if it's completely broken. I'm really afraid it won't fix the 'dictionary changed size' problem though..
https://github.com/olysonek/tuned/tree/hotplug-races

Comment 3 Luiz Capitulino 2019-01-24 20:04:04 UTC
Here you go. It took a bit to reproduce, and for some reason I'm now
only reproducing it when stopping the profile (although I did get it
when starting the profile in comment 1).

2019-01-24 14:58:26,409 ERROR    tuned.units.manager: BUG: Unhandled exception in stop_tuning: dictionary changed size during iteration
2019-01-24 14:58:26,409 ERROR    tuned.units.manager: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tuned/units/manager.py", line 75, in _try_call
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/instance/instance.py", line 82, in unapply_tuning
    self._plugin.instance_unapply_tuning(self, full_rollback)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/base.py", line 291, in instance_unapply_tuning
    self._instance_unapply_static(instance, full_rollback)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_scheduler.py", line 418, in _instance_unapply_static
    self._restore_ps_affinity()
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_scheduler.py", line 395, in _restore_ps_affinity
    ps = self.get_processes()
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_scheduler.py", line 145, in get_processes
    ps.reload_threads()
  File "/usr/lib/python3.6/site-packages/procfs/procfs.py", line 462, in reload_threads
    for pid in self.processes.keys():
RuntimeError: dictionary changed size during iteration

Comment 4 Ondřej Lysoněk 2019-01-24 21:12:12 UTC
Thanks, Luiz. It's indeed a different problem. It turned out to be a bug in python-linux-procfs, reported as bug#1669294. There's a patch, which you may want to try out.

*** This bug has been marked as a duplicate of bug 1669294 ***