Bug 1666680 - tuned fails due to "RuntimeError: Set changed size during iteration" [NEEDINFO]
Summary: tuned fails due to "RuntimeError: Set changed size during iteration"
Keywords:
Status: CLOSED DUPLICATE of bug 1669294
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: tuned
Version: 8.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.0
Assignee: Jaroslav Škarvada
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On: 1592743
Blocks: 1640832 1665995
TreeView+ depends on / blocked
 
Reported: 2019-01-16 11:07 UTC by Ondřej Lysoněk
Modified: 2019-01-24 21:12 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1592743
Environment:
Last Closed: 2019-01-24 21:12:12 UTC
Type: Bug
Target Upstream Version:
lcapitulino: needinfo? (jskarvad)


Attachments (Terms of Use)

Comment 1 Luiz Capitulino 2019-01-24 18:07:21 UTC
We seem to be triggering this issue with tuned-2.10.0-14.el8
when applying the realtime-virtual-host profile:

2019-01-14 09:01:12,156 ERROR    tuned.units.manager: BUG: Unhandled exception in start_tuning: dictionary changed size during iteration

It seems that the realtime-virtual-host profile is not fully
applied because of this bug.

Do you think it's the same issue? If yes, do we have a plan to
get it fixed?

Comment 2 Ondřej Lysoněk 2019-01-24 19:17:35 UTC
Oh wow, I misread the error message. I thought it's the same as in this bug, sorry! Could you try to reproduce the issue with the following patch applied to /usr/lib/python3.6/site-packages/tuned/units/manager.py? Alternatively, you can pass the '-D' option to tuned in /usr/lib/systemd/system/tuned.service. Please provide a new log after doing that. I'll provide a test build, if this is any trouble.

diff --git a/tuned/units/manager.py b/tuned/units/manager.py
index 62ed0a8..2e9928e 100644
--- a/tuned/units/manager.py
+++ b/tuned/units/manager.py
@@ -77,7 +77,7 @@ class Manager(object):
 			trace = traceback.format_exc()
 			log.error("BUG: Unhandled exception in %s: %s"
 					% (caller, str(e)))
-			log.debug(trace)
+			log.error(trace)
 			return exc_ret
 
 	def destroy_all(self):
--

Anyway, I've been working on the 'Set changed size' problem. Here's a WIP branch. I haven't tested the patch yet, so I'm sorry if it's completely broken. I'm really afraid it won't fix the 'dictionary changed size' problem though..
https://github.com/olysonek/tuned/tree/hotplug-races

Comment 3 Luiz Capitulino 2019-01-24 20:04:04 UTC
Here you go. It took a bit to reproduce, and for some reason I'm now
only reproducing it when stopping the profile (although I did get it
when starting the profile in comment 1).

2019-01-24 14:58:26,409 ERROR    tuned.units.manager: BUG: Unhandled exception in stop_tuning: dictionary changed size during iteration
2019-01-24 14:58:26,409 ERROR    tuned.units.manager: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tuned/units/manager.py", line 75, in _try_call
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/instance/instance.py", line 82, in unapply_tuning
    self._plugin.instance_unapply_tuning(self, full_rollback)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/base.py", line 291, in instance_unapply_tuning
    self._instance_unapply_static(instance, full_rollback)
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_scheduler.py", line 418, in _instance_unapply_static
    self._restore_ps_affinity()
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_scheduler.py", line 395, in _restore_ps_affinity
    ps = self.get_processes()
  File "/usr/lib/python3.6/site-packages/tuned/plugins/plugin_scheduler.py", line 145, in get_processes
    ps.reload_threads()
  File "/usr/lib/python3.6/site-packages/procfs/procfs.py", line 462, in reload_threads
    for pid in self.processes.keys():
RuntimeError: dictionary changed size during iteration

Comment 4 Ondřej Lysoněk 2019-01-24 21:12:12 UTC
Thanks, Luiz. It's indeed a different problem. It turned out to be a bug in python-linux-procfs, reported as bug#1669294. There's a patch, which you may want to try out.

*** This bug has been marked as a duplicate of bug 1669294 ***


Note You need to log in before you can comment on or make changes to this bug.