Bug 1669294

Summary: RuntimeError is raised if a process vanishes while pidstats.reload_threads() executes
Product: Red Hat Enterprise Linux 8 Reporter: Ondřej Lysoněk <olysonek>
Component: python-linux-procfsAssignee: John Kacur <jkacur>
Status: CLOSED CURRENTRELEASE QA Contact: Qiao Zhao <qzhao>
Severity: high Docs Contact:
Priority: high    
Version: 8.0CC: bhu, hwkernel-mgr, jlelli, jwboyer, lcapitulino, pezhang
Target Milestone: rcKeywords: Patch, Regression
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python3-linux-procfs-0.6-6.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-14 01:49:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1640832, 1649545, 1665995    
Attachments:
Description Flags
Fix refreshing cache none

Description Ondřej Lysoněk 2019-01-24 21:09:53 UTC
Description of problem:
If a process disappears while iterating the loop in pidstats.reload_threads(), we get a RuntimeError as shown below. This is because we cannot remove an entry from a dictionary while iterating the dictionary.

This problem is the cause of the following Tuned error reported in bug#1666562:
2019-01-16 03:22:34,859 ERROR    tuned.units.manager: BUG: Unhandled exception in start_tuning: dictionary changed size during iteration

Version-Release number of selected component (if applicable):
python3-linux-procfs-0.6-5.el8.noarch

How reproducible:
always

Steps to Reproduce:
1. Add the following line to the beginning of pidstats.reload_threads() in /usr/lib/python3.6/site-packages/procfs/procfs.py:
import pdb; pdb.set_trace()
2. Start some process
3. Start the python interpreter and proceed as follows:
# python3
Python 3.6.8 (default, Jan 11 2019, 02:17:16) 
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import procfs
>>> ps = procfs.pidstats()
>>> ps.reload_threads()
> /usr/lib/python3.6/site-packages/procfs/procfs.py(463)reload_threads()
-> for pid in self.processes.keys():
(Pdb) next
> /usr/lib/python3.6/site-packages/procfs/procfs.py(464)reload_threads()
-> try:
(Pdb) 

At this point, terminate the process started in step 2. Return to the
python interpreter:

(Pdb) continue
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/procfs/procfs.py", line 465, in reload_threads
    self.processes[pid].load_threads()
RuntimeError: dictionary changed size during iteration


Actual results:
RuntimeError is raised

Expected results:
reload_threads() finishes without errors

Additional info:
A fix for the problem has been sent upstream:
https://lists.fedorahosted.org/archives/list/tuna-devel@lists.fedorahosted.org/thread/PK354CZETOV23PJPMJ7CZCK7SOOV2JX6/

Comment 1 Ondřej Lysoněk 2019-01-24 21:12:12 UTC
*** Bug 1666680 has been marked as a duplicate of this bug. ***

Comment 2 Luiz Capitulino 2019-01-25 13:36:59 UTC
I'm raising severity because when this issue triggers the realtime tuned profiles fail to apply.

Comment 3 John Kacur 2019-01-25 16:47:27 UTC
Created attachment 1523545 [details]
Fix refreshing cache

We should consider applying this one too

Comment 10 Luiz Capitulino 2019-01-30 17:42:18 UTC
FWIW, I quickly verified the reproducer found when debugging bug 1665995
and it works for me too.

Thanks everyone for fixing this one so quickly!

Comment 11 Luiz Capitulino 2019-02-13 14:59:46 UTC
Can we set the "Fixed In Version" field so that we know when this was fixed? Thanks.

Comment 12 Beth Uptagrafft 2019-02-13 20:32:51 UTC
(In reply to Luiz Capitulino from comment #11)
> Can we set the "Fixed In Version" field so that we know when this was fixed?
> Thanks.

Done - python3-linux-procfs-0.6-6.el8

Comment 13 Ondřej Lysoněk 2019-02-19 14:44:55 UTC
*** Bug 1676962 has been marked as a duplicate of this bug. ***