RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1432003 - After logrotate , dynamic looping call gets 'ValueError: I/O operation on closed file' on self.stream.flush() in /usr/lib64/python2.7/logging/handlers.py
Summary: After logrotate , dynamic looping call gets 'ValueError: I/O operation on clo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: python
Version: 7.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: pre-dev-freeze
: ---
Assignee: Charalampos Stratakis
QA Contact: Branislav Náter
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-14 09:55 UTC by Masaki Furuta ( RH )
Modified: 2020-05-14 15:45 UTC (History)
20 users (show)

Fixed In Version: python-2.7.5-58.el7
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-08-01 20:25:03 UTC
Target Upstream Version:
Embargoed:
vstinner: needinfo+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Python 21742 0 None None None 2017-05-03 12:54:01 UTC
Red Hat Product Errata RHSA-2017:1868 0 normal SHIPPED_LIVE Moderate: python security and bug fix update 2017-08-01 18:24:21 UTC

Comment 5 Victor Stinner 2017-03-31 14:33:37 UTC 
The problem is unrelated to Nova, but Oslo Log. In RHOS 6 (Juno), Nova still used "Oslo Incubator": copy of Oslo Log in openstack/common/.

The problem is that an external program (logrotate) renames the log file while Nova has an open file descriptor. 

nova/openstack/common/log.py adds a logging.handlers.WatchedFileHandler() log handler:
https://github.com/openstack/nova/blob/juno-eol/nova/openstack/common/log.py#L490

This doesn't support when the open file is moved/renamed. The bug is already known:
https://bugs.launchpad.net/oslo.log/+bug/1437444

It was fixed upstream with the addition of a new log handler which detects and handles moved file: FastWatchedFileHandler.
https://review.openstack.org/#/c/208366/

Two similar bugzilla issues:

* https://bugzilla.redhat.com/show_bug.cgi?id=1432003 (reported at 2014-07-25, no component) -> this one was reported upstream, see the launchpad issue above
* https://bugzilla.redhat.com/show_bug.cgi?id=1123298 (reported at 2014-07-25, component: nova)
Comment 8 Victor Stinner 2017-04-03 14:03:50 UTC 
Mike Burns confirmed that python-inotify is already in RHEL 7, so it is ok to use it in RHOS 6.
Comment 15 Charalampos Stratakis 2017-04-19 15:49:56 UTC 
(In reply to Petr Viktorin from comment #12)
> Harris or Tomáš, can you please confirm that RHEL 7 uses 2.7.5 and that
> https://hg.python.org/cpython/rev/bb8b0c7fefd0 fix was not backported?

I can confirm that the python version in RHEL 7 (2.7.5), doesn't include this fix.

Victor, by backporting the patch in python, the issue will be resolved?
Comment 16 Victor Stinner 2017-04-19 16:03:50 UTC 
> Victor, by backporting the patch in python, the issue will be resolved?

Sorry, that's hard to say at this point:

"How reproducible: Once on customer site only."

and then "Actual results: Can't get any log entry to pursue the cause in nova-compute.log after Feb 05, due to 'ValueError: I/O operation on closed file' on self.stream.flush() in /usr/lib64/python2.7/logging/handlers.py, after logrotate."

Funny or not, I need missing logs to prove that the logging bugfix will fix the logging issue...

At least, according to the customer traceback and to the CPython upstream bug report: they seem *very* similar.

The real mystery is why the nova failed to reopen the nova file while logrotate was rotating the log file? Likely a race condition, likely hard to reproduce.
Comment 20 Charalampos Stratakis 2017-04-19 16:45:14 UTC 
Scratch builds with the fix on top of python-2.7.5-39

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13042090
Comment 34 errata-xmlrpc 2017-08-01 20:25:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1868
Note You need to log in before you can comment on or make changes to this bug.