Bug 813902

Summary:	Excessive "stat"-ing by Python
Product:	[Fedora] Fedora	Reporter:	Dave Malcolm <dmalcolm>
Component:	python2	Assignee:	Charalampos Stratakis <cstratak>
Status:	CLOSED UPSTREAM	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	bkabrda, cstratak, dmalcolm, ivazqueznet, jonathansteffan, mhroncok, pviktori, rkuska, tomspur, torsava
Target Milestone:	---	Keywords:	FutureFeature
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-07 11:27:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Dave Malcolm 2012-04-18 17:54:46 UTC

Description of problem:
I received this report about a problem on a network filesystem in which:
> strace of a python app on his systems shows an fstat() after every open().
> It's the python built-in open() that does the fstat().

I believe that the fstat being seen is due to this code in Objects/fileobject.c (line 126 below), which is called by builtin-open (actually by file_init) when the fopen() succeeds:

   115  /* On Unix, fopen will succeed for directories.
   116     In Python, there should be no file objects referring to
   117     directories, so we need a check.  */
   118  
   119  static PyFileObject*
   120  dircheck(PyFileObject* f)
   121  {
   122  #if defined(HAVE_FSTAT) && defined(S_IFDIR) && defined(EISDIR)
   123      struct stat buf;
   124      if (f->f_fp == NULL)
   125          return f;
>>>126      if (fstat(fileno(f->f_fp), &buf) == 0 &&
   127          S_ISDIR(buf.st_mode)) {
   128          char *msg = strerror(EISDIR);
   129          PyObject *exc = PyObject_CallFunction(PyExc_IOError, "(isO)",
   130                                                EISDIR, msg, f->f_name);
   131          PyErr_SetObject(PyExc_IOError, exc);
   132          Py_XDECREF(exc);
   133          return NULL;
   134      }
   135  #endif
   136      return f;
   137  }

Given that we control the stack (kernel and glibc), perhaps we can patch things to avoid this extra syscall, with some kind of fopen that fails with EISDIR 

Version-Release number of selected component (if applicable):
python-2.7.3-1

Comment 1 Dave Malcolm 2012-04-18 18:02:01 UTC

The dircheck() call was added by upstream in response to:
  http://bugs.python.org/issue487277

Initial commit was:
  http://hg.python.org/cpython/rev/9ba9d1238804
then removed in:
  http://hg.python.org/cpython/rev/aaf5ee3b8f93
then fixed in:
  http://hg.python.org/cpython/rev/acf5ecd806af

Comment 3 Fedora End Of Life 2013-04-03 14:38:01 UTC

This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Comment 4 Fedora Admin XMLRPC Client 2013-05-10 04:59:23 UTC

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 5 Fedora Admin XMLRPC Client 2013-05-10 05:00:35 UTC

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 6 Bohuslav "Slavek" Kabrda 2014-06-05 14:00:10 UTC

So, I'd first like to concentrate on Python 3 regarding this, since I found out it does 6 (!) syscalls on opening a file (2x fstat, 2x ioctl and 2xlseek). I'm currently working on a patch that would bring this number down. I'll keep noting my progress in this bug. Python 2 is not that interesting right now, since we'll be switching Fedora to Python 3 as a default, so Python 3 should get all the work.

Comment 7 Bohuslav "Slavek" Kabrda 2014-06-05 14:55:18 UTC

(In reply to Bohuslav "Slavek" Kabrda from comment #6)

> out it does 6 (!) syscalls on opening a file (2x fstat, 2x ioctl and

(That was incorrect, Python 3.4 only does 5 - there's just 1 lseek)

Comment 8 Bohuslav "Slavek" Kabrda 2014-07-03 10:34:37 UTC

So, my first patch for this finally got accepted, removing one of the fstat calls:
http://bugs.python.org/issue21679
http://hg.python.org/cpython/rev/3b5279b5bfd1

Comment 9 Fedora End Of Life 2015-01-09 21:57:55 UTC

This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 10 Fedora Admin XMLRPC Client 2015-05-12 12:01:52 UTC

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 11 Fedora Admin XMLRPC Client 2016-01-29 13:04:08 UTC

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 12 Fedora Admin XMLRPC Client 2017-01-10 18:48:24 UTC

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.