Bug 1976169

Summary: mod_wsgi segfaults when python program outputs a warning message
Product: Red Hat Enterprise Linux 9 Reporter: Alfredo Moralejo <amoralej>
Component: mod_wsgiAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED CURRENTRELEASE QA Contact: Branislav Náter <bnater>
Severity: high Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: apevec, bnater, bstinson, jpena, jwboyer, luhliari
Target Milestone: betaKeywords: AutoVerified, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mod_wsgi-4.7.1-7.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-12-07 22:04:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alfredo Moralejo 2021-06-25 11:52:06 UTC 
Description of problem:

When testing OpenStack on current compose of CentOS Stream 9 we are hitting the problem reported in:

https://github.com/GrahamDumpleton/mod_wsgi/issues/650

The program we are running under wsgi creates following warnings:

/usr/lib/python3.9/site-packages/oslo_utils/fnmatch.py:24: DeprecationWarning: Using the oslo.utils's 'fnmatch' module is deprecated, please use the stdlib 'fnmatch' module.                
  debtcollector.deprecate(                                                                                                                                                                   
/usr/lib/python3.9/site-packages/aodh/storage/impl_sqlalchemy.py:112: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade                       
  self._engine_facade = db_session.EngineFacade(self.dress_url(url),                                                                                                                         
/usr/lib/python3.9/site-packages/aodh/storage/impl_sqlalchemy.py:112: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade                       
  self._engine_facade = db_session.EngineFacade(self.dress_url(url),             

and mod_wsgi crashes with following stack:

 Process 73780 (httpd) of user 978 dumped core.
                                                   
                                                   Stack trace of thread 73810:
                                                   #0  0x00007f9d04be6395 Log_write (mod_wsgi_python3.so + 0xe395)
                                                   #1  0x00007f9d049a0469 method_vectorcall_VARARGS (libpython3.9.so.1.0 + 0x126469)
                                                   #2  0x00007f9d049a20df PyObject_VectorcallMethod (libpython3.9.so.1.0 + 0x1280df)
                                                   #3  0x00007f9d049d406e _textiowrapper_writeflush (libpython3.9.so.1.0 + 0x15a06e)
                                                   #4  0x00007f9d04a29ba9 _io_TextIOWrapper_write (libpython3.9.so.1.0 + 0x1afba9)
                                                   #5  0x00007f9d04997a77 method_vectorcall_O (libpython3.9.so.1.0 + 0x11da77)
                                                   #6  0x00007f9d049839f0 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1099f0)
                                                   #7  0x00007f9d04990373 function_code_fastcall (libpython3.9.so.1.0 + 0x116373)
                                                   #8  0x00007f9d0498378d _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10978d)
                                                   #9  0x00007f9d04990373 function_code_fastcall (libpython3.9.so.1.0 + 0x116373)
                                                   #10 0x00007f9d04a01223 PyObject_CallOneArg.lto_priv.22 (libpython3.9.so.1.0 + 0x187223)
                                                   #11 0x00007f9d04a4b09a warn_explicit (libpython3.9.so.1.0 + 0x1d109a)
                                                   #12 0x00007f9d04a4a938 do_warn (libpython3.9.so.1.0 + 0x1d0938)
                                                   #13 0x00007f9d04a4a7fc PyErr_WarnEx (libpython3.9.so.1.0 + 0x1d07fc)
                                                   #14 0x00007f9d048e2d5f convertsimple.cold (libpython3.9.so.1.0 + 0x68d5f)
                                                   #15 0x00007f9d0498c002 vgetargs1_impl (libpython3.9.so.1.0 + 0x112002)
                                                   #16 0x00007f9d0498be1a PyArg_ParseTuple (libpython3.9.so.1.0 + 0x111e1a)
                                                   #17 0x00007f9d04be635b Log_write (mod_wsgi_python3.so + 0xe35b)
                                                   #18 0x00007f9d049a0469 method_vectorcall_VARARGS (libpython3.9.so.1.0 + 0x126469)
                                                   #19 0x00007f9d049a20df PyObject_VectorcallMethod (libpython3.9.so.1.0 + 0x1280df)
                                                   #20 0x00007f9d049d406e _textiowrapper_writeflush (libpython3.9.so.1.0 + 0x15a06e)
                                                   #21 0x00007f9d04a29ba9 _io_TextIOWrapper_write (libpython3.9.so.1.0 + 0x1afba9)
                                                   #22 0x00007f9d04997a77 method_vectorcall_O (libpython3.9.so.1.0 + 0x11da77)
                                                   #23 0x00007f9d049839f0 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1099f0)



Version-Release number of selected component (if applicable):


python3-mod_wsgi-4.7.1-4.el9.x86_64

How reproducible:

allways with current openstack which provides some warnings on wsgi services.

Steps to Reproduce:

There is a simple example of reproducer in https://github.com/GrahamDumpleton/mod_wsgi/issues/650

apparently https://github.com/GrahamDumpleton/mod_wsgi/commit/2c9d1b3e725ad2a072e9fef4cb1cb0bb9ae2d540 may be the fix. I'm working to build and test 4.7.1-4 with this patch.

Note this may be a critical issue for many services running on mod_wsgi.
Comment 1 Alfredo Moralejo 2021-06-25 12:14:02 UTC 
I've just tested that applying following patch on top of mod_wsgi-4.7.1-6.el9 (last build in cs9 koji), it works fine:

# cat ../SOURCES/mod_wsgi_fix_deprecation.patch
--- a/src/server/wsgi_logger.c  2021-06-25 11:59:23.936000000 +0000
+++ b/src/server/wsgi_logger.c  2021-06-25 12:00:10.257000000 +0000
@@ -223,7 +223,7 @@
     return Py_False;
 }
 
-static void Log_queue(LogObject *self, const char *msg, int len)
+static void Log_queue(LogObject *self, const char *msg, Py_ssize_t len)
 {
     const char *p = NULL;
     const char *q = NULL;
@@ -330,7 +330,7 @@
 static PyObject *Log_write(LogObject *self, PyObject *args)
 {
     const char *msg = NULL;
-    int len = -1;
+    Py_ssize_t len = -1;
 
     WSGIThreadInfo *thread_info = NULL;
 
--- a/src/server/wsgi_python.h  2021-06-25 12:00:38.400000000 +0000
+++ b/src/server/wsgi_python.h  2021-06-25 12:01:15.686000000 +0000
@@ -21,6 +21,8 @@
 
 /* ------------------------------------------------------------------------- */
 
+#define PY_SSIZE_T_CLEAN 1
+
 #include <Python.h>
 
 #if !defined(PY_VERSION_HEX)
Comment 2 Javier Peña 2021-06-25 13:17:27 UTC 
Just FTR, the patch mentioned by Alfredo is https://github.com/GrahamDumpleton/mod_wsgi/commit/2c9d1b3e725ad2a072e9fef4cb1cb0bb9ae2d540
Comment 6 Alfredo Moralejo 2021-07-07 11:19:49 UTC 
mod_wsgi-4.7.1-7.el9 is not in last compose, it seems blocked by some gate?
Comment 8 Alan Pevec 2021-07-16 22:30:02 UTC 
Can be moved to ON_QA, not sure what is the workflow for CS9 bugs, build is in CS9 compose:
https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/Packages/python3-mod_wsgi-4.7.1-7.el9.x86_64.rpm

It works in Puppet OpenStack integration test scenarios.