Created attachment 1132160 [details]
backtrace

Created attachment 1132258 [details]
httpd error_log

This might be known issue with mod_wsgi, https://github.com/GrahamDumpleton/mod_wsgi/issues/11

What version of mod_wsgi, Apache and Python are you using?

Graham, you are the maintainer and expert on mod_wsgi. Can you have a look please? The machine is running mod_wsgi-3.4-12.el7.

The big question is whether this is happening on any attempt to shutdown a process or when handling a specific request.

If it is only on process shutdown there could be various issues that could cause it, including:

* The referenced issue 11 from mod_wsgi GitHub issue tracker.
* The fact that in a process where multithreading is used, you can't be certain that there aren't background threads still running when destroying the Python interpreter.
* Reference counting bugs in how Python is used, or misuse of Apache memory pools that only show up on process shutdown.

Without a stack trace from a core dump where at least function name symbols aren't stripped, impossible to even speculate.

That there are some corner cases that can cause core dumps on process shutdown is therefore known and not too much that can do about it unless have good details on how to cause it or where it is occurring. Trying to uncover the causes can be very very hard and even in latest mod_wsgi I myself experience some crashes on shutdown in testing but as soon as try and insert gdb to debug the problems they don't occur. I know it relates to incorrect use of memory due to memory checking libraries warning me first, but no closer to working it out.

If instead the core dump is occurring when a request is being handled, that is a bit easier to work out the problem.

A common cause of processes crashing when handling requests is the result of C extensions for Python modules not being implemented to work in Python sub interpreters. The solution to that is to force the use of the main Python interpreter context using the 'WSGIApplicationGroup`` directive for mod_wsgi in Apache configuration. This may necessitate also using mod_wsgi daemon mode if running multiple Django web applications as you can't run more than one of those in a process, thus need to delegate to different mod_wsgi daemon process groups.

So for this case based on details given, can't really identify any specific problem. If it is infrequent and the hosted web applications are otherwise working fine, would suggest it relates to process shutdown issues.

If application is impacted, would start by looking at how mod_wsgi is configured, whether using embedded or daemon mode and whether use of main Python interpreter is being forced using 'WSGIApplicationGroup' with the value '%{GLOBAL}'.

*** Bug 1317376 has been marked as a duplicate of this bug. ***

Moving to mod_wsgi, the workaround:
  WSGIApplicationGroup' with the value '%{GLOBAL}'.

is suboptimal.

IPA management framework, KDC proxy and future Custodia service(will be introduced in 7.3) needs to be separated.

Therefore proposing a fix on mod_wsgi side.

If you have shown that setting:

WSGIApplicationGroup %{GLOBAL}

fixes the problem being seen, then it means the problem is in a third party Python package which includes a C extension component which has not been implemented to work correctly in sub interpreters.

In other words, it has got nothing to do with mod_wsgi and that third party Python package needs to be fixed.

That said, the recommended way of using mod_wsgi is to using separate daemon process groups for each separate hosted application and also forcing use of the main interpreter in those respective processes using WSGIApplicationGroup with value %{GLOBAL}.

Preferably, even if using embedded mode, you should still use WSGIApplicationGroup with that value. Using embedded mode is very much not recommended though because the typical Apache configuration is not set up properly for Python applications and using embedded mode will result in poor performance and excessive memory usage. This is because the sub optional configuration of Apache, not being tailored for Python web applications, will exacerbate issues arising around arguable design flaws in Apache process management. 

Why are you saying it is sub optimal when using it is the recommended way of doing things with mod_wsgi?

If you have been able to pin down further the scenario under which it occurs, please provide additional details. To date, insufficient information has been provided about how Apache has been configured to run the WSGI application using mod_wsgi, such as whether embedded mode or daemon mode is used, and how daemon mode and application groups are set up if it is used.

Thanks a lot Graham for providing such details feedback!

Sudhir, others: assigning application crashes to a web server is a bit like blaming a C compiler bug for your application not behaving right.  It's not impossible that's true, but you need to do a lot of work to demonstrate it.  Without narrowing this down to a very specific repro case there is not a lot we can do about it.

FWIW, I have been working on changes to very latest mod_wsgi which will in the first instance provide a flag which allows you to say not to try and destroy the Python interpreter on process shutdown. It isn't a straight forward change as I need to still go through process of trying to stop threads and call Python atexit callbacks so an application has a chance to clean up stuff. These are things normally down when the interpreter is destroyed.

With the change it would at least be possible to hopefully silence the process crashes given that they appear to be occurring during destruction of the Python interpreter. These changes are unlikely to be able to be back ported to 3.4 and would also hesitate contemplating doing that for 4.3 either.

Hello

We are seeing this error in CentOS 8 as well.

2021-11-14T12:38:16.065076+00:00 bepcrfoam01 kernel: [10652415.515561] httpd[1744656]: segfault at 8 ip 00007f3f453cb984 sp 00007f3f3663da60 error 4 in libpython2.7.so.1.0[7f3f45348000+186000]

We are not sure how to reproduce it.
Can someone help to reopen the BUG.

(In reply to Iqbal Singh Aulakh from comment #29)
> Hello
> 
> We are seeing this error in CentOS 8 as well.
> 
> 2021-11-14T12:38:16.065076+00:00 bepcrfoam01 kernel: [10652415.515561]
> httpd[1744656]: segfault at 8 ip 00007f3f453cb984 sp 00007f3f3663da60 error
> 4 in libpython2.7.so.1.0[7f3f45348000+186000]
> 
> We are not sure how to reproduce it.
> Can someone help to reopen the BUG.

The original bug was about RHEL7. On CentOS 8 default platform Python version is 3.6, not 2.7.

If you are using something else, please open a different bug with more details.