Bug 1163516
| Summary: | RHEL7 httpd crash in new_threadstate during ipactl restart with | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Scott Poore <spoore> | ||||||||||||||||||
| Component: | mod_wsgi | Assignee: | Luboš Uhliarik <luhliari> | ||||||||||||||||||
| Status: | CLOSED DUPLICATE | QA Contact: | BaseOS QE - Apps <qe-baseos-apps> | ||||||||||||||||||
| Severity: | unspecified | Docs Contact: | |||||||||||||||||||
| Priority: | unspecified | ||||||||||||||||||||
| Version: | 7.0 | CC: | jkaluza, jkejda, jorton, sgoveas | ||||||||||||||||||
| Target Milestone: | rc | ||||||||||||||||||||
| Target Release: | --- | ||||||||||||||||||||
| Hardware: | Unspecified | ||||||||||||||||||||
| OS: | Unspecified | ||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||||
| Last Closed: | 2017-07-13 16:34:28 UTC | Type: | Bug | ||||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||
| Embargoed: | |||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||
|
Description
Scott Poore
2014-11-12 21:57:55 UTC
Created attachment 956881 [details]
abrt output email for the crash
Curious crash here. Backtrace is: :#0 PyGrammar_RemoveAccelerators (g=<optimized out>) at /usr/src/debug/Python-2.7.5/Parser/acceler.c:48 :#1 0x00007f9b4134e3bb in Py_Finalize () at /usr/src/debug/Python-2.7.5/Python/pythonrun.c:547 :#2 0x00007f9b41627482 in wsgi_python_term () at mod_wsgi.c:5901 :#3 0x00007f9b41627a1b in wsgi_python_child_cleanup (data=<optimized out>) at mod_wsgi.c:6942 :#4 0x00007f9b4f4061ae in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2352 :#5 apr_pool_destroy (pool=0x7f9b5127d038) at memory/unix/apr_pools.c:814 :#6 0x00007f9b41631e50 in wsgi_start_process (p=p@entry=0x7f9b5102f158, daemon=<optimized out>) at mod_wsgi.c:11958 which says tehre is a Python interpreter being *destroyed* during startup... confusing. From the coredump, it looks like Python interpreter is destroyed, but before mod_wsgi process is stopped completely, its deadlock watchdog thread is executed. This thread does not check if the Python interpreter is initialized and presumes it's always there, but it's not in this case, therefore it crashes. Proposed patch fixes that by: 1. Using apr_atomic_* for wsgi_python_initialized to be able to use it from multiple threads. 2. Setting wsgi_python_initialized to 0 *before* calling Py_Finalize();, so the deadlock watcher thread won't call Python methods during the Py_Finalize call. 3. Checking for wsgi_python_initialized in deadlock watcher thread. Created attachment 957145 [details]
proposed patch
Scott, if you see this crashes regularly or you are able to reproduce them, we can provide a test package with the patch applied, so you could verify it really fixes the issue you see. Jan, Unfortunately, I'm not certain about the frequency. I haven't seen this particular crash occur again since I opened the bug. I have also attempted to manually reproduce this by restarting ipa repeatedly. So far, no luck. Think I may have better luck reproducing if I just restarted httpd? I could certainly take a look at a test rpm and add to a repo and test with that but, as of right now, I'm not sure about reproducing this one properly. Thanks, Scott Created attachment 958317 [details]
abrt output email for another crash
I may have spoken too soon. I just saw this crash occur. It looks a little different but, I want to make sure. Is this crash the same?
This is the same crash. I will build the test rpm for you today. Ok got the test rpm and running some tests now. Hopefully, will see (or not see) some results today. I'm also asking my team to do the same if they see the crash. Thanks, Scott Ok, I don't think we've seen this crash anywhere again when the test package was used. I don't know how many jobs that is in total though. The only place I've seen the crash was a test not using the test rpm. Thanks, Scott Created attachment 979219 [details]
Crash email 1
Received crash in the same function while ipa stop start on 2 different servers. I have attached the complete abrt emails and coredumps
Created attachment 979221 [details]
Coredump 1
Created attachment 979222 [details]
Crash email 2
Created attachment 979223 [details]
Coredump 2
Steeve, are these new crashes with the patched mod_wsgi or not? (In reply to Joe Orton from comment #17) > Steeve, are these new crashes with the patched mod_wsgi or not? mod_wsgi-3.4-12.el7_0.x86_64 was the version used on these servers. I will try mod_wsgi-3.4-13.el7_0.x86_64 and report back. Joe, I don't think we saw crashes when using -13 but, I'll let Steeve confirm since he's seeing them more readily at the moment. Will that fix make it into RHEL7.1? Thanks, Scott Joe, Scott, I tested with the fixed version in multiple jobs. There were no crashes seen. Thanks, -Steeve Created attachment 1069579 [details]
abrt email for kdcproxy crash
This looks similar and does appears to have occurred for the affected version:
mod_wsgi-3.4-12.el7_0.x86_64
Attaching the abrt email we got for this one.
*** This bug has been marked as a duplicate of bug 1445540 *** |