Bug 2092160
| Summary: | jq runs out of memory and causes mod_auth_openidc to seqfault | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Kevin Fenzi <kfenzi> |
| Component: | jq | Assignee: | Tomas Halman <thalman> |
| Status: | VERIFIED --- | QA Contact: | Scott Poore <spoore> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.6 | CC: | aboscatt, kevin, spoore |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | sync-to-jira | ||
| Fixed In Version: | jq-1.6-7.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi Kevin, could you provide your (sanitized?) httpd configuration so I can try to reproduce the issue? Do you need everything ? Or just the mod_auth_openidc ? auth_openidc.conf: ``` OIDCOAuthIntrospectionEndpoint https://id.fedoraproject.org/openidc/TokenInfo OIDCOAuthClientID src-verifier OIDCOAuthClientSecret redacted OIDCOAuthIntrospectionEndpointAuth client_secret_basic OIDCOAuthIntrospectionEndpointMethod POST OIDCOAuthTokenIntrospectionInterval 60 OIDCOAuthSSLValidateServer On OIDCOAuthAcceptTokenAs basic ``` httpush.conf: ``` <Location /> AuthType oauth20 Require all granted </Location> <LocationMatch ".*/git-receive-pack"> AuthType oauth20 Require claims_expr '(.scope | index("https://src.fedoraproject.org/push") != null)' </LocationMatch> <LocationMatch ".*/info/refs"> <If "%{QUERY_STRING} =~ /service=git-upload-pack/"> Require all granted </If> <Else> AuthType oauth20 Require claims_expr '(.scope | index("https://src.fedoraproject.org/push") != null)' </Else> </LocationMatch> SuexecUserGroup pagure package ``` I can get the rest if you want the entire thing... Hi Kevin, thank you, this is enough for apache httpd config. So far I was not able to reproduce the issue. Can you provide the coredump from the crash? It would be good to see how the claims/scopes looks in your case. T. Proved to be difficult to get httpd to give a coredump... but I think I got one. ;) https://infrastructure.fedoraproject.org/infra/tmp/core.httpd.48.a5f728fa134843f98c38f66e860c6613.4056581.16548800650 Hi Kevin, Unfortunately the link doesn't work. Could you provide exact rpm version of involved components? (something like `rpm -qa | grep -e jq -e httpd -e mod_auth`) How does the claim/scope looks like in your setup? Tomas Sigh. It should be: https://infrastructure.fedoraproject.org/infra/tmp/core.httpd.48.a5f728fa134843f98c38f66e860c6613.4056581.1654880065000000.lz4 (Unless bugzilla is cutting it off?) httpd-2.4.37-47.module+el8.6.0+14529+083145da.1.x86_64 httpd-filesystem-2.4.37-47.module+el8.6.0+14529+083145da.1.noarch httpd-tools-2.4.37-47.module+el8.6.0+14529+083145da.1.x86_64 jq-1.6-3.el8.x86_64 jq-debuginfo-1.6-3.el8.x86_64 jq-debugsource-1.6-3.el8.x86_64 mod_auth_gssapi-1.6.1-7.1.el8.x86_64 mod_auth_openidc-2.3.7-11.module+el8.6.0+14082+b6f23e95.x86_64 pagure-web-apache-httpd-5.13.3-2.el8.noarch redhat-logos-httpd-84.5-1.el8.noarch Require claims_expr '(.scope | index("https://src.fedoraproject.org/push") != null)' Thank you Kevin(In reply to Kevin Fenzi from comment #6) > Sigh. It should be: > > https://infrastructure.fedoraproject.org/infra/tmp/core.httpd.48. > a5f728fa134843f98c38f66e860c6613.4056581.1654880065000000.lz4 > > (Unless bugzilla is cutting it off?) > > httpd-2.4.37-47.module+el8.6.0+14529+083145da.1.x86_64 > httpd-filesystem-2.4.37-47.module+el8.6.0+14529+083145da.1.noarch > httpd-tools-2.4.37-47.module+el8.6.0+14529+083145da.1.x86_64 > jq-1.6-3.el8.x86_64 > jq-debuginfo-1.6-3.el8.x86_64 > jq-debugsource-1.6-3.el8.x86_64 > mod_auth_gssapi-1.6.1-7.1.el8.x86_64 > mod_auth_openidc-2.3.7-11.module+el8.6.0+14082+b6f23e95.x86_64 > pagure-web-apache-httpd-5.13.3-2.el8.noarch > redhat-logos-httpd-84.5-1.el8.noarch > > Require claims_expr '(.scope | index("https://src.fedoraproject.org/push") > != null)' Thank you Kevin Happy to help, hopefully you can duplicate it. Let me know if I can provide any more info. Hi Kevin, thank you for your patience. Finally I have a reproducer so I can start working on fix. Jq has an issue when it is used in multi-threading environment. The workaround for you might be to set Apache Httpd MPM configuration to use forking instead of threads. HTH Tom Upstream patch https://github.com/stedolan/jq/pull/2546 Tom Verified. Version :: jq-1.6-7.el8.x86_64 Results :: gating passed no segfault: # gcc -o segfault_test segfault_in_multi_threaded_env.c -lpthread -ljq # ./segfault_test jq_init jq: 0x7f2dd4000b60 prg: .data jq_init jq: 0x7f2dcc000b60 prg: .data compiled dumped: 1 compiled dumped: 1 |
With jq-1.6-3.el8 and using mod_auth_openidc with apache httpd we are getting segfaults. Here's the trace: #0 0x0000000000000000 in ?? () #1 0x00007f1c98bc1019 in jv_nomem_handler (handler=0x0, data=0x0) at src/jv_alloc.c:82 #2 0x00007f1c98bba10d in jq_compile_args (jq=<optimized out>, str=<optimized out>, args=...) at src/execute.c:1162 #3 0x00007f1c994cab6d in oidc_authz_match_claims_expr () from target:/etc/httpd/modules/mod_auth_openidc.so #4 0x00007f1c994cafbf in oidc_authz_worker24 () from target:/etc/httpd/modules/mod_auth_openidc.so #5 0x00007f1c994f2eaa in oidc_authz_checker () from target:/etc/httpd/modules/mod_auth_openidc.so #6 0x00007f1ca5755d61 in apply_authz_sections () from target:/etc/httpd/modules/mod_authz_core.so #7 0x00007f1ca5755e45 in apply_authz_sections () from target:/etc/httpd/modules/mod_authz_core.so #8 0x00007f1ca575611a in authorize_user_core () from target:/etc/httpd/modules/mod_authz_core.so #9 0x00005637546a82e8 in ap_run_auth_checker () #10 0x00005637546aa8bc in ap_process_request_internal () #11 0x00005637546c9840 in ap_process_async_request () #12 0x00005637546c5ce0 in ap_process_http_connection () #13 0x00005637546bc0c8 in ap_run_process_connection () #14 0x00007f1c9dd5fa47 in process_socket () from target:/etc/httpd/modules/mod_mpm_event.so #15 0x00007f1c9dd603ea in worker_thread () from target:/etc/httpd/modules/mod_mpm_event.so #16 0x00007f1ca903f1cf in start_thread () from target:/lib64/libpthread.so.0 #17 0x00007f1ca8aa7d83 in clone () from target:/lib64/libc.so.6 Our config has: Require claims_expr '(.scope | index("https://src.fedoraproject.org/push") != null)' and that seems to cause jq to run out of memory now? It may also have something to do with our httpd setup, but we aren't limiting things that I can think of. Downgrading back to jq-1.5-12.el8.x86_64 and restarting httpd (note: just reloading doesn't do it) seems to work around the issue. Happy to gather more info, etc.