Bug 2192665
| Summary: | cargo downloads fail randomly in f38 | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Enrico Scholz <rh-bugzilla> |
| Component: | curl | Assignee: | Kamil Dudka <kdudka> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 38 | CC: | amulhern, igor.raits, jistone, kdudka, lzaoral, msekleta, paul, rust-sig, svashisht, TicoTimo, vmihalko |
| Target Milestone: | --- | Keywords: | Regression |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | curl-8.0.1-1.fc38 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-05-05 01:28:52 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Enrico Scholz
2023-05-02 16:58:01 UTC
Other errors | Caused by: | failed to verify the checksum of `windows-sys v0.45.0` | Caused by: | failed to verify the checksum of `windows_x86_64_msvc v0.48.0` | Downloaded security-framework v2.8.2 | free(): invalid pointer | Aborted (core dumped) Which version of cargo / rust is installed on your system?
(i.e. please paste the output of this command: "rpm -q rust cargo")
We discovered some code generation problems with rust 1.69 when built against LLVM 16, so a follow-up update reverted it to using LLVM 15, but the follow-up update hasn't been pushed to the stable ("fedora") repositories yet: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7fbc906f3c
If Rust 1.69/LLVM 16 is indeed the source of your problem, the pending update might fix this problem (or might not).
I updated to # rpm -qa cargo\* rust\* rust-srpm-macros-24-1.fc38.noarch cargo-1.69.0-2.fc38.x86_64 rust-std-static-1.69.0-2.fc38.x86_64 rust-1.69.0-2.fc38.x86_64 from https://koji.fedoraproject.org/koji/buildinfo?buildID=2194853 and problem still persists. valgrind shows lot of "invalid read" errors (which occur too deep in free'd blocks to be the usual optimized strlen()/memcpy()) ==1036== Invalid read of size 2 ==1036== at 0x484B660: memmove (vg_replace_strmem.c:1398) ==1036== by 0x4A4038E: UnknownInlinedFun (vtls.c:517) ==1036== by 0x4A4038E: ossl_new_session_cb (openssl.c:3009) ==1036== by 0x4AB5E94: ssl_update_cache (ssl_lib.c:3768) ==1036== by 0x4AE32C0: UnknownInlinedFun (statem_clnt.c:2618) ==1036== by 0x4AE32C0: ossl_statem_client_process_message (statem_clnt.c:1054) ==1036== by 0x4ADD61E: UnknownInlinedFun (statem.c:647) ==1036== by 0x4ADD61E: state_machine (statem.c:442) ==1036== by 0x4ACD35D: ssl3_read_bytes (rec_layer_s3.c:1716) ==1036== by 0x4AA76A4: ssl3_read_internal (s3_lib.c:4463) ==1036== by 0x4AADF4A: SSL_read (ssl_lib.c:1885) ==1036== by 0x4A4BE19: ossl_recv.lto_priv.0 (openssl.c:4624) ==1036== by 0x4A4A043: ssl_cf_recv.lto_priv.0 (vtls.c:1575) ==1036== by 0x4A06231: http2_recv (http2.c:1785) ==1036== by 0x4A2876D: Curl_read (sendf.c:743) ==1036== Address 0x116f5fb8 is 8 bytes inside a block of size 10 free'd ==1036== at 0x48440E4: free (vg_replace_malloc.c:884) ==1036== by 0x4A40CDB: reuse_conn.lto_priv.0 (url.c:3411) ==1036== by 0x4A2169D: UnknownInlinedFun (url.c:3771) ==1036== by 0x4A2169D: UnknownInlinedFun (url.c:3970) ==1036== by 0x4A2169D: multi_runsingle (multi.c:1934) ==1036== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==1036== by 0x9C02DA: ??? (in /usr/bin/cargo) ==1036== by 0x4B31B9: ??? (in /usr/bin/cargo) ==1036== by 0x4AE2B1: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== by 0x29216C: ??? (in /usr/bin/cargo) ==1036== Block was alloc'd at ==1036== at 0x484186F: malloc (vg_replace_malloc.c:393) ==1036== by 0x50EEFEE: strdup (strdup.c:42) ==1036== by 0x4A1F295: UnknownInlinedFun (url.c:1892) ==1036== by 0x4A1F295: UnknownInlinedFun (url.c:3496) ==1036== by 0x4A1F295: UnknownInlinedFun (url.c:3970) ==1036== by 0x4A1F295: multi_runsingle (multi.c:1934) ==1036== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==1036== by 0x9C02DA: ??? (in /usr/bin/cargo) ==1036== by 0x4B31B9: ??? (in /usr/bin/cargo) ==1036== by 0x4AE2B1: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== by 0x29216C: ??? (in /usr/bin/cargo) and then sometimes ==1036== Invalid free() / delete / delete[] / realloc() ==1036== at 0x48440E4: free (vg_replace_malloc.c:884) ==1036== by 0x4A363AC: Curl_free_request_state (url.c:2103) ==1036== by 0x4A571AE: Curl_close.isra.0 (url.c:411) ==1036== by 0x49F12AB: curl_easy_cleanup (easy.c:798) ==1036== by 0x8092C4: ??? (in /usr/bin/cargo) ==1036== by 0x4425C7: ??? (in /usr/bin/cargo) ==1036== by 0x4AE37F: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== by 0x29216C: ??? (in /usr/bin/cargo) ==1036== Address 0x6a92118 is 24 bytes inside a block of size 872 free'd ==1036== at 0x48440E4: free (vg_replace_malloc.c:884) ==1036== by 0x4A049C3: UnknownInlinedFun (http_proxy.c:236) ==1036== by 0x4A049C3: http_proxy_cf_detach_data.lto_priv.0 (http_proxy.c:1147) ==1036== by 0x4A14E6B: UnknownInlinedFun (cfilters.c:477) ==1036== by 0x4A14E6B: Curl_detach_connection (multi.c:957) ==1036== by 0x4A1505F: multi_done.lto_priv.0 (multi.c:661) ==1036== by 0x4A15787: UnknownInlinedFun (multi.c:819) ==1036== by 0x4A15787: curl_multi_remove_handle (multi.c:768) ==1036== by 0x9C0379: ??? (in /usr/bin/cargo) ==1036== by 0x80929C: ??? (in /usr/bin/cargo) ==1036== by 0x4425C7: ??? (in /usr/bin/cargo) ==1036== by 0x4AE37F: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== Block was alloc'd at ==1036== at 0x4846464: calloc (vg_replace_malloc.c:1340) ==1036== by 0x4A09145: UnknownInlinedFun (http_proxy.c:142) ==1036== by 0x4A09145: http_proxy_cf_connect.lto_priv.0 (http_proxy.c:1078) ==1036== by 0x4A4AA0B: UnknownInlinedFun (vtls.c:1512) ==1036== by 0x4A4AA0B: ssl_cf_connect.lto_priv.0 (vtls.c:1493) ==1036== by 0x4A1EF0C: UnknownInlinedFun (cfilters.c:367) ==1036== by 0x4A1EF0C: multi_runsingle (multi.c:2070) ==1036== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==1036== by 0x9C02DA: ??? (in /usr/bin/cargo) ==1036== by 0x4B31B9: ??? (in /usr/bin/cargo) ==1036== by 0x4AE2B1: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== Might be problem with curl or openssl libcurl-7.87.0-8.fc38.x86_64 openssl-libs-3.0.8-2.fc38.x86_64 affects also usual download during 'cargo build'; not only 'cargo vendor' downgrade to curl-7.85.0-8.fc37.x86_64 seems to fix the problem; reassigning to curl I also tried libcurl-7.86.0-4.fc38.x86_64 and had no valgrind errors, while libcurl-7.87.0-1.fc38.x86_64 does show errors. There are also no valgrind errors with rawhide's initial upgrade to libcurl-8.0.0-1.fc39, nor the current libcurl-8.0.1-3.fc39. (all other packages remaining on f38) very likely http proxy related; I can not reproduce the 'cargo' download problems on a system without proxy. I have tested libcurl-7.86.0-4.fc38.x86_64 OK libcurl-7.87.0-1.fc38.x86_64 broken libcurl-7.87.0-8.fc38.x86_64 broken libcurl-7.88.0-2.fc39.x86_64 broken libcurl-7.88.1-1.fc39.x86_64 broken libcurl-8.0.0-1.fc39.x86_64 OK Although 'valgrind' is silent with 7.88.0 + .1, 'cargo' fails there (just for reference) with | Caused by: | [2] Failed initialization ([CONN-3-0] send: no filter connected) I am not able to reproduce the bug in my testing environment. Using a local squid and/or valgrind did not help. Based on the stack trace in comment #3, I believe it could have been fixed with this upstream commit: https://github.com/curl/curl/commit/f8da4f2f2d0451dc0a126ae3e5077b4527ccdc86 I will shortly prepare a build with the above fix backported for testing. Fedora commit: https://src.fedoraproject.org/rpms/curl/c/472ff9ff394deee558414aea9cd405309e2642b9?branch=f38 FEDORA-2023-46c1b96671 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-46c1b96671 libcurl-7.87.0-9.fc38.x86_64 does **not** fix the download problem; valgrind is still noisy with it ==5363== Invalid write of size 8 ==5363== at 0x4A187BA: UnknownInlinedFun (mime.c:1308) ==5363== by 0x4A187BA: Curl_mime_cleanpart (mime.c:1178) ==5363== by 0x49FD47D: Curl_http_done (http.c:1615) ==5363== by 0x4A15006: multi_done.lto_priv.0 (multi.c:646) ==5363== by 0x4A1F521: multi_runsingle (multi.c:2125) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== Address 0x58dbd98 is 488 bytes inside a block of size 872 free'd ==5363== at 0x48440E4: free (vg_replace_malloc.c:884) ==5363== by 0x4A049C3: UnknownInlinedFun (http_proxy.c:236) ==5363== by 0x4A049C3: http_proxy_cf_detach_data.lto_priv.0 (http_proxy.c:1147) ==5363== by 0x4A14E6B: UnknownInlinedFun (cfilters.c:477) ==5363== by 0x4A14E6B: Curl_detach_connection (multi.c:957) ==5363== by 0x4A1505F: multi_done.lto_priv.0 (multi.c:661) ==5363== by 0x4A1F56A: multi_runsingle (multi.c:2544) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== Block was alloc'd at ==5363== at 0x4846464: calloc (vg_replace_malloc.c:1340) ==5363== by 0x4A09145: UnknownInlinedFun (http_proxy.c:142) ==5363== by 0x4A09145: http_proxy_cf_connect.lto_priv.0 (http_proxy.c:1078) ==5363== by 0x4A4AA6B: UnknownInlinedFun (vtls.c:1536) ==5363== by 0x4A4AA6B: ssl_cf_connect.lto_priv.0 (vtls.c:1517) ==5363== by 0x4A1EF0C: UnknownInlinedFun (cfilters.c:367) ==5363== by 0x4A1EF0C: multi_runsingle (multi.c:2070) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== warning: spurious network error (2 tries remaining): [56] Failure when receiving data from the peer (Proxy CONNECT aborted) ==5363== Invalid free() / delete / delete[] / realloc() ==5363== at 0x48440E4: free (vg_replace_malloc.c:884) ==5363== by 0x4A363AC: Curl_free_request_state (url.c:2103) ==5363== by 0x4A1F0DC: UnknownInlinedFun (url.c:3978) ==5363== by 0x4A1F0DC: multi_runsingle (multi.c:1934) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== by 0x215563: ??? (in /usr/bin/cargo) ==5363== Address 0x58dbbc8 is 24 bytes inside a block of size 872 free'd ==5363== at 0x48440E4: free (vg_replace_malloc.c:884) ==5363== by 0x4A049C3: UnknownInlinedFun (http_proxy.c:236) ==5363== by 0x4A049C3: http_proxy_cf_detach_data.lto_priv.0 (http_proxy.c:1147) ==5363== by 0x4A14E6B: UnknownInlinedFun (cfilters.c:477) ==5363== by 0x4A14E6B: Curl_detach_connection (multi.c:957) ==5363== by 0x4A1505F: multi_done.lto_priv.0 (multi.c:661) ==5363== by 0x4A1F56A: multi_runsingle (multi.c:2544) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== Block was alloc'd at ==5363== at 0x4846464: calloc (vg_replace_malloc.c:1340) ==5363== by 0x4A09145: UnknownInlinedFun (http_proxy.c:142) ==5363== by 0x4A09145: http_proxy_cf_connect.lto_priv.0 (http_proxy.c:1078) ==5363== by 0x4A4AA6B: UnknownInlinedFun (vtls.c:1536) ==5363== by 0x4A4AA6B: ssl_cf_connect.lto_priv.0 (vtls.c:1517) ==5363== by 0x4A1EF0C: UnknownInlinedFun (cfilters.c:367) ==5363== by 0x4A1EF0C: multi_runsingle (multi.c:2070) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== Upstream https://github.com/curl/curl/commit/3f3ddee0665176040b3eaf89a912a922726ecb18 (7.88.0) sounds like a good candidate for these errors. But does not explain why cargo download is still broken until 8.0 Thanks for feedback! The stack trace is now different though. I will try to backport the other upstream commit. This would be much easier to debug if I had a local reproducer... Fedora commit: https://src.fedoraproject.org/rpms/curl/c/23fee3d38607b49d63fbbc7e2f374f489b7be184?branch=f38 git bisect shows that
commit af22c2a546ab862ab577c8d9d3609af0de178974 (HEAD)
Author: Stefan Eissing <stefan>
Date: Tue Nov 22 09:55:41 2022 +0100
vtls: localization of state data in filters
is the first commit which broke 'cargo'.
It went ok by
commit 821f6e2a89de8aec1c7da3c0f381b92b2b801efc
Author: Stefan Eissing <stefan>
Date: Thu Feb 9 16:07:34 2023 +0100
CURLOPT_PIPEWAIT: allow waited reuse also for subsequent connections
Oops, forgot to commit the patch: https://src.fedoraproject.org/rpms/curl/c/ac9ca2137b4a7ce3b44ce16dedabe950e183e933?branch=f38 Thanks! Could you please check whether commit 821f6e2a89de8aec1c7da3c0f381b92b2b801efc alone fixes it? The changes in /lib seem to apply fine. (In reply to Kamil Dudka from comment #16) > Oops, forgot to commit the patch: > https://src.fedoraproject.org/rpms/curl/c/ > ac9ca2137b4a7ce3b44ce16dedabe950e183e933?branch=f38 ... and it does not build anyway because the change depends on the following massive rewrite, which introduced bunch of other bugs: https://github.com/curl/curl/commit/71b7e0161032927cdfb4e75ea40f65b8898b3956 At this point I think it would be safer to rebase curl in f38 to curl-8.0.1. Any objections? I have prepared the rebase here: https://src.fedoraproject.org/rpms/curl/pull-request/17 rebase to 8.0.1 is ok with me. 821f6e2a89de8aec1c7da3c0f381b92b2b801efc on 7.87.0 makes 'cargo' work again, but valgrind is still noisy with this combination. After f8da4f2f2d0451dc0a126ae3e5077b4527ccdc86 (applied by you in libcurl-7.87.0-9), valgrind becomes silent. Thank you for testing it! I will proceed with the rebase. Although it looks like a new major release, there is no SONAME bump or any known ABI breaking changes. Fedora commit: https://src.fedoraproject.org/rpms/curl/c/f854769ba434c9fffd92498b590b8d0e29c9a616?branch=f38 FEDORA-2023-46c1b96671 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-46c1b96671 FEDORA-2023-46c1b96671 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report. |