Running `cargo vendor` fails randomly in F38 while working fine in F37 and before. | $ cargo vendor | error: failed to sync | | Caused by: | failed to download packages | | Caused by: | failed to download from `https://crates.io/api/v1/crates/url/2.3.1/download` | | Caused by: | [1] Unsupported protocol (Received HTTP/0.9 when not allowed) 'strace' shows | getpeername(11, {sa_family=AF_INET, sin_port=htons(3128), sin_addr=inet_addr("192.168.12.182")}, [128 => 16]) = 0 | getsockname(11, {sa_family=AF_INET, sin_port=htons(46534), sin_addr=inet_addr("10.0.2.100")}, [128 => 16]) = 0 | sendto(11, "CONNECT static.crates.io:443 HTT"..., 125, MSG_NOSIGNAL, NULL, 0) = 125 | poll([{fd=11, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout) | writev(2, [{iov_base="munmap_chunk(): invalid pointer", iov_len=31}, {iov_base="\n", iov_len=1}], 2munmap_chunk(): invalid pointer | ) = 32 | mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2dcb8eb000 | rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0 | gettid() = 621 | getpid() = 621 | tgkill(621, 621, SIGABRT) = 0 | --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=621, si_uid=505} --- | +++ killed by SIGABRT (core dumped) +++ Reproducible: Always Steps to Reproduce: $ podman run -it --rm registry.fedoraproject.org/fedora:38 # dnf install cargo m4 git-core # cd /tmp # git clone https://github.com/ensc/r-tftpd.git # cd r-tftpd/ # make # cargo vendor Actual Results: | Caused by: | [1] Unsupported protocol (Received HTTP/0.9 when not allowed) on random crates. Tests are done with systems behind an http proxy. Used repository (r-tftpd) is arbitrary; seen at least with 'grcov' too.
Other errors | Caused by: | failed to verify the checksum of `windows-sys v0.45.0` | Caused by: | failed to verify the checksum of `windows_x86_64_msvc v0.48.0` | Downloaded security-framework v2.8.2 | free(): invalid pointer | Aborted (core dumped)
Which version of cargo / rust is installed on your system? (i.e. please paste the output of this command: "rpm -q rust cargo") We discovered some code generation problems with rust 1.69 when built against LLVM 16, so a follow-up update reverted it to using LLVM 15, but the follow-up update hasn't been pushed to the stable ("fedora") repositories yet: https://bodhi.fedoraproject.org/updates/FEDORA-2023-7fbc906f3c If Rust 1.69/LLVM 16 is indeed the source of your problem, the pending update might fix this problem (or might not).
I updated to # rpm -qa cargo\* rust\* rust-srpm-macros-24-1.fc38.noarch cargo-1.69.0-2.fc38.x86_64 rust-std-static-1.69.0-2.fc38.x86_64 rust-1.69.0-2.fc38.x86_64 from https://koji.fedoraproject.org/koji/buildinfo?buildID=2194853 and problem still persists. valgrind shows lot of "invalid read" errors (which occur too deep in free'd blocks to be the usual optimized strlen()/memcpy()) ==1036== Invalid read of size 2 ==1036== at 0x484B660: memmove (vg_replace_strmem.c:1398) ==1036== by 0x4A4038E: UnknownInlinedFun (vtls.c:517) ==1036== by 0x4A4038E: ossl_new_session_cb (openssl.c:3009) ==1036== by 0x4AB5E94: ssl_update_cache (ssl_lib.c:3768) ==1036== by 0x4AE32C0: UnknownInlinedFun (statem_clnt.c:2618) ==1036== by 0x4AE32C0: ossl_statem_client_process_message (statem_clnt.c:1054) ==1036== by 0x4ADD61E: UnknownInlinedFun (statem.c:647) ==1036== by 0x4ADD61E: state_machine (statem.c:442) ==1036== by 0x4ACD35D: ssl3_read_bytes (rec_layer_s3.c:1716) ==1036== by 0x4AA76A4: ssl3_read_internal (s3_lib.c:4463) ==1036== by 0x4AADF4A: SSL_read (ssl_lib.c:1885) ==1036== by 0x4A4BE19: ossl_recv.lto_priv.0 (openssl.c:4624) ==1036== by 0x4A4A043: ssl_cf_recv.lto_priv.0 (vtls.c:1575) ==1036== by 0x4A06231: http2_recv (http2.c:1785) ==1036== by 0x4A2876D: Curl_read (sendf.c:743) ==1036== Address 0x116f5fb8 is 8 bytes inside a block of size 10 free'd ==1036== at 0x48440E4: free (vg_replace_malloc.c:884) ==1036== by 0x4A40CDB: reuse_conn.lto_priv.0 (url.c:3411) ==1036== by 0x4A2169D: UnknownInlinedFun (url.c:3771) ==1036== by 0x4A2169D: UnknownInlinedFun (url.c:3970) ==1036== by 0x4A2169D: multi_runsingle (multi.c:1934) ==1036== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==1036== by 0x9C02DA: ??? (in /usr/bin/cargo) ==1036== by 0x4B31B9: ??? (in /usr/bin/cargo) ==1036== by 0x4AE2B1: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== by 0x29216C: ??? (in /usr/bin/cargo) ==1036== Block was alloc'd at ==1036== at 0x484186F: malloc (vg_replace_malloc.c:393) ==1036== by 0x50EEFEE: strdup (strdup.c:42) ==1036== by 0x4A1F295: UnknownInlinedFun (url.c:1892) ==1036== by 0x4A1F295: UnknownInlinedFun (url.c:3496) ==1036== by 0x4A1F295: UnknownInlinedFun (url.c:3970) ==1036== by 0x4A1F295: multi_runsingle (multi.c:1934) ==1036== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==1036== by 0x9C02DA: ??? (in /usr/bin/cargo) ==1036== by 0x4B31B9: ??? (in /usr/bin/cargo) ==1036== by 0x4AE2B1: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== by 0x29216C: ??? (in /usr/bin/cargo) and then sometimes ==1036== Invalid free() / delete / delete[] / realloc() ==1036== at 0x48440E4: free (vg_replace_malloc.c:884) ==1036== by 0x4A363AC: Curl_free_request_state (url.c:2103) ==1036== by 0x4A571AE: Curl_close.isra.0 (url.c:411) ==1036== by 0x49F12AB: curl_easy_cleanup (easy.c:798) ==1036== by 0x8092C4: ??? (in /usr/bin/cargo) ==1036== by 0x4425C7: ??? (in /usr/bin/cargo) ==1036== by 0x4AE37F: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== by 0x29216C: ??? (in /usr/bin/cargo) ==1036== Address 0x6a92118 is 24 bytes inside a block of size 872 free'd ==1036== at 0x48440E4: free (vg_replace_malloc.c:884) ==1036== by 0x4A049C3: UnknownInlinedFun (http_proxy.c:236) ==1036== by 0x4A049C3: http_proxy_cf_detach_data.lto_priv.0 (http_proxy.c:1147) ==1036== by 0x4A14E6B: UnknownInlinedFun (cfilters.c:477) ==1036== by 0x4A14E6B: Curl_detach_connection (multi.c:957) ==1036== by 0x4A1505F: multi_done.lto_priv.0 (multi.c:661) ==1036== by 0x4A15787: UnknownInlinedFun (multi.c:819) ==1036== by 0x4A15787: curl_multi_remove_handle (multi.c:768) ==1036== by 0x9C0379: ??? (in /usr/bin/cargo) ==1036== by 0x80929C: ??? (in /usr/bin/cargo) ==1036== by 0x4425C7: ??? (in /usr/bin/cargo) ==1036== by 0x4AE37F: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== Block was alloc'd at ==1036== at 0x4846464: calloc (vg_replace_malloc.c:1340) ==1036== by 0x4A09145: UnknownInlinedFun (http_proxy.c:142) ==1036== by 0x4A09145: http_proxy_cf_connect.lto_priv.0 (http_proxy.c:1078) ==1036== by 0x4A4AA0B: UnknownInlinedFun (vtls.c:1512) ==1036== by 0x4A4AA0B: ssl_cf_connect.lto_priv.0 (vtls.c:1493) ==1036== by 0x4A1EF0C: UnknownInlinedFun (cfilters.c:367) ==1036== by 0x4A1EF0C: multi_runsingle (multi.c:2070) ==1036== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==1036== by 0x9C02DA: ??? (in /usr/bin/cargo) ==1036== by 0x4B31B9: ??? (in /usr/bin/cargo) ==1036== by 0x4AE2B1: ??? (in /usr/bin/cargo) ==1036== by 0x3EC1AF: ??? (in /usr/bin/cargo) ==1036== by 0x3EB68C: ??? (in /usr/bin/cargo) ==1036== by 0x23A609: ??? (in /usr/bin/cargo) ==1036== by 0x24FFD2: ??? (in /usr/bin/cargo) ==1036== Might be problem with curl or openssl
libcurl-7.87.0-8.fc38.x86_64 openssl-libs-3.0.8-2.fc38.x86_64 affects also usual download during 'cargo build'; not only 'cargo vendor'
downgrade to curl-7.85.0-8.fc37.x86_64 seems to fix the problem; reassigning to curl
I also tried libcurl-7.86.0-4.fc38.x86_64 and had no valgrind errors, while libcurl-7.87.0-1.fc38.x86_64 does show errors.
There are also no valgrind errors with rawhide's initial upgrade to libcurl-8.0.0-1.fc39, nor the current libcurl-8.0.1-3.fc39. (all other packages remaining on f38)
very likely http proxy related; I can not reproduce the 'cargo' download problems on a system without proxy. I have tested libcurl-7.86.0-4.fc38.x86_64 OK libcurl-7.87.0-1.fc38.x86_64 broken libcurl-7.87.0-8.fc38.x86_64 broken libcurl-7.88.0-2.fc39.x86_64 broken libcurl-7.88.1-1.fc39.x86_64 broken libcurl-8.0.0-1.fc39.x86_64 OK Although 'valgrind' is silent with 7.88.0 + .1, 'cargo' fails there (just for reference) with | Caused by: | [2] Failed initialization ([CONN-3-0] send: no filter connected)
I am not able to reproduce the bug in my testing environment. Using a local squid and/or valgrind did not help. Based on the stack trace in comment #3, I believe it could have been fixed with this upstream commit: https://github.com/curl/curl/commit/f8da4f2f2d0451dc0a126ae3e5077b4527ccdc86 I will shortly prepare a build with the above fix backported for testing.
Fedora commit: https://src.fedoraproject.org/rpms/curl/c/472ff9ff394deee558414aea9cd405309e2642b9?branch=f38
FEDORA-2023-46c1b96671 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-46c1b96671
libcurl-7.87.0-9.fc38.x86_64 does **not** fix the download problem; valgrind is still noisy with it ==5363== Invalid write of size 8 ==5363== at 0x4A187BA: UnknownInlinedFun (mime.c:1308) ==5363== by 0x4A187BA: Curl_mime_cleanpart (mime.c:1178) ==5363== by 0x49FD47D: Curl_http_done (http.c:1615) ==5363== by 0x4A15006: multi_done.lto_priv.0 (multi.c:646) ==5363== by 0x4A1F521: multi_runsingle (multi.c:2125) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== Address 0x58dbd98 is 488 bytes inside a block of size 872 free'd ==5363== at 0x48440E4: free (vg_replace_malloc.c:884) ==5363== by 0x4A049C3: UnknownInlinedFun (http_proxy.c:236) ==5363== by 0x4A049C3: http_proxy_cf_detach_data.lto_priv.0 (http_proxy.c:1147) ==5363== by 0x4A14E6B: UnknownInlinedFun (cfilters.c:477) ==5363== by 0x4A14E6B: Curl_detach_connection (multi.c:957) ==5363== by 0x4A1505F: multi_done.lto_priv.0 (multi.c:661) ==5363== by 0x4A1F56A: multi_runsingle (multi.c:2544) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== Block was alloc'd at ==5363== at 0x4846464: calloc (vg_replace_malloc.c:1340) ==5363== by 0x4A09145: UnknownInlinedFun (http_proxy.c:142) ==5363== by 0x4A09145: http_proxy_cf_connect.lto_priv.0 (http_proxy.c:1078) ==5363== by 0x4A4AA6B: UnknownInlinedFun (vtls.c:1536) ==5363== by 0x4A4AA6B: ssl_cf_connect.lto_priv.0 (vtls.c:1517) ==5363== by 0x4A1EF0C: UnknownInlinedFun (cfilters.c:367) ==5363== by 0x4A1EF0C: multi_runsingle (multi.c:2070) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== warning: spurious network error (2 tries remaining): [56] Failure when receiving data from the peer (Proxy CONNECT aborted) ==5363== Invalid free() / delete / delete[] / realloc() ==5363== at 0x48440E4: free (vg_replace_malloc.c:884) ==5363== by 0x4A363AC: Curl_free_request_state (url.c:2103) ==5363== by 0x4A1F0DC: UnknownInlinedFun (url.c:3978) ==5363== by 0x4A1F0DC: multi_runsingle (multi.c:1934) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== by 0x215563: ??? (in /usr/bin/cargo) ==5363== Address 0x58dbbc8 is 24 bytes inside a block of size 872 free'd ==5363== at 0x48440E4: free (vg_replace_malloc.c:884) ==5363== by 0x4A049C3: UnknownInlinedFun (http_proxy.c:236) ==5363== by 0x4A049C3: http_proxy_cf_detach_data.lto_priv.0 (http_proxy.c:1147) ==5363== by 0x4A14E6B: UnknownInlinedFun (cfilters.c:477) ==5363== by 0x4A14E6B: Curl_detach_connection (multi.c:957) ==5363== by 0x4A1505F: multi_done.lto_priv.0 (multi.c:661) ==5363== by 0x4A1F56A: multi_runsingle (multi.c:2544) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== Block was alloc'd at ==5363== at 0x4846464: calloc (vg_replace_malloc.c:1340) ==5363== by 0x4A09145: UnknownInlinedFun (http_proxy.c:142) ==5363== by 0x4A09145: http_proxy_cf_connect.lto_priv.0 (http_proxy.c:1078) ==5363== by 0x4A4AA6B: UnknownInlinedFun (vtls.c:1536) ==5363== by 0x4A4AA6B: ssl_cf_connect.lto_priv.0 (vtls.c:1517) ==5363== by 0x4A1EF0C: UnknownInlinedFun (cfilters.c:367) ==5363== by 0x4A1EF0C: multi_runsingle (multi.c:2070) ==5363== by 0x4A21E1D: curl_multi_perform (multi.c:2690) ==5363== by 0x9C241A: ??? (in /usr/bin/cargo) ==5363== by 0x4B9613: ??? (in /usr/bin/cargo) ==5363== by 0x4B4B91: ??? (in /usr/bin/cargo) ==5363== by 0x2E3D68: ??? (in /usr/bin/cargo) ==5363== by 0x792A7B: ??? (in /usr/bin/cargo) ==5363== by 0x79198C: ??? (in /usr/bin/cargo) ==5363== by 0x791884: ??? (in /usr/bin/cargo) ==5363== Upstream https://github.com/curl/curl/commit/3f3ddee0665176040b3eaf89a912a922726ecb18 (7.88.0) sounds like a good candidate for these errors. But does not explain why cargo download is still broken until 8.0
Thanks for feedback! The stack trace is now different though. I will try to backport the other upstream commit. This would be much easier to debug if I had a local reproducer...
Fedora commit: https://src.fedoraproject.org/rpms/curl/c/23fee3d38607b49d63fbbc7e2f374f489b7be184?branch=f38
git bisect shows that commit af22c2a546ab862ab577c8d9d3609af0de178974 (HEAD) Author: Stefan Eissing <stefan> Date: Tue Nov 22 09:55:41 2022 +0100 vtls: localization of state data in filters is the first commit which broke 'cargo'. It went ok by commit 821f6e2a89de8aec1c7da3c0f381b92b2b801efc Author: Stefan Eissing <stefan> Date: Thu Feb 9 16:07:34 2023 +0100 CURLOPT_PIPEWAIT: allow waited reuse also for subsequent connections
Oops, forgot to commit the patch: https://src.fedoraproject.org/rpms/curl/c/ac9ca2137b4a7ce3b44ce16dedabe950e183e933?branch=f38
Thanks! Could you please check whether commit 821f6e2a89de8aec1c7da3c0f381b92b2b801efc alone fixes it? The changes in /lib seem to apply fine.
(In reply to Kamil Dudka from comment #16) > Oops, forgot to commit the patch: > https://src.fedoraproject.org/rpms/curl/c/ > ac9ca2137b4a7ce3b44ce16dedabe950e183e933?branch=f38 ... and it does not build anyway because the change depends on the following massive rewrite, which introduced bunch of other bugs: https://github.com/curl/curl/commit/71b7e0161032927cdfb4e75ea40f65b8898b3956 At this point I think it would be safer to rebase curl in f38 to curl-8.0.1. Any objections?
I have prepared the rebase here: https://src.fedoraproject.org/rpms/curl/pull-request/17
rebase to 8.0.1 is ok with me. 821f6e2a89de8aec1c7da3c0f381b92b2b801efc on 7.87.0 makes 'cargo' work again, but valgrind is still noisy with this combination. After f8da4f2f2d0451dc0a126ae3e5077b4527ccdc86 (applied by you in libcurl-7.87.0-9), valgrind becomes silent.
Thank you for testing it! I will proceed with the rebase. Although it looks like a new major release, there is no SONAME bump or any known ABI breaking changes.
Fedora commit: https://src.fedoraproject.org/rpms/curl/c/f854769ba434c9fffd92498b590b8d0e29c9a616?branch=f38
FEDORA-2023-46c1b96671 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.