Bug 1830512
| Summary: | zchunk does not work when going through transparent proxy "WebProxy/1.0" | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | 247 <mad_fer_it> | ||||||||||||||||||||||
| Component: | zchunk | Assignee: | Jonathan Dieter <jonathan> | ||||||||||||||||||||||
| Status: | CLOSED EOL | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||||||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||||||||||||||
| Priority: | unspecified | ||||||||||||||||||||||||
| Version: | 32 | CC: | alciregi, jonathan | ||||||||||||||||||||||
| Target Milestone: | --- | Keywords: | Reopened | ||||||||||||||||||||||
| Target Release: | --- | ||||||||||||||||||||||||
| Hardware: | x86_64 | ||||||||||||||||||||||||
| OS: | Linux | ||||||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||||||||
| Last Closed: | 2021-05-25 17:10:06 UTC | Type: | Bug | ||||||||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||||||
| Embargoed: | |||||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||||
Thanks for this. The log file I need for further details is in /var/log/dnf.librepo.log. If you could upload that, it would really help to track down the problem. Created attachment 1684286 [details]
the correct log for the bug
ok i just attacched the correct dnf log for the bug, hope it helps... I'm afraid that's still the wrong log. I need the log file that contains librepo in the name, /var/log/dnf.librepo.log. Thanks! Created attachment 1684438 [details]
dnf librepo log
Created attachment 1684439 [details]
dnf librepo log
oh sorry, attached the right one now... Ok, I'm still unable to reproduce. Can you please upload your DNF cache for updates-modular? I'm hoping that if I have that I can reproduce. Please run "tar -czvf updates-modular-cache.tar.gz /var/cache/dnf/updates-modular*" and attach the results. Also, just to check, could you please run "rpm -q zchunk libcurl librepo" and post the result? Created attachment 1684520 [details]
fedora modular
considering the updates modular folder was empty, i decides to upload also the modular folder, maybe it can help also, the result of rpm -q zchunk libcurl librepo is zchunk package was not installed libcurl-7.69.1-3.fc32.x86_64 libcurl-7.69.1-3.fc32.i686 librepo-1.11.3-1.fc32.x86_64 ......not installed?maybe this is the culprit?i will try to install it so installed zchunk with dnf install zchunk (after having set zchunk false again) and the result of rpm -q zchunk libcurl librepo is zchunk-1.1.5-2.fc32.x86_64 libcurl-7.69.1-3.fc32.x86_64 libcurl-7.69.1-3.fc32.i686 librepo-1.11.3-1.fc32.x86_64 the reult with dnf update anyway is still the same, still errors... Ok, I'm completely baffled. Unfortunately the "fedora-modular" tarball doesn't help since it's the "update-modular" repository that's causing problems. It looks to me like librepo is successfully downloading and writing the zchunk header, but, when writing the body, is hitting problems. Unfortunately, we don't seem to be getting any zchunk errrors in "dnf.librepo.log" to explain why. I'm unable to reproduce locally, which makes it very difficult to debug. If you could run "rm -rf /var/cache/dnf/*" to completely clear the DNF cache, that might be worth a shot. If that doesn't fix the problem, then I think I'll need to provide you with a custom-build librepo with extra logging turned on, to see if we can get more details. If I built it, would you be willing to use it for testing? done, and that solved the error, it's quite strange because i had already deleted the folder manually with no results at all, i guess i will need to wait 2 or 3 days before i can close this bug ticket anyway, what do you think? Well, glad that worked. Go ahead and leave this open for the next few days of updates. If you don't see the problem re-appear, then feel free to close it. sorry but it seems that the problem is back, done an update today and still had the same error...guess you need to provide me that custom librepo... oh, i am actually surprised because gnome software showed me the same error but found some update, i guess we should investigate this anyway? edit : terminal gives me the error without finding anything...we really need to investigate this... Ok, please update using my scratch build of librepo: sudo dnf update https://kojipkgs.fedoraproject.org//work/tasks/580/44130580/librepo-1.11.3-2.fc32.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/580/44130580/python3-librepo-1.11.3-2.fc32.x86_64.rpm Then, attempt to update: sudo dnf --verbose --disablerepo=* --enablerepo=updates-modular list updates Upload the *full* DNF output. We'll take it from there. To revert to the original librepo, run: sudo dnf -y distro-sync "*librepo*" if you were talking about the log inside the terminal i've attached it in txt, if you were talking about lirepo.dnf log it's here too... Created attachment 1685452 [details]
dnf.librepo custom log
Created attachment 1685453 [details]
dnf.librepo dnf custom log txt
According to the logs, the command succeeded this time when you ran it. I'll need both logs (but especially the terminal log) from a failure to be able to debug further. ok so, 2 days ago, i've tried to update with gnome software and everything went ok, yesterday i tried to update and both software and terminal gave me the usual error, i am trying again now but it seems that both terminal and software are working, i'm honestly more baffled than you...will try again in the next days... so, done 4 updates in 4 days via gnome software and everything worked as expected, i honestly don't know what to say at this point, maybe installing zchunk was really the solution?or cleaning the cache via terminal?i don't know i will now revert back to the default librepo to see what happen anyway... reverted to default librepo tried to update 3 times with gnome software and 1 attempt failed with the zchunk error, the other 2 were succesful, tried 1 time with terminal and it was working at this point it's not completely solved but i do guess it's working at least, i do guess we can close this one? Yeah, I don't know what to say. It sounds like an odd intermittent problem, but without the detailed logs of the failure, I don't think we can fix it. I'm going to close this as WORKSFORME, but if you see it happen again and can get me the full logs with all the libcurl and zchunk debugging turned on, feel free to re-open. hi, considering this problem is persisting even after a fresh install i decided to reopen the bug report so i can send you the correct log for the bug Created attachment 1690175 [details]
terminal output for failed update
Created attachment 1690176 [details]
dnf librepo with failed update
Created attachment 1690183 [details]
dnf librepo with failed update
Ok, out of curiosity, are you using a 4G connection while you do your updates? It looks like you're being proxied by something called "WebProxy/1.0 Pre-Alpha:74451F86D20366078660B4197F5845A2". A bit of Googling shows that it's most common on certain 4G networks and isn't very reliable. What happens if you drop down to H+ or 3G? What i am actually using is a tp-link 4g router with a 4g sim card inside My computer anyway is connected through wifi to this 4g router (it's 4g ok but i still connect through wifi althought i guess it does not matter) Anyway i don't know as 3g and h+ signals are very bad in my area, so i never use them, want me to try? I read somewhere that some mobile providers only use that proxy for 4G, so I wondered if 3G / H+ might fix the problem for you. Just to be clear, due to the rarity of this bug, the fact that it doesn't seem to actually be in zchunk, but rather in a rare third party proxy, my inability to reproduce the bug (I've tried with Vodafone Ireland - everything works as expected), and the availability of an easy workaround (turning off zchunk in dnf.conf), I'm afraid I won't be putting much more effort into fixing this. well ok, at least it's working in some ways, i'll revert back to default librepo Jonathan, out of curiosity, does librepo is used by packagekit as well? I believe it is. Please take a look to BZ#1779104, it seems related. Again, out of curiosity, why some repodata files are downloaded using https and other ones are downloaded using plain http? It's random that the failed ones are downloaded using http? Hello again. Just trying to understand something. If I download a zchunk file using the 4G connection /usr/bin/zckdl -v http://fedora.mirror.garr.it/fedora/linux/releases/32/Everything/x86_64/os/repodata/26477913ba3649764be912df1d1f3ef91a876bf78390b76a13e52dbbe6fb152a-filelists.xml.zck It works. So, why the aforementioned proxy should create issues with zck downloads only when using dnf or packagekit? Presumably because you're not using an older version of filelists.xml.zck as the source, so zchunk is downloading the file as one big chunk: zckdl -s oldfileslists.xml.zck -v http://<...>/filelists.xml.zck It seems the proxy is running into issues when trying to download only some chunks. For me to debug the problem, I need access to a system behind that proxy. Without that access, there's really not much I can do. Hello Jonathan, and thank you. So, just to speak, if the dnf or the packagekit caches are clean, are empty, it should work in theory. Because zchunk is downloading the file as one big chunk, isn't it? Or libdnf and so on work in a different way compared to using the zckdl command? I'd have to look back at the code I wrote, but I don't think so. When librepo is downloading the file, I believe it will always first download the zchunk header, and then the rest of the file because it doesn't look for files with matching chunks until it's already downloaded the header. IIRC, zckdl has an optimization where it will just do a normal download if there are no files submitted as the source. In addition, does libdnf or librepo (sorry, as you can see I don't really know how it works) or packagekit use something like zckdl, or they actually use curl/libcurl? So the behaviour is different? (In reply to Alessio from comment #44) > In addition, does libdnf or librepo (sorry, as you can see I don't really > know how it works) or packagekit use something like zckdl, or they actually > use curl/libcurl? So the behaviour is different? Because the error that makes me think is this: Fatal error: cannot update repo 'fedora': Yum repo downloading error: Downloading error(s): repodata/b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47-comps-Everything.x86_64.xml.zck - Download failed: Curl error (23): Failed writing received data to disk/application for http://fedora.mirror.garr.it/fedora/linux/releases/32/Everything/x86_64/os/repodata/b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47-comps-Everything.x86_64.xml.zck [Failed writing body (0 != 8328)]; Last error: Curl error (23): Failed writing received data to disk/application for http://fedora.mirror.garr.it/fedora/linux/releases/32/Everything/x86_64/os/repodata/b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47-comps-Everything.x86_64.xml.zck [Failed writing body (0 != 8328)] packagekit uses libdnf which uses librepo to get the repository metadata, which uses libzck to figure out what to download, which uses libcurl to actually do the download. In the same way, zckdl uses libzck to figure out what to download, which uses libcurl to actually do the download. (In reply to Jonathan Dieter from comment #46) > packagekit uses libdnf which uses librepo to get the repository metadata, > which uses libzck to figure out what to download, which uses libcurl to > actually do the download. In the same way, zckdl uses libzck to figure out > what to download, which uses libcurl to actually do the download. Mmm. It seems that packagekit behaves differently. Is it possible? If I add this patch to the librepo package (note: I'm not a programmer, I am a copy and paste expert :-) --- librepo-1.12.0/librepo/handle.c.orig 2020-05-15 09:43:49.000000000 +0200 +++ librepo-1.12.0/librepo/handle.c 2020-07-06 08:20:33.117100180 +0200 @@ -52,8 +52,10 @@ lr_get_curl_handle() { CURL *h; + FILE* logfile; lr_global_init(); + logfile = fopen("dump.txt", "wb"); h = curl_easy_init(); curl_easy_setopt(h, CURLOPT_FOLLOWLOCATION, 1); @@ -65,6 +67,9 @@ curl_easy_setopt(h, CURLOPT_SSL_VERIFYPEER, 1); curl_easy_setopt(h, CURLOPT_FTP_USE_EPSV, LRO_FTPUSEEPSV_DEFAULT); curl_easy_setopt(h, CURLOPT_FILETIME, 0); + curl_easy_setopt(h, CURLOPT_NOPROGRESS, 1L); + curl_easy_setopt(h, CURLOPT_VERBOSE, 1L); + curl_easy_setopt(h, CURLOPT_STDERR, logfile); return h; } And I build the librepo rpm. When I run dnf, then the dump.txt file is created and there are the curl logs. If I run "pkcon refresh", the dump.txt file is not created/feeded, like if pkcon invokes librepo in some other way or it doesn't use it at all? :thinking: It doesn't use it at all, I mean this part of code. If that's the case, I'd suggest continuing the investigation in #1779104. This particular bug is for the fact that zchunk doesn't work behind a specific transparent proxy used by certain 4G providers. I've updated the title accordingly. Whoops, nevermind. I had to restart packagekit service. Now the dump.txt file is generated as well with pkcon. Sorry. (In reply to Jonathan Dieter from comment #49) > If that's the case, I'd suggest continuing the investigation in #1779104. > This particular bug is for the fact that zchunk doesn't work behind a > specific transparent proxy used by certain 4G providers. I've updated the > title accordingly. Thanks. This message is a reminder that Fedora 32 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '32'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 32 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. |
Created attachment 1684077 [details] the log of dnf --verbose update Description of problem: well simply enough zchunk does not work at all, nor in terminal nor in gnome software centre Version-Release number of selected component (if applicable): How reproducible: don't know, i just try to update and it does not work Steps to Reproduce: 1. 2. 3. Actual results: zchunk does not work Expected results: to just update Additional info: i can solve this problem by adding the string zchunk=False to the dnf conf file, but what's the point of zchunk then? anyway no i am not under a proxy and updates were working well under fedora 31 i tried running the update with the verbose mode and i've attacched the result on here i was told to attach the file librepo.log on here but unfortunately i do not know where to find the file, so i just created a txt with the output...