Bug 1830512 - zchunk does not work when going through transparent proxy "WebProxy/1.0"
Summary: zchunk does not work when going through transparent proxy "WebProxy/1.0"
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: zchunk
Version: 32
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Jonathan Dieter
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-02 11:25 UTC by 247
Modified: 2021-05-25 17:10 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-25 17:10:06 UTC
Type: Bug


Attachments (Terms of Use)
the log of dnf --verbose update (5.21 KB, text/plain)
2020-05-02 11:25 UTC, 247
no flags Details
the correct log for the bug (744.79 KB, text/plain)
2020-05-02 22:12 UTC, 247
no flags Details
dnf librepo log (4.27 MB, text/plain)
2020-05-03 14:36 UTC, 247
no flags Details
dnf librepo log (4.27 MB, text/plain)
2020-05-03 14:38 UTC, 247
no flags Details
fedora modular (4.29 MB, application/x-xz)
2020-05-03 17:51 UTC, 247
no flags Details
dnf.librepo custom log (4.45 MB, text/plain)
2020-05-05 21:57 UTC, 247
no flags Details
dnf.librepo dnf custom log txt (409.33 KB, text/plain)
2020-05-05 21:59 UTC, 247
no flags Details
terminal output for failed update (426.61 KB, text/plain)
2020-05-20 11:37 UTC, 247
no flags Details
dnf librepo with failed update (181.26 KB, text/plain)
2020-05-20 11:39 UTC, 247
no flags Details
dnf librepo with failed update (181.26 KB, text/plain)
2020-05-20 11:44 UTC, 247
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github zchunk zchunk issues 25 0 None closed zchunk not working in fedora 32 2021-02-18 21:11:38 UTC
Red Hat Bugzilla 1779104 0 low CLOSED PackageKit: loading of MD_TYPE_PRIMARY has failed. 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1851841 0 medium CLOSED zchunk issue with packagekit 2021-02-22 00:41:40 UTC

Internal Links: 1779104

Description 247 2020-05-02 11:25:21 UTC
Created attachment 1684077 [details]
the log of dnf --verbose update

Description of problem:
well simply enough zchunk does not work at all, nor in terminal nor in gnome software centre

Version-Release number of selected component (if applicable):


How reproducible:
don't know, i just try to update and it does not work

Steps to Reproduce:
1.
2.
3.

Actual results:
zchunk does not work

Expected results:
to just update

Additional info:
i can solve this problem by adding the string zchunk=False to the dnf conf file, but what's the point of zchunk then?

anyway no i am not under a proxy and updates were working well under fedora 31

i tried running the update with the verbose mode and i've attacched the result on here

i was told to attach the file librepo.log on here but unfortunately i do not know where to find the file, so i just created a txt with the output...

Comment 1 Jonathan Dieter 2020-05-02 19:29:15 UTC
Thanks for this.  The log file I need for further details is in /var/log/dnf.librepo.log.  If you could upload that, it would really help to track down the problem.

Comment 2 247 2020-05-02 22:12:28 UTC
Created attachment 1684286 [details]
the correct log for the bug

Comment 3 247 2020-05-02 22:13:05 UTC
ok i just attacched the correct dnf log for the bug, hope it helps...

Comment 4 Jonathan Dieter 2020-05-03 13:19:32 UTC
I'm afraid that's still the wrong log.  I need the log file that contains librepo in the name, /var/log/dnf.librepo.log.  Thanks!

Comment 5 247 2020-05-03 14:36:29 UTC
Created attachment 1684438 [details]
dnf librepo log

Comment 6 247 2020-05-03 14:38:08 UTC
Created attachment 1684439 [details]
dnf librepo log

Comment 7 247 2020-05-03 14:38:44 UTC
oh sorry, attached the right one now...

Comment 8 Jonathan Dieter 2020-05-03 15:30:28 UTC
Ok, I'm still unable to reproduce.  Can you please upload your DNF cache for updates-modular?  I'm hoping that if I have that I can reproduce.  Please run "tar -czvf updates-modular-cache.tar.gz /var/cache/dnf/updates-modular*" and attach the results.

Also, just to check, could you please run "rpm -q zchunk libcurl librepo" and post the result?

Comment 9 247 2020-05-03 17:51:39 UTC
Created attachment 1684520 [details]
fedora modular

Comment 10 247 2020-05-03 17:54:53 UTC
considering the updates modular folder was empty, i decides to upload also the modular folder, maybe it can help

also, the result of rpm -q zchunk libcurl librepo is

zchunk package was not installed
libcurl-7.69.1-3.fc32.x86_64
libcurl-7.69.1-3.fc32.i686
librepo-1.11.3-1.fc32.x86_64


......not installed?maybe this is the culprit?i will try to install it

Comment 11 247 2020-05-03 18:31:21 UTC
so installed zchunk with dnf install zchunk (after having set zchunk false again) and the result of rpm -q zchunk libcurl librepo is

zchunk-1.1.5-2.fc32.x86_64
libcurl-7.69.1-3.fc32.x86_64
libcurl-7.69.1-3.fc32.i686
librepo-1.11.3-1.fc32.x86_64

the reult with dnf update anyway is still the same, still errors...

Comment 12 Jonathan Dieter 2020-05-03 19:12:47 UTC
Ok, I'm completely baffled.  Unfortunately the "fedora-modular" tarball doesn't help since it's the "update-modular" repository that's causing problems.  It looks to me like librepo is successfully downloading and writing the zchunk header, but, when writing the body, is hitting problems.  Unfortunately, we don't seem to be getting any zchunk errrors in "dnf.librepo.log" to explain why.

I'm unable to reproduce locally, which makes it very difficult to debug.

If you could run "rm -rf /var/cache/dnf/*" to completely clear the DNF cache, that might be worth a shot.

If that doesn't fix the problem, then I think I'll need to provide you with a custom-build librepo with extra logging turned on, to see if we can get more details.  If I built it, would you be willing to use it for testing?

Comment 13 247 2020-05-03 20:34:27 UTC
done, and that solved the error, it's quite strange because i had already deleted the folder manually with no results at all, i guess i will need to wait 2 or 3 days before i can close this bug ticket anyway, what do you think?

Comment 14 Jonathan Dieter 2020-05-04 20:23:18 UTC
Well, glad that worked.  Go ahead and leave this open for the next few days of updates.  If you don't see the problem re-appear, then feel free to close it.

Comment 15 247 2020-05-05 11:12:14 UTC
sorry but it seems that the problem is back, done an update today and still had the same error...guess you need to provide me that custom librepo...

Comment 16 247 2020-05-05 11:16:29 UTC
oh, i am actually surprised because gnome software showed me the same error but found some update, i guess we should investigate this anyway?

edit : terminal gives me the error without finding anything...we really need to investigate this...

Comment 17 Jonathan Dieter 2020-05-05 19:20:43 UTC
Ok, please update using my scratch build of librepo:

sudo dnf update https://kojipkgs.fedoraproject.org//work/tasks/580/44130580/librepo-1.11.3-2.fc32.x86_64.rpm https://kojipkgs.fedoraproject.org//work/tasks/580/44130580/python3-librepo-1.11.3-2.fc32.x86_64.rpm

Then, attempt to update:

sudo dnf --verbose --disablerepo=* --enablerepo=updates-modular list updates

Upload the *full* DNF output.  We'll take it from there.

Comment 18 Jonathan Dieter 2020-05-05 19:22:56 UTC
To revert to the original librepo, run:

sudo dnf -y distro-sync "*librepo*"

Comment 19 247 2020-05-05 21:56:39 UTC
if you were talking about the log inside the terminal i've attached it in txt, if you were talking about lirepo.dnf log it's here too...

Comment 20 247 2020-05-05 21:57:41 UTC
Created attachment 1685452 [details]
dnf.librepo custom log

Comment 21 247 2020-05-05 21:59:47 UTC
Created attachment 1685453 [details]
dnf.librepo dnf custom log txt

Comment 22 Jonathan Dieter 2020-05-06 19:06:44 UTC
According to the logs, the command succeeded this time when you ran it.  I'll need both logs (but especially the terminal log) from a failure to be able to debug further.

Comment 23 247 2020-05-06 21:10:14 UTC
ok so, 2 days ago, i've tried to update with gnome software and everything went ok, yesterday i tried to update and both software and terminal gave me the usual error, i am trying again now but it seems that both terminal and software are working, i'm honestly more baffled than you...will try again in the next days...

Comment 24 247 2020-05-10 11:10:17 UTC
so, done 4 updates in 4 days via gnome software and everything worked as expected, i honestly don't know what to say at this point, maybe installing zchunk was really the solution?or cleaning the cache via terminal?i don't know

i will now revert back to the default librepo to see what happen anyway...

Comment 25 247 2020-05-13 11:47:37 UTC
reverted to default librepo

tried to update 3 times with gnome software and 1 attempt failed with the zchunk error, the other 2 were succesful, tried 1 time with terminal and it was working

at this point it's not completely solved but i do guess it's working at least, i do guess we can close this one?

Comment 26 Jonathan Dieter 2020-05-13 19:34:47 UTC
Yeah, I don't know what to say.  It sounds like an odd intermittent problem, but without the detailed logs of the failure, I don't think we can fix it.  I'm going to close this as WORKSFORME, but if you see it happen again and can get me the full logs with all the libcurl and zchunk debugging turned on, feel free to re-open.

Comment 27 247 2020-05-20 11:36:34 UTC
hi, considering this problem is persisting even after a fresh install i decided to reopen the bug report so i can send you the correct log for the bug

Comment 28 247 2020-05-20 11:37:11 UTC
Created attachment 1690175 [details]
terminal output for failed update

Comment 29 247 2020-05-20 11:39:21 UTC
Created attachment 1690176 [details]
dnf librepo with failed update

Comment 30 247 2020-05-20 11:44:38 UTC
Created attachment 1690183 [details]
dnf librepo with failed update

Comment 31 Jonathan Dieter 2020-05-20 20:01:06 UTC
Ok, out of curiosity, are you using a 4G connection while you do your updates?  It looks like you're being proxied by something called "WebProxy/1.0 Pre-Alpha:74451F86D20366078660B4197F5845A2".  A bit of Googling shows that it's most common on certain 4G networks and isn't very reliable.  What happens if you drop down to H+ or 3G?

Comment 32 247 2020-05-21 09:38:05 UTC
What i am actually using is a tp-link 4g router with a 4g sim card inside

My computer anyway is connected through wifi to this 4g router (it's 4g ok but i still connect through wifi althought i guess it does not matter)

Comment 33 247 2020-05-21 09:41:50 UTC
Anyway i don't know as 3g and h+ signals are very bad in my area, so i never use them, want me to try?

Comment 34 Jonathan Dieter 2020-05-23 17:28:49 UTC
I read somewhere that some mobile providers only use that proxy for 4G, so I wondered if 3G / H+ might fix the problem for you.  Just to be clear, due to the rarity of this bug, the fact that it doesn't seem to actually be in zchunk, but rather in a rare third party proxy, my inability to reproduce the bug (I've tried with Vodafone Ireland - everything works as expected), and the availability of an easy workaround (turning off zchunk in dnf.conf), I'm afraid I won't be putting much more effort into fixing this.

Comment 35 247 2020-05-24 09:20:05 UTC
well ok, at least it's working in some ways, i'll revert back to default librepo

Comment 36 Alessio 2020-06-27 18:34:57 UTC
Jonathan, out of curiosity, does librepo is used by packagekit as well?

Comment 37 Jonathan Dieter 2020-06-27 18:36:36 UTC
I believe it is.

Comment 38 Alessio 2020-06-27 19:04:33 UTC
Please take a look to BZ#1779104, it seems related.

Comment 39 Alessio 2020-06-28 14:32:01 UTC
Again, out of curiosity, why some repodata files are downloaded using https and other ones are downloaded using plain http?
It's random that the failed ones are downloaded using http?

Comment 40 Alessio 2020-06-30 09:11:46 UTC
Hello again.
Just trying to understand something.

If I download a zchunk file using the 4G connection

/usr/bin/zckdl -v http://fedora.mirror.garr.it/fedora/linux/releases/32/Everything/x86_64/os/repodata/26477913ba3649764be912df1d1f3ef91a876bf78390b76a13e52dbbe6fb152a-filelists.xml.zck

It works.
So, why the aforementioned proxy should create issues with zck downloads only when using dnf or packagekit?

Comment 41 Jonathan Dieter 2020-06-30 19:29:48 UTC
Presumably because you're not using an older version of filelists.xml.zck as the source, so zchunk is downloading the file as one big chunk:

zckdl -s oldfileslists.xml.zck -v http://<...>/filelists.xml.zck

It seems the proxy is running into issues when trying to download only some chunks.

For me to debug the problem, I need access to a system behind that proxy.  Without that access, there's really not much I can do.

Comment 42 Alessio 2020-07-01 06:49:01 UTC
Hello Jonathan, and thank you.
So, just to speak, if the dnf or the packagekit caches are clean, are empty, it should work in theory. Because zchunk is downloading the file as one big chunk, isn't it? Or libdnf and so on work in a different way compared to using the zckdl command?

Comment 43 Jonathan Dieter 2020-07-01 06:54:48 UTC
I'd have to look back at the code I wrote, but I don't think so.  When librepo is downloading the file, I believe it will always first download the zchunk header, and then the rest of the file because it doesn't look for files with matching chunks until it's already downloaded the header.  IIRC, zckdl has an optimization where it will just do a normal download if there are no files submitted as the source.

Comment 44 Alessio 2020-07-01 07:32:01 UTC
In addition, does libdnf or librepo (sorry, as you can see I don't really know how it works) or packagekit use something like zckdl, or they actually use curl/libcurl? So the behaviour is different?

Comment 45 Alessio 2020-07-01 07:44:42 UTC
(In reply to Alessio from comment #44)
> In addition, does libdnf or librepo (sorry, as you can see I don't really
> know how it works) or packagekit use something like zckdl, or they actually
> use curl/libcurl? So the behaviour is different?

Because the error that makes me think is this:

Fatal error: cannot update repo 'fedora': Yum repo downloading error: Downloading error(s): repodata/b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47-comps-Everything.x86_64.xml.zck - Download failed: Curl error (23): Failed writing received data to disk/application for http://fedora.mirror.garr.it/fedora/linux/releases/32/Everything/x86_64/os/repodata/b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47-comps-Everything.x86_64.xml.zck [Failed writing body (0 != 8328)]; Last error: Curl error (23): Failed writing received data to disk/application for http://fedora.mirror.garr.it/fedora/linux/releases/32/Everything/x86_64/os/repodata/b7e1566c87075299c4a199737b341e7d15fbf1a4d49c4c2cbe43368437b02c47-comps-Everything.x86_64.xml.zck [Failed writing body (0 != 8328)]

Comment 46 Jonathan Dieter 2020-07-04 11:04:17 UTC
packagekit uses libdnf which uses librepo to get the repository metadata, which uses libzck to figure out what to download, which uses libcurl to actually do the download.  In the same way, zckdl uses libzck to figure out what to download, which uses libcurl to actually do the download.

Comment 47 Alessio 2020-07-06 06:36:56 UTC
(In reply to Jonathan Dieter from comment #46)
> packagekit uses libdnf which uses librepo to get the repository metadata,
> which uses libzck to figure out what to download, which uses libcurl to
> actually do the download.  In the same way, zckdl uses libzck to figure out
> what to download, which uses libcurl to actually do the download.

Mmm.
It seems that packagekit behaves differently. Is it possible?

If I add this patch to the librepo package (note: I'm not a programmer, I am a copy and paste expert :-)


--- librepo-1.12.0/librepo/handle.c.orig        2020-05-15 09:43:49.000000000 +0200
+++ librepo-1.12.0/librepo/handle.c     2020-07-06 08:20:33.117100180 +0200
@@ -52,8 +52,10 @@
 lr_get_curl_handle()
 {
     CURL *h;
+    FILE* logfile;

     lr_global_init();
+    logfile = fopen("dump.txt", "wb");

     h = curl_easy_init();
     curl_easy_setopt(h, CURLOPT_FOLLOWLOCATION, 1);
@@ -65,6 +67,9 @@
     curl_easy_setopt(h, CURLOPT_SSL_VERIFYPEER, 1);
     curl_easy_setopt(h, CURLOPT_FTP_USE_EPSV, LRO_FTPUSEEPSV_DEFAULT);
     curl_easy_setopt(h, CURLOPT_FILETIME, 0);
+    curl_easy_setopt(h, CURLOPT_NOPROGRESS, 1L);
+    curl_easy_setopt(h, CURLOPT_VERBOSE, 1L);
+    curl_easy_setopt(h, CURLOPT_STDERR, logfile);

     return h;
 }


And I build the librepo rpm.
When I run dnf, then the dump.txt file is created and there are the curl logs.

If I run "pkcon refresh", the dump.txt file is not created/feeded, like if pkcon invokes librepo in some other way or it doesn't use it at all? :thinking:

Comment 48 Alessio 2020-07-06 06:40:55 UTC
It doesn't use it at all, I mean this part of code.

Comment 49 Jonathan Dieter 2020-07-06 06:53:34 UTC
If that's the case, I'd suggest continuing the investigation in #1779104.  This particular bug is for the fact that zchunk doesn't work behind a specific transparent proxy used by certain 4G providers.  I've updated the title accordingly.

Comment 50 Alessio 2020-07-06 07:24:37 UTC
Whoops, nevermind. I had to restart packagekit service. Now the dump.txt file is generated as well with pkcon.
Sorry.

Comment 51 Alessio 2020-07-06 07:25:00 UTC
(In reply to Jonathan Dieter from comment #49)
> If that's the case, I'd suggest continuing the investigation in #1779104. 
> This particular bug is for the fact that zchunk doesn't work behind a
> specific transparent proxy used by certain 4G providers.  I've updated the
> title accordingly.

Thanks.

Comment 52 Fedora Program Management 2021-04-29 16:48:10 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 53 Ben Cotton 2021-05-25 17:10:06 UTC
Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.