While trying to check whether https://github.com/rpm-software-management/librepo/pull/158 fix works, I see there are no logs for librepo now :-( From copr build: https://copr.fedorainfracloud.org/coprs/praiskup/ping/build/995509/ .... Copr repository 14 kB/s | 2.1 kB 00:00 BaseOS 738 kB/s | 8.6 MB 00:11 updates 939 kB/s | 8.6 MB 00:09 epel 24 kB/s | 5.3 kB 00:00 Failed to download metadata for repo 'epel' Error: Failed to download metadata for repo 'epel' INFO: chroot_scan: 3 files copied to /var/lib/copr-rpmbuild/results/chroot_scan INFO: /var/lib/mock/995509-epel-6-x86_64-1565066903.507321/root/var/log/dnf.log /var/lib/mock/995509-epel-6-x86_64-1565066903.507321/root/var/log/dnf.librepo.log /var/lib/mock/995509-epel-6-x86_64-1565066903.507321/root/var/log/dnf.rpm.log ... Though the dnf.librepo.log is empty. Can you please suggest work-around?
Hi Pavel, you can either run dnf with --verbose option, or set debuglevel configuration option to something higher than 2. This will enable DEBUG level logging in dnf.librepo.log and hawkey.log. It's currently disabled by default due to bug 1580022 and bug 1355764.
I see that you disabled "DEBUG" output, but again - this is *failure*, and still the log file *is created*, but without any useful info. I'm not saying you should revert the change -- just saying that you should probably convert some DEBUG output into ERROR output.
Just few words why I need this; sure I can use --verbose, but it would be cumbersome to enable this for everyone in Fedora Copr (the whole point of the bugs cited in #1 would be lost). The thing is that we face some teporary issues with repositories during the rush repo hours (or minutes): https://pagure.io/fedora-infrastructure/issue/7987 https://pagure.io/fedora-infrastructure/issue/7301 And it is really important to know what failed and why. For the rest of the day, we can have the log file empty.
Pavel, I agree a lot of the log messages have a wrong level of DEBUG, but it may not be straightforward to set the right level for some of them. What are the messages you are getting that should have a higher (ERROR) level?
It has been suggested in a different bug (https://bugzilla.redhat.com/show_bug.cgi?id=1355764) we should be logging the URLs from which packages and repo data are being downloaded ("Effective url" logged in dnf.librepo.log). These can be a bit verbose themselves, but I understand the users might want to see them. (it would mean to switch those to the INFO level so they are logged by default)
I propose output / logging levels should at least have: 1. quiet (essential information: what package transactions succeeded, what failed) 2. normal (includes: if yum/dnf is aware of using a proxy, which proxy was used [no comment if no proxy], URLs of attempted downloads, download statistics, and if it failed, why. I think yum used to indicate which plugins it was using, a line with a list of plugins was handy for knowing what was already being tried. 3. verbose-basic (Anything that you would help a user not expert in rpms, to diagnose basic problems.) Consider some users have challenging network environments. If possible include IP addresses, unless they are handled at a different layer invisible to dnf. I'm not sure what problems are common and easy to fix but certainly include: network failures (maybe wrong network, maybe wrong proxy settings, maybe network is too slow.) Key signing issues? Out of space errors? Probably anything here should show some error message in the normal verbosity, and more here if there's a lot to share. * If you want to eliminate the category verbose-basic, items probably should go into 'normal'. 4. verbose-all (everything you might need to debug the most difficult problems.) If you have a list of error conditions or statuses that you are unsure about categorizing, why not ask here?
Thanks for the write-up, Phil, it is useful and we'll take it into consideration. However, I don't think we have a facility in place to accommodate the first 3 verbosity levels you've mentioned (as we only have one INFO level and this would kind of require three). But I do tend to agree we need more than one regular verbosity level. Your verbosity level 4. is basically DEBUG, which we do have. To implement something like this will require a significant logging revamp, which is planned, but there are things higher on the priority list... Currently we might have to resort to some compromise between logging some information on the default verbosity level while not taking up too much space and the rest will be there with --verbose.
(In reply to Lukáš Hrázký from comment #4) > Pavel, I agree a lot of the log messages have a wrong level of DEBUG, but it > may not be straightforward to set the right level for some of them. What are > the messages you are getting that should have a higher (ERROR) level? I don't receive any messages, that's the problem. We can not afford using --verbose. But in our case - it would be good to use ERROR message for any **failed** download attempt error (server error 404, 500, timeouts, etc...): ERROR: download attempt failed <domain1>/<uri1>, err. 404 ERROR: download attempt failed <domain2>/<uri2>, err. 404 For baseurl, where we re-try the same url several times: ERROR: download attempt failed <domain1>/<uri1>, err 500 ERROR: download attempt failed <domain1>/<uri2>, err 500 ... Even if the re-try mechanism succeeds eventually, it's worth logging the previous attempt failures so we can analyze the log files and see what is broken and why.
FWIW, the problem is that one mirror wasn't synchronized for some time and it was thus broken. Unfortunately, librepo chose the broken mirror for downloading repomd.xml file with outdated references to metadata, and all the attempts to download the metadata failed with 404. Somewhat useful info was in 'dnf.log' itself (404 errors), but I wasn't able to deduce where the rempomd.xml file was downloaded from so where the problem is. So please consider changing those messages as INFO: 2019-08-16T08:14:18Z DEBUG select_next_target: Selecting mirror for: repodata/repomd.xml 2019-08-16T08:14:18Z DEBUG prepare_next_transfer: URL: http://mirror.cogentco.com/pub/linux/epel/7/aarch64/repodata/repomd.xml More info here: https://pagure.io/fedora-infrastructure/issue/8096
PR: https://github.com/rpm-software-management/librepo/pull/164 I've bumped the log level of error messages (I hope I got it right, I couldn't really test most of them easily) to either WARNING for local errors (mostly filesystem related) and to INFO for HTTP errors. I've also set a single regular INFO level message like this: INFO Downloading: <URL> So it logs a single message per downloaded file with the full URL. That should provide enough information for the user while not making the log grow too much. Any issues during the download etc. should now also be logged by default.
FEDORA-2019-049b4836a6 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-049b4836a6
dnf-4.2.11-2.fc30, dnf-plugins-core-4.0.10-1.fc30, libdnf-0.35.5-2.fc30, librepo-1.10.6-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-049b4836a6
Can you provide F31 update?
We will do an update after F31 final release, but if that's too late for you we can do a backport now?
Ok, I can build myself. But it is really weird to see F30 updates but no F31 updates. Why are you waiting actually?
We are just being cautious, we don't want to do a big rebase now and potentially break something so close to the freeze.
(a) You don't have to push the update to stable, and (b) you already provided F30 update (so breakage there would be much more serious issue than in F31). No pressure, just non-standard workflow (historically this wouldn't be according guidelines, packages in Fedora N had to have higher NVR than Fedora N-1).
dnf-4.2.11-2.fc30, dnf-plugins-core-4.0.10-1.fc30, libdnf-0.35.5-2.fc30, librepo-1.10.6-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.