Bug 1275275

It would help if you could provide a reproducer review link.

See bug 1265628.

I can confirm this behaviour, see https://bugzilla.redhat.com/show_bug.cgi?id=1244797  repoquery stuff takes hours.

Maybe?

10.1. Enabling the Caches
To configure yum to retain downloaded files rather than discarding them, set the keepcache option in /etc/yum.conf to 1: 

https://docs.fedoraproject.org/en-US/Fedora_Core/5/html/Software_Management_Guide/sn-yum-caching.html

Is that also implemented rightly into dnf?

IMHO this bug should get more severity as there's a risk to loose actual package review due to lack of time, so people get annoyed and leave.

It's really annoying if a simple review needs more than 3 hours, almost all time spent in repoqueries.

Another "funny" issue I already mentioned in the devel-list:

Between the (repoquery -q -C --requires --resolve xxxx) messages there are several lines of this type (in german):
"11-22 19:05 root         DEBUG    Running: dnf repoquery -q -C --
requires --resolve Die letzte Prüfung auf abgelaufene Metadaten wurde
vor 0:12:50 auf Sun Nov 22 16:03:37 2015 ausgeführt"

It tells me that the last metadata check is 0:12:50 ago and was done at
16:03:37, the message has a timestamp of 19:05.
The timestamp is correct, the time for the metadata-check is most
likely also correct, but the time span is obviously wrong.

Some notes:

f-r is coded with the nowadays wrong assumption that 'repoquery --resolve --requires <pkg>' only accepts a single argument. Since dnf now accepts multiple arguments deps.py:list_deps should be changed to use this fact instead of invoking a separate dnf call for each package to get the deps for.

dnf is very slow compared to yum, and this is the root cause. See bug #1285360.

I have patches available which deduplicates some of the tested package sets. With those, this review completes in about 14 minutes on my i5. Of this, more than 9 minutes is spent in a single command 'dnf repoquery -q -C --requires --resolve bash glibc systemd systemd-libs dhcpcd' which is ridiculous; it completes in seconds using yum.

For now, the work-around is using -x CheckOwnDirs. Using this option f-r completes in about 3-4 minutes for me. This also means that directory ownership needs to be checked manually.

Pushed fix upstream: https://git.fedorahosted.org/cgit/FedoraReview.git/commit/?h=devel&id=8c109cc5139283e01932c1

Please refer to https://fedorahosted.org/FedoraReview/wiki/UseDevelopmentVersion for running the upstream version. We need to settle the DNF issues a little before making a new release.

dnf-1.1.5-1.fc23.noarch have any improvement ?

Hello , testing with develop version (git HEAD)


rm -rf 3912-lives/ ; time /home/sergio/fedora-scm/fedora-review/FedoraReview/try-fedora-review  --other-bz https://bugzilla.rpmfusion.org -b 3912 -m fedora-23-x86_64-rpmfusion_free


real    62m30.408s
user    52m15.072s
sys     1m42.055s

98% of the time is after: INFO: Active plugins: Generic, Shell-api, C/C++, Perl

what can I do to debug or workaround the issue ? 


Thanks

Debugging seems to be a bit tricky and should IMHO probably be done by the dnf maintainers. As stated in comment #8, it should be possible to walk-around the problem using the '-x CheckOwnDirs' option.

(In reply to Alec Leamas from comment #12)
> As stated in comment #8, it should be possible to
> walk-around the problem using the '-x CheckOwnDirs' option.

real    13m51.181s
user    3m52.999s
sys     1m8.201s

Maybe we should default '-x CheckOwnDir' option until dnf fix his bug ( all my test was with dnf-1.1.5-1.fc23.noarch ) 

Thanks, Merry Christmas and a happy New Year

(In reply to Sergio Monteiro Basto from comment #13)
…
> Maybe we should default '-x CheckOwnDir' option until dnf fix his bug ( all
> my test was with dnf-1.1.5-1.fc23.noarch ) 

No, my vote goes against this disable as default. It's nothing about a b0rken feature here but a request to improve performance (non-functional requirement).

Someone could use a customer wrapper script in its local system that includes the '-x CheckOwnDir' option.

(In reply to Raphael Groner from comment #14)
> (In reply to Sergio Monteiro Basto from comment #13)
> …
> > Maybe we should default '-x CheckOwnDir' option until dnf fix his bug ( all
> > my test was with dnf-1.1.5-1.fc23.noarch ) 
> 
> No, my vote goes against this disable as default. It's nothing about a
> b0rken feature here but a request to improve performance (non-functional
> requirement).
> 
> Someone could use a customer wrapper script in its local system that
> includes the '-x CheckOwnDir' option.

Until dnf bug is not fixed , CheckOwnDir will be painful .

(In reply to Sergio Monteiro Basto from comment #15)
 
> Until dnf bug is not fixed , CheckOwnDir will be painful .

Indeed. In the meantime, double negations also causes some pain...

That said, I will not do something like disabling CheckOwnDirs in the f-r defaults. This is a bug a bug in dnf, and we should IMHO not sweep it under the carpet. What we could do is to release a new version with the work-around in the release notes. However, I hesitate to do this until I hear from the dnf folks in bug #1279538. Their complete silence here is a major headache.

(In reply to Alec Leamas from comment #16)

> That said, I will not do something like disabling CheckOwnDirs in the f-r
> defaults. This is a bug a bug in dnf, and we should IMHO not sweep it under
> the carpet. 

I suggest disabling only temporarily "until dnf fix his bug"

(In reply to Sergio Monteiro Basto from comment #17)
> (In reply to Alec Leamas from comment #16)
> 
> > That said, I will not do something like disabling CheckOwnDirs in the f-r
> > defaults. This is a bug a bug in dnf, and we should IMHO not sweep it under
> > the carpet. 
> 
> I suggest disabling only temporarily "until dnf fix his bug"

As said already in comment #14, that disable would fool all reviews in the false assumption fedora-review is right about its telling to folder ownership, that is only possible with inclusion of all dependencies resolved via dnf.

This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase

*** Bug 1322365 has been marked as a duplicate of this bug. ***

Any progress on this?

I tried fedora-review for the first time in a long time yesterday and it took 11 *hours* to run, failing at the end with the following error:

04-13 04:58 root         DEBUG    Running check: CheckNoNameConflict
04-13 04:58 root         DEBUG    Exception down the road...
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/FedoraReview/review_helper.py", line 237, in run
    self._do_run(outfile)
  File "/usr/lib/python2.7/site-packages/FedoraReview/review_helper.py", line 226, in _do_run
    self._do_report(outfile)
  File "/usr/lib/python2.7/site-packages/FedoraReview/review_helper.py", line 99, in _do_report
    self._run_checks(self.bug.spec_file, self.bug.srpm_file, outfile)
  File "/usr/lib/python2.7/site-packages/FedoraReview/review_helper.py", line 118, in _run_checks
    writedown=not Settings.no_report)
  File "/usr/lib/python2.7/site-packages/FedoraReview/checks.py", line 378, in run_checks
    run_check(name)
  File "/usr/lib/python2.7/site-packages/FedoraReview/checks.py", line 352, in run_check
    check.run()
  File "/usr/lib/python2.7/site-packages/FedoraReview/plugins/generic.py", line 1536, in run
    import pycurl
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (nss)
04-13 04:58 root         ERROR    Exception down the road...(logs in /home/adam/.cache/fedora-review.log)
04-13 04:58 root         DEBUG    Report completed:  42460.951 seconds

This is when reviewing Slurm at https://bugzilla.redhat.com/show_bug.cgi?id=1149566

@adam: Please file a separate bug on this issue, it's something completely different. Doing so, please also attach the ~/.cache/fedora-review.log file.

According to conversation with jsilhan from dnf team it doesn't seem they will accelerate the query much (e.g. 500x or so) in the near future, so the caching is probably the right way to go.

Alec - fair enough. See https://bugzilla.redhat.com/show_bug.cgi?id=1326622

The salient point for this bug is that the run took 11 hours before it failed.

@jaroslaw: Caching?! What's done in f-r upstream is de-duplicating most of the queries, which of course is a good thing.  But what could be cached here, really?

Depressing news from the dns maintainers, indeed. To tolerate performance like this makes the statement that dnf is a proper replacement for yum questionable. 

Can yum-utils be retired given this state of dnf? Is the proper solution to back down to using yum-utils until the bug is fixed? "Scratching my head"

(In reply to Alec Leamas from comment #26)
> @jaroslaw: Caching?! What's done in f-r upstream is de-duplicating most of
> the queries, which of course is a good thing.  But what could be cached
> here, really?
> 
I suggested caching just not to do duplicate dnf queries. It would be simple hack which could be realised by some function wrapper. I suggested it because I haven't checked the f-r code and I didn't know how hard would be to fully de-duplicate, which is of course the right way to go.

> Depressing news from the dns maintainers, indeed. To tolerate performance
> like this makes the statement that dnf is a proper replacement for yum
> questionable. 
> 
> Can yum-utils be retired given this state of dnf? Is the proper solution to
> back down to using yum-utils until the bug is fixed? "Scratching my head"
>
+1

DNF folks resolved the performance issue, see bug 1279538 comment 22, bug 1279538 comment 24, i.e.:
8m38.467s vs 0m3.179s

yup. Time for a fedora-review release. Will look into it,

fedora-review-0.6.1-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-c502551b96

fedora-review-0.6.1-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-cdc0c4f35b

fedora-review-0.6.1-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-c502551b96

fedora-review-0.6.1-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-cdc0c4f35b

fedora-review-0.6.1-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

fedora-review-0.6.1-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.