Doing depsolving (more carefully, analyzing requires/obsoletes/etc.) by hands is not very good idea. For example, if there's rich dependency, you will always miss broken dependency because it happens only in some circumstances. There are many different corner-cases. If you use libsolv, you can easily perform "job"s and analyze solver results. And after all, it's much faster to use libsolv directly. Also, hawkey is basically dead and python-hawkey bindings exist only because DNF needs them (real and supported API is only DNF's one). I would not expect to have proper libdnf implementation at least in near future, so libsolv seems more optimal solution.
rpmdeplint already uses librepo and hawkey exclusively for dependency solving, it doesn't do any of that "by hand". Are you suggesting that we should change to use libsolv calls directly instead of going through hawkey? Or something else? Do you have some specific part of the code in mind which is not doing things the best way, which you could point at so I can better understand what is being suggested?
(In reply to Dan Callaghan from comment #1) > rpmdeplint already uses librepo and hawkey exclusively for dependency > solving, it doesn't do any of that "by hand". > > Are you suggesting that we should change to use libsolv calls directly > instead of going through hawkey? Or something else? Yes, exactly. > > Do you have some specific part of the code in mind which is not doing things > the best way, which you could point at so I can better understand what is > being suggested? Actually, codebase of rpmdeplint is not huge, so for example implementation of repoclosure throught hawkey.Query() is not good. Also find_upgrade_problems() is not done in good way. Regarding repoclosure, you can check proper implementation here: https://github.com/openSUSE/libsolv/blob/master/tools/installcheck.c#L288-L351
Looked at: https://github.com/openSUSE/libsolv/blob/master/doc/libsolv-bindings.txt which provides some examples. I've also seen python*-solv packages in Fedora. I guess a change to libsolv would be something for a 2.x release. From looking at the code, the change might not be as heavyweight. Not sure whenever we will find time to implement that tho.
Started playing around with this in some spare time, it looks fairly easy and will let us solve some showstoppers with the repodata caching code in bug 1343247.
One problem is that we have some unit tests in rpmdeplint which are currently using hawkey's TestSack functionality, which uses libsolv's testcase_add_testtags under the hood to load fake repos in a nice minimal plaintext format like this: =Ver: 2.0 # =Pkg: lemmon 1 3 noarch =Prv: lemmon-juice ... The testcase_add_testtags function in libsolv is not exposed in the Python bindings.
(In reply to Dan Callaghan from comment #5) > One problem is that we have some unit tests in rpmdeplint which are > currently using hawkey's TestSack functionality, which uses libsolv's > testcase_add_testtags under the hood to load fake repos in a nice minimal > plaintext format like this: > > =Ver: 2.0 > # > =Pkg: lemmon 1 3 noarch > =Prv: lemmon-juice > ... > > The testcase_add_testtags function in libsolv is not exposed in the Python > bindings. https://github.com/openSUSE/libsolv/issues/new
(In reply to Igor Gnatenko from comment #6) I decided to just switch those test cases over to using real packages generated by rpmfluff, rather than loading fake repodata. We're already using rpmfluff in the acceptance tests, and I prefer that because it doesn't rely on libsolv internal test stuff.
(In reply to Igor Gnatenko from comment #2) > Regarding repoclosure, you can check proper implementation here: > https://github.com/openSUSE/libsolv/blob/master/tools/installcheck.c#L288- > L351 So I have studied that installcheck.c and as far as I can see, it is *not* actually an implementation of repoclosure. Or at least not a useful one. First it is solving "weak" install jobs for every package in the pool, and then only keeping ones which were installed. I assume this is to filter out older packages so they are not tested. For example if the repos already have a-0.1-1 and a-0.1-2 then we only need to check a-0.1-2, we can forget about a-0.1-1. However, it still leaves those old/obsoleted packages in the pool, and then just proceeds to solve an install job for every candidate package. *But* this does not catch the number 1 most important problem in repoclosure: where a new version of a package no longer provides something that an older version used to provide, and it breaks some requirement in another package. That's test_catches_soname_change here: https://pagure.io/rpmdeplint/blob/rpmdeplint-1.3/f/acceptance_tests/test_check_repoclosure.py#_13 I reimplemented the algorithm from installcheck.c and it fails to find that problem, because libsolv will happily satisfy the missing dependency by installing the *old* version of the package which we are updating. So it is true that libsolv is finding a valid solution (just install the older version) and dnf would do that too, but it defeats the purpose of this check. I think what I really want is to solve an install job for every package but with older/obsoleted packages completed excluded from the solution.
This is what I have so far: https://gerrit.beaker-project.org/5688 tests: use rpmfluff instead of hawkey.test.TestSackMixin https://gerrit.beaker-project.org/5689 tests: lemon only has one M https://gerrit.beaker-project.org/5690 remove unused methods on DependencyAnalyzer and DependencySet https://gerrit.beaker-project.org/5691 tests: extra cases covering old/obsoleted packages for repoclosure https://gerrit.beaker-project.org/5692 use libsolv directly instead of hawkey The last commit is *almost* complete but it is not passing the repoclosure tests, due to the issues I described in comment 8. There would also be one more commit needed to tidy up the hawkey arch handling stuff which should no longer be needed.
(In reply to Dan Callaghan from comment #8) > (In reply to Igor Gnatenko from comment #2) > > Regarding repoclosure, you can check proper implementation here: > > https://github.com/openSUSE/libsolv/blob/master/tools/installcheck.c#L288- > > L351 > > So I have studied that installcheck.c and as far as I can see, it is *not* > actually an implementation of repoclosure. Or at least not a useful one. > > First it is solving "weak" install jobs for every package in the pool, and > then only keeping ones which were installed. I assume this is to filter out > older packages so they are not tested. For example if the repos already have > a-0.1-1 and a-0.1-2 then we only need to check a-0.1-2, we can forget about > a-0.1-1. that's not true, it keeps trying to solve until nothing passes, then it goes one by one and does solve + reporting problems. [...] > I reimplemented the algorithm from installcheck.c and it fails to find that > problem, because libsolv will happily satisfy the missing dependency by > installing the *old* version of the package which we are updating. SOLVER_FORCEBEST In the meantime I realized that we have tool which is very similar by use-cases -- https://github.com/fedora-modularity/depchase I'm in process of rewriting it, so switching from DNF API to libsolv makes it to work 20m -> 3s Probably we should cooperate to not create 2 almost same tools..
(In reply to Igor Gnatenko from comment #10) > In the meantime I realized that we have tool which is very similar by > use-cases -- https://github.com/fedora-modularity/depchase > > I'm in process of rewriting it, so switching from DNF API to libsolv makes > it to work 20m -> 3s > > Probably we should cooperate to not create 2 almost same tools.. Hmm yeah I hadn't seen that depchase before. It seems to be covering basically the same ground as rpmdeplint list-deps? I am just not sure what its --selfhost and --hint options are for...
(In reply to Igor Gnatenko from comment #10) > SOLVER_FORCEBEST So I added SOLVER_FORCEBEST to the job on line 184 in https://gerrit.beaker-project.org/c/5692/1/rpmdeplint/__init__.py and it did not make any difference. The solver still just picked the old package version to satisfy the requirement, instead of complaining that the new package has changed soname... so I think that means I am still not understanding your suggestion.
I also compiled installcheck.c and tried it on the same test case -- it reports no problems. So I think this means what I wrote in comment 8 is correct -- the algorithm in installcheck.c is not adequate to actually catch this case, because it will just let the older package satisfy the requirement. And of course it is true that DNF will do that too, but that's not the point -- it's still a problem that we want to catch.
Okay, I posted v2 of the patch here: https://gerrit.beaker-project.org/c/5692/2 This version actually passes all our test cases. :-) The _select_obsoleted_by method here: https://gerrit.beaker-project.org/c/5692/2/rpmdeplint/__init__.py#152 is building a solv.Selection which filters out old/obsoleted packages, and then in the find_repoclosure_problems we explicitly exclude those by adding SOLVER_ERASE jobs when solving. That seems to be what installcheck.c is trying to do (when it builds its "cand" queue) but the difference is installcheck.c never actually excludes those packages from the solution.
Ahhh I just realised there is a huge spanner in the works here -- we are supporting rpmdeplint on RHEL7 but libsolv on RHEL7 does not have any Python bindings. So this change would effectively mean rpmdeplint no longer runs on RHEL, only Fedora...
Switching to libsolv does let us make a number of nice improvements/fixes in rpmdeplint though. So I think we will have to call this 2.0 and continue maintaining a 1.x branch for RHEL7, based on the existing code using hawkey.
(In reply to Dan Callaghan from comment #15) > Ahhh I just realised there is a huge spanner in the works here -- we are > supporting rpmdeplint on RHEL7 but libsolv on RHEL7 does not have any Python > bindings. So this change would effectively mean rpmdeplint no longer runs on > RHEL, only Fedora... and requesting it is not an option, right? ;)
(In reply to Igor Gnatenko from comment #17) > and requesting it is not an option, right? ;) Filed a RHEL7 RFE to add libsolv Python bindings: bug 1467169.
The libsolv patch is merged now, for rpmdeplint 2.0. However there are some (unrelated) issues with the caching stuff in bug 1343247 which I will need to fix before we can release it.
FEDORA-2023-a965252f36 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a965252f36
FEDORA-2023-a965252f36 has been pushed to the Fedora 39 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-a965252f36` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-a965252f36 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-a965252f36 has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.