Bug 1698034

cloning for RHEL-8 so we can address it in 8.1 eventually

+++ This bug was initially created as a clone of Bug #1534123 +++

⋊> ~/P/f/r/rust-openssl on master ⨯ for p in (sudo dnf -q repoquery --whatrequires rust-bitflags0.9-devel); echo $p; and sudo dnf -q repoquery --requires $p | grep bitflags; end                          12:38:47
rust-atk-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-clap-devel-0:2.29.1-1.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-devicemapper-devel-0:0.15.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gdk-devel-0:0.7.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gdk-pixbuf-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gdk-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gio-devel-0:0.3.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gio-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-git2-devel-0:0.6.11-1.fc28.noarch
(crate(bitflags) >= 0.9.0 with crate(bitflags) < 0.10.0)
rust-glib-devel-0:0.4.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-glib-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gobject-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gtk-devel-0:0.3.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gtk-source-sys-devel-0:0.5.0-1.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-gtk-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-nix-devel-0:0.9.0-4.fc28.noarch
(crate(bitflags) >= 0.9.0 with crate(bitflags) < 0.10.0)
rust-openssl-devel-0:0.9.23-2.fc28.noarch
(crate(bitflags) >= 0.9.0 with crate(bitflags) < 0.10.0)
rust-pango-devel-0:0.3.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-pango-sys-devel-0:0.5.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-pulldown-cmark-devel-0:0.1.0-2.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
rust-syntex_syntax-devel-0:0.59.1-3.fc28.noarch
(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0)
⋊> ~/P/f/r/rust-openssl on master ⨯ dnf --version                                                                                                                                                          12:39:42
2.7.5
  Installed: dnf-0:2.7.5-5.fc28.noarch at Fri 05 Jan 2018 04:36:53 PM GMT
  Built    : Fedora Project at Wed 03 Jan 2018 10:56:10 AM GMT

  Installed: rpm-0:4.14.0-5.fc28.x86_64 at Thu 16 Nov 2017 10:44:14 AM GMT
  Built    : Fedora Project at Tue 07 Nov 2017 11:56:33 AM GMT

---

This is blocking me to rewrite some part of infrastructure which relies on yum.

--- Additional comment from Igor Gnatenko on 2018-01-13 11:43:26 UTC ---

For those who didn't understand issue -- dnf returns list of 21 packages which it thinks depend on rust-bitflags0.9-devel while truth is that only 3 packages depend on it.

--- Additional comment from Fedora End Of Life on 2018-02-20 15:32:03 UTC ---

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

--- Additional comment from Jaroslav Mracek on 2018-10-01 16:10:06 UTC ---

Please if the problem is based on that a query always return for condition in richdeps true this is not a bug, but by design. Thanks a lot for understanding.

--- Additional comment from Igor Gnatenko on 2018-10-01 21:01:44 UTC ---

How come it is by design?

Returning package which has nothing to do with requirement in command line is a bug.

I don't know how exactly it is implemented in repoquery, but livsolv returns proper result.

Also end builddep works fine, so this is definitely a bug in repoquery.

--- Additional comment from Jaroslav Mracek on 2018-10-02 07:36:22 UTC ---

Probably you can provide a solution, because you are an author of rich deps implementation in libdnf (Query). Please can you help us?

--- Additional comment from Igor Gnatenko on 2018-10-02 09:43:31 UTC ---

(In reply to Jaroslav Mracek from comment #5)
> Probably you can provide a solution, because you are an author of rich deps
> implementation in libdnf (Query). Please can you help us?

I never designed libdnf's Query and I never liked it. For reference, this libsolv code works fine:

#!/usr/bin/python3
import solv

pool = solv.Pool()
pool.setarch('x86_64')

repo = pool.add_repo('available')

bitflags_compat = repo.add_solvable()
bitflags_compat.name = 'rust-bitflags0.9-devel'
bitflags_compat.evr = '0.9.8-1'
bitflags_compat.arch = 'noarch'
bitflags_compat.add_deparray(solv.SOLVABLE_PROVIDES, pool.Dep('crate(bitflags)').Rel(solv.REL_EQ, pool.Dep('0.9.8')))
bitflags = repo.add_solvable()
bitflags.name = 'rust-bitflags-devel'
bitflags.evr = '1.1.1-1'
bitflags.arch = 'noarch'
bitflags.add_deparray(solv.SOLVABLE_PROVIDES, pool.Dep('crate(bitflags)').Rel(solv.REL_EQ, pool.Dep('1.1.1')))
clap = repo.add_solvable()
clap.name = 'rust-clap-devel'
clap.evr = '1-1'
clap.arch = 'noarch'
clap.add_deparray(solv.SOLVABLE_REQUIRES, pool.Dep('crate(bitflags)').Rel(solv.REL_GT | solv.REL_EQ, pool.Dep('1.0.0')).Rel(solv.REL_WITH, pool.Dep('crate(bitflags)').Rel(solv.REL_LT, pool.Dep('2.0.0'))))
openssl = repo.add_solvable()
openssl.name = 'rust-openssl-devel'
openssl.evr = '1-1'
openssl.arch = 'noarch'
openssl.add_deparray(solv.SOLVABLE_REQUIRES, pool.Dep('crate(bitflags)').Rel(solv.REL_GT | solv.REL_EQ, pool.Dep('0.9.0')).Rel(solv.REL_WITH, pool.Dep('crate(bitflags)').Rel(solv.REL_LT, pool.Dep('1.0.0'))))

pool.createwhatprovides()

for pkg in (clap, openssl):
    for dep in pkg.lookup_deparray(solv.SOLVABLE_REQUIRES):
        print(f'{dep}: {pool.whatprovides(dep)}')


(crate(bitflags) >= 1.0.0 with crate(bitflags) < 2.0.0): [<Solvable #3 rust-bitflags-devel-1.1.1-1.noarch>]
(crate(bitflags) >= 0.9.0 with crate(bitflags) < 1.0.0): [<Solvable #2 rust-bitflags0.9-devel-0.9.8-1.noarch>]


DNF must be doing something wrong. I'm not sure what exactly.

--- Additional comment from Igor Gnatenko on 2018-10-02 12:07:09 UTC ---

> /usr/lib/python3.7/site-packages/dnf/cli/commands/repoquery.py(317)_get_recursive_deps_query()
    316         t = t.union(query_in.filter(requires=set_requires))

Bug happens here.

ipdb> [str(x) for x in set_requires if not isinstance(x, str)]
['crate(bitflags) = 0.9.1', 'crate(bitflags/default) = 0.9.1', 'crate(bitflags/example_generated) = 0.9.1', 'crate(bitflags/unstable_testing) = 0.9.1', 'rust-bitflags0.9-devel = 0.9.1-4.fc29']

But query_in.filter(requires=…) is broken here.

For instance,
>>> query_in.filter(requires=hawkey.Reldep(self.base.sack, 'crate(bitflags) = 2.0.0')).run()

Roughly looking at libdnf's code, seems that depSplitter works fine, but probably filtering doesn't.

I'm not profecient in C++ and I don't have time to waste, so please just fix this bug.

---

You can still reproduce this bug on rawhide. Just run `dnf repoquery --whatrequires rust-bitflags0.9-devel` which would return things like rust-clap-devel, but that one depends on rust-bitflags-devel (not the rust-bitflags0.9-devel).

--- Additional comment from Igor Gnatenko on 2018-10-02 12:11:32 UTC ---

void
Query::Impl::filterRcoReldep(const Filter & f, Map *m)
{
    assert(f.getMatchType() == _HY_RELDEP);

    Pool *pool = dnf_sack_get_pool(sack);
    Id rco_key = reldep_keyname2id(f.getKeyname());
    Queue rco;
    auto resultPset = result.get();

    queue_init(&rco);
    Id resultId = -1;
    while ((resultId = resultPset->next(resultId)) != -1) {
        Solvable *s = pool_id2solvable(pool, resultId );
        for (auto match : f.getMatches()) {
            Id reldepFilterId = match.reldep;

            queue_empty(&rco);
            solvable_lookup_idarray(s, rco_key, &rco);
            for (int j = 0; j < rco.count; ++j) {
                Id reldepIdFromSolvable = rco.elements[j];

                if (pool_match_dep(pool, reldepFilterId, reldepIdFromSolvable )) {
                    MAPSET(m, resultId );
                    goto nextId;
                }
            }
        }
        nextId:;
    }
    queue_free(&rco);
}


I think bug would be in this function, because `pool_match_dep()` returns true in pessimistic case (aka some part of reldep is matching). You should not us this function for this case.

--- Additional comment from Igor Gnatenko on 2018-10-02 12:14:49 UTC ---

Michael, correct me if I'm wrong.

--- Additional comment from Michael Schröder on 2018-10-04 10:04:29 UTC ---

Well, you're somewhat wrong ;)

Actually libsolv has a 'solvable_matchesdep()' function that pretty much looks like the above code.

But let's talk a bit of what to expect from a 'whatrequires' query. I want to start with the argument being a capability and not a package name, i.e. what rpm and zypper do.
Basically the question is "If this capability is provided, which packages require it?". (As provides can't use rich deps you also can't use a rich dep as whatprovides query argument.)

Now say you have got kernel packages that provide different flavors, and you have a provide for every flavor. Some kernel module has: "Requires: (kernel with flavor = default)".
You then do a whatrequires match for "kernel". Clearly you want to see the kernel module in the output.

So this is what libsolv implements. It returns the package even if only the first part of the "with" statement is true. (And it can't check the second part, because there was no way to specify the flavor capability.)

This does not work well if the "with" op is used to specify an interval, like in this bug report. I could somewhat special case this, but I'm not really convinced that it makes sense because things will get pretty inconsistent.

But lets step back a bit: I'm somewhat confused about the repoquery in this bug: it looks to me like you specified a package and then dnf does a dependency match for all the provides of the package? In that case there would be a way out: libsolv would need to provide a dependency match where not just one capability, but a set of capabilities is provided as argument. In that case we could implement the with/without rich ops in a correct way.

--- Additional comment from Igor Gnatenko on 2018-10-04 10:20:21 UTC ---

> But lets step back a bit: I'm somewhat confused about the repoquery in this bug: it looks to me like you specified a package and then dnf does a dependency match for all the provides of the package? In that case there would be a way out: libsolv would need to provide a dependency match where not just one capability, but a set of capabilities is provided as argument. In that case we could implement the with/without rich ops in a correct way.

Yes. It goes through all provides of a package and tries to find what requires them.

For things like and/or, I would say that behavior is correct, but with/without case I think should be special.

--- Additional comment from Igor Gnatenko on 2018-10-31 12:46:56 UTC ---

So Michael implemented pool_whatmatchessolvable(). And now it is available in rawhide.

Can you fix implementation now?

--- Additional comment from Neal Gompa on 2019-02-02 23:59:45 UTC ---

This is a seriously big problem as we keep using rich dependencies. Even dnfdragora uses rich dependencies now, and we have the entire Rust ecosystem relying on it. If we can't repoquery them reasonably well, this is going to cause major headaches. And we still need this working for spam-o-matic and other aspects of releng check scripts.

Daniel, can you please look at this and prioritize it?

--- Additional comment from Daniel Mach on 2019-03-20 11:34:55 UTC ---

I believe the bug is still valid on F30, moving release accordingly.
I also replaced 'depends on' with 'blocks' dnf-community tracker.