Bug 1156007

Hi, we could do that but we have to solve more urgent bugs now.

Hi Jan. This is actually fairly high priority for Fedora overall, as this is essential to our overall promotion and marketing strategy. Thanks!

Hello Matt. I think Jan could have phrased his comment better. Rest assured that we fully recognize the importance of this feature for Fedora and we do plan to deliver it as soon as possible. However, now we need to focus on things that block dnf from being default in Fedora. The justification for this decision is that the benefits of this feature would be significantly reduced without dnf being on as many systems as possible.

Thanks Jan. I'm going to e-mail the yum devel list about coming up with a solution for F21, and then presumably that will need to be ported forward into F22.

Although we'll also want something for software-center-based package management in F21 -- I'm not actually even sure of the details of how that works. Hmmm; where is the best place to coordinate all of this?

> Hmmm; where is the best place to coordinate all of this?

Let's try the thread you started on yum-devel mailing list. I believe all the stakeholders are subscribed there.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

The bug is here for more then 2 years, therefore probably requested feature is out of scope. Please if the request is still relevant and has high impact, don't hesitate to reopen the bug report.

This would still be really helpful and I regard it as high priority even though it has not been addressed in two years.

What I'd like is ID, VERSION_ID, and VARIANT_ID from /etc/os-release.

I am not sure this is exactly an HTTP header as much as adding additional information to the User Agent String.

24.194.145.233 - - [09/May/2017:04:45:35 +0000] "GET /metalink?repo=updates-released-f24&arch=x86_64 HTTP/1.1" 200 8747 "-" "dnf/1.1.10"

to 

24.194.145.233 - - [09/May/2017:04:45:35 +0000] "GET /metalink?repo=updates-released-f24&arch=x86_64 HTTP/1.1" 200 8747 "-" "dnf/1.1.10 fedora 25 server"

This looks like updating dnf-2.4.1/const.py or updating 
./dnf/repo.py:        self.useragent = dnf.const.USER_AGENT
./dnf/util.py:    handle.useragent = dnf.const.USER_AGENT

to have that additional data in it. I expect it is non-trivial but not impossible?

> It would be nice if dnf would either send a custom http header which we could log to mirror manager, or set the user agent, or something, based on the presence of fedora-release-workstation, fedora-release-server, fedora-release-cloud — or the absence of any of these.

Maybe better solution is to add requested information to the metalink.
Eg. add '&rel=fedora-server'

How to do this?
You want information based on presence of 'fedora-release-[server,workstation,...]-<version>.noarch.rpm' package. (Or information from '/etc/os-release' file. But content of this file is provided by 'fedora-release-[server,workstation,...]-<version>.noarch.rpm'.)

There is package 'fedora-repos-<version>.noarch.rpm'. But we can use the same logic as above. That means create 'fedora-repos-[server,workstation,...]-<version>.noarch.rpm' packages. These packages will be contain metalinks with &rel=requested_information. 

This solution is DNF (YUM, ...) independent.

(In reply to Jaroslav Rohel from comment #11)
> Maybe better solution is to add requested information to the metalink.
> Eg. add '&rel=fedora-server'


That could work too. Smooge, what do you think?

@jrohel, it is not only about metalinks, but also for direct baseurl access I suppose...

Matt, what are you trying to achieve?

(In reply to Igor Gnatenko from comment #13)
> Matt, what are you trying to achieve?

To get a general sense of the popularity of each edition (and possibly spin) in a low-impact way. I'll defer to Smooge on best implementation from the server side.

One of the big issues that comes up every release is "What editions are being used?" and how much. Knowing this has multiple benefits to Fedora (and various downstreams)

* This is to help allow groups like QA and Release Engineering know which spins, editions, containers, etc. are getting used the most so that they can work out "OK XYZ container isn't building so we need to slip, ABC container didn't build so we don't need to slip"

* This is to help edition groups to get an idea if their edition makes any sense. While no one wants to know if their release is a "flop" they need to know if they want to improve on the next iteration. 

* This needs to be configurable in some way that survives proxy systems and mirrormanager mangling as we get the data from apache logs.

My current mirrormanager scripts do parse the /metalink?repo=updates-released-f21&arch=x86_64 type lines so a release= or container= for however that is done should be possible to add to this. I wasn't sure how it would affect mirrormanager (or CentOS's mirroring system or the RHEL one) so had gone with putting the data in User Agent as that was less likely to tools I could not fix.

Please let me know how I can give better information for this.

We discussed this internally in the DNF team and we prefer adding the argument to the metalink.

Both sides
- metalink in repo file
- metalink parser on server
are under control of fedora infra/releng.

Other distros don't have to use this at all.

We recommend using 'variant' variable with values such as 'server', 'workstation', etc.
Keep it close to fedora composes.

Ok, but we would still need DNF support for that. Currently, the metalink line an repo file looks like:

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch

Where $releasever and $basearch are filled in by DNF. We would need to add a $releasevariant or something to that.

Dear Matthew, I would like to explain the solution little bit in more detail. We propose following:

The package fedora-repos will be splitted into 3 packages (fedora-server-repos, fedora-workstations-repos, and fedora-cloud-repos). Each of them will have /etc/yum.repos.d/fedora.repo with modification in metalink, where you can use something like for fedora-workstations-repos:

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch&releasevariant=workstation 

or for fedora-server-repos: 

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch&releasevariant=server 

That means that there will be no additional variable or requirement from dnf site, but only infrastructure solution. And this is exactly what you need, because your request is Fedora specific (cannot be applied for RHEL), therefore it has to be solved by Fedora delivery approach of distro variant. Hope that it helps.

(In reply to Jaroslav Mracek from comment #18)
> The package fedora-repos will be splitted into 3 packages
> (fedora-server-repos, fedora-workstations-repos, and fedora-cloud-repos).
> Each of them will have /etc/yum.repos.d/fedora.repo with modification in
> metalink, where you can use something like for fedora-workstations-repos:

I'm not automatically opposed to this, but an upgrade path which replaces the repo definition files seems scary.

Also this is compounded by https://apps.fedoraproject.org/packages/fedora-workstation-repositories, which adds select third-party repositories.

Why not just add support for custom substitution variables? It's been requested by other folks from time to time.

See bug 1251774 and the libdnf GitHub issue[1].

[1]: https://github.com/rpm-software-management/libdnf/issues/170

I'd like to leave this bug open for tracking. It might be something we want to solve elsewhere than DNF, but clearly it isn't addressed by *anything* now.

The custom variables are supported by dnf like in yum (in /etc/dnf/vars), and the functionality is enhanced by https://github.com/rpm-software-management/dnf/pull/938 .

What is the status of this being accepted into rawhide?  Would be awesome to see this part of the yum4 efforts to understand usage and interest.

I created a proposal about solution. It requires changes in fedora-release and fedora-repos packages. 
https://src.fedoraproject.org/fork/jmracek/rpms/fedora-release/branch/variant
https://pagure.io/fork/jmracek/fedora-repos/branch/variant

Please what do you think?

I am just mentioning, that the change in baseurl will create incompatibility issue with packagekit.

Additionally I create an alternative solution (https://github.com/rpm-software-management/dnf/pull/1039), but it requires https://src.fedoraproject.org/fork/jmracek/rpms/fedora-release/branch/variant .

What solution would be better (Comment 25 ,Comment 27, or something else)?

I like the second one. It looks like it would not affect PackageKit but they could add it to their software? It would be also fairly simple to have other tools use without causing URL changes (if I understand the first correctly).

I think Matthew Miller needs to weigh in to clear the needinfo bit.

Please if anyone can try package fedora-release with commit https://src.fedoraproject.org/fork/jmracek/rpms/fedora-release/c/a7f840d9e864bb23ea36b321420fa3179395cc9d?branch=variant I will be very happy.

I like the first option. However the fedora-release changes seem a tad incomplete and the openh264 repo should not have any changes for this. There will be nothing for $releasever unless someone happens to be using an edition, potentially we can work around this for most caes, livecds's containers, images, etc.  We will need dnf, mirrormanager etc to work when an empty variable is passed on.

The reason I am not a fan of the second option is that the header will be sent to every repo used. We will be giving the edition used data to every mirror as well as all third party repos that our users enable. I can see people not being entirely happy about it.

Unfortunately the first solution breaks packagekit and probably some others that cannot use yum variables. From my point of view I don't like both solution. Both due that we will provide in both cases information about releasevariant to anyone who can provide .repo file or can read headers of curl calls. Additional the first solution will break things.

OK is it possible to look at my original idea? We stick the data in the User Agent string. This means PackageKit can do something similar at a different time and the parsing of metalinks doesn't get touched. This would allow for the EL clients to put in something for their product lines so that workstation/server/etc clients can be seen in EPEL (which is one thing I have been asked if I could give data about but can't.)

This could also be configurable to make it silent for people who don't want to report it. Or it could be configured to be used by a client for their own needs.

Many sites (including Red Hat) talk to other parts of the company via proxies which lose data about which client is talking about what. Now while satellite allows for this sort of reporting, a lot of infrastructure is based off of downloading a bunch of packages, putting them into a repo, and have a webserver sit in front. Then they are asked to see if XYZ division actually checked in and got an update... which they can't. If the client can set up something like:


/etc/yum.repos.d/foobar.repo
...
Agent_Addition="Company X, Division Y, Release Blah"

/etc/yum.repos.d/epel.repo
Agent_Addition="distributor=%distributor, release=%release, codename=%stuff"  

which it gets from lsb_release data

Then what shows up in the logs is 
"dnf/x.x.x (distributor=centos release=7.4.1708 codename=core)"
"dnf/x.x.x (distributor=fedora release=28 codename=workstation)"
"dnf/x.x.x (distributor=rhel release=8.1.2201 codename=server)"

smooge that will present that information to every mirror, to every third party repo.

Yeah, I guess enable_extra_agent=true/false per repo would not help here.

Related PR: https://github.com/rpm-software-management/libdnf/pull/684

*** Bug 1542060 has been marked as a duplicate of this bug. ***

New PR: https://github.com/rpm-software-management/libdnf/pull/768

We're extending the User-Agent field from "libdnf" to "libdnf/0.35.2 (Fedora 31; server; Linux.x86_64)".