Bug 1156007 - [rfe] info about fedora in http header
[rfe] info about fedora in http header
Status: POST
Product: Fedora
Classification: Fedora
Component: dnf (Show other bugs)
rawhide
Unspecified Unspecified
low Severity unspecified
: ---
: ---
Assigned To: Jaroslav Mracek
Fedora Extras Quality Assurance
: FutureFeature, Reopened, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2014-10-23 07:47 EDT by Matthew Miller
Modified: 2018-04-08 00:56 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-19 08:23:00 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Matthew Miller 2014-10-23 07:47:25 EDT
It appears that our three products will be all drawing from the same "Everything" tree on the mirrors, at least for F21 and probably for F22.

With this plan, there is no way for us to easily count how many of which product are in use. It would be nice if dnf would either send a custom http header which we could log to mirror manager, or set the user agent, or something, based on the presence of fedora-release-workstation, fedora-release-server, fedora-release-cloud — or the absence of any of these.
Comment 1 Honza Silhan 2014-10-31 09:18:53 EDT
Hi, we could do that but we have to solve more urgent bugs now.
Comment 2 Matthew Miller 2014-10-31 13:45:00 EDT
Hi Jan. This is actually fairly high priority for Fedora overall, as this is essential to our overall promotion and marketing strategy. Thanks!
Comment 3 Jan Zeleny 2014-11-03 02:42:00 EST
Hello Matt. I think Jan could have phrased his comment better. Rest assured that we fully recognize the importance of this feature for Fedora and we do plan to deliver it as soon as possible. However, now we need to focus on things that block dnf from being default in Fedora. The justification for this decision is that the benefits of this feature would be significantly reduced without dnf being on as many systems as possible.
Comment 4 Matthew Miller 2014-11-03 08:52:02 EST
Thanks Jan. I'm going to e-mail the yum devel list about coming up with a solution for F21, and then presumably that will need to be ported forward into F22.

Although we'll also want something for software-center-based package management in F21 -- I'm not actually even sure of the details of how that works. Hmmm; where is the best place to coordinate all of this?
Comment 5 Jan Zeleny 2014-11-03 09:03:17 EST
> Hmmm; where is the best place to coordinate all of this?

Let's try the thread you started on yum-devel mailing list. I believe all the stakeholders are subscribed there.
Comment 6 Fedora Admin XMLRPC Client 2016-07-08 05:24:50 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 7 Jaroslav Mracek 2017-05-18 05:09:16 EDT
The bug is here for more then 2 years, therefore probably requested feature is out of scope. Please if the request is still relevant and has high impact, don't hesitate to reopen the bug report.
Comment 8 Matthew Miller 2017-05-18 08:53:40 EDT
This would still be really helpful and I regard it as high priority even though it has not been addressed in two years.
Comment 9 Matthew Miller 2017-05-18 09:07:23 EDT
What I'd like is ID, VERSION_ID, and VARIANT_ID from /etc/os-release.
Comment 10 Stephen John Smoogen 2017-05-18 15:41:16 EDT
I am not sure this is exactly an HTTP header as much as adding additional information to the User Agent String.

24.194.145.233 - - [09/May/2017:04:45:35 +0000] "GET /metalink?repo=updates-released-f24&arch=x86_64 HTTP/1.1" 200 8747 "-" "dnf/1.1.10"

to 

24.194.145.233 - - [09/May/2017:04:45:35 +0000] "GET /metalink?repo=updates-released-f24&arch=x86_64 HTTP/1.1" 200 8747 "-" "dnf/1.1.10 fedora 25 server"

This looks like updating dnf-2.4.1/const.py or updating 
./dnf/repo.py:        self.useragent = dnf.const.USER_AGENT
./dnf/util.py:    handle.useragent = dnf.const.USER_AGENT

to have that additional data in it. I expect it is non-trivial but not impossible?
Comment 11 Jaroslav Rohel 2017-06-01 06:12:01 EDT
> It would be nice if dnf would either send a custom http header which we could log to mirror manager, or set the user agent, or something, based on the presence of fedora-release-workstation, fedora-release-server, fedora-release-cloud — or the absence of any of these.

Maybe better solution is to add requested information to the metalink.
Eg. add '&rel=fedora-server'

How to do this?
You want information based on presence of 'fedora-release-[server,workstation,...]-<version>.noarch.rpm' package. (Or information from '/etc/os-release' file. But content of this file is provided by 'fedora-release-[server,workstation,...]-<version>.noarch.rpm'.)

There is package 'fedora-repos-<version>.noarch.rpm'. But we can use the same logic as above. That means create 'fedora-repos-[server,workstation,...]-<version>.noarch.rpm' packages. These packages will be contain metalinks with &rel=requested_information. 

This solution is DNF (YUM, ...) independent.
Comment 12 Matthew Miller 2017-06-01 07:11:51 EDT
(In reply to Jaroslav Rohel from comment #11)
> Maybe better solution is to add requested information to the metalink.
> Eg. add '&rel=fedora-server'


That could work too. Smooge, what do you think?
Comment 13 Igor Gnatenko 2017-06-01 07:16:29 EDT
@jrohel, it is not only about metalinks, but also for direct baseurl access I suppose...

Matt, what are you trying to achieve?
Comment 14 Matthew Miller 2017-06-01 08:22:24 EDT
(In reply to Igor Gnatenko from comment #13)
> Matt, what are you trying to achieve?

To get a general sense of the popularity of each edition (and possibly spin) in a low-impact way. I'll defer to Smooge on best implementation from the server side.
Comment 15 Stephen John Smoogen 2017-06-01 10:53:23 EDT
One of the big issues that comes up every release is "What editions are being used?" and how much. Knowing this has multiple benefits to Fedora (and various downstreams)

* This is to help allow groups like QA and Release Engineering know which spins, editions, containers, etc. are getting used the most so that they can work out "OK XYZ container isn't building so we need to slip, ABC container didn't build so we don't need to slip"

* This is to help edition groups to get an idea if their edition makes any sense. While no one wants to know if their release is a "flop" they need to know if they want to improve on the next iteration. 

* This needs to be configurable in some way that survives proxy systems and mirrormanager mangling as we get the data from apache logs.

My current mirrormanager scripts do parse the /metalink?repo=updates-released-f21&arch=x86_64 type lines so a release= or container= for however that is done should be possible to add to this. I wasn't sure how it would affect mirrormanager (or CentOS's mirroring system or the RHEL one) so had gone with putting the data in User Agent as that was less likely to tools I could not fix.

Please let me know how I can give better information for this.
Comment 16 Jaroslav Rohel 2017-06-08 07:38:33 EDT
We discussed this internally in the DNF team and we prefer adding the argument to the metalink.

Both sides
- metalink in repo file
- metalink parser on server
are under control of fedora infra/releng.

Other distros don't have to use this at all.

We recommend using 'variant' variable with values such as 'server', 'workstation', etc.
Keep it close to fedora composes.
Comment 17 Matthew Miller 2017-06-08 08:02:27 EDT
Ok, but we would still need DNF support for that. Currently, the metalink line an repo file looks like:

  metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch

Where $releasever and $basearch are filled in by DNF. We would need to add a $releasevariant or something to that.
Comment 18 Jaroslav Mracek 2017-06-19 08:23:00 EDT
Dear Matthew, I would like to explain the solution little bit in more detail. We propose following:

The package fedora-repos will be splitted into 3 packages (fedora-server-repos, fedora-workstations-repos, and fedora-cloud-repos). Each of them will have /etc/yum.repos.d/fedora.repo with modification in metalink, where you can use something like for fedora-workstations-repos:

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch&releasevariant=workstation 

or for fedora-server-repos: 

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-debug-$releasever&arch=$basearch&releasevariant=server 

That means that there will be no additional variable or requirement from dnf site, but only infrastructure solution. And this is exactly what you need, because your request is Fedora specific (cannot be applied for RHEL), therefore it has to be solved by Fedora delivery approach of distro variant. Hope that it helps.
Comment 19 Matthew Miller 2017-06-19 09:42:36 EDT
(In reply to Jaroslav Mracek from comment #18)
> The package fedora-repos will be splitted into 3 packages
> (fedora-server-repos, fedora-workstations-repos, and fedora-cloud-repos).
> Each of them will have /etc/yum.repos.d/fedora.repo with modification in
> metalink, where you can use something like for fedora-workstations-repos:

I'm not automatically opposed to this, but an upgrade path which replaces the repo definition files seems scary.
Comment 20 Matthew Miller 2017-06-19 09:45:22 EDT
Also this is compounded by https://apps.fedoraproject.org/packages/fedora-workstation-repositories, which adds select third-party repositories.
Comment 21 Neal Gompa 2017-06-19 09:54:40 EDT
Why not just add support for custom substitution variables? It's been requested by other folks from time to time.

See bug 1251774 and the libdnf GitHub issue[1].

[1]: https://github.com/rpm-software-management/libdnf/issues/170
Comment 22 Matthew Miller 2017-07-28 09:40:18 EDT
I'd like to leave this bug open for tracking. It might be something we want to solve elsewhere than DNF, but clearly it isn't addressed by *anything* now.
Comment 23 Jaroslav Mracek 2017-10-03 15:38:59 EDT
The custom variables are supported by dnf like in yum (in /etc/dnf/vars), and the functionality is enhanced by https://github.com/rpm-software-management/dnf/pull/938 .
Comment 24 Terry Bowling 2018-03-09 05:56:56 EST
What is the status of this being accepted into rawhide?  Would be awesome to see this part of the yum4 efforts to understand usage and interest.
Comment 25 Jaroslav Mracek 2018-03-14 04:27:46 EDT
I created a proposal about solution. It requires changes in fedora-release and fedora-repos packages. 
https://src.fedoraproject.org/fork/jmracek/rpms/fedora-release/branch/variant
https://pagure.io/fork/jmracek/fedora-repos/branch/variant

Please what do you think?
Comment 26 Jaroslav Mracek 2018-03-14 05:17:33 EDT
I am just mentioning, that the change in baseurl will create incompatibility issue with packagekit.
Comment 27 Jaroslav Mracek 2018-03-14 06:07:24 EDT
Additionally I create an alternative solution (https://github.com/rpm-software-management/dnf/pull/1039), but it requires https://src.fedoraproject.org/fork/jmracek/rpms/fedora-release/branch/variant .
Comment 28 Jaroslav Mracek 2018-03-14 06:11:07 EDT
What solution would be better (Comment 25 ,Comment 27, or something else)?
Comment 29 Stephen John Smoogen 2018-03-14 12:54:15 EDT
I like the second one. It looks like it would not affect PackageKit but they could add it to their software? It would be also fairly simple to have other tools use without causing URL changes (if I understand the first correctly).

I think Matthew Miller needs to weigh in to clear the needinfo bit.
Comment 30 Jaroslav Mracek 2018-03-16 11:09:33 EDT
Please if anyone can try package fedora-release with commit https://src.fedoraproject.org/fork/jmracek/rpms/fedora-release/c/a7f840d9e864bb23ea36b321420fa3179395cc9d?branch=variant I will be very happy.
Comment 31 Dennis Gilmore 2018-03-16 14:38:17 EDT
I like the first option. However the fedora-release changes seem a tad incomplete and the openh264 repo should not have any changes for this. There will be nothing for $releasever unless someone happens to be using an edition, potentially we can work around this for most caes, livecds's containers, images, etc.  We will need dnf, mirrormanager etc to work when an empty variable is passed on.

The reason I am not a fan of the second option is that the header will be sent to every repo used. We will be giving the edition used data to every mirror as well as all third party repos that our users enable. I can see people not being entirely happy about it.
Comment 32 Jaroslav Mracek 2018-03-19 07:11:33 EDT
Unfortunately the first solution breaks packagekit and probably some others that cannot use yum variables. From my point of view I don't like both solution. Both due that we will provide in both cases information about releasevariant to anyone who can provide .repo file or can read headers of curl calls. Additional the first solution will break things.
Comment 33 Stephen John Smoogen 2018-03-19 10:36:07 EDT
OK is it possible to look at my original idea? We stick the data in the User Agent string. This means PackageKit can do something similar at a different time and the parsing of metalinks doesn't get touched. This would allow for the EL clients to put in something for their product lines so that workstation/server/etc clients can be seen in EPEL (which is one thing I have been asked if I could give data about but can't.)

This could also be configurable to make it silent for people who don't want to report it. Or it could be configured to be used by a client for their own needs.

Many sites (including Red Hat) talk to other parts of the company via proxies which lose data about which client is talking about what. Now while satellite allows for this sort of reporting, a lot of infrastructure is based off of downloading a bunch of packages, putting them into a repo, and have a webserver sit in front. Then they are asked to see if XYZ division actually checked in and got an update... which they can't. If the client can set up something like:


/etc/yum.repos.d/foobar.repo
...
Agent_Addition="Company X, Division Y, Release Blah"

/etc/yum.repos.d/epel.repo
Agent_Addition="distributor=%distributor, release=%release, codename=%stuff"  

which it gets from lsb_release data

Then what shows up in the logs is 
"dnf/x.x.x (distributor=centos release=7.4.1708 codename=core)"
"dnf/x.x.x (distributor=fedora release=28 codename=workstation)"
"dnf/x.x.x (distributor=rhel release=8.1.2201 codename=server)"
Comment 34 Dennis Gilmore 2018-03-22 13:51:28 EDT
smooge that will present that information to every mirror, to every third party repo.
Comment 35 Stephen John Smoogen 2018-03-22 13:54:22 EDT
Yeah, I guess enable_extra_agent=true/false per repo would not help here.

Note You need to log in before you can comment on or make changes to this bug.