Bug 375111

Unlike channel configuration, I'm not sure if it's possible to ship smart mirror
lists as files in a package.

Another thing is that it doesn't really make sense to include them in a package
because of their dynamic nature, mirrors get added and removed and we don't want
to be updating the configuration package because of this.

I suppose it'd be best if mirrormanager was taught to output smart mirror lists
in addition to yum ones, then people could run "smart mirror --sync
http://some.where/mirrormanager-for-smart" and we could include that info in
this package somehow.

http://fedoraproject.org/wiki/Infrastructure/MirrorManager
https://hosted.fedoraproject.org/projects/mirrormanager/

This issue has been around since at least fedora core 6 and it bugs me a lot.

I have tried a number of things to get smart working on mirrors.

1) smart channel --add type=mirrors
2) smart channel --add type=up2date-mirrors
3) bash script which parses the dynamic/localised mirror list returned by the
normal yum mirror list url.

The bash script works well but it needs to be run occasionally to update the
list.  Smart will favour mirrors which are available more often than not so it
doesn't need to be run that often. It could be a good prototype for a
fedora-mirror.py which can be run from /etc/smart/distro.d.

On the other hand the mirrormanager output only needs to do one of two things:

1) put the baseurl as the first line and then a tab before all the other urls
(the actual mirrors). this has the disadvantage that yum will use the redhat
repos as a near mirror (which is bad of course)

2) do the above but only for a url passed with an option of &smart=yes
e.g. http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-8&arch=i386&smart=yes

this might be the simplest option as it would only need a change in the script
which generates the mirrorlist sent to the client.
with this the mirrors could be added using a command something like:
smart channel --add "Fedora 8 - i386" type=mirrors alias=fedora enabled=yes
url=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-8&arch=i386&smart=yes

3) Not sure if the up2date mirror lists are still produced for fedora 8
(probably not, the wiki has a request to remove this information). However smart
 can still use this information.
smart channel --add "Fedora 8 - i386" type=up2date-mirrors alias=fedora
enabled=yes url=??????
(not sure how the url would look as I can't find much info for up2date. This
used to be done so if the code is still there it might be easy to put it back.

My preference would be to do 2 but 3 might be ok as well. I hope something gets
done because I use smart all the time and it *is* only using the fedora servers.
Somehow this makes me feel bad :(

BTW I think Selinux is denying smart access to some operations which I think is
quietly failing those operations. To see this do a smart upgrade or smart
install and then a tail messages and you will see entries relating to selinux
denying access to the temp files in /tmp/TMPXXXXX which smart generates. As you
guys might be more familiar with smart than I am, do you think this is worthy of
opening a bug?

I would love to see smart be able to automatically use mirrors. It is a
beautiful thing to see smart doing multiple downloads from different mirrors and
taking half the time of yum to do the downloads.

I should have said 'one of three things in my first post :)

Some additional info on using the output returned by the current mirrorlist
script. I believe that the comment line that is output with information about
the mirrors being supplied actually makes a mirror import fail in smart(using
the channel --add type=up2date-mirrors option). It's either that or because
something else in the format is not up2date mirror format. Probably the latter
but documentation on what an up2date mirrorlist should look like is pretty thin
(non-existant?)

Mirrormanager improvement ideas and requests would be much more likely to be
addressed when communicated through their ticket system instead of here in
Bugzilla, see https://hosted.fedoraproject.org/projects/mirrormanager/

Thats true. However I wanted to discuss it with people who know and use smart 
on fedora. What is a good proposition to take to the mirrormanager guys that 
would meet the needs of a Smart Package Manager user? As they say, there's more 
than one way to skin a cat.

Re comment 2 by Rehan Kahn: If the bash script was published, it might alleviate
the current situation were, if the definitions provided by the RPM are used, all
smart usage on Fedora defaults to accessing the Red Hat repositories. This is
probably not a "good thing" from the perspective of efficient network usage.

Yes it's a very bad thing. That was why I wrote the script. I planned to 
publish it somehow. However it's a sprawling mess and mutated into a kind of 
repository manager for yum and smart.

If I remember correctly the only thing it actually did with reference to smart 
and mirrors was wget the normal mirror list (e.g. the mirrorlist url from the 
yum conf file) into a file, export only the lines beginning with ftp and http 
(the comment lines used to confuse smart) and then parse the generated file and 
import the mirrors. This worked quite well.

Let me see if I can dig it up.
R

Well, I also have a two liner for producing mirror lists for smart out of yum's
mirrors URL. But that isn't really scaling well: It would need to run before
every smart update or smart something --update call, e.g. it would have to
become a plugin to make sense. Furthermore the reimporting of mirrors kills the
scores, so it's competing to smart's own mirroring system.

So ATM we don't have a clear path of what is right for the end user. I think
this issue is better presented upstream.

Just for reference the two-liner I use is:

releasever=8; basearch=x86_64
grep -B1 mirrorlist /etc/yum.repos.d/fedora*repo | sed -e's,.*#,,'
-e's@.*mirrorlist=\(.*\)$@wget -nv -O - '\''\1'\'' | grep -v \\# | sed
-e"s,^,smart mirror --add $baseurl ,"@' -e"s,\$releasever,$releasever,g"
-e"s,\$basearch,$basearch,g" | grep -v -- ^-- | sh | sh

I also have a script for converting yum configs (including mirror lists)
available at http://scop.fedorapeople.org/scripts/yum2smart.py .  Not sure if
it's any more useful in this particular case than Axel's "two-liner", but I've
found it useful in converting repo configs in general.  I've intended to submit
is as a contrib script to smart upstream, but haven't got around to doing it yet.

Ville, I like your script as with some plugin magic it could be made the plugin
mentioned above.

Maybe Gustavo and friends will poke on it to massage it into a plugin if you
post it on list or labbix. Thanks!

I think one can safely assume that mirrors don't get a lot of churn. On that 
basis perhaps it is better that a cron job runs once a week or month updating 
the mirrors in smart. Smart can then sort out the 'bad' mirrors from the good 
as it does this anyway. The fallback is always the baseurl during periods when 
every mirror goes down :) This way there is no additional overhead for updating 
the mirrors each time smart runs.

A plugin has been posted on the Smart bugtracker which might solve this issue.

http://tracker.labix.org/issue350

A new mirror channel type (mirrorlist) might be able to solve this problem. As 
it is a native Smart channel it integrates well with how Smart likes to do 
things. 

See https://bugs.launchpad.net/smart/+bug/247351

I have tested this on Fedora 8 and it works well. If more feedback is received 
this might get into upstream pretty soon as it is sorely needed.

I saw that there are conflicting suggestions on which one of the plugins/code patches to use for mirrorlist support. First merge reviews have been made, and it looks like an alternative approach with different end-user visible syntax will be used, but even this didn't make it into smart 1.0 (which has been packaged up and awaits the infrastructure systems to get back).

We shouldn't make any hasty assumptions on what upstream will choose to merge into the mainline code. It's good to see that this is being worked on and we'll get tis into Fedora as soon as there is some official blessing.

It's a shame neither implementation made it into the latest release. Just to clarify the two implementations. The new mirrorlist channel definition is currently the 'approved' method of storing dynamic mirrors (dynamic mirrors should not be editable by end users). The second method adds an import of the mirrors to the user editable internal smart channel format. Both work quite well with the only difference being that the mirrors are editable within smart in the second method.

As no agreement has been reached in approving the second method yet you should be ok using the mirrorlist channel (first method). It just adds two files to Smart and makes no other changes to the code. Smart automatically integrates the channel. There is an example of Fedora 8 mirrorlist .channel files in the upstream bugtracker.

(In reply to comment #15)
> The new mirrorlist channel definition is currently the 'approved' method of storing dynamic mirrors

Can you point me to the approval? If it is approved, why isn't it in the sources?

> As no agreement has been reached in approving the second method yet you should
> be ok using the mirrorlist channel (first method).

Actually I just found out that you are the author of the first method, and upstream seems to favour the other one (there were some comments that seem to have been resolved and are waiting for a new merge review by upstream).

I'm sorry, Fedora should stay close to upstream and if upstream looks like going one direction Fedora shouldn't be going the other way. Please note that I'm not commenting on whether method A or B is better, and your work on improving smart for Fedora is acknowledged!

(In reply to comment #16)
> (In reply to comment #15)
> > The new mirrorlist channel definition is currently the 'approved' method of storing dynamic mirrors
> Can you point me to the approval? If it is approved, why isn't it in the
> sources?

Good question! Don't know is the answer. Actually, although not well documented Smart is plugin based. The idea being that for certain additional functionality the end user can just add a plugin for that functionality. this applies to backends, channels, mirrors etc. There are some guidelines which are only discovered by asking on irc atm :) So adding a mirrorlist channel is implicitally approved as it is just another plugin. This is the way method one works. Method two on the other hand actually changes the way Smart works with rpm-md (rpm metadata) channels (from a higher level design point). Method 2 is a better way from the end user perspective as you can see and edit the mirrors. I use method 2 preferentially over method 1.

> > As no agreement has been reached in approving the second method yet you should
> > be ok using the mirrorlist channel (first method).
> Actually I just found out that you are the author of the first method, and
> upstream seems to favour the other one (there were some comments that seem to
> have been resolved and are waiting for a new merge review by upstream).
> I'm sorry, Fedora should stay close to upstream and if upstream looks like
> going one direction Fedora shouldn't be going the other way. Please note that
> I'm not commenting on whether method A or B is better, and your work on
> improving smart for Fedora is acknowledged!

Thanks, I'm not pushing either method though, you know from a self agrandisment perspective :) I am suggesting that Fedora use method one as a plugin suitable for Fedora (mirrorlist channel), right now, as it provides sorely needed functionality and does not alter the way Smart inherently works so it won't break future compatibility  (so is upstream approved) . Should method two become incorporated in the base code then move to that in future as it is a nicer way of doing the same thing (mirror information is kept with repo information, mirrors are user editable, etc). Using either method in fedora will then increase it's likelyhood of being included in future Smart releases.

To summarise (the above is a bit abstract), Using a mirrorlist channel is implicitly approved (by upstream) as it is a plugin (method 1) and adding mirror functionality to rpm-md channels (method 2) needs explicit approval as it modifies the base code and might break backwards compatibility.

If you want I can provide the bits neccessary for both if you want.

This message is a reminder that Fedora 8 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 8.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '8'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 8's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 8 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 8 changed to end-of-life (EOL) status on 2009-01-07. Fedora 8 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.