Bug 126800 - Test harness for zeroing in on component with yum problems
Summary: Test harness for zeroing in on component with yum problems
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: yum
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Jeremy Katz
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2004-06-27 06:30 UTC by Bob Gustafson
Modified: 2014-01-21 22:49 UTC (History)
0 users

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2005-07-29 19:12:31 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Output from running script ruby yum_inst.rb (3.41 KB, text/plain)
2004-06-27 20:24 UTC, Bob Gustafson
no flags Details
wrapper script for yum - to install everything (with exceptions) (1.11 KB, text/plain)
2004-09-04 16:47 UTC, Bob Gustafson
no flags Details

Description Bob Gustafson 2004-06-27 06:30:17 UTC
Description of problem:

  If you do 'yum install \*', quite often this command will fail
because one (or more) components has a problem.

  If you then do 'yum --exclude=problem install \*', it will still
fail, because either --exclude does not work, or there is a failure in
some other component which only makes it look like 'problem' has a
problem.

  My approach was to run 'yum install a\*', yum install b\*', etc. 
This at least installs components with different first letters than
the problem component.

  I got tired of typing 26 different command lines, so I wrote a small
test harness that does this programatically. By making the process
recursive, when it comes to a letter with a problem component - for
example 'yum install g\*', it will back off and try 'yum install
ga\*', 'yum install gb\*', etc. Only two letters are tried in the
attached program, but a change of one number will make it recurse further.

If you name the attached test program 'yum_batch.rb', then the command
to run it is 'ruby yum_batch.rb'

Version-Release number of selected component (if applicable):

  Latest

How reproducible:

  Try it to see

Steps to Reproduce:
1. ruby yum_batch.rb

Actual results:

ruby yum_batch.rb
...
...
Doing testph re\*
Resolving dependencies
.Package xerces-j needs katana, this is not available.
Doing testph rf\*
Doing testph rg\*
Doing testph rh\*
Resolving dependencies
..
..

Expected results:

  Should install almost all of the FC2 components.

Additional info:

  This is the program:

# yum_batch.rb by Bob Gustafson 27-Jun-2004
def chk(phrase,letter)
  testph = phrase[0..-3] + letter + phrase[-2..-1]
  puts "Doing testph #{testph}"
  outs = `yum -y install #{testph} 2>&1`.split(/\n/)
  outs.each do |line|
    if    line.match(/Gathering header/)
    elsif line.match(/Server: /)
    elsif line.match(/Finding /)
    elsif line.match(/Downloading /)
    elsif line.match(/Cannot find/)
    elsif line.match(/No actions /)
    elsif line.match(/this is not available/)
      if testph.length > 3
        puts line
        return
      end
      ('a'..'z').each do |c|
        chk(testph,c)
      end
    else
      puts line
    end
  end
end
 
('a'..'z').each do |c|
  chk("\\*",c)
end

Comment 1 Bob Gustafson 2004-06-27 06:46:08 UTC
Whoops, needed one more line.

# yum_batch.rb by Bob Gustafson 27-Jun-2004 v2
def chk(phrase,letter)
  testph = phrase[0..-3] + letter + phrase[-2..-1]
  puts "Doing testph #{testph}"
  outs = `yum -y install #{testph} 2>&1`.split(/\n/)
  outs.each do |line|
    if    line.match(/Gathering header/)
    elsif line.match(/Server: /)
    elsif line.match(/Finding /)
    elsif line.match(/Downloading /)
    elsif line.match(/Cannot find/)
    elsif line.match(/No actions /)
    elsif line.match(/this is not available/)
      if testph.length > 3
        puts line
        return
      end
      ('a'..'z').each do |c|
        chk(testph,c)
      end
      return
    else
      puts line
    end
  end
end
 
('a'..'z').each do |c|
  chk("\\*",c)
end

Comment 2 Seth Vidal 2004-06-27 15:27:06 UTC
why on earth would you ever run yum install \* ?

And what times did you see --exclude not work?



Comment 3 Bob Gustafson 2004-06-27 18:12:44 UTC
On problems with --exclude, see bug 126387, and bug 126777

-----

Yes - life is not all 'yum install \*'.

I got to thinking that I would like to exclude the -debuginfo
packages. I can see where they would be useful to folks who are
actually into the code with gdb, but the download of 480MB
kernel-debuginfo was cramping my updates..

So I read the man page for yum and looked at the 'yum list ..'
options. (This section is not so clear though - at least for me).

With this new knowledge, I wrote another script. (I also noticed that
package names contain capital letters, hyphens, numbers, etc., which
the previous script did not take into account..).

---- run by 'ruby yum_inst.rb'

# yum_inst.rb by Bob Gustafson 27-Jun-2004
def get_list_tobe_installed
  names = []
  candidates = `yum list 2>&1`.split(/\n/)
  flag = false
  candidates.each do |cand|
    if flag
      name = cand.split(/\s+/)[0]
      if name == nil then next end
      if !name.match("debuginfo") then names << name end
    elsif cand.match("----")
      flag = true
    end
  end
  names
end
 
get_list_tobe_installed.each do |name|
  puts "Doing #{name}"
  outs = `yum -y install #{name} 2>&1 `.split(/\n/)
  outs.each do |line|
    if    line.match(/Gathering header/)
    elsif line.match(/Server: /)
    elsif line.match(/Finding /)
    elsif line.match(/Downloading /)
    elsif line.match(/Cannot find/)
    elsif line.match(/No actions /)
      puts line
    end
  end
end

----

You may criticize the use of 'yum install \*', but it does seem to
show bugs that the above script delicately steps over. See below:

[root@hoho2 root]# yum install \*
Gathering header information file(s) from server(s)
Server: Fedora Core 2 - Development Tree
Finding updated packages
Downloading needed headers
kernel-smp is installed and is the latest version.
openssl-debuginfo is installed and is the latest version.
glibc-debuginfo is installed and is the latest version.
openssl is installed and is the latest version.
glibc is installed and is the latest version.
kernel-debuginfo is installed and is the latest version.
kernel is installed and is the latest version.
Resolving dependencies
.Package xerces-j needs katana, this is not available.
Package xalan-j needs katana, this is not available.
Package jakarta-regexp needs katana, this is not available.
Package bcel needs katana, this is not available.
Package jaf needs katana, this is not available.
Package javamail needs katana, this is not available.
Package ecj needs katana, this is not available.
Package junit needs katana, this is not available.
Package cup needs katana, this is not available.
Package xalan-j-xsltc needs katana, this is not available.
Package ant-core needs katana, this is not available.
Package commons-logging needs katana, this is not available.
Package ant-optional needs katana, this is not available.
Package katana-build needs katana = 1.0.4-4, this is not available.
Package jlex needs katana, this is not available.
Package gnome-utils needs /usr/bin/gtk-query-immodules-2.0, this is
not available.
[root@hoho2 root]#

Comment 4 Bob Gustafson 2004-06-27 20:24:27 UTC
Created attachment 101454 [details]
Output from running script   ruby yum_inst.rb

Comment 5 Bob Gustafson 2004-06-27 20:26:53 UTC
Whoops, one more line needed

(The attachment above is from running this updated version)

# yum_inst.rb by Bob Gustafson 27-Jun-2004 v2
def get_list_tobe_installed
  names = []
  candidates = `yum list 2>&1`.split(/\n/)
  flag = false
  candidates.each do |cand|
    if flag
      name = cand.split(/\s+/)[0]
      if name == nil then next end
      if !name.match("debuginfo") then names << name end
    elsif cand.match("----")
      flag = true
    end
  end
  names
end
 
get_list_tobe_installed.each do |name|
  puts "Doing #{name}"
  outs = `yum -y install #{name} 2>&1 `.split(/\n/)
  outs.each do |line|
    if    line.match(/Gathering header/)
    elsif line.match(/Server: /)
    elsif line.match(/Finding /)
    elsif line.match(/Downloading /)
    elsif line.match(/Cannot find/)
    elsif line.match(/No actions /)
    else
      puts line
    end
  end
end

Comment 6 Bob Gustafson 2004-07-24 19:53:11 UTC
The command:

yum -y --exclude=\*debuginfo\* install \*

Works just about as well as the script.

Comment 7 Bob Gustafson 2004-09-04 16:43:44 UTC
Since this is a wrapper for yum, whenever the yum output changes, the
parsing within this script needs to be changed.

See attachment for latest yum_inst4.rb script.

Comment 8 Bob Gustafson 2004-09-04 16:47:52 UTC
Created attachment 103468 [details]
wrapper script for yum - to install everything (with exceptions)

Comment 9 Rahul Sundaram 2005-06-20 21:28:38 UTC
Bob,

I would suggest writing test suites in the language in which Yum is written in -
Python. It needs to be written in the same way if you are thinking about having
this test suite included by default in the yum build process.

It doesnt make much sense to me to have a different language dependency just for
a test suite not that I dont like Ruby or anything like that

Comment 10 Bob Gustafson 2005-06-21 17:13:33 UTC
Everyone has their own hammer. Mine is Ruby at the moment.

yum/up2date works pretty well now. The only problems occur with a user system
which has not been updated for awhile. The number of updates overwhelms yum/up2date.

Because the 'checking for dependencies' phase takes much longer when there are a
large number of components to change, it makes sense to 'switch modes' so to
speak: When the number of updateable components is large. Do some of the
components completely (dependencies, download, install, etc.) before going on to
the next batch. Otherwise, failures of some components to download or install
will kill the whole process - requiring a restart from the beginning.

My hack in Ruby is a wrapper to accomplish this to some extent.

Comment 11 Seth Vidal 2005-06-21 17:17:34 UTC
Actually, I'm curious - why not make this script function as a yum shell script.

yum shell
> install a*
> run
> install b*
> run

etc etc etc.
also I think you'll find that the yum depsolving routines in yum 2.3.X to be
much much faster
and cvs-HEAD has even more speedups.



Comment 12 Seth Vidal 2005-07-29 19:12:31 UTC
closing b/c I doubt we'll bring in the ruby script for yum.


Note You need to log in before you can comment on or make changes to this bug.