Bug 973383 - Upgrade from RHEV-M 3.1 to 3.2 failed with 'GroupsError: No Groups Available in any repository'
Upgrade from RHEV-M 3.1 to 3.2 failed with 'GroupsError: No Groups Available ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup (Show other bugs)
3.2.0
Unspecified Unspecified
urgent Severity unspecified
: ---
: 3.3.0
Assigned To: Alon Bar-Lev
Jiri Belka
integration
: Triaged, ZStream
Depends On:
Blocks: 975022
  Show dependency treegraph
 
Reported: 2013-06-11 15:59 EDT by Petr Spacek
Modified: 2015-09-22 09 EDT (History)
14 users (show)

See Also:
Fixed In Version: is6
Doc Type: Bug Fix
Doc Text:
Previously, local replication of a yum repository would sometimes fail to correctly replicate groups depending on the tool used, resulting in no groups being replicated. This would result in yum issuing an exception when attempting to search for groups instead of returning an empty set, causing engine-setup to fail. With this update, such exceptions are now ignored and an empty set is returned, making it possible for engine-setup to proceed.
Story Points: ---
Clone Of:
: 975022 (view as bug list)
Environment:
Last Closed: 2014-01-21 12:27:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ovirt-engine-upgrade_2013_06_11_21_52_28.log (8.79 KB, text/plain)
2013-06-11 15:59 EDT, Petr Spacek
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 15665 None None None Never
oVirt gerrit 15854 None None None Never
Red Hat Product Errata RHSA-2014:0038 normal SHIPPED_LIVE Important: Red Hat Enterprise Virtualization Manager 3.3.0 update 2014-01-21 17:03:06 EST

  None (edit)
Description Petr Spacek 2013-06-11 15:59:05 EDT
Created attachment 759805 [details]
ovirt-engine-upgrade_2013_06_11_21_52_28.log

Description of problem:
Attempt to upgrade rhevm-3.1.0-53.el6ev.noarch -> RHEV-M 3.2.0 failed with following error message:

# rhevm-upgrade --no-yum-rollback

Checking for updates... (This may take several minutes)...[ ERROR ]
Starting ovirt-engine service...                     [ DONE ]
Error: Upgrade failed.
please check log at /var/log/ovirt-engine/ovirt-engine-upgrade_2013_06_11_21_52_28.log

Version-Release number of selected component (if applicable):
rhevm-setup-3.2.0-11.30.el6ev.noarch

How reproducible:
100 %

Steps to Reproduce:
1. Find a machine with rhevm-3.1.0-53.el6ev.noarch
2. Configure a yum repo with RHEV-M 3.2.0
3. Do yum upgrade (to get newest rhevm-upgrade script)
4. Run rhevm-upgrade

Actual results:
Upgrade failed.

Excerpt from ovirt-engine-upgrade_2013_06_11_21_52_28.log:
2013-06-11 21:52:32::DEBUG::common_utils::345::root:: YUM: VERB: Downloading: repomdh1KR1Ntmp.xml 951 (100%)
2013-06-11 21:40:52::ERROR::common_utils::353::root:: YUM: FAIL: No Groups Available in any repository
2013-06-11 21:40:52::DEBUG::common_utils::345::root:: YUM: VERB: Performing rollback
2013-06-11 21:40:52::DEBUG::common_utils::1438::root:: Locking rpms in yum-version-lock
2013-06-11 21:40:53::DEBUG::rhevm-upgrade::585::root:: starting ovirt-engine service.
2013-06-11 21:40:53::DEBUG::common_utils::453::root:: Executing command --> '/sbin/service ovirt-engine start' in working directory '/'
2013-06-11 21:40:53::DEBUG::common_utils::491::root:: output = Starting engine-service:ESC[60G [  ESC[0;32mOKESC[0;39m  ]

2013-06-11 21:40:53::DEBUG::common_utils::492::root:: stderr = 
2013-06-11 21:40:53::DEBUG::common_utils::493::root:: retcode = 0
2013-06-11 21:40:53::ERROR::rhevm-upgrade::1366::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1359, in <module>
    main(options)
  File "/usr/bin/rhevm-upgrade", line 1190, in main
    runFunc([rhyum.begin], MSG_INFO_CHECK_UPDATE)
  File "/usr/bin/rhevm-upgrade", line 622, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 305, in begin
    for group in self._miniyum.queryGroups():
  File "/usr/share/ovirt-engine/scripts/miniyum.py", line 801, in queryGroups
    installed, available = self._yb.doGroupLists()
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 2829, in doGroupLists
    if self.comps.compscount == 0:
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 910, in <lambda>
    comps = property(fget=lambda self: self._getGroups(),
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 832, in _getGroups
    raise Errors.GroupsError, _('No Groups Available in any repository')
GroupsError: No Groups Available in any repository

Expected results:
Installation works.
Comment 2 Petr Spacek 2013-06-11 16:18:58 EDT
I found that the problem disappears if I register the system to public RHN:
# rhn-channel -l
jbappplatform-6-x86_64-server-6-rpm
rhel-x86_64-server-6
rhel-x86_64-server-6-rhevm-3.2

How can I track down the problem in RHN->yum repo mirroring script? What exactly the upgrade script need to see in the repo?
Comment 4 Vladimir Vasilev 2013-06-12 03:46:30 EDT
I did the yum repo mirror. But yum groups needs to be created manually. 
Can you tell me the name of the group and which packages are there?
Comment 5 Itamar Heim 2013-06-13 02:52:39 EDT
moran - please check with GSS if the KB for offline install/upgrade needs to be updated as well
Comment 6 Alon Bar-Lev 2013-06-13 08:32:52 EDT
packaging: setup: handle a yum error when no groups


apparently, when querying yum for group, if no groups at all it raises
an exception instead of return empty set. this is unexpected behavior of
an api.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=973383
Change-Id: Ifb1f196006662d6571c7ca80554e0a9a17aaec2a
Signed-off-by: Alon Bar-Lev <alonbl@redhat.com>
Comment 7 Alon Bar-Lev 2013-06-13 08:34:41 EDT
Hello Petr,

Can you please checkout this patch?

You can manually change this on system at /usr/share/ovirt-engine/scripts/miniyum.py

Thanks!
Comment 8 Alon Bar-Lev 2013-06-13 08:36:48 EDT
Moran,

Can you please take care to include the group in the channel?

And tell me if you like the fix in miniyum for z stream, I don't think it is required if there is a group...

Thanks,
Alon
Comment 9 Moran Goldboim 2013-06-13 08:49:56 EDT
sure, though i'm quite sure we validated it's working without group definition, on the transition to groups in the repo, was something changed on our code or yum?
Comment 10 Alon Bar-Lev 2013-06-13 08:53:02 EDT
(In reply to Moran Goldboim from comment #9)
> sure, though i'm quite sure we validated it's working without group
> definition, on the transition to groups in the repo, was something changed
> on our code or yum?

As I wrote above, the problem is with yum raising an error when querying available groups when no group is available in system. This probably was not tested, as it is unexpected.
Comment 11 Petr Spacek 2013-06-13 08:55:06 EDT
I tested the patch but it didn't work:

2013-06-13 14:51:38::ERROR::rhevm-upgrade::1366::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1359, in <module>
    main(options)
  File "/usr/bin/rhevm-upgrade", line 1190, in main
    runFunc([rhyum.begin], MSG_INFO_CHECK_UPDATE)
  File "/usr/bin/rhevm-upgrade", line 622, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 305, in begin
    for group in self._miniyum.queryGroups():
  File "/usr/share/ovirt-engine/scripts/miniyum.py", line 819, in queryGroups
    except Errors.GroupsError as e:
NameError: global name 'Errors' is not defined

I modified the patch in this way:
except Errors.GroupsError as e: -> except yum.Errors.GroupsError as e


And now I got different error:

2013-06-13 14:53:23::ERROR::rhevm-upgrade::1366::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1359, in <module>
    main(options)
  File "/usr/bin/rhevm-upgrade", line 1190, in main
    runFunc([rhyum.begin], MSG_INFO_CHECK_UPDATE)
  File "/usr/bin/rhevm-upgrade", line 622, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 305, in begin
    for group in self._miniyum.queryGroups():
TypeError: 'NoneType' object is not iterable
Comment 12 Petr Spacek 2013-06-13 09:00:38 EDT
(In reply to Alon Bar-Lev from comment #10)
> As I wrote above, the problem is with yum raising an error when querying
> available groups when no group is available in system. This probably was not
> tested, as it is unexpected.

Note that some users use 'mrepo' tool for RHN->yum repo conversion (e.g. in combination with Spacewalk etc.). It was also presented on Summit 2010:

See http://www.redhat.com/promo/summit/2010/presentations/summit/decoding-the-code/wed/cshabazi-530-more/MORE-Kickstart-Tips-and-Tricks.pdf and search for 'mrepo'.
Comment 13 Alon Bar-Lev 2013-06-13 09:07:25 EDT
Sorry, my bad! Fixed in gerrit.

Can you please also attach the output of:

# yum groups list

Just to make sure no groups is available or something else is broken, Thanks!
Comment 14 Petr Spacek 2013-06-13 09:26:12 EDT
(In reply to Alon Bar-Lev from comment #13)
> Sorry, my bad! Fixed in gerrit.
Great, now it works!

# rhevm-upgrade -c
Checking for updates... (This may take several minutes)...[ DONE ]
No updates available
Starting ovirt-engine service...                     [ DONE ]

> Can you please also attach the output of:
> # yum groups list
> 
> Just to make sure no groups is available or something else is broken, Thanks!

# yum grouplist
Setting up Group Process
Error: No group data available for configured repositories
Comment 15 Alon Bar-Lev 2013-06-13 09:30:59 EDT
(In reply to Petr Spacek from comment #14)
> # rhevm-upgrade -c
> Checking for updates... (This may take several minutes)...[ DONE ]
> No updates available
> Starting ovirt-engine service...                     [ DONE ]

This is not right... there should have been an update, right?

Can you please attach the upgrade log?
Comment 16 Petr Spacek 2013-06-13 11:31:11 EDT
I'm sorry for the confusion. I tried 'rhevm-upgrade -c' on already upgraded machine (upgraded via RHN).

The point is that without the patch the 'rhevm-upgrade -c' died:
> Checking for updates... (This may take several minutes)...[ FAIL ...

Unfortunatelly, I don't have another machine for upgrade at the moment.
Comment 18 Vladimir Vasilev 2013-06-14 04:34:11 EDT
The patch works for me too, we just did a small correction:


--- a/miniyum.py        2013-06-14 10:33:09.737877468 +0200
+++ b/miniyum.py        2013-06-14 10:32:52.445863772 +0200
@@ -815,7 +815,7 @@
                         'uservisible': grp.user_visible
                     })
 
-       except Errors.GroupsError as e:
+       except yum.Errors.GroupsError as e:
                # rhbz#973383 empty groups raises an exception
                self._sink.verbose('Ignoring group error: %s' % e)
         except Exception as e:
Comment 19 Alon Bar-Lev 2013-06-14 04:41:40 EDT
(In reply to Vladimir Vasilev from comment #4)
> I did the yum repo mirror. But yum groups needs to be created manually. 
> Can you tell me the name of the group and which packages are there?

After discussion with Vladimir Vasilev who use mrepo, I can interpret that better.

mrepo does not support yum groups, it does not replicate these. I found a patch that should support that[1] at upstream, it did not added to the actual codebase.

Options:

1. Apply the patch to ignore yum error when no groups are available. This will avoid error and update ovirt-engine package and dependencies to match the ovirt-engine minimum requirements.

2. Revert bug#883530 and re-add explicit yum upgrade to each satellite package hard coded within ovirt-engine package, so these are upgraded to latest version (beyond the minimum requirements of ovirt-engine).

[1] https://github.com/dagwieers/mrepo/blob/master/patches/yam-0.8.0-yumgroups.patch
Comment 20 Andrew Cathrow 2013-06-14 07:11:46 EDT
(In reply to Alon Bar-Lev from comment #19)
> (In reply to Vladimir Vasilev from comment #4)
> > I did the yum repo mirror. But yum groups needs to be created manually. 
> > Can you tell me the name of the group and which packages are there?
> 
> After discussion with Vladimir Vasilev who use mrepo, I can interpret that
> better.
> 
> mrepo does not support yum groups, it does not replicate these. I found a
> patch that should support that[1] at upstream, it did not added to the
> actual codebase.
> 
> Options:
> 
> 1. Apply the patch to ignore yum error when no groups are available. This
> will avoid error and update ovirt-engine package and dependencies to match
> the ovirt-engine minimum requirements.
> 
> 2. Revert bug#883530 and re-add explicit yum upgrade to each satellite
> package hard coded within ovirt-engine package, so these are upgraded to
> latest version (beyond the minimum requirements of ovirt-engine).
> 
> [1]
> https://github.com/dagwieers/mrepo/blob/master/patches/yam-0.8.0-yumgroups.
> patch

Is there an RFE to add this patch to RHEL?
Comment 21 Alon Bar-Lev 2013-06-14 08:28:24 EDT
(In reply to Andrew Cathrow from comment #20)
> (In reply to Alon Bar-Lev from comment #19)
> > (In reply to Vladimir Vasilev from comment #4)
> > > I did the yum repo mirror. But yum groups needs to be created manually. 
> > > Can you tell me the name of the group and which packages are there?
> > 
> > After discussion with Vladimir Vasilev who use mrepo, I can interpret that
> > better.
> > 
> > mrepo does not support yum groups, it does not replicate these. I found a
> > patch that should support that[1] at upstream, it did not added to the
> > actual codebase.
> > [1]
> > https://github.com/dagwieers/mrepo/blob/master/patches/yam-0.8.0-yumgroups.
> > patch
> 
> Is there an RFE to add this patch to RHEL?

as far as I see mrepo is not part of RHEL nor the older name yam.
Comment 23 Jiri Belka 2013-10-25 08:01:03 EDT
ok, is20.

# cat -n /usr/lib/python2.6/site-packages/otopi/miniyum.py | sed -n '/def queryGroups/,/def/p'
   853      def queryGroups(self):
   854          ret = []
   855
   856          try:
   857              with self._disableOutput:
   858                  installed, available = self._yb.doGroupLists()
   859
   860                  for grp in installed:
   861                      ret.append({
   862                          'operation': 'installed',
   863                          'name': grp.name,
   864                          'uservisible': grp.user_visible
   865                      })
   866                  for grp in available:
   867                      ret.append({
   868                          'operation': 'available',
   869                          'name': grp.name,
   870                          'uservisible': grp.user_visible
   871                      })
   872          except yum.Errors.GroupsError as e:
   873              # rhbz#973383 empty groups raises an exception
   874              self._sink.verbose('Ignoring group error: %s' % e)
   875          except Exception as e:
   876              self._sink.error(e)
   877              raise
   878
   879          return ret
   880
   881      def queryPackages(self, pkgnarrow='all', patterns=None, showdups=None):
Comment 24 Charlie 2013-11-27 19:23:58 EST
This bug is currently attached to errata RHEA-2013:15231. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.
* Consequence: What happens when the bug presents.
* Fix: What was done to fix the bug.
* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes 

Thanks in advance.
Comment 25 errata-xmlrpc 2014-01-21 12:27:01 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0038.html

Note You need to log in before you can comment on or make changes to this bug.