Bug 862880

Summary: katello-agent throws trace, won't let user install packages on RHEL 6.1 system
Product: Red Hat Satellite Reporter: Corey Welton <cwelton>
Component: katello-agentAssignee: Brad Buckingham <bbuckingham>
Status: CLOSED CURRENTRELEASE QA Contact: Og Maciel <omaciel>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.0.1CC: cpelland, jlaska, omaciel
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: UnusedFlags: cwelton: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: yum and yum-rhn-plugins included in default/base RHEL 6.1 clients are not compatible with CFSE Consequence: installing/removing packages using the CFSE web ui fail Fix: update yum and yum-rhn-packages to latest available version Result: installing/removing packages from the CFSE web ui works after packages are updated.
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-12 15:23:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
rhsm.log none

Description Corey Welton 2012-10-03 20:05:31 UTC
Description of problem:
When trying to install packages from katello on a rhel 6.1 system, errors are thrown in the agent log and packages fail to install.

katello-agent-1.1.2-1.el6cf.noarch

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.  sync and promote RHEL 6Server content
2.  register a RHEL 6.1 system to katello.  note that you may have to get a more recent subscription-manager package than the one that comes default with RHEL 6.1.  Preferably do this with an activation key or just manually subscribe the system to RHEL
3. Systems > $system > Content > Packages
4. Choose a package (such as "abrt") from the content and attempt to update.
  
Actual results:
update/install fails:

2012-10-03 15:58:20,464 [ERROR][worker-0] __call__() @ dispatcher.py:488 - Values instance has no attribute 'repos'
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/gofer/rmi/dispatcher.py", line 485, in __call__
    retval = method(*args, **keywords)
  File "/usr/lib/gofer/plugins/katelloplugin.py", line 197, in update
    updated = p.update(names)
  File "/usr/lib64/gofer/plugins/package.py", line 165, in update
    yb.update(pattern=info)
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 3676, in update
    m = self.pkgSack.returnNewestByNameArch(patterns=pats)
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 882, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 668, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.6/site-packages/yum/repos.py", line 265, in populateSack
    self.doSetup()
  File "/usr/lib/python2.6/site-packages/yum/repos.py", line 72, in doSetup
    self.ayum.plugins.run('prereposetup')
  File "/usr/lib/python2.6/site-packages/yum/plugins.py", line 184, in run
    func(conduitcls(self, self.base, conf, **kwargs))
  File "/usr/share/yum-plugins/rhnplugin.py", line 199, in prereposetup_hook
    for opt, repoexp in opts.repos:
AttributeError: Values instance has no attribute 'repos'
2012-10-03 15:58:20,522 [INFO][worker-0] sendreply() @ rmi.py:158 - 67ba7ebd-37dc-4706-9b7d-9fa1459abc17 processed in: 778 (ms)
2012-10-03 15:58:20,643 [INFO][PathMonitor1] changed() @ katelloplugin.py:135 - changed: /etc/yum.repos.d/redhat.repo
2012-10-03 15:58:20,723 [INFO][PathMonitor1] report_enabled() @ katelloplugin.py:382 - reporting: {'enabled_repos': {'repos': []}}


Expected results:
should be able to install packages

Additional info:
Note: this seems specific to 6.1, and/or maybe earlier.  I don't know the full range of distros this affects, but it doesn't seem to occur on 6.3, etc.

I have tried this on at least two 6.1 boxes -- one old, one a fresh provision -- and get this error both places.  I think I've tried it on another 6.1 too, but I can't remember.

Comment 1 Corey Welton 2012-10-04 16:25:17 UTC
it should be noted that after i updated this box to rhel 6.3 and restarted goferd,  package(s) installed fine.


2012-10-04 11:57:52,405 [INFO][02013477-4fc4-44d3-8705-f21f6dfed26b] connect() @ broker.py:103 - {se-rhelbox.usersys.redhat.com:5671} connected to AMQP
2012-10-04 11:58:04,349 [INFO][worker-0] dispatch() @ dispatcher.py:530 - dispatching:{
  "args": [
    [
      "atmel-firmware", 
      "acl"
    ]
  ], 
  "auth": {
    "secret": "b3f5f02e05849afb9fa60ee847295bd48bde015dd42eea19039a2ac5025a5276", 
    "pam": null, 
    "uuid": "02013477-4fc4-44d3-8705-f21f6dfed26b"
  }, 
  "classname": "Packages", 
  "kws": {}, 
  "cntr": null, 
  "method": "update"
}
2012-10-04 11:58:08,245 [INFO][worker-0] update() @ katelloplugin.py:198 - Packages updated: {'resolved': [], 'deps': []}
2012-10-04 11:58:08,245 [INFO][worker-0] sendreply() @ rmi.py:158 - 32c1a212-1aac-46bf-bc83-a97239a819d2 processed in: 3.954 (seconds)

Comment 2 Brad Buckingham 2012-10-16 17:24:09 UTC
I tried to reproduce the error; however, server initiated installs/updates worked successfully.

The following was the basic scenario:

- on my katello server
  - create RHEL 6 provider/product/repo
  - sync and promote RHEL 6

- client
  - create RHEL 6.1 client (used our snap-guest config)
    - # cat /etc/redhat-release 
      Red Hat Enterprise Linux Server release 6.1 (Santiago)

  - register and subscribe the client to RHN
  - yum repolist
  - update subscription manager to (subscription-manager-0.99.19.4-1.el6_3.x86_64)
  - install katello-agent (katello-agent-1.1.3-1.el6.noarch)
  - unregister the client from RHN

  - install bootstrap rpm from my katello server (e.g. candlepin-cert-consumer-myserver.com-1.0-1.noarch.rpm)
  - service goferd start
  - register the client to my katello server
  - subscribe the client to my rhel6 repo
  - yum repolist

At this point, I was able to successfully install packages (e.g. xterm) as well as update packages (e.g. abrt) from the server.

Packages installed on server:
-----------------------------
katello-all-1.1.12-14.el6cf.noarch
katello-cli-1.1.8-7.el6cf.noarch
katello-selinux-1.1.1-1.el6cf.noarch
katello-qpid-client-key-pair-1.0-1.noarch
katello-glue-candlepin-1.1.12-14.el6cf.noarch
katello-qpid-broker-key-pair-1.0-1.noarch
katello-certs-tools-1.1.8-1.el6cf.noarch
katello-common-1.1.12-14.el6cf.noarch
katello-1.1.12-14.el6cf.noarch
katello-candlepin-cert-key-pair-1.0-1.noarch
katello-configure-1.1.9-7.el6cf.noarch
katello-cli-common-1.1.8-7.el6cf.noarch
katello-glue-pulp-1.1.12-14.el6cf.noarch

pulp-admin-1.1.13-1.el6cf.noarch
m2crypto-0.21.1.pulp-7.el6.x86_64
python-oauth2-1.5.170-2.pulp.el6.noarch
pulp-1.1.13-1.el6cf.noarch
pulp-client-lib-1.1.13-1.el6cf.noarch
mod_wsgi-3.3-3.pulp.el6.x86_64
pulp-common-1.1.13-1.el6cf.noarch
pulp-selinux-server-1.1.13-1.el6cf.noarch
katello-glue-pulp-1.1.12-14.el6cf.noarch

Comment 3 Brad Buckingham 2012-10-16 21:19:15 UTC
Based on discussion with Corey, we are going to move this to ON_QA to see if we can get it reproduced.

Comment 5 Og Maciel 2012-10-17 10:03:06 UTC
Created attachment 628636 [details]
rhsm.log

This issue still persists on RHEL 6.1 client. Attaching rhsm.log

Comment 6 Og Maciel 2012-10-17 10:04:45 UTC
Sadly this is still affecting RHEL 6.1 clients.

Client:

* subscription-manager-0.99.19.4-1.el6_3.x86_64

Server:

* candlepin-0.7.8.1-1.el6cf.noarch
* candlepin-selinux-0.7.8.1-1.el6cf.noarch
* candlepin-tomcat6-0.7.8.1-1.el6cf.noarch
* katello-1.1.12-16.el6cf.noarch
* katello-all-1.1.12-16.el6cf.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.1.8-1.el6cf.noarch
* katello-cli-1.1.8-7.el6cf.noarch
* katello-cli-common-1.1.8-7.el6cf.noarch
* katello-common-1.1.12-16.el6cf.noarch
* katello-configure-1.1.9-7.el6cf.noarch
* katello-glue-candlepin-1.1.12-16.el6cf.noarch
* katello-glue-pulp-1.1.12-16.el6cf.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-1.1.1-1.el6cf.noarch
* pulp-1.1.13-1.el6cf.noarch
* pulp-admin-1.1.13-1.el6cf.noarch
* pulp-client-lib-1.1.13-1.el6cf.noarch
* pulp-common-1.1.13-1.el6cf.noarch
* pulp-selinux-server-1.1.13-1.el6cf.noarch

Comment 8 Brad Buckingham 2012-10-17 20:13:53 UTC
I installed a new rhel6.1 client from an internal repo and was still unable to reproduce the error using my CFSE server.  

I then went to look at the logs associated with the error and decided to register my rhel6.1 client to the QA CFSE server.  Using this client I am still unable to reproduce the error. (i.e. I can install/update packages to my client using the QA CFSE server).

I then looked at the QA rhel6.1 client and in the /var/log/gofer/agent.log, I see that the initial failed attempts had the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/gofer/rmi/dispatcher.py", line 485, in __call__
    retval = method(*args, **keywords)
  File "/usr/lib/gofer/plugins/katelloplugin.py", line 175, in install
    installed = p.install(names)
  File "/usr/lib64/gofer/plugins/package.py", line 176, in install
    yb.install(pattern=info)
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 3338, in install
    mypkgs = self.pkgSack.returnPackages(patterns=pats,
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 882, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 668, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.6/site-packages/yum/repos.py", line 260, in populateSack
    self.doSetup()
  File "/usr/lib/python2.6/site-packages/yum/repos.py", line 72, in doSetup
    self.ayum.plugins.run('prereposetup')
  File "/usr/lib/python2.6/site-packages/yum/plugins.py", line 184, in run
    func(conduitcls(self, self.base, conf, **kwargs))
  File "/usr/share/yum-plugins/rhnplugin.py", line 197, in prereposetup_hook
    for opt, repoexp in opts.repos:
AttributeError: Values instance has no attribute 'repos'

The files at the bottom of the stack are part of the yum and yum-rhn-plugin rpms:
# rpm -qf /usr/lib/python2.6/site-packages/yum/__init__.py /usr/lib/python2.6/site-packages/yum/repos.py /usr/lib/python2.6/site-packages/yum/plugins.py /usr/share/yum-plugins/rhnplugin.py
yum-3.2.29-30.el6.noarch
yum-3.2.29-30.el6.noarch
yum-3.2.29-30.el6.noarch
yum-rhn-plugin-0.9.1-40.el6.noarch

I see on the QA client, that it was updated to rhel6.3 prior to being able to successfully install the vim rpm from the QA CFSE server.  From the /var/log/yum.log, it appears that both yum and yum-rhn-plugin were part of this update.

I suspect the issue raised by this bugzilla is actually that the yum and yum-rhn-plugin packages included in the base rhel6.1 client that the test infrastructure is creating is not compatibile with CFSE.  On my client, the install included yum-3.2.29 and yum-rhn-plugin-0.9.1-40.

I'd suggest the following:
- create the rhel 6.1 client using the same procedures that demonstrates the issue
- rpm -q yum yum-rhn-plugin
- yum update yum yum-rhn-plugin

And then try to perform the package actions again.  If this demonstrates the issue, we will know that they yum pkgs need to be updated.  If it does not, then we'll need to look at the other packages that were updated on the client (in /var/log/yum.log) to determine which might be the issue.

Comment 9 Og Maciel 2012-10-22 16:07:59 UTC
# rpm -q yum yum-rhn-plugin subscription-manager
yum-3.2.29-17.el6.noarch
yum-rhn-plugin-0.9.1-26.el6.noarch
subscription-manager-0.95.11-1.el6.x86_64

# yum update yum yum-rhn-plugin subscription-manager

# rpm -q yum yum-rhn-plugin subscription-manager
yum-3.2.29-30.el6.noarch
yum-rhn-plugin-0.9.1-40.el6.noarch
subscription-manager-0.99.19.4-1.el6_3.x86_64

# yum -y install http://[CFSE]/pub/candlepin-cert-consumer-[CFSE]-1.0-1.noarch.rpm
# subscription-manager --force register --username admin --password admin --org Starbucks --env Demi --autosubscribe

Proceeded to install katello-agent and start goferd service. Then, from web ui was able to install packages.

Comment 10 Og Maciel 2012-10-22 16:22:16 UTC
Verified that this is caused by older versions of yum and yum-rhn-plugin from RHEL 6.1 base installations. Updating them solves the problem. Filed https://bugzilla.redhat.com/show_bug.cgi?id=868984 to deal with documenting it.

Comment 11 Corey Welton 2012-10-22 18:27:43 UTC
clearing needinfo flag.

Comment 12 James Laska 2012-12-12 15:23:56 UTC
Marking CLOSED CURRENTRELEASE based on comment#10.  Customers are advised to update their RHEL-6.1 systems prior to remotely scheduling package operations from System Engine.

Comment 13 Mike McCune 2013-08-16 18:14:15 UTC
getting rid of 6.0.0 version since that doesn't exist