Bug 487112

Summary: New cx_Oracle causes rhn-satellite-activate failures
Product: Red Hat Satellite 5 Reporter: Jan Pazdziora <jpazdziora>
Component: ServerAssignee: Michael Mráka <mmraka>
Status: CLOSED CURRENTRELEASE QA Contact: Steve Salevan <ssalevan>
Severity: medium Docs Contact:
Priority: low    
Version: 530CC: bperkins
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: sat530 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-10 20:33:38 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 456985    

Description Jan Pazdziora 2009-02-24 10:16:45 UTC
Description of problem:

Installation of Satellite ends with

Where is your satellite certificate file? /root/cperry-lots-of-slots.cert?rev=181349
** Loading RHN Satellite Certificate.
** Verifying certificate locally.
** Activating RHN Satellite.
There was a problem validating the satellite certificate: 1

and there's error in rhn-installation.log:

ERROR: unhandled exception occurred:
Traceback (most recent call last):
  File "/usr/bin/rhn-satellite-activate", line 45, in ?
    sys.exit(abs(mod.main() or 0))
  File "/usr/share/rhn/satellite_tools/rhn_satellite_activate.py", line 574, in main
    rhnSQL.initDB(CFG.DEFAULT_DB)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 98, in initDB
    __init__DB(backend, host, port, username, password, database)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 52, in __init__DB
    __DB.connect()
  File "/usr/share/rhn/server/rhnSQL/driver_cx_Oracle.py", line 316, in connect
    self.dbh = self._connect()
  File "/usr/share/rhn/server/rhnSQL/driver_cx_Oracle.py", line 342, in _connect
    dbh = cx_Oracle.Connection(self.dsn)
RuntimeError: Unable to acquire Oracle environment handle

When I run the /usr/bin/rhn-satellite-activate on the command line, I get the same error:

# /usr/bin/rhn-satellite-activate --disconnected
NOTE: using backup cert as default: /etc/sysconfig/rhn/rhn-entitlement-cert.xml

ERROR: unhandled exception occurred:
Traceback (most recent call last):
  File "/usr/bin/rhn-satellite-activate", line 45, in ?
    sys.exit(abs(mod.main() or 0))
  File "/usr/share/rhn/satellite_tools/rhn_satellite_activate.py", line 574, in main
    rhnSQL.initDB(CFG.DEFAULT_DB)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 98, in initDB
    __init__DB(backend, host, port, username, password, database)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 52, in __init__DB
    __DB.connect()
  File "/usr/share/rhn/server/rhnSQL/driver_cx_Oracle.py", line 316, in connect
    self.dbh = self._connect()
  File "/usr/share/rhn/server/rhnSQL/driver_cx_Oracle.py", line 342, in _connect
    dbh = cx_Oracle.Connection(self.dsn)
RuntimeError: Unable to acquire Oracle environment handle

Using older cx_Oracle rpm seems to fix the problem for me.

Version-Release number of selected component (if applicable):

# rpm -q cx_Oracle
cx_Oracle-4.2.1-6.el5sat

How reproducible:

Deterministic.

Steps to Reproduce:
1. Run /usr/bin/rhn-satellite-activate --disconnected
  
Actual results:

NOTE: using backup cert as default: /etc/sysconfig/rhn/rhn-entitlement-cert.xml

ERROR: unhandled exception occurred:
Traceback (most recent call last):
  File "/usr/bin/rhn-satellite-activate", line 45, in ?
    sys.exit(abs(mod.main() or 0))
  File "/usr/share/rhn/satellite_tools/rhn_satellite_activate.py", line 574, in main
    rhnSQL.initDB(CFG.DEFAULT_DB)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 98, in initDB
    __init__DB(backend, host, port, username, password, database)
  File "/usr/share/rhn/server/rhnSQL/__init__.py", line 52, in __init__DB
    __DB.connect()
  File "/usr/share/rhn/server/rhnSQL/driver_cx_Oracle.py", line 316, in connect
    self.dbh = self._connect()
  File "/usr/share/rhn/server/rhnSQL/driver_cx_Oracle.py", line 342, in _connect
    dbh = cx_Oracle.Connection(self.dsn)
RuntimeError: Unable to acquire Oracle environment handle


Expected results:

No error, activated OK.

Additional info:

Reverting to cx_Oracle-4.2.1-4.el5.i386.rpm from Satellite-5.3.0-RHEL5-re20090213.1 allows the rhn-satellite-activate to proceed OK.

Comment 1 Jan Pazdziora 2009-02-24 10:18:58 UTC
With the new cx_Oracle, the strace end with

15010 stat64("/usr/share/rhn/server/rhnSQL/os", 0xbfa607a4) = -1 ENOENT (No such file or directory)
15010 open("/usr/share/rhn/server/rhnSQL/os.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
15010 open("/usr/share/rhn/server/rhnSQL/osmodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
15010 open("/usr/share/rhn/server/rhnSQL/os.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
15010 open("/usr/share/rhn/server/rhnSQL/os.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
15010 close(3)                          = 0
15010 mmap2(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7a48000
15010 open("/usr/lib/oracle/10.2.0.4/client//lib/libclntsh.so.10.1/libociei.so", O_RDONLY) = -1 ENOTDIR (Not a directory)
15010 open("/usr/lib/oracle/10.2.0.4/client//lib/libclntsh.so.10.1/libociicus.so", O_RDONLY) = -1 ENOTDIR (Not a directory)
15010 write(2, "\nERROR: unhandled exception occu"..., 38) = 38
15010 write(2, "Traceback (most recent call last"..., 35) = 35
15010 open("/usr/bin/rhn-satellite-activate", O_RDONLY|O_LARGEFILE) = 3

Note the double slash in the path -- that double slash is also present in previous (successful) accesses like

15010 stat64("/usr/lib/oracle/10.2.0.4/client//lib/i686", 0xbfa600c0) = -1 ENOENT (No such file or directory)
15010 open("/usr/lib/oracle/10.2.0.4/client//lib/sse2/libclntsh.so.10.1", O_RDONLY) = -1 ENOENT (No such file or directory)
15010 stat64("/usr/lib/oracle/10.2.0.4/client//lib/sse2", 0xbfa600c0) = -1 ENOENT (No such file or directory)
15010 open("/usr/lib/oracle/10.2.0.4/client//lib/libclntsh.so.10.1", O_RDONLY) = 5
15010 read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\344o\177\0064\0\0\0"..., 512) = 512

With the older cx_Oracle package, strace looks like this

14994 stat64("/usr/share/rhn/server/rhnSQL/os", 0xbfc809c4) = -1 ENOENT (No such file or directory)
14994 open("/usr/share/rhn/server/rhnSQL/os.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
14994 open("/usr/share/rhn/server/rhnSQL/osmodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
14994 open("/usr/share/rhn/server/rhnSQL/os.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
14994 open("/usr/share/rhn/server/rhnSQL/os.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
14994 close(3)                          = 0
14994 mmap2(NULL, 143360, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb79bc000
14994 open("/usr/lib/oracle/10.2.0.4/client/lib/libociei.so", O_RDONLY) = 3
14994 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P&\0\0004\0\0\0"..., 512) = 512
14994 fstat64(3, {st_mode=S_IFREG|0755, st_size=72674548, ...}) = 0
14994 mmap2(NULL, 72677524, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x19ed000
14994 mmap2(0x5f3b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x454d) = 0x5f3b000

Comment 2 Jan Pazdziora 2009-02-24 13:42:37 UTC
This was observed on on i386 compose, FYI.

Comment 3 Michael Mráka 2009-02-24 15:53:49 UTC
Fixed in spacewalk git repo
commit c1b3a06bae71a1ea1481919e4a55a4ec26ea4938
    Automatic commit of package [cx_Oracle] minor release [4.2.1-7].
commit 4e1255a858ba6c6361543b37d6b2e882f6c8a6b5
    487112 - changed build flags and environment
commit 6f7a3a8d72155c1ae194e404ba4beb1ed5e7cfe4
    487112 - removed trailing / from ORACLE_HOME

Comment 4 Michael Mráka 2009-03-02 09:31:00 UTC
Moved ON_QA for Satellite-5.3.0-RHEL5-re20090227.1.

Comment 5 Steve Salevan 2009-03-10 18:48:11 UTC
I saw no errors upon the installation of my latest Satellite (530-re20090306.2), and I was unable to reproduce the traceback with several permutations of the rhn-satellite-activate command on two separate Satellites.  Things look good, so I'm moving this bug to VERIFIED.

Comment 6 Brandon Perkins 2009-08-25 21:16:35 UTC
Release Pending.  No Satellite Activation failures seen.

Comment 7 Brandon Perkins 2009-09-10 20:33:38 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1434.html