This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 838033

Summary: patch cluster deployment fails on solaris 10 clients
Product: [Community] Spacewalk Reporter: pierre.casenove
Component: ClientsAssignee: Jan Pazdziora <jpazdziora>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satellite-qa-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 1.7CC: jpazdziora
Target Milestone: ---   
Target Release: ---   
Hardware: sparc   
OS: Solaris   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-06 13:34:16 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 917805    
Attachments:
Description Flags
Python trace when opening the zip file
none
Patch to avoid using ZipFile
none
Correct patch to avoid using ZipFile none

Description pierre.casenove 2012-07-06 03:19:07 EDT
Description of problem:
Using solaris2mpm on a RHEL 6 box, I've generated the mpm files associated with the latest patch cluster for Solaris 10.
This patch cluster is a zip file requiring ZIP64 extension.
I've rhnpushed these mpm files on my spacewalk server, version 1.7, postgresql 8.4, RHEL 5.
When scheduling the patch cluster deployment on a SOLARIS 10 SPARC client, here is the output of rhn_check:
D: do_call solarispkgs.patchClusterInstall
([[['patch-cluster-solaris-', '20120412', '1',
'sparc-solaris-patch-cluster', 'solaris-10-sparc-patches'], {}]],)
Installing patch cluster:  [[['patch-cluster-solaris-', '20120412',
'1', 'sparc-solaris-patch-cluster', 'solaris-10-sparc-patches'], {}]]
Loading cache...
Updating cache...
#####################################################################
[100%]
Updating cache...
#####################################################################
[100%]
Updating cache...
#####################################################################
[100%]

Computing transaction...
Committing transaction...
D: Sending back response (102, 'Patch cluster install failed: Bad
magic number for central directory', {'version': 0, 'name':
'solarispkgs.patchClusterInstall'})
D: do_call solarispkgs.checkNeedUpdate ('rhnsd=1',)
Doing checkNeedUpdate

Traceback (most recent call last):
  File "/opt/redhat/rhn/solaris/usr/sbin/rhn_check", line 317, in ?
    run_local_actions()
  File "/opt/redhat/rhn/solaris/usr/sbin/rhn_check", line 77, in
run_local_actions
    (status, message, data) = run_action(method, params)
  File "/opt/redhat/rhn/solaris/usr/sbin/rhn_check", line 137, in run_action
    (status, message, data) = rhn.actions.do_call(method, params)
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/actions/__init__.py",
line 18, in do_call
    return apply(func, params)
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/actions/solarispkgs.py",
line 189, in checkNeedUpdate
    return refresh_list()
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/actions/solarispkgs.py",
line 197, in refresh_list
    exitdata = iface.run(command)
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/interfaces/up2date/interface.py",
line 159, in run
    pkglist = self.getPackages()
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/interfaces/up2date/interface.py",
line 50, in getPackages
    if reload: self._ctrl.reloadChannels()
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/control.py",
line 351, in reloadChannels
    if not channel.fetch(self._fetcher, progress):
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/channels/solaris_rhn.py",
line 51, in fetch
    pkgs = rhnPackages.listAllAvailablePackagesComplete(channelList)
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/client/rhnPackages.py",
line 80, in listAllAvailablePackagesComplete
    tmplist = s.listAllPackagesComplete(channelName, channelVersion)
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/rpclib.py",
line 563, in __call__
    result = self._send(self._name, args)
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/rpclib.py",
line 308, in _request
    verbose=self._verbose
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/transports.py",
line 168, in request
    headers, fd = req.send_http(host, handler)
  File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/transports.py",
line 702, in send_http
    response.status, response.reason, response.msg)
xmlrpclib.ProtocolError: <ProtocolError for server
/XMLRPC/$RHN/solaris-10-sparc-patches/listAllPackagesComplete/20120704082526:
401 Authorization Required>

2 errors can be seen:
First: D: Sending back response (102, 'Patch cluster install failed: Bad
magic number for central directory', {'version': 0, 'name':
'solarispkgs.patchClusterInstall'})
and: response.status, response.reason, response.msg)
xmlrpclib.ProtocolError: <ProtocolError for server
/XMLRPC/$RHN/solaris-10-sparc-patches/listAllPackagesComplete/20120704082526:
401 Authorization Required>


Version-Release number of selected component (if applicable):
1.7

How reproducible:
rhnpsuh latest patch_cluster available from Oracle and schedule deployment on the client, from a spacewalk 1.7 server

Steps to Reproduce:
1.run solaris2mpm on 10_recommended.zip
2.rhnpush all mpm files
3.schedule patch cluster deployment
  
Actual results:
Deployment fails with error: Patch cluster install failed: Bad
magic number for central directory

Expected results:
patch cluster deployed

Additional info:
Comment 1 pierre.casenove 2012-08-02 05:38:26 EDT
I've traced python calls to identify the exact issue:
In pm.py, the unzip command is called:
pm.py(62):                 cmdstr = "unzip -u %s -d %s" % (path, tdir)
pm.py(63):                 ret, x = commands.getstatusoutput(cmdstr)

This call returns correctly.
Then, still in pm.py, a ZipFile instance is created:
pm.py(65):                 if ret != 0:
pm.py(69):                 zf = zipfile.ZipFile(path)

And this call fails, as ZIP64 extensions are missing:
--- modulename: zipfile, funcname: __init__
zipfile.py(184):         if compression == ZIP_STORED:
zipfile.py(185):             pass
zipfile.py(192):         self.debug = 0  # Level of printing: 0 through 3
zipfile.py(193):         self.NameToInfo = {}    # Find file info given name
zipfile.py(194):         self.filelist = []      # List of ZipInfo instances for archive
zipfile.py(195):         self.compression = compression  # Method of compression
zipfile.py(196):         self.mode = key = mode.replace('b', '')[0]
zipfile.py(199):         if isinstance(file, basestring):
zipfile.py(200):             self._filePassed = 0
zipfile.py(201):             self.filename = file
zipfile.py(202):             modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'}
zipfile.py(203):             self.fp = open(file, modeDict[mode])
zipfile.py(209):         if key == 'r':
zipfile.py(210):             self._GetContents()
 --- modulename: zipfile, funcname: _GetContents
zipfile.py(229):         try:
zipfile.py(230):             self._RealGetContents()
 --- modulename: zipfile, funcname: _RealGetContents
zipfile.py(239):         fp = self.fp
zipfile.py(240):         endrec = _EndRecData(fp)
 --- modulename: zipfile, funcname: _EndRecData
zipfile.py(83):     fpin.seek(-22, 2)               # Assume no archive comment.
zipfile.py(84):     filesize = fpin.tell() + 22     # Get file size
zipfile.py(85):     data = fpin.read()
zipfile.py(86):     if data[0:4] == stringEndArchive and data[-2:] == "\000\000":
zipfile.py(87):         endrec = struct.unpack(structEndArchive, data)
zipfile.py(88):         endrec = list(endrec)
zipfile.py(89):         endrec.append("")               # Append the archive comment
zipfile.py(90):         endrec.append(filesize - 22)    # Append the record start offset
zipfile.py(91):         return endrec
zipfile.py(241):         if not endrec:
zipfile.py(243):         if self.debug > 1:
zipfile.py(245):         size_cd = endrec[5]             # bytes in central directory
zipfile.py(246):         offset_cd = endrec[6]   # offset of central directory
zipfile.py(247):         self.comment = endrec[8]        # archive comment
zipfile.py(249):         x = endrec[9] - size_cd
zipfile.py(251):         concat = x - offset_cd
zipfile.py(252):         if self.debug > 2:
zipfile.py(255):         self.start_dir = offset_cd + concat
zipfile.py(256):         fp.seek(self.start_dir, 0)
zipfile.py(257):         total = 0
zipfile.py(258):         while total < size_cd:
zipfile.py(259):             centdir = fp.read(46)
zipfile.py(260):             total = total + 46
zipfile.py(261):             if centdir[0:4] != stringCentralDir:
zipfile.py(262):                 raise BadZipfile, "Bad magic number for central directory"


I attach the part of the trace showing the error.
Comment 2 pierre.casenove 2012-08-02 05:39:05 EDT
Created attachment 601912 [details]
Python trace when opening the zip file
Comment 3 pierre.casenove 2012-08-02 05:52:16 EDT
One last comment, looking at pm.py code, I think the problem can be adressed in two ways.
The problem is in these lines:

cmdstr = "unzip -u %s -d %s" % (path, tdir)
ret, x = commands.getstatusoutput(cmdstr)

if ret != 0:
   raise UnzipException("patch %s: unzip of %s into %s failed: %s" % \
          pkg.name, path, tdir, x))
zf = zipfile.ZipFile(path)
pd = zf.namelist()[0].split('/')[0]
zf.close()
pkgdir = os.path.join(tdir, pd)

If I understand well, the ZipFile is only created to list the patches included in the patch cluster.
One note: this is time consuming! Just after launching unzip command, the code recreates zipfile object from a 2GB file!

In my point of view, it can be adressed in two ways:
1) upgrade to python 2.6 in /opt/redhat, and remove unzip command to use python 2.6 ZipFile.extract command, and then ZipFile.namelist()
2) using os module, it should be possible to create the pkglist

I cn test any patch to test the correction on my solaris test boxes
Comment 4 Jan Pazdziora 2012-11-19 12:34:37 EST
If the unzip extracts the content of the "path" zipfile to "tdir" directory, can't we get the list of the patches (the "pd") directly from the directory? Or perhaps do another call to unzip to do it for us?
Comment 5 pierre.casenove 2012-11-19 12:47:57 EST
This was my second proposition in comment number 3.
You've set the status to NEEDINFO. What information do you need from me?
All I can say is that when the client crashes, the complete archive has been extracted to tdir. So it should be possible to get a list of patches, without using ZipFile.
Comment 6 Jan Pazdziora 2012-11-19 13:15:01 EST
Well yes, I kinda asked what we can do with just unzip and that extracted directory. I don't have Solaris around so I'm just assuming here, looking at the code. Is the only purpose of that zf to get the name of the sole directory which should be in that zip?
Comment 7 pierre.casenove 2012-11-20 03:40:27 EST
OK, so I've been looking around a bit more.
ZipFile objects are used twice:
- in pm.py, the zf object is created only to get the name of the directory
- in loader.py, in function getPatchData, which is called only when a "patch" is detected

So, I've been doing some more tests.
1) I've extracted using unzip command the 10_recommended.zip file (the patch cluster so)
unzip -d /data/recom/ /tmp/10_Recommended.zip
ls -l /data/recom
drwxr-xr-x 3 casenovp casenovp 4096 Apr 13  2012 10_Recommended

2) I've been executing a sample python script with only this code:
zf = zipfile.ZipFile("/tmp/10_Recommended.zip")
pd = zf.namelist()[0].split('/')[0]
print pd
zf.close()

Which gives: 10_Recommended

This is equivalent to 
pd= os.listdir(tdir)
print pd[0]

If you are okay with this, I can modify pm.py on my setup to call os.listdir instead of creating the ZipFile object and try to deploy the patch cluster.
Comment 8 Jan Pazdziora 2012-11-20 03:57:55 EST
Nod, this is exactly what I had in mind.

At the same time, the

                if not os.path.isdir(pkgdir):
                    raise UnzipException( \
                        "patch %s contained more than 1 directory: %s" % \
                            (pkg.name, pkgdir))

part seems suspicious as well -- the patch contains more than one
directory exactly if the os.listdir returns multiple directory
entries. So it could benefit from the full os.listdir list too.
Comment 9 pierre.casenove 2012-11-20 04:06:53 EST
I'll modify the client code and get back to you ASAP (not before tomorrow though).
But I'm not sure I understood exactly your last point: do you want to modify the python code so that if os.listdir returns more than 1 directory, the exception is raised?
I can do that as well.
Comment 10 Jan Pazdziora 2012-11-20 04:27:20 EST
(In reply to comment #9)
> But I'm not sure I understood exactly your last point: do you want to modify
> the python code so that if os.listdir returns more than 1 directory, the
> exception is raised?

Yes, that's what that exception seems to be about.
Comment 11 pierre.casenove 2012-11-21 07:41:32 EST
I've implemented the corrections listed above:
pd= os.listdir(tdir)
pkgdir = os.path.join(tdir, pd)
if len(pd) > 1:
    raise UnzipException( \
           "patch %s contained more than 1 directory: %s" % \
                (pkg.name, pkgdir))

I attach the patch to the post.

And now, I get the following error:

D: Sending back response (102, "Patch cluster install failed: 'list' object has no attribute 'startswith'", {'version': 0, 'name': 'solarispkgs.patchClusterInstall'})

The unzip command went well and the process crashed further. I need to trace python once again to track down the issue!
Comment 12 pierre.casenove 2012-11-21 07:42:21 EST
Created attachment 649197 [details]
Patch to avoid using ZipFile
Comment 13 pierre.casenove 2012-11-22 02:09:20 EST
My patch was buggy in fact.
I attach the correct patch.
Could you regenerate the Solaris packages and the tar files at http://spacewalk.redhat.com/solaris/ ?
Comment 14 pierre.casenove 2012-11-22 02:09:53 EST
Created attachment 649550 [details]
Correct patch to avoid using ZipFile
Comment 15 Jan Pazdziora 2012-11-23 02:49:35 EST
(In reply to comment #14)
> Created attachment 649550 [details]
> Correct patch to avoid using ZipFile

Committed to Spacewalk master, 23b4dc4de7760f2ecda937cd08b3283f4c12064b.
Comment 16 Stephen Herr 2013-03-01 12:06:54 EST
Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.
Comment 17 pierre.casenove 2013-03-05 15:27:19 EST
The patch is in spacewalk master, but the issue is in the spacewalk Solaris clients.
The packages available for download on http://spacewalk.redhat.com/solaris/ have not been regenerated.
So the BZ is not completely corrected, until the .pkg files gets updated
Comment 18 Stephen Herr 2013-03-06 13:34:16 EST
Spacewalk 1.9 has been released.

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19