Description of problem: Using solaris2mpm on a RHEL 6 box, I've generated the mpm files associated with the latest patch cluster for Solaris 10. This patch cluster is a zip file requiring ZIP64 extension. I've rhnpushed these mpm files on my spacewalk server, version 1.7, postgresql 8.4, RHEL 5. When scheduling the patch cluster deployment on a SOLARIS 10 SPARC client, here is the output of rhn_check: D: do_call solarispkgs.patchClusterInstall ([[['patch-cluster-solaris-', '20120412', '1', 'sparc-solaris-patch-cluster', 'solaris-10-sparc-patches'], {}]],) Installing patch cluster: [[['patch-cluster-solaris-', '20120412', '1', 'sparc-solaris-patch-cluster', 'solaris-10-sparc-patches'], {}]] Loading cache... Updating cache... ##################################################################### [100%] Updating cache... ##################################################################### [100%] Updating cache... ##################################################################### [100%] Computing transaction... Committing transaction... D: Sending back response (102, 'Patch cluster install failed: Bad magic number for central directory', {'version': 0, 'name': 'solarispkgs.patchClusterInstall'}) D: do_call solarispkgs.checkNeedUpdate ('rhnsd=1',) Doing checkNeedUpdate Traceback (most recent call last): File "/opt/redhat/rhn/solaris/usr/sbin/rhn_check", line 317, in ? run_local_actions() File "/opt/redhat/rhn/solaris/usr/sbin/rhn_check", line 77, in run_local_actions (status, message, data) = run_action(method, params) File "/opt/redhat/rhn/solaris/usr/sbin/rhn_check", line 137, in run_action (status, message, data) = rhn.actions.do_call(method, params) File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/actions/__init__.py", line 18, in do_call return apply(func, params) File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/actions/solarispkgs.py", line 189, in checkNeedUpdate return refresh_list() File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/actions/solarispkgs.py", line 197, in refresh_list exitdata = iface.run(command) File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/interfaces/up2date/interface.py", line 159, in run pkglist = self.getPackages() File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/interfaces/up2date/interface.py", line 50, in getPackages if reload: self._ctrl.reloadChannels() File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/control.py", line 351, in reloadChannels if not channel.fetch(self._fetcher, progress): File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/smart/channels/solaris_rhn.py", line 51, in fetch pkgs = rhnPackages.listAllAvailablePackagesComplete(channelList) File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/client/rhnPackages.py", line 80, in listAllAvailablePackagesComplete tmplist = s.listAllPackagesComplete(channelName, channelVersion) File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/rpclib.py", line 563, in __call__ result = self._send(self._name, args) File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/rpclib.py", line 308, in _request verbose=self._verbose File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/transports.py", line 168, in request headers, fd = req.send_http(host, handler) File "/opt/redhat/rhn/solaris/lib/python2.4/site-packages/rhn/transports.py", line 702, in send_http response.status, response.reason, response.msg) xmlrpclib.ProtocolError: <ProtocolError for server /XMLRPC/$RHN/solaris-10-sparc-patches/listAllPackagesComplete/20120704082526: 401 Authorization Required> 2 errors can be seen: First: D: Sending back response (102, 'Patch cluster install failed: Bad magic number for central directory', {'version': 0, 'name': 'solarispkgs.patchClusterInstall'}) and: response.status, response.reason, response.msg) xmlrpclib.ProtocolError: <ProtocolError for server /XMLRPC/$RHN/solaris-10-sparc-patches/listAllPackagesComplete/20120704082526: 401 Authorization Required> Version-Release number of selected component (if applicable): 1.7 How reproducible: rhnpsuh latest patch_cluster available from Oracle and schedule deployment on the client, from a spacewalk 1.7 server Steps to Reproduce: 1.run solaris2mpm on 10_recommended.zip 2.rhnpush all mpm files 3.schedule patch cluster deployment Actual results: Deployment fails with error: Patch cluster install failed: Bad magic number for central directory Expected results: patch cluster deployed Additional info:
I've traced python calls to identify the exact issue: In pm.py, the unzip command is called: pm.py(62): cmdstr = "unzip -u %s -d %s" % (path, tdir) pm.py(63): ret, x = commands.getstatusoutput(cmdstr) This call returns correctly. Then, still in pm.py, a ZipFile instance is created: pm.py(65): if ret != 0: pm.py(69): zf = zipfile.ZipFile(path) And this call fails, as ZIP64 extensions are missing: --- modulename: zipfile, funcname: __init__ zipfile.py(184): if compression == ZIP_STORED: zipfile.py(185): pass zipfile.py(192): self.debug = 0 # Level of printing: 0 through 3 zipfile.py(193): self.NameToInfo = {} # Find file info given name zipfile.py(194): self.filelist = [] # List of ZipInfo instances for archive zipfile.py(195): self.compression = compression # Method of compression zipfile.py(196): self.mode = key = mode.replace('b', '')[0] zipfile.py(199): if isinstance(file, basestring): zipfile.py(200): self._filePassed = 0 zipfile.py(201): self.filename = file zipfile.py(202): modeDict = {'r' : 'rb', 'w': 'wb', 'a' : 'r+b'} zipfile.py(203): self.fp = open(file, modeDict[mode]) zipfile.py(209): if key == 'r': zipfile.py(210): self._GetContents() --- modulename: zipfile, funcname: _GetContents zipfile.py(229): try: zipfile.py(230): self._RealGetContents() --- modulename: zipfile, funcname: _RealGetContents zipfile.py(239): fp = self.fp zipfile.py(240): endrec = _EndRecData(fp) --- modulename: zipfile, funcname: _EndRecData zipfile.py(83): fpin.seek(-22, 2) # Assume no archive comment. zipfile.py(84): filesize = fpin.tell() + 22 # Get file size zipfile.py(85): data = fpin.read() zipfile.py(86): if data[0:4] == stringEndArchive and data[-2:] == "\000\000": zipfile.py(87): endrec = struct.unpack(structEndArchive, data) zipfile.py(88): endrec = list(endrec) zipfile.py(89): endrec.append("") # Append the archive comment zipfile.py(90): endrec.append(filesize - 22) # Append the record start offset zipfile.py(91): return endrec zipfile.py(241): if not endrec: zipfile.py(243): if self.debug > 1: zipfile.py(245): size_cd = endrec[5] # bytes in central directory zipfile.py(246): offset_cd = endrec[6] # offset of central directory zipfile.py(247): self.comment = endrec[8] # archive comment zipfile.py(249): x = endrec[9] - size_cd zipfile.py(251): concat = x - offset_cd zipfile.py(252): if self.debug > 2: zipfile.py(255): self.start_dir = offset_cd + concat zipfile.py(256): fp.seek(self.start_dir, 0) zipfile.py(257): total = 0 zipfile.py(258): while total < size_cd: zipfile.py(259): centdir = fp.read(46) zipfile.py(260): total = total + 46 zipfile.py(261): if centdir[0:4] != stringCentralDir: zipfile.py(262): raise BadZipfile, "Bad magic number for central directory" I attach the part of the trace showing the error.
Created attachment 601912 [details] Python trace when opening the zip file
One last comment, looking at pm.py code, I think the problem can be adressed in two ways. The problem is in these lines: cmdstr = "unzip -u %s -d %s" % (path, tdir) ret, x = commands.getstatusoutput(cmdstr) if ret != 0: raise UnzipException("patch %s: unzip of %s into %s failed: %s" % \ pkg.name, path, tdir, x)) zf = zipfile.ZipFile(path) pd = zf.namelist()[0].split('/')[0] zf.close() pkgdir = os.path.join(tdir, pd) If I understand well, the ZipFile is only created to list the patches included in the patch cluster. One note: this is time consuming! Just after launching unzip command, the code recreates zipfile object from a 2GB file! In my point of view, it can be adressed in two ways: 1) upgrade to python 2.6 in /opt/redhat, and remove unzip command to use python 2.6 ZipFile.extract command, and then ZipFile.namelist() 2) using os module, it should be possible to create the pkglist I cn test any patch to test the correction on my solaris test boxes
If the unzip extracts the content of the "path" zipfile to "tdir" directory, can't we get the list of the patches (the "pd") directly from the directory? Or perhaps do another call to unzip to do it for us?
This was my second proposition in comment number 3. You've set the status to NEEDINFO. What information do you need from me? All I can say is that when the client crashes, the complete archive has been extracted to tdir. So it should be possible to get a list of patches, without using ZipFile.
Well yes, I kinda asked what we can do with just unzip and that extracted directory. I don't have Solaris around so I'm just assuming here, looking at the code. Is the only purpose of that zf to get the name of the sole directory which should be in that zip?
OK, so I've been looking around a bit more. ZipFile objects are used twice: - in pm.py, the zf object is created only to get the name of the directory - in loader.py, in function getPatchData, which is called only when a "patch" is detected So, I've been doing some more tests. 1) I've extracted using unzip command the 10_recommended.zip file (the patch cluster so) unzip -d /data/recom/ /tmp/10_Recommended.zip ls -l /data/recom drwxr-xr-x 3 casenovp casenovp 4096 Apr 13 2012 10_Recommended 2) I've been executing a sample python script with only this code: zf = zipfile.ZipFile("/tmp/10_Recommended.zip") pd = zf.namelist()[0].split('/')[0] print pd zf.close() Which gives: 10_Recommended This is equivalent to pd= os.listdir(tdir) print pd[0] If you are okay with this, I can modify pm.py on my setup to call os.listdir instead of creating the ZipFile object and try to deploy the patch cluster.
Nod, this is exactly what I had in mind. At the same time, the if not os.path.isdir(pkgdir): raise UnzipException( \ "patch %s contained more than 1 directory: %s" % \ (pkg.name, pkgdir)) part seems suspicious as well -- the patch contains more than one directory exactly if the os.listdir returns multiple directory entries. So it could benefit from the full os.listdir list too.
I'll modify the client code and get back to you ASAP (not before tomorrow though). But I'm not sure I understood exactly your last point: do you want to modify the python code so that if os.listdir returns more than 1 directory, the exception is raised? I can do that as well.
(In reply to comment #9) > But I'm not sure I understood exactly your last point: do you want to modify > the python code so that if os.listdir returns more than 1 directory, the > exception is raised? Yes, that's what that exception seems to be about.
I've implemented the corrections listed above: pd= os.listdir(tdir) pkgdir = os.path.join(tdir, pd) if len(pd) > 1: raise UnzipException( \ "patch %s contained more than 1 directory: %s" % \ (pkg.name, pkgdir)) I attach the patch to the post. And now, I get the following error: D: Sending back response (102, "Patch cluster install failed: 'list' object has no attribute 'startswith'", {'version': 0, 'name': 'solarispkgs.patchClusterInstall'}) The unzip command went well and the process crashed further. I need to trace python once again to track down the issue!
Created attachment 649197 [details] Patch to avoid using ZipFile
My patch was buggy in fact. I attach the correct patch. Could you regenerate the Solaris packages and the tar files at http://spacewalk.redhat.com/solaris/ ?
Created attachment 649550 [details] Correct patch to avoid using ZipFile
(In reply to comment #14) > Created attachment 649550 [details] > Correct patch to avoid using ZipFile Committed to Spacewalk master, 23b4dc4de7760f2ecda937cd08b3283f4c12064b.
Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.
The patch is in spacewalk master, but the issue is in the spacewalk Solaris clients. The packages available for download on http://spacewalk.redhat.com/solaris/ have not been regenerated. So the BZ is not completely corrected, until the .pkg files gets updated
Spacewalk 1.9 has been released. https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19