Bug 749667 - Regression importing Red Hat distros with 'cobbler import
Regression importing Red Hat distros with 'cobbler import
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: cobbler (Show other bugs)
16
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: James C.
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-10-27 16:31 EDT by Cole Robinson
Modified: 2012-04-21 09:18 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-04-21 09:18:11 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Fix various issues importing RH distros (3.43 KB, text/plain)
2011-10-27 16:44 EDT, Cole Robinson
no flags Details

  None (edit)
Description Cole Robinson 2011-10-27 16:31:48 EDT
The command:

cobbler import --name=xdist-rhel6-latest --path=/mnt/data/devel/trees/rhel6-latest --available-as=http://192.168.1.149/trees/rhel6-latest

No longer works for me with latest f16 cobbler. It worked with cobbler 2.0 IIRC.

The following patch fixes a few issues, pasted inline for explaination:

--- cobbler/modules/manage_import_redhat.py	2011-10-05 20:49:16.000000000 -0400
+++ /usr/lib/python2.7/site-packages/cobbler/modules/manage_import_redhat.py	2011-10-27 16:24:47.905975596 -0400
@@ -246,7 +246,7 @@ class ImportRedhatManager:
 
             if not self.network_root.endswith("/"):
                 self.network_root = self.network_root + "/"
-            self.path = os.path.normpath( self.mirror )
+            self.path = self.rootdir = os.path.normpath( self.mirror )
             valid_roots = [ "nfs://", "ftp://", "http://" ]
             for valid_root in valid_roots:
                 if self.network_root.startswith(valid_root):

rootdir needs to match self.path here, otherwise rootdir was pointing to somewhere under /var, when we need it to point to the media being imported.

@@ -273,6 +273,10 @@ class ImportRedhatManager:
             # FIXME: this automagic is not possible (yet) without mirroring
             self.repo_finder(distros_added)
 
+        if not distros_added:
+            utils.die(self.logger,
+                      "Failed to detect any distros in %s" % self.path)
+
         # find the most appropriate answer files for each profile object

         self.logger.info("associating kickstarts")

This makes 'cobbler import' appropriately fail if no distros are actually successfully detected from the passed tree.

@@ -281,11 +285,12 @@ class ImportRedhatManager:
         # ensure bootloaders are present
         self.api.pxegen.copy_bootloaders()
 
+
         return True
 
     # required function for import modules
     def get_valid_arches(self):
-        return ["i386", "ia64", "ppc", "ppc64", "s390", "s390x", "x86_64", "x86",]
+        return ["i386", "ia64", "ppc", "ppc64", "s390", "s390x", "x86_64"]
 

     # required function for import modules
     def get_valid_breeds(self):

Ignoring the accidental whitespace change, this removes x86 from the expected package arch list. AFAIK red hat has never used 'x86' in RPM package strings. It's a problem anyways, because x86 will always also match x86_64, so importing an x86_64 distro will cause problems since cobbler detects it as a multiarch tree.

self.get_entry should also be changed to never return None (which it does in 2 error paths), since callers expect a list for a return value. The offending areas should probably just use utils.die.
Comment 1 Cole Robinson 2011-10-27 16:44:47 EDT
Created attachment 530565 [details]
Fix various issues importing RH distros
Comment 2 Cole Robinson 2011-10-27 16:48:37 EDT
A couple more fixes were needed for importing an F16 tree. The first basic one is that fedora16 needs to be added to the whitelist in cobblers/codes.py

Second problem is that scanning an x86_64 tree is again detecting 2 architectures, because the kernel package scanning is picking up an i686 kernel-tools package. This patch fixes that:

@@ -835,12 +844,22 @@ class ImportRedhatManager:
         """
         Is the given filename a kernel filename?
         """
+        blacklist = ["kernel-tools"]
+        whitelist = ["kernel-header", "kernel-source", "kernel-smp",
+                     "kernel-largesmp", "kernel-hugemem", "linux-headers-",
+                     "kernel-devel", "kernel-"]
 
         if not filename.endswith("rpm") and not filename.endswith("deb"):
             return False
-        for match in ["kernel-header", "kernel-source", "kernel-smp", "kernel-largesmp", "kernel-hugemem", "linux-headers-", "kernel-devel", "kernel-"]:
+
+        for match in blacklist:
+            if filename.find(match) != -1:
+                return False
+
+        for match in whitelist:
             if filename.find(match) != -1:
                 return True
+
         return False
 
     def scan_pkg_filename(self, rpm):



Yall should probably be parsing .treeinfo files for red hat distros, it's much much easier. Would also be nice to have some logic that makes an unknown fedora value just default to the highest supported fedora OS. We do similar things in python-virtinst:

http://git.fedorahosted.org/git/?p=python-virtinst.git;a=blob_plain;f=virtinst/OSDistro.py;hb=HEAD
Comment 3 Fedora Admin XMLRPC Client 2012-04-11 18:00:03 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 4 James C. 2012-04-16 22:12:19 EDT
I believe this has been fixed in the most recent release (2.2.x). Can you please verify if this is still an issue? I have imported SL6 on a F16 system running cobbler without issue.
Comment 5 Cole Robinson 2012-04-17 11:28:04 EDT
Things don't obviously fail like previously, but now there is a very annoying change in behavior.



I have a RHEL6 tree on my local machine. I want cobbler to learn about it and advertise a kickstart for it. I don't want it to copy it to it's preferred location.

sudo cobbler import --name=xdist-rhel6-latest --path=/mnt/data/devel/trees/rhel6-latest --available-as=http://192.168.1.149/trees/rhel6-latest

Previously the above invocation was a very quick operation. With current version from updates-testing, it begins rsyncing the distro tree to /var/www/cobbler/ks_mirror/xdist-rhel6-latest which would leave 2 copies on my physical machine.

Maybe there was a regression and import just isn't taking into account --available-as ?
Comment 6 Cole Robinson 2012-04-17 12:05:09 EDT
Actually it looks like the change was deliberate (and done by you) :)

https://github.com/cobbler/cobbler/commit/494d36ff5053958a90fc71c48a5b038b6b1907b0

But the commit message provides little justification for this change. All the cobbler documentation for --available-as touted its main feature that it didn't try and mirror the distro to another location. So this change is a regression and an API break since the cli tool is the admin's API to cobbler.

IMO this change should be reverted. If you need a way to pass to specify network_root but still do the mirroring, what you want is a new cli option to force mirroring, not changing existing option semantics.

My workflow has me regularly blow away and reimport distros with cobbler (I'm a virt developer and use pxe to advertise multiple distros).
Comment 7 James C. 2012-04-17 18:36:55 EDT
Yes, that is correct. That commit was to address a chicken & egg problem with the new import modules, which have to scan files before they can figure out what distro to import. Doing the full rsync was supposed to be a stop-gap measure so that --available-as imports didn't fail outright (which was the situation before).

I will start working on correcting this situation, which is going to involve using a file containing a whitelist of just the files it needs to rsync while excluding the rest. This bug will be closed as fixed, and I'll open a new issue on github to track the fix if that works for you.
Comment 8 James C. 2012-04-17 23:39:00 EDT
Got this working for RedHat-based distros: http://pastebin.com/qSYeW973

As you can see, the downloaded data for Centos 5.8 was just 78M - much better than the full DVD (or more). I should get the rest of the distros fixed up the same way, and will aim to get fix in 2.2.3.
Comment 9 Cole Robinson 2012-04-18 08:40:15 EDT
Sounds good, thanks James. I'd also appreciate a redhat bugzilla so I can track when the issue is fixed in fedora.
Comment 10 James C. 2012-04-18 08:59:40 EDT
That's fine, I'll just leave this open as well. The github issue is https://github.com/cobbler/cobbler/issues/138
Comment 11 James C. 2012-04-21 09:18:11 EDT
Code merged into master to correct this: https://github.com/cobbler/cobbler/pull/145

If there's enough testing we may be able to get it into 2.2.3, otherwise it will be in 2.2.4.

Note You need to log in before you can comment on or make changes to this bug.