Bug 1276896

Summary: koan fails because virt-install --version reports on stderr, not stdout
Product: [Fedora] Fedora EPEL Reporter: Brian J. Murrell <brian.murrell>
Component: cobblerAssignee: James C. <jimi>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: epel7CC: awood, bruno.travouillon, extras-orphan, jeckersb, jimi, orion, scott, vanmeeuwen+fedora, yuri
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: cobbler-2.6.11-9.gitf78af86.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-26 00:19:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brian J. Murrell 2015-11-01 00:39:33 UTC
Description of problem:
When trying to use koan, it fails with an error:

expected version format differs
  File "/usr/lib/python2.7/site-packages/koan/app.py", line 274, in main
    k.run()
   File "/usr/lib/python2.7/site-packages/koan/app.py", line 424, in run
    self.virt()
   File "/usr/lib/python2.7/site-packages/koan/app.py", line 845, in virt
    return self.net_install(after_download)
   File "/usr/lib/python2.7/site-packages/koan/app.py", line 637, in net_install
    if len(version_str) == 0 and not utils.check_version_greater_or_equal(version_str.strip(), "0.2.0"):
   File "/usr/lib/python2.7/site-packages/koan/utils.py", line 592, in check_version_greater_or_equal
    raise Exception("expected version format differs")

Version-Release number of selected component (if applicable):
2.6.9-1

How reproducible:
100%

Steps to Reproduce:
1. Set up a cobbler server
2. Try to create a VM with something like "koan --virt --profile test-el6.7-x86_64 --virt-name centos6.7 --force-path --virt-path /var/lib/libvirt/images/centos6.7-disk0-tmp
3. Observe failure

Actual results:
expected version format differs
  File "/usr/lib/python2.7/site-packages/koan/app.py", line 274, in main
    k.run()
   File "/usr/lib/python2.7/site-packages/koan/app.py", line 424, in run
    self.virt()
   File "/usr/lib/python2.7/site-packages/koan/app.py", line 845, in virt
    return self.net_install(after_download)
   File "/usr/lib/python2.7/site-packages/koan/app.py", line 637, in net_install
    if len(version_str) == 0 and not utils.check_version_greater_or_equal(version_str.strip(), "0.2.0"):
   File "/usr/lib/python2.7/site-packages/koan/utils.py", line 592, in check_version_greater_or_equal
    raise Exception("expected version format differs")

Expected results:
Should provision VM

Additional info:
This is due to https://github.com/cobbler/koan/issues/8.  TL;DR: app.py needs to account for virt-install --version emitting on stderr rather than stdout the way that virtinstall.py does.

Here's a patch that fixes it:

--- /usr/lib/python2.7/site-packages/koan/app.py.dist	2015-10-31 17:29:58.039167678 -0700
+++ /usr/lib/python2.7/site-packages/koan/app.py	2015-10-31 17:31:29.656164095 -0700
@@ -631,7 +631,7 @@
                 if not os.path.exists("/usr/bin/qemu-img"):
                     raise InfoException("qemu package needs to be installed")
                 # is libvirt new enough?
-                rc, version_str = utils.subprocess_get_response(shlex.split('/usr/bin/virt-install --version'), True)
+                rc, stdout, version_str = utils.subprocess_get_response(shlex.split('/usr/bin/virt-install --version'), True, True)
                 print "version_str = %s, %s" % (version_str, utils.subprocess_get_response(shlex.split('/usr/bin/virt-install --version')))
                 version_str = version_str.strip()
                 if len(version_str) == 0 and not utils.check_version_greater_or_equal(version_str.strip(), "0.2.0"):
--- /usr/lib/python2.7/site-packages/koan/utils.py.dist	2015-10-31 17:32:44.778161157 -0700
+++ /usr/lib/python2.7/site-packages/koan/utils.py	2015-10-31 17:36:41.142151913 -0700
@@ -173,7 +173,7 @@
         raise InfoException, "command failed (%s)" % rc
     return rc
 
-def subprocess_get_response(cmd, ignore_rc=False):
+def subprocess_get_response(cmd, ignore_rc=False, get_stderr=False):
     """
     Wrapper around subprocess.check_output(...)
     """
@@ -181,8 +181,11 @@
     rc = 0
     result = ""
     if not ANCIENT_PYTHON:
-        p = sub_process.Popen(cmd, stdout=sub_process.PIPE)
-        result = p.communicate()[0]
+        if get_stderr:
+            p = sub_process.Popen(cmd, stdout=sub_process.PIPE, stderr=sub_process.PIPE)
+        else:
+            p = sub_process.Popen(cmd, stdout=sub_process.PIPE)
+        result, stderr_result = p.communicate()
         rc = p.wait()
     else:
         cmd = string.join(cmd, " ")
@@ -190,6 +193,8 @@
         rc = os.system(cmd)
     if not ignore_rc and rc != 0:
         raise InfoException, "command failed (%s)" % rc
+    if get_stderr:
+        return rc, result, stderr_result
     return rc, result
 
 def input_string_or_hash(options,delim=None,allow_multiples=True):

Understood that this needs a fix upstream, but can we please patch the existing 2.6.9 and issue a hotfix release until upstream gets a release together with this needed change as this prevents us from deploying VMs on EL 7.1.

Comment 1 Yuri Arabadji 2016-07-13 16:57:55 UTC
Can we finally push this to EPEL or whatever so Koan actually works on EL7 nodes? Tnx.

Comment 2 Orion Poplawski 2016-07-13 17:02:24 UTC
koan is part of the cobbler package.

Comment 3 Yuri Arabadji 2016-07-13 17:03:28 UTC
Thanks, cap. Does that mean we'll have to wait another year to get the fix pushed?

Comment 4 Fedora Update System 2016-07-13 18:22:39 UTC
cobbler-2.6.11-5.git13b035f.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-ed7d53c99a

Comment 5 Fedora Update System 2016-07-15 07:47:06 UTC
cobbler-2.6.11-6.git95749a6.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-ed7d53c99a

Comment 6 Brian J. Murrell 2016-07-20 19:00:53 UTC
(In reply to Fedora Update System from comment #4)
> cobbler-2.6.11-5.git13b035f.el7 has been submitted as an update to Fedora
> EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-ed7d53c99a

I tried leaving a comment on bodhi about this build but it wouldn't take it for whatever reason.  So here it is:

I think I have to give this a big -1:

    # koan --virt --profile=basic-fc24-x86_64 --virt-name fc24
    - looking for Cobbler at http://cobbler:80/cobbler_api
    - reading URL: http://cobbler/cblr/svc/op/ks/profile/basic-fc24-x86_64
    install_tree: http://cobbler/cblr/links/Fedora-24-x86_64
    - ['/usr/bin/virt-install', '--version']
    - ['virt-install', '--version']
    - ['virt-install', '--os-variant', 'list']
    ERROR    
    --name is required
    --memory amount in MiB is required
    --disk storage must be specified (override with --disk none)
    An install method must be specified
    (--location URL, --cdrom CD/ISO, --pxe, --import, --boot hd|cdrom|...)
    - ['osinfo-query', 'os']
    - warning: virt-install doesn't know this os_version, defaulting to generic26
    - adding disk: /var/lib/libvirt/images//fc24-disk0 of size 20 (driver type=raw)
    - ['virt-install', '--connect', 'qemu:///system', '--name', 'fc24', '--ram', '1024', '--vcpus', '1', '--autostart', '--vnc', '--virt-type', 'kvm', '--machine', 'pc', '--location', 'http://cobbler/cblr/links/Fedora-24-x86_64/', '--extra-args=ksdevice=link lang= console=tty0 console=ttyS0,115200 text net.ifnames=0 ks=http://cobbler/cblr/svc/op/ks/profile/basic-fc24-x86_64 kssendmac inst.sshd=1 ', '--arch', 'x86_64', '--os-variant', 'generic26', '--disk', 'path=/var/lib/libvirt/images//fc24-disk0,size=20,format=raw', '--network', 'bridge=br0', '--wait', '0', '--noautoconsole']
    ERROR    Error validating install location: Distro 'generic26' does not exist in our dictionary
    command failed (1)

While it generally fixes the particular bug, it's as much a step backward as it is forward.  As can be seen by the failure above.

We really should be striving for a koan and a virt-install that are compatible and don't complain about each other like they do above.

Comment 7 Fedora Update System 2016-07-21 18:03:16 UTC
cobbler-2.6.11-9.gitf78af86.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-ed7d53c99a

Comment 8 Orion Poplawski 2016-07-21 18:07:00 UTC
New update should fix the unknown os-variant issue.

FWIW, I tend to keep the os versions for my distros capped at the latest version explicitly supported (i.e. that osinfo-query reports), which seems to be fedora22 for EL7.2.  The default fallback of "virtio26" is an alias for "fedora10".  Not sure if there are any new features that can be taken advantage of between fedora10 and fedora22.

Comment 9 Fedora Update System 2016-07-23 00:47:57 UTC
cobbler-2.6.11-9.gitf78af86.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-ed7d53c99a

Comment 10 Fedora Update System 2017-01-26 00:19:07 UTC
cobbler-2.6.11-9.gitf78af86.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.