Bug 1253464

Summary: [Centos7] Provisioning to KVM on CentOS7 creates VMs as QEMU instead of KVM.
Product: [Community] Spacewalk Reporter: mriswithe
Component: WebUIAssignee: Jan Dobes <jdobes>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.3CC: jdobes, jhutar, tlestach
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: spacewalk-koan-2.4.2-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-08 13:26:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1267654    
Attachments:
Description Flags
KVM host sos report
none
spacewalk server sos report
none
spacewalk server debug (from spacewalk application) none

Description mriswithe 2015-08-13 18:16:31 UTC
Description of problem:
*** Note this is on CentOS7 on all involved machines ***

Running current releases of CentOS7 there are a couple problems when provisioning to a registered KVM host.

1. You must be running virt-install-0.10.0-20.el7, if you upgrade to the current, virt-install-1.1.0-12.el7.noarch, Koan fails no matter what you do. I am opening a separate bug for that issue, as it is separate, though relevant. 

2. When you attempt to provision a VM from the spacewalk web GUI, it will create the VM as QEMU even if your host is capable of KVM and it is set to use KVM in the cobbler config, and spacewalk config. 

Version-Release number of selected component (if applicable):
--Spacewalk Server--
spacewalk-backend-sql-2.3.52-1.el7.noarch
spacewalk-backend-config-files-tool-2.3.52-1.el7.noarch
spacewalk-base-minimal-2.3.53-1.el7.noarch
spacewalk-java-postgresql-2.3.179-1.el7.noarch
spacewalk-search-2.3.7-1.el7.noarch
spacewalk-setup-2.3.14-1.el7.noarch
spacewalk-backend-xml-export-libs-2.3.52-1.el7.noarch
spacewalk-backend-applet-2.3.52-1.el7.noarch
spacewalk-setup-postgresql-2.3.1-1.el7.noarch
spacewalk-backend-libs-2.3.52-1.el7.noarch
spacewalk-config-2.3.17-1.el7.noarch
spacewalk-backend-sql-postgresql-2.3.52-1.el7.noarch
spacewalk-backend-server-2.3.52-1.el7.noarch
spacewalk-backend-xmlrpc-2.3.52-1.el7.noarch
spacewalk-backend-iss-2.3.52-1.el7.noarch
rhn-org-httpd-ssl-key-pair-spacewalk.internal-1.0-1.noarch
spacewalk-admin-2.3.4-1.el7.noarch
spacewalk-setup-jabberd-2.3.2-1.el7.noarch
spacewalk-selinux-2.3.2-1.el7.noarch
spacewalk-taskomatic-2.3.179-1.el7.noarch
spacewalk-backend-app-2.3.52-1.el7.noarch
spacewalk-backend-iss-export-2.3.52-1.el7.noarch
spacewalk-certs-tools-2.3.3-1.el7.noarch
spacewalk-base-2.3.53-1.el7.noarch
spacewalk-html-2.3.53-1.el7.noarch
spacewalk-jpp-workaround-2.3.2-1.el7.noarch
spacewalk-backend-tools-2.3.52-1.el7.noarch
spacewalk-common-2.3.4-1.el7.noarch
spacewalk-reports-2.3.7-1.el7.noarch
spacewalk-java-config-2.3.179-1.el7.noarch
spacewalk-java-lib-2.3.179-1.el7.noarch
spacewalk-doc-indexes-2.3.3-1.el7.noarch
spacewalk-schema-2.3.44-1.el7.noarch
spacewalk-postgresql-2.3.4-1.el7.noarch
spacewalk-backend-config-files-common-2.3.52-1.el7.noarch
spacewalk-backend-package-push-server-2.3.52-1.el7.noarch
spacewalk-branding-2.3.25-1.el7.noarch
spacewalk-repo-2.3-4.el7.noarch
spacewalk-utils-2.3.37-1.el7.noarch
spacewalk-java-2.3.179-1.el7.noarch
spacewalk-backend-2.3.52-1.el7.noarch
spacewalk-backend-config-files-2.3.52-1.el7.noarch
spacewalk-base-minimal-config-2.3.53-1.el7.noarch

cobbler2-2.0.11-42.el7.noarch
cobbler-loaders-1.0.3-1.el7.noarch
cobbler20-2.0.11-42.el7.noarch


---KVM Host---
rhn-virtualization-common-5.4.55-1.el7.noarch
libvirt-daemon-driver-network-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-nodedev-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-lxc-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-storage-1.2.8-16.el7_1.3.x86_64
libvirt-1.2.8-16.el7_1.3.x86_64
libvirt-glib-0.1.7-3.el7.x86_64
virt-what-1.13-5.el7.x86_64
virt-manager-common-0.10.0-20.el7.noarch
libvirt-python-1.2.8-7.el7_1.1.x86_64
rhn-virtualization-host-5.4.55-1.el7.noarch
libvirt-daemon-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-nwfilter-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-interface-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-secret-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-config-network-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-kvm-1.2.8-16.el7_1.3.x86_64
perl-Sys-Virt-1.2.8-6.el7.x86_64
virt-install-0.10.0-20.el7.noarch
libvirt-client-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-driver-qemu-1.2.8-16.el7_1.3.x86_64
libvirt-daemon-config-nwfilter-1.2.8-16.el7_1.3.x86_64
spacewalk-koan-2.3.2-1.el7.noarch
koan-2.6.9-1.el7.noarch


How reproducible: 100%


Steps to Reproduce:
1. Spacewalk 2.3 server on CentOS7
2. KVM host on CentOS7 running virt-install-0.10.0-20.el7
3. Register KVM host to Spacewalk server in CentOS7 channels
4. Sync kickstart for CentOS7 base channel
5. Create distribution for Centos7 from official CentOS channels (as created by spacewalk-common-channels). Choose Installer Generation RHEL7
6. Create Kickstart profile from official CentOS7 channels, choose virtualization type "KVM Virtualized guest"
7. Select your system that has either virtualization or virtualization platform entitlements
8. Go to Virtualizaton -> Provisioning and choose your kickstart image
9. Choose a name and click schedule kickstart and finish.
10. On KVM host, either after running rhn_check or letting OSAD get the update, see the virtual machine is created either with virsh or virt-manager. 
11. Check the hardware information and see it is specifically QEMU, not KVM.
12. Kickstart/install time with no additional packages is roughly 95 minutes, versus if a KVM VM is created instead (by hand) and just PXE booted, it takes around 10 minutes. 



Actual results:
VM is created with the --virt-type qemu. Exact command given by Koan/cobbler/spacewalk can be seen by running rhn_check -v on the host (instead of using OSAD). The important part is the --virt-type qemu. 

# rhn_check -v
- looking for Cobbler at http://spacewalk.internal.mriswithe.com:443/cobbler_api
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/koan/utils.py", line 570, in __try_connect
    xmlrpc_server.ping()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1312, in single_request
    response.msg,
ProtocolError: <ProtocolError for spacewalk.internal.mriswithe.com:443/cobbler_api: 400 Bad Request>
- looking for Cobbler at https://spacewalk.internal.mriswithe.com:443/cobbler_api
- reading URL: http://spacewalk.internal.mriswithe.com/cblr/svc/op/ks/system/node2.internal.mriswithe.com:1:bugzilla_test
install_tree: http://spacewalk.internal.mriswithe.com/ty/F5b7yA17
- ['/usr/bin/virt-install', '--version']
- ['virt-install', '--version']
- ['virt-install', '--os-variant', 'list']
- ['osinfo-query', 'os']
- ['qemu-img', 'create', '-f', 'raw', '/var/lib/libvirt/images/bugzilla_test', '5G']
Formatting '/var/lib/libvirt/images/bugzilla_test', fmt=raw size=5368709120 
- adding disk: /var/lib/libvirt/images/bugzilla_test of size 5 (driver type=raw)
- ['virt-install', '--connect', 'qemu:///system', '--name', 'bugzilla_test', '--ram', '1024', '--vcpus', '1', '--autostart', '--vnc', '--virt-type', 'qemu', '--machine', 'pc', '--hvm', '--location', 'http://spacewalk.internal.mriswithe.com/ty/F5b7yA17/', '--extra-args=ks=http://spacewalk.internal.mriswithe.com/cblr/svc/op/ks/system/node2.internal.mriswithe.com:1:bugzilla_test ksdevice=link kssendmac lang= text ', '--arch', 'x86_64', '--os-variant', 'rhel7', '--disk', 'path=/var/lib/libvirt/images/bugzilla_test,size=5,format=raw', '--network', 'bridge=bridge0.300,mac=00:16:3e:13:44:e4', '--wait', '0', '--noautoconsole']

[root@node2 ~]# virsh dumpxml bugzilla_test | grep domain\ type
<domain type='qemu' id='2'>

It looks like virt-install at current version, virt-install-1.1.0-12.el7.noarch, will interpret --virt-type qemu as QEMU as well:

[root@node1 kvm]# rpm -q virt-install
virt-install-1.1.0-12.el7.noarch

[root@node1 kvm]# virt-install --connect qemu:///system  --ram 1024 --vcpus 1 --autostart --virt-type qemu --hvm --pxe --arch x86_64 --os-variant rhel7 --disk size=5 --network bridge=bridge0.300 --wait 0 --noautoconsole --name cli_test2

Starting install...
Allocating 'cli_test2.qcow2'                                                                                                                                                                                          | 5.0 GB  00:00:00     
Creating domain...                                                                                                                                                                                                    |    0 B  00:00:00     
Domain installation still in progress. You can reconnect to 
the console to complete the installation process.

[root@node1 kvm]# virsh dumpxml cli_test2 | grep domain\ type
<domain type='qemu'>


Expected results:
VM created using KVM full virtualization instead of QEMU. 

[root@node2 ~]# virsh dumpxml testthepxe | grep domain\ type
<domain type='kvm'>


Additional info:
Also of note is that the command specifies RAW format for the disk, and like the KVM/qemu command output, I don't see a way to fix/change this option in the GUI or any config files. 

CentOS6 running python-virtinst interprets the virt-install command the same way it appears.

[root@cent6-virtinstall ~]# rpm -q python-virtinst
python-virtinst-0.600.0-29.el6.noarch

[root@cent6-virtinstall ~]# virt-install --connect qemu:///system  --ram 1024 --vcpus 1 --autostart --virt-type qemu --machine pc --hvm  --location http://spacewalk.internal.mriswithe.com/ks/dist/org/1/centos7 --extra-args="ks=http://172.16.30.3/cblr/svc/op/ks/profile/CentOS_7-KVM:1:SpacewalkDefaultOrganization ksdevice=link kssendmac lang= text"  --arch x86_64 --os-variant rhel7 --disk path=/var/lib/libvirt/images/cli_test,size=5 --network bridge=virbr0 --wait 0 --noautoconsole --name cli_test -d
Thu, 13 Aug 2015 18:10:21 DEBUG    Launched with command line:
/usr/sbin/virt-install --connect qemu:///system --ram 1024 --vcpus 1 --autostart --virt-type qemu --machine pc --hvm --location http://spacewalk.internal.mriswithe.com/ks/dist/org/1/centos7 --extra-args=ks=http://172.16.30.3/cblr/svc/op/ks/profile/CentOS_7-KVM:1:SpacewalkDefaultOrganization ksdevice=link kssendmac lang= text --arch x86_64 --os-variant rhel7 --disk path=/var/lib/libvirt/images/cli_test,size=5 --network bridge=virbr0 --wait 0 --noautoconsole --name cli_test -d
Thu, 13 Aug 2015 18:10:21 DEBUG    Requesting libvirt URI qemu:///system
Thu, 13 Aug 2015 18:10:21 DEBUG    Received libvirt URI qemu:///system
Thu, 13 Aug 2015 18:10:21 DEBUG    Requesting virt method 'hvm', hv type 'qemu'.
Thu, 13 Aug 2015 18:10:21 DEBUG    Received virt method 'hvm'
Thu, 13 Aug 2015 18:10:21 DEBUG    Hypervisor name is 'qemu'

Comment 1 mriswithe 2015-08-13 18:35:06 UTC
Created attachment 1062779 [details]
KVM host sos report

Comment 2 mriswithe 2015-08-13 18:36:11 UTC
Created attachment 1062780 [details]
spacewalk server sos report

Comment 3 mriswithe 2015-08-13 18:37:03 UTC
Created attachment 1062781 [details]
spacewalk server debug (from spacewalk application)

Comment 4 Jan Hutaƙ 2015-08-25 10:25:44 UTC
I can see the issue as well on SW nightly (as of 2015-08-24) and official RHEL7 channel.


# rhn_check -vv
D: check_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>packages.update</methodName>\n<params>\n<param>\n<value><array><data>\n<value><array><data>\n<value><string>spacewalk-koan</string></value>\n<value><string>2.3.0</string></value>\n<value><string>2.el6sat</string></value>\n<value><string></string></value>\n<value><string>noarch</string></value>\n</data></array></value>\n</data></array></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 14}
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1440494913.3 with expiration of 1440498513.3 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010000, 'X-RHN-Auth-Server-Time': '1440494913.27', 'X-RHN-Auth': 'QVrt7TccC8Hp7cmjXQV0hoP4U8OWQ2mGLPo/z2UVURw=', 'X-RHN-Auth-Channels': [['rhel-x86_64-server-6', '20150824165909', '1', '1'], ['rhn-tools-rhel-x86_64-server-6', '20150824213929', '0', '1']], 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
D: handle_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>packages.update</methodName>\n<params>\n<param>\n<value><array><data>\n<value><array><data>\n<value><string>spacewalk-koan</string></value>\n<value><string>2.3.0</string></value>\n<value><string>2.el6sat</string></value>\n<value><string></string></value>\n<value><string>noarch</string></value>\n</data></array></value>\n</data></array></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 14}
D: handle_action actionid = 14, version = 2
D: do_call packages.update([['spacewalk-koan', '2.3.0', '2.el6sat', '', 'noarch']],){'cache_only': None}
Loaded plugins: product-id, rhnplugin
D: rpcServer: Calling XMLRPC up2date.listChannels
This system is receiving updates from RHN Classic or Red Hat Satellite.
D: Called update[['spacewalk-koan', '2.3.0', '2.el6sat', '', 'noarch']]
D: More recent version of package spacewalk-koan-0:2.3.0-2.el6sat.noarch is already installed
D: Sending back response(0, 'Requested packages already installed', {})
D: check_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>kickstart_guest.initiate</methodName>\n<params>\n<param>\n<value><string>[fqdn]</string></value>\n</param>\n<param>\n<value><string>[fqdn]:1:jhutar2</string></value>\n</param>\n<param>\n<value><string>qemu</string></value>\n</param>\n<param>\n<value><int>4</int></value>\n</param>\n<param>\n<value><string>jhutar2</string></value>\n</param>\n<param>\n<value><int>1024</int></value>\n</param>\n<param>\n<value><int>1</int></value>\n</param>\n<param>\n<value><int>3</int></value>\n</param>\n<param>\n<value><string>virbr0</string></value>\n</param>\n<param>\n<value><string>/var/lib/libvirt/images/jhutar2</string></value>\n</param>\n<param>\n<value><string> </string></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 15}
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1440494914.34 with expiration of 1440498514.34 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010000, 'X-RHN-Auth-Server-Time': '1440494914.32', 'X-RHN-Auth': 'W7nH+WNRjBoM0VeZsd2VYTGCMogigTslZuRI4dXurLI=', 'X-RHN-Auth-Channels': [['rhel-x86_64-server-6', '20150824165909', '1', '1'], ['rhn-tools-rhel-x86_64-server-6', '20150824213929', '0', '1']], 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
D: handle_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>kickstart_guest.initiate</methodName>\n<params>\n<param>\n<value><string>[fqdn]</string></value>\n</param>\n<param>\n<value><string>[fqdn]:1:jhutar2</string></value>\n</param>\n<param>\n<value><string>qemu</string></value>\n</param>\n<param>\n<value><int>4</int></value>\n</param>\n<param>\n<value><string>jhutar2</string></value>\n</param>\n<param>\n<value><int>1024</int></value>\n</param>\n<param>\n<value><int>1</int></value>\n</param>\n<param>\n<value><int>3</int></value>\n</param>\n<param>\n<value><string>virbr0</string></value>\n</param>\n<param>\n<value><string>/var/lib/libvirt/images/jhutar2</string></value>\n</param>\n<param>\n<value><string> </string></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 15}
D: handle_action actionid = 15, version = 2
D: do_call kickstart_guest.initiate('[fqdn]', '[fqdn]:1:jhutar2', 'qemu', 4, 'jhutar2', 1024, 1, 3, 'virbr0', '/var/lib/libvirt/images/jhutar2', ' '){'cache_only': None}
- looking for Cobbler at http://[fqdn]:443/cobbler_api
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/koan/utils.py", line 570, in __try_connect
    xmlrpc_server.ping()
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1387, in _parse_response
    p.feed(response)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 601, in feed
    self._parser.Parse(data, 0)
ExpatError: syntax error: line 1, column 49
- looking for Cobbler at https://[fqdn]:443/cobbler_api
- reading URL: http://[fqdn]/cblr/svc/op/ks/system/[fqdn]:1:jhutar2
install_tree: http://[fqdn]/ty/Z3QLHVt1
- ['/usr/bin/virt-install', '--version']
libvirtd (pid  7636) is running...
- ['virt-install', '--version']
- ['qemu-img', 'create', '-f', 'raw', '/var/lib/libvirt/images/jhutar2', '3G']
Formatting '/var/lib/libvirt/images/jhutar2', fmt=raw size=3221225472 
- adding disk: /var/lib/libvirt/images/jhutar2 of size 3 (driver type=raw)
- ['virt-install', '--connect', 'qemu:///system', '--name', 'jhutar2', '--ram', '1024', '--vcpus', '1', '--autostart', '--vnc', '--virt-type', 'qemu', '--machine', 'pc', '--hvm', '--location', 'http://[fqdn]/ty/Z3QLHVt1/', '--extra-args=ks=http://[fqdn]/cblr/svc/op/ks/system/[fqdn]:1:jhutar2 ksdevice=link kssendmac lang= text ', '--arch', 'x86_64', '--os-variant', 'rhel7', '--disk', 'path=/var/lib/libvirt/images/jhutar2,size=3,format=raw', '--network', 'bridge=virbr0,mac=00:16:3e:48:94:77', '--wait', '0', '--noautoconsole']

Starting install...
Retrieving file .treeinfo...                                                                                                                                                                                          | 4.2 kB     00:00 ... 
Retrieving file vmlinuz...                                                                                                                                                                                            | 9.3 MB     00:00 ... 
Retrieving file initrd.img...                                                                                                                                                                                         |  68 MB     00:00 ... 
Creating domain...                                                                                                                                                                                                    |    0 B     00:00     
Domain installation still in progress. You can reconnect to 
the console to complete the installation process.
- polling for virt completion
- install is still running, sleeping for 1 minute (0)
[...]


When I have overwrote virt_type in spacewalk-koan code, it worked (i.e. I got guest with VirtIO HDD and NIC):


# diff -u /usr/share/rhn/actions/kickstart_guest.py.ORIG /usr/share/rhn/actions/kickstart_guest.py
--- /usr/share/rhn/actions/kickstart_guest.py.ORIG	2015-08-25 06:23:57.080360979 -0400
+++ /usr/share/rhn/actions/kickstart_guest.py	2015-08-25 05:33:48.749076008 -0400
@@ -43,6 +43,7 @@
     error_messages = {}
 
     log_notify_handler = KickstartLogNotifyHandler(ks_session_id)
+    virt_type = 'kvm'
 
     return  spacewalkkoan.initiate_guest(kickstart_host, cobbler_system_name,
                 virt_type, name, mem_kb, vcpus, disk_gb, virt_bridge, disk_path, extra_append, log_notify_handler)

Comment 5 Tomas Lestach 2015-08-27 15:11:56 UTC
Just tried on provision a RHEL7.1 KVM guest on a RHEL7.1 host and worked for me:

rhn_check -vvv
D: opening  db environment /var/lib/rpm cdb:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db index       /var/lib/rpm/Providename 0x400 mode=0x0
D: check_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>packages.update</methodName>\n<params>\n<param>\n<value><array><data>\n<value><array><data>\n<value><string>spacewalk-koan</string></value>\n<value><string>2.3.0</string></value>\n<value><string>2.el7sat</string></value>\n<value><string></string></value>\n<value><string>noarch</string></value>\n</data></array></value>\n</data></array></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 282}
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1440678693.63 with expiration of 1440682293.63 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010010, 'X-RHN-Auth-Server-Time': '1440678693.63', 'X-RHN-Auth': '5JiAPwRpsfLLoXHxbp6h9wXGdLjr5Ap6iipKxtdjqy4=', 'X-RHN-Auth-Channels': [['rhel-x86_64-server-7', '20150814101056', '1', '1'], ['rhn-tools-rhel-x86_64-server-7', '20150803183912', '0', '1']], 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
D: handle_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>packages.update</methodName>\n<params>\n<param>\n<value><array><data>\n<value><array><data>\n<value><string>spacewalk-koan</string></value>\n<value><string>2.3.0</string></value>\n<value><string>2.el7sat</string></value>\n<value><string></string></value>\n<value><string>noarch</string></value>\n</data></array></value>\n</data></array></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 282}
D: handle_action actionid = 282, version = 2
D: do_call packages.update([['spacewalk-koan', '2.3.0', '2.el7sat', '', 'noarch']],){'cache_only': None}
D: opening  db environment /var/lib/rpm cdb:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: opening  db index       /var/lib/rpm/Name 0x400 mode=0x0
D: added key gpg-pubkey-fd431d51-4ae0493b to keyring
D: added key gpg-pubkey-2fa658e0-45700c69 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
D: opening  db index       /var/lib/rpm/Providename 0x400 mode=0x0
D: closed   db index       /var/lib/rpm/Providename
D: closed   db index       /var/lib/rpm/Name
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
Loaded plugins: product-id, rhnplugin
Config time: 0.072
D: rpcServer: Calling XMLRPC up2date.listChannels
This system is receiving updates from RHN Classic or Red Hat Satellite.
rpmdb time: 0.000
repo time: 0.014
Setting up Package Sacks
pkgsack time: 0.042
D: Called update[['spacewalk-koan', '2.3.0', '2.el7sat', '', 'noarch']]
D: opening  db environment /var/lib/rpm cdb:0x401
D: opening  db index       /var/lib/rpm/Packages 0x400 mode=0x0
D: opening  db index       /var/lib/rpm/Name 0x400 mode=0x0
D:  read h#     695 Header sanity check: OK
D: Package spacewalk-koan-0:2.3.0-2.el7sat.noarch already installed
D: Sending back response(0, 'Requested packages already installed', {})
D: check_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>kickstart_guest.initiate</methodName>\n<params>\n<param>\n<value><string><spacewalk_fqdn></string></value>\n</param>\n<param>\n<value><string><fqdn>:1:kvm0</string></value>\n</param>\n<param>\n<value><string>qemu</string></value>\n</param>\n<param>\n<value><int>10</int></value>\n</param>\n<param>\n<value><string>kvm0</string></value>\n</param>\n<param>\n<value><int>1024</int></value>\n</param>\n<param>\n<value><int>1</int></value>\n</param>\n<param>\n<value><int>3</int></value>\n</param>\n<param>\n<value><string>virbr0</string></value>\n</param>\n<param>\n<value><string>/var/lib/libvirt/images/kvm0</string></value>\n</param>\n<param>\n<value><string> </string></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 283}
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1440678694.65 with expiration of 1440682294.65 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010010, 'X-RHN-Auth-Server-Time': '1440678694.64', 'X-RHN-Auth': 'Wbghgdl+yKmyunN1RjjoZT9LUd4XtFkTzdn0cLf4icg=', 'X-RHN-Auth-Channels': [['rhel-x86_64-server-7', '20150814101056', '1', '1'], ['rhn-tools-rhel-x86_64-server-7', '20150803183912', '0', '1']], 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
D: handle_action{'action': "<?xml version='1.0'?>\n<methodCall>\n<methodName>kickstart_guest.initiate</methodName>\n<params>\n<param>\n<value><string><spacewalk_fqdn></string></value>\n</param>\n<param>\n<value><string><fqdn>:1:kvm0</string></value>\n</param>\n<param>\n<value><string>qemu</string></value>\n</param>\n<param>\n<value><int>10</int></value>\n</param>\n<param>\n<value><string>kvm0</string></value>\n</param>\n<param>\n<value><int>1024</int></value>\n</param>\n<param>\n<value><int>1</int></value>\n</param>\n<param>\n<value><int>3</int></value>\n</param>\n<param>\n<value><string>virbr0</string></value>\n</param>\n<param>\n<value><string>/var/lib/libvirt/images/kvm0</string></value>\n</param>\n<param>\n<value><string> </string></value>\n</param>\n</params>\n</methodCall>\n", 'version': 2, 'id': 283}
D: handle_action actionid = 283, version = 2
D: do_call kickstart_guest.initiate('<spacewalk_fqdn>', '<fqdn>:1:kvm0', 'qemu', 10, 'kvm0', 1024, 1, 3, 'virbr0', '/var/lib/libvirt/images/kvm0', ' '){'cache_only': None}
- looking for Cobbler at http://<spacewalk_fqdn>/cobbler_api
- reading URL: http://<spacewalk_fqdn>/cblr/svc/op/ks/system/<fqdn>:1:kvm0
install_tree: http://<spacewalk_fqdn>/ty/2BrngZtT
- ['rpm', '-q', 'virt-install']
- ['virt-install', '--version']
- ['virt-install', '--os-variant', 'list']
- ['osinfo-query', 'os']
- adding disk: /var/lib/libvirt/images/kvm0 of size 3 (driver type=None)
- ['virt-install', '--connect', 'qemu:///system', '--name', 'kvm0', '--ram', '1024', '--vcpus', '1', '--autostart', '--vnc', '--virt-type', 'kvm', '--machine', 'pc', '--extra-args=ks=http://<spacewalk_fqdn>/cblr/svc/op/ks/system/<fqdn>:1:kvm0 ksdevice=link kssendmac lang= text ', '--location', 'http://<spacewalk_fqdn>/ty/2BrngZtT/', '--os-variant', 'rhel7.0', '--disk', 'path=/var/lib/libvirt/images/kvm0,size=3', '--network', 'bridge=virbr0,mac=00:16:3e:27:32:c8', '--wait', '0', '--noautoconsole']

Starting install...
Retrieving file vmlinuz...                               | 9.6 MB     00:00 !!! 
Retrieving file initrd.img...                            |  70 MB     00:00 !!! 
Allocating 'kvm0'                                        | 3.0 GB     00:00     
Creating domain...                                       |    0 B     00:00     
Domain installation still in progress. You can reconnect to 
the console to complete the installation process.
None
- polling for virt completion
- install is still running, sleeping for 1 minute (0)
- install is still running, sleeping for 1 minute (1)
- install is still running, sleeping for 1 minute (2)
- install is still running, sleeping for 1 minute (3)
- install is still running, sleeping for 1 minute (4)
- install is still running, sleeping for 1 minute (5)
- install is still running, sleeping for 1 minute (6)
- install is still running, sleeping for 1 minute (7)
- install is still running, sleeping for 1 minute (8)
- install is still running, sleeping for 1 minute (9)
- install is still running, sleeping for 1 minute (10)
- install is still running, sleeping for 1 minute (11)
- install is still running, sleeping for 1 minute (12)
- install is still running, sleeping for 1 minute (13)
- install is still running, sleeping for 1 minute (14)
- install is still running, sleeping for 1 minute (15)
- install is still running, sleeping for 1 minute (16)
- shutdown VM detected, is the install done?  Restarting!
D: rpcServer: Calling XMLRPC registration.virt_notify
D: Sending back response(0, 'Virtual kickstart initiate succeeded', {})
D: do_call packages.checkNeedUpdate('rhnsd=1',){}
D: Called refresh_rpmlist
Updating package profile
D: rpcServer: Calling XMLRPC registration.welcome_message
D: rpcServer: Calling XMLRPC registration.update_packages
D: local action status: (0, 'rpmlist refreshed', {})
D: rpcServer: Calling XMLRPC registration.welcome_message
D: closed   db index       /var/lib/rpm/Name
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
D: closed   db index       /var/lib/rpm/Providename
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm

I see, JanH had a rhel6 host.

Comment 6 mriswithe 2015-08-27 17:45:47 UTC
(In reply to Tomas Lestach from comment #5)

This is very interesting because I have had actually a number of people who have stood up the same environment I am run into the identical issue. So there must be something different/wrong that we are all falling into the trap on. 

One thing I need to make sure I state is that I am running CentOS7 on all hosts involved here. My Spacewalk server, and KVM host.

The main difference I see between the outputs is that both JanH and I had a Python error after this part....

Your output (working):

> D: handle_action actionid = 283, version = 2
> D: do_call kickstart_guest.initiate('<spacewalk_fqdn>', '<fqdn>:1:kvm0', 'qemu', 10, 'kvm0', 1024, 1, 3, 'virbr0', '/var/lib/libvirt/images/kvm0', ' '){'cache_only': None}
> - looking for Cobbler at http://<spacewalk_fqdn>/cobbler_api
> - reading URL: http://<spacewalk_fqdn>/cblr/svc/op/ks/system/<fqdn>:1:kvm0


Her output (Failing):
> D: handle_action actionid = 15, version = 2
> D: do_call kickstart_guest.initiate('[fqdn]', '[fqdn]:1:jhutar2', 'qemu', 4, 'jhutar2', 1024, 1, 3, 'virbr0', '/var/lib/libvirt/images/jhutar2', ' '){'cache_only': None}
> - looking for Cobbler at http://[fqdn]:443/cobbler_api
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/site-packages/koan/utils.py", line 570, in __try_connect
>     xmlrpc_server.ping()
>   File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
>     return self.__send(self.__name, args)
>   File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
>     verbose=self.__verbose
>   File "/usr/lib64/python2.6/xmlrpclib.py", line 1253, in request
>     return self._parse_response(h.getfile(), sock)
>   File "/usr/lib64/python2.6/xmlrpclib.py", line 1387, in _parse_response
>     p.feed(response)
>   File "/usr/lib64/python2.6/xmlrpclib.py", line 601, in feed
>     self._parser.Parse(data, 0)
> ExpatError: syntax error: line 1, column 49
> - looking for Cobbler at https://[fqdn]:443/cobbler_api
> - reading URL: http://[fqdn]/cblr/svc/op/ks/system/[fqdn]:1:jhutar2

My output (Failing):

> D: handle_action actionid = 305, version = 2
> D: do_call kickstart_guest.initiate('[SPACEWALK-FQDN]', '[KVM-FQDN]:1:test-spacewalk-prov', 'qemu', 36, 'test-spacewalk-prov', 1024, 1, 5, 'bridge0.300', '/var/lib/libvirt/images/test-spacewalk-prov', ' '){'cache_only': None}
> - looking for Cobbler at http://[SPACEWALK-FQDN]:443/cobbler_api
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/koan/utils.py", line 570, in __try_connect
>     xmlrpc_server.ping()
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
>     return self.__send(self.__name, args)
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
>     verbose=self.__verbose
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
>     return self.single_request(host, handler, request_body, verbose)
>   File "/usr/lib64/python2.7/xmlrpclib.py", line 1312, in single_request
>     response.msg,
> ProtocolError: <ProtocolError for [SPACEWALK-FQDN]:443/cobbler_api: 400 Bad Request>
> - looking for Cobbler at https://[SPACEWALK-FQDN]:443/cobbler_api
> - reading URL: http://[SPACEWALK-FQDN]/cblr/svc/op/ks/system/[KVM-FQDN]:1:test-spacewalk-prov


I would be interested to see what cobbler and koan versions you are using on your spacewalk server and KVM node. There is cobbler2(or 20)/cobbler-epel available.These are what I am running:

SPACEWALK# rpm -qa | egrep -i cobbler\|koan
cobbler2-2.0.11-42.el7.noarch
koan-2.6.9-1.el7.noarch
cobbler-loaders-1.0.3-1.el7.noarch
cobbler20-2.0.11-42.el7.noarch

KVM# rpm -qa | egrep -i cobbler\|koan
koan-2.6.9-1.el7.noarch
spacewalk-koan-2.3.2-1.el7.noarch

Would it be possible for you to provide a list of RPMs from your spacewalk and KVM servers? Or compare against mine from my SOS reports? I assume a list of RPMs is there. 

Also, another thought is your /etc/rhn/rhn.conf and /etc/cobbler/settings

There is something different about your environment than mine. I followed the install process listed here: https://fedorahosted.org/spacewalk/wiki/HowToInstall

following the instructions to use the postgres database on the same machine. 


Are you aware of any steps you took outside of that process ? 

Thanks,
Mriswithe

Comment 7 Tomas Lestach 2015-08-28 07:51:56 UTC
Sure,
SPACEWALK# rpm -qa | egrep -i cobbler\|koan
cobbler20-2.0.11-42.el6.noarch
cobbler2-2.0.11-42.el6.noarch

KVM# rpm -qa | egrep -i cobbler\|koan
koan-2.0.7-54.el7sat.noarch
spacewalk-koan-2.3.0-2.el7sat.noarch

So, the difference is I use koan and spacewalk-koan on the kvm host from rhn-tools-rhel-x86_64-server-7. I need to try it again with spacewalk packages.

Comment 8 Jan Dobes 2015-09-14 12:44:38 UTC
Reproduced with Spacewalk master packages + EPEL Koan. The problem is, --virt-type parameter is exactly the same value as defined in:

spaceschema=# select name, label from rhnKickstartVirtualizationType;
            name             |   label   
-----------------------------+-----------
 Para-Virtualized Host       | para_host
 XEN Fully-Virtualized Guest | xenfv
 XEN Para-Virtualized Guest  | xenpv
 None                        | none
 KVM Virtualized Guest       | qemu
(5 rows)

Comment 9 Jan Dobes 2015-09-14 13:38:30 UTC
(In reply to Tomas Lestach from comment #7)
> Sure,
> SPACEWALK# rpm -qa | egrep -i cobbler\|koan
> cobbler20-2.0.11-42.el6.noarch
> cobbler2-2.0.11-42.el6.noarch
> 
> KVM# rpm -qa | egrep -i cobbler\|koan
> koan-2.0.7-54.el7sat.noarch
> spacewalk-koan-2.3.0-2.el7sat.noarch
> 
> So, the difference is I use koan and spacewalk-koan on the kvm host from
> rhn-tools-rhel-x86_64-server-7. I need to try it again with spacewalk
> packages.

I investigated Koan code and it seems, the checking for KVM capability and changing 'qemu' to 'kvm' is not there in current version. It was there years ago and this is why it works with old Koan.

There are some solutions:

1. If it isn't intended behaviour, then propose patch for Koan and wait for their next release.
2. Fix it in our spacewalk-koan wrapper, switch to KVM if it's possible.
3. Fix it by dividing "KVM Virtualized Guest | qemu" in Spacewalk into "KVM Virtualized Guest | kvm" and "QEMU Virtualized Guest | qemu".

Comment 10 Jan Dobes 2015-09-16 11:37:36 UTC
I used variant 2, fixed in spacewalk master:

e2ecb3bb3d5c461ba9b73dc8efd44ad48ebbd3d4

Comment 11 Jan Dobes 2015-10-08 13:26:21 UTC
Spacewalk 2.4 has been released.