Bug 1098645

Summary:	Smartcard device failures prevent VM from booting with error message "Changed state to Down: internal error ifname "vnet1" not in key map"
Product:	Red Hat Enterprise Linux 6	Reporter:	Jake Hunsaker <jhunsaker>
Component:	libvirt	Assignee:	John Ferlan <jferlan>
Status:	CLOSED DUPLICATE	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	6.5	CC:	acathrow, codong, dyuan, iheim, jhunsaker, lcui, lpeer, mzhan, Rhev-m-bugs, yeylon
Target Milestone:	pre-dev-freeze
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-05-27 18:29:42 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jake Hunsaker 2014-05-16 19:34:56 UTC

Description of problem:

This is related to BZ 1098638 in that we found duplicate smartcard devices being created which prevented the VM from booting. However, we only found that after removing all vNICs from the VM as prior to their removal, the error message being presented was:

libvirtError: internal error ifname "vnet1" not in key map
Thread-97::DEBUG::2014-05-02 16:23:16,101::vm::2657::vm.Vm::(setDownStatus) vmId=`e3704160-28c2-48e4-8882-d7cb8ec25bb9`::Changed state to Down: internal error ifname "vnet1" not in key map

Version-Release number of selected component (if applicable):
 rhevm-3.3.2-0.50

How reproducible:
Reliably

Steps to Reproduce:
1. Create a VM off a template that was modified similar to BZ 1098638
2. Try and start the VM
3.

Actual results:
VM fails to boot with above error

Expected results:
If smartcard device errors prevent the VM from booting, the error message should indicate the issue is with the smartcard device, not a vnet device.

Additional info:

While I realize the root issue is in fact BZ 1098638, this bug is being opened to handle the incorrect error messages specifically. Pertinent logs are attached to that BZ.

Comment 3 John Ferlan 2014-05-20 11:30:14 UTC

Any chance you can provide the generated xml for the original guest and then perhaps XML or a virsh command sequence in order to trigger the bug?  Essentially some way without having rhevm/ovirt or some gui involved.  It'll make things much easier to diagnose.

Comment 4 John Ferlan 2014-05-21 15:31:19 UTC

I dug through the vdsm log of the related bz and found the XML of the created guest with two smartcard devices listed:


<domain type="kvm">
	<name>mine</name>
	<uuid>85a2a9fa-aef6-4fef-b35f-0bf4909bcd11</uuid>
	<memory>2097152</memory>
	<currentMemory>2097152</currentMemory>
	<vcpu>2</vcpu>
	<memtune>
		<min_guarantee>1048576</min_guarantee>
	</memtune>
	<devices>
		<channel type="unix">
			<target name="com.redhat.rhevm.vdsm" type="virtio"/>
			<source mode="bind" path="/var/lib/libvirt/qemu/channels/85a2a9fa-aef6-4fef-b35f-0bf4909bcd11.com.redhat.rhevm.vdsm"/>
		</channel>
		<channel type="unix">
			<target name="org.qemu.guest_agent.0" type="virtio"/>
			<source mode="bind" path="/var/lib/libvirt/qemu/channels/85a2a9fa-aef6-4fef-b35f-0bf4909bcd11.org.qemu.guest_agent.0"/>
		</channel>
		<input bus="ps2" type="mouse"/>
		<channel type="spicevmc">
			<target name="com.redhat.spice.0" type="virtio"/>
		</channel>
		<graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice">
			<channel mode="secure" name="main"/>
			<channel mode="secure" name="inputs"/>
			<channel mode="secure" name="cursor"/>
			<channel mode="secure" name="playback"/>
			<channel mode="secure" name="record"/>
			<channel mode="secure" name="display"/>
			<channel mode="secure" name="usbredir"/>
			<channel mode="secure" name="smartcard"/>
		</graphics>
		<smartcard mode="passthrough" type="spicevmc"/>
		<smartcard mode="passthrough" type="spicevmc"/>
...

So although I don't necessarily need this information any more - a general rule of thumb to ease bug report resolution is to provide XML and a sequence of commands or API calls that will reproduce symptoms seen.  Digging through other bug reports and having to determine command sequences, configurations, or API's to call is overhead I'd prefer to avoid.

NB: The description of a <smartcard> device in the libvirt documentation (http://libvirt.org/formatdomain.html) states in the last sentence:

"For now, qemu only supports at most one smartcard, with an address of bus=0 slot=0. "

When modifying a guest though there is no assumption of underlying hypervisor.

If I take one of my guests and add two smartcards entries as above to it via 'virsh edit <guest>', then save it - things are fine.  However, when I go to start it I get:

# virsh start f18
error: Failed to start domain f18
error: unsupported configuration: this QEMU binary lacks multiple smartcard support

#

A similar failure occurs if I 'virsh dumpxml f18 > f18-test.xml', then edit f18-test.xml to change the name of the 'vm' and remove the <uuid> entry, and perform a 'virsh create f18-test.xml'.

I searched for the error message from the vdsm.log from the other bz, but did not it in the libvirt sources - so I have to assume that message is not sourced by libvirt.

Support for the above error message was added to libvirt-0.10.2-20.el6 for bz 975751 as part of commit id '5efc6dcb' towards the end of qemuBuildCommandLine() within the "if (def->nsmartcards) {" condition.

Since the original report doesn't indicate which libvirt version you have installed and only indicates "rhevm-3.3.2-0.50" - I will certainly need some more information in order to proceed.

At this point I'm inclined to mark this as a duplicate, but I will await your feedback.

Comment 5 John Ferlan 2014-05-21 18:58:41 UTC

Finally found the 'ifname "vnet1" not in key map' message...

It's removed in libvirt-0.10.2-34 as a means of resolving bz 903480 (both in 6.5 and 6.6).  Since that's more recent than the libvirt-0.10.2-20 build where the code was added to message the failure to start due to not allowing more than 1 smart card, I'll assume that rhevm-3.3.2-0.50 doesn't include the -34 changes.

I'm still in the camp of this is not a bug, but will await your feedback.

Comment 6 Jake Hunsaker 2014-05-21 19:13:08 UTC

libvirt in use by customer is libvirt-0.10.2-29.el6_5.5.x86_64. 

If it's already removed in 0.10.2-34 I think we may be able to close this BZ since RHEV should pull this updated libvirt package in with a future release.

Comment 7 John Ferlan 2014-05-27 18:29:42 UTC

I am going to close this as a duplicate of 903480 since a fix already exists in 0.10.2-34.

*** This bug has been marked as a duplicate of bug 903480 ***