Bug 576497

What flags appear in /proc/cpuinfo?

Please ignore comment 2, it is not correct. (Although the 2 bugs were tested on the same host, but actually different cpu flags under different kernel)

The flags is as below:
# fgrep flags /proc/cpuinfo 
flags		: fpu tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est ssse3 cx16 sse4_1 lahf_lm
flags		: fpu tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est ssse3 cx16 sse4_1 lahf_lm
flags		: fpu tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est ssse3 cx16 sse4_1 lahf_lm
flags		: fpu tsc msr pae cx8 apic mtrr cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc pni est ssse3 cx16 sse4_1 lahf_lm

No svm/vmx/smx flags 

This bug is reproduced on the following 2 groups
- rhel55-server-x86_64-xen
- v7-1.1-22.el5
- kernel-xen-2.6.18-194.el5
- xen-3.0.3-105.el5

- rhel55-server-x86_64-xen
- v7-1.1-23.el5
- kernel-xen-2.6.18-194.el5
- xen-3.0.3-105.el5

And as I said in Description
This bug cannot reproduce with v7-1.1-22.el5/v7-1.1-23.el5 on rhel55-server-x86_64-kvm (kernel-2.6.18-194.el5)

To me this appears to be a kernel-xen bug, how is it suggested to move it ahead ? also will any workarounds be required to be opened issue for, for v7 ?

Although it maybe of interest to know what stuff kernel-xen does to probe VT functionality. However xen_caps line output of "xm info" shows the VT (hvm) capability properly, and to me it sense that v7 can also be patched to use that for a system booted on kernel-xen: 

<snip>
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
<snip> 

thoughts ?

(In reply to comment #7)

In reply to my own comment in /usr/share/v7/lib/v7/fvtest.py after trying to check flags from /proc/cpuinfo it also tries /sys/hypervisor/properties/capabilities but even if hvm capability is listed it is not able to make use of it, I don't yet understand why. 

Output of /sys/hypervisor/properties/capabilities on my system is same as xen_caps line in previous comment :
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64

1st we should determine if a work around is available. Typically this is done by manually planning a test where auto detection did not function as expected.  What happens if fv_* is planned manually and then run via 'v7 run --test=fv_*'?

According to comment 9, I tested manually planning a fv_memory test ahdn then run via `v7 run -t fv_memory`, seems cannot work around this bug.
1. # v7 plan
2. # v7 print
No fv_* is listed out
3. # v7 plan --add -t fv_memory
loaded results /var/v7/results.xml
Tested OS: Red Hat Enterprise Linux Server 5 (Tikanga)
Kernel RPM: kernel-xen-2.6.18-194.el5
v7 version 1.1, release 23
Tested OS: Red Hat Enterprise Linux Server 5 (Tikanga)
Kernel RPM: kernel-xen-2.6.18-194.el5
v7 version 1.1, release 23
Warning: No device specified with --udi <device udi> or --device <logical device name>
Are you sure the test does not require a specific device? (y|n) y
response: y
Added test
saved test plan to /var/v7/results.xml
4. # v7 print
Test Plan:
----------------------------------------------------------------
usb                                                        
network    eth0       net_00_25_64_a6_fe_df                
info                                                       
fv_memory     
......
5. # v7 run -t fv_memory
Running fv_memory:
---------------------------------
fv_memory                                                  
info                                                       

running fv_memory on 
mkdir -p /tmp/v7-fv_memory-PUmiHL/mnt/tests/V7/v7/fv_memory
cp -a testinfo.desc runtest.sh fv_memory.py Makefile /tmp/v7-fv_memory-PUmiHL/mnt/tests/V7/v7/fv_memory
install -m 0755 runtest.sh /tmp/v7-fv_memory-PUmiHL/mnt/tests/V7/v7/fv_memory
make OUTPUTFILE=/var/log/v7/runs/1/fv_memory/output.log RUNMODE=normal UDI= DEVICE= TESTSERVER=unknown run
chmod a+x ./runtest.sh ./fv_memory.py
./runtest.sh
/tmp/v7-fv_memory-PUmiHL/mnt/tests/V7/v7/fv_memory/fv_memory.py
<output>
Running ./fv_memory.py:
Verified that guest v7x86_64 is not running
Traceback (most recent call last):
  File "./fv_memory.py", line 34, in ?
    returnValue = test.do(sys.argv)
  File "/usr/share/v7/lib/v7/test.py", line 255, in do
    return self.run()
  File "/usr/share/v7/lib/v7/fvtest.py", line 435, in run
    if not self.verifyGuest():
  File "/usr/share/v7/lib/v7/fvtest.py", line 142, in verifyGuest
    return self.verifyGuestFiles()
  File "/usr/share/v7/lib/v7/fvtest.py", line 146, in verifyGuestFiles
    if not self.verifyFile(self.guestImageDirectory, self.dataImageFile):
AttributeError: FvMemoryTest instance has no attribute 'guestImageDirectory'
...finished running ./fv_memory.py, exit code=1
</output>
recovered exit code=1
..................

Tested on:
- rhel55-server-x86_64-xen
- v7-1.1-23.el5
- kernel-xen-2.6.18-194.el5
- xen-3.0.3-105.el5

I met ditto behaviour, perhaps we need to have Greg take a look and guide us.

Manual planning the fv tests isn't going to work with R23 - you'll run into the above traceback because v7 doesn't know whether to do xen or kvm virtualization,
since it doesn't detect the cpuflags.

We'd need to change the logic a bit to support manual planning - i.e. use xen on xen kernels, kvm otherwise.

I hit this problem today. This is going to block cert on all machines experiencing the problem, so we need a workaround as soon as possible. I opened a BZ for the missing flags under the Xen kernel:

https://bugzilla.redhat.com/show_bug.cgi?id=592114

The kernel is _correct_ in not reporting the flags, because from the point of view of dom0 the machine does not support FV.  If it did, for example you could run KVM from dom0.  Instead, you need to ask Xen to create a new domain, and it will handle VMX for you.

Comment 8 says:

> In reply to my own comment in /usr/share/v7/lib/v7/fvtest.py after trying to
> check flags from /proc/cpuinfo it also tries
> /sys/hypervisor/properties/capabilities but even if hvm capability is listed it
> is not able to make use of it, I don't yet understand why. 

and it seems that it would be the right thing to do.  Even better would be to first check /sys/hypervisor, and then check /proc/cpuinfo (the opposite order would fail in a hypothetical future version of Xen that support nested virtual machines, but that's very unlikely to happen in RHEL5 anyway).

I guess the other question to ask here is: *why* is this coming up now?  Xen has always behaved in this manner, and we are already past 5.5, so it's not like this is something new or different.  What has changed that is causing these problems with certification now?

Chris Lalancette

(In reply to comment #15)
> I guess the other question to ask here is: *why* is this coming up now?

Good question, we should probably answer this last.  1st question is how do we unblock certs today, 2nd question is how do we create an interim solution, (v7 having unique code to determine for itself the flags of a cpu seems silly). then we can get back to the and how do we not get here again.

> Xen has always behaved in this manner, and we are already past 5.5, so it's not
> like this is something new or different.

Not quite accurate, 5.4 on the same boxes works proven by multiple vendors on different machines, as well I tested a 5.5 pre-release which worked fine (Lenovo T60) and benl's group is responsible for testing the RHEL releases against the stable v7 releases and they did not report a problem.

>  What has changed that is causing these problems with certification now?

Good question, seems to be the -xen kernel in some late 5.5 cycle; but again we should come back to this after solving the immediate and intermediate issues.
> 
> Chris Lalancette

(In reply to comment #16)
> (In reply to comment #15)
> > I guess the other question to ask here is: *why* is this coming up now?
> 
> Good question, we should probably answer this last.  1st question is how do we
> unblock certs today, 2nd question is how do we create an interim solution, (v7
> having unique code to determine for itself the flags of a cpu seems silly).
> then we can get back to the and how do we not get here again.
> 
> > Xen has always behaved in this manner, and we are already past 5.5, so it's not
> > like this is something new or different.
> 
> Not quite accurate, 5.4 on the same boxes works proven by multiple vendors on
> different machines, as well I tested a 5.5 pre-release which worked fine
> (Lenovo T60) and benl's group is responsible for testing the RHEL releases
> against the stable v7 releases and they did not report a problem.

Ah, this was the important point that I forgot about.  So the situation is that Xen has always hidden some flags from guests, but prior to 5.5, it wasn't any flags that we cared about.  In particular, prior to 5.5 we did not hide the VMX/SVM flags from guests.  In 5.5 we got much more aggressive about hiding flags from the guests (though I don't remember explicitly disabling the VMX/SVM flag, it's possible that this was done).

The reason we did this is that you don't really want to export flags to a guest that you have no reasonable way to support.  That is, if you think about from a guest point-of-view, it makes no sense to report the VMX flag; the guest can't possibly make use of it, so it really shouldn't know it was there.

The unfortunate part is that "guests" include dom0, and so now it seems that we have masked VMX/SVM from dom0.  That is *still* correct, from a purely theoretical standpoint; dom0 itself can't do anything with the VMX flag either (it's the hypervisor that is responsible for starting guests).  However, from a backwards-compatible point-of-view, this is wrong; many people have been depending on the presence/absence of the VMX/SVM flag in the dom0 to detect whether their particular machine supports hardware virtualization.

So, I think we need to do a few things:
1)  Just to make absolutely sure that the current machine you are trying to cert does support the VMX flag, can you run the certification on 5.4, and then run the certification on 5.5, using the exact same machine?  If it passes on 5.4 and fails on 5.5, that tells us that we did indeed introduce a bug here.
2)  We need to come up with a short-term workaround for cert, while we fix the actual bug.  There must be a way to ask the hypervisor about this directly, though it's not coming to mind at present.  I'll see what I can dig up.

Chris Lalancette

(In reply to comment #17)

> 
> So, I think we need to do a few things:
> 1)  Just to make absolutely sure that the current machine you are trying to
> cert does support the VMX flag, can you run the certification on 5.4, and then
> run the certification on 5.5, using the exact same machine?  If it passes on
> 5.4 and fails on 5.5, that tells us that we did indeed introduce a bug here.

This has been demonstrated already although it is not an option for all systems as some require 5.5.

> 2)  We need to come up with a short-term workaround for cert, while we fix the
> actual bug.  There must be a way to ask the hypervisor about this directly,
> though it's not coming to mind at present.  I'll see what I can dig up.

On the plus side, the solution for any immediate certs is to start the guest by hand and run the test suite within the guest as was done prior to adding the fv_* tests.

That leaves us with an intermediate issue given testers will tire of this manual operation quickly so we need to update v7 with at least a workaround for the 5.5ga kernel (thoughts on what this might be?) even if the flag is to return in a later kernel.

I'd prefer to see us update v7 once for this, so if there is a standard or more accepted method of determining FV capability that libvirt or some of the other virt tools use, it seems to me that we should call that instead of coding a unique to v7 solution; especially if it can work on RHEL6 as well.


> Chris Lalancette

(In reply to comment #18)
> (In reply to comment #17)
> 
> > 
> > So, I think we need to do a few things:
> > 1)  Just to make absolutely sure that the current machine you are trying to
> > cert does support the VMX flag, can you run the certification on 5.4, and then
> > run the certification on 5.5, using the exact same machine?  If it passes on
> > 5.4 and fails on 5.5, that tells us that we did indeed introduce a bug here.
> 
> This has been demonstrated already although it is not an option for all systems
> as some require 5.5.

OK, I just wanted to make sure.  Like I said, I don't remember explicitly disabling the VMX/SVM flag, so maybe it was done inadvertently.

<snip>

> I'd prefer to see us update v7 once for this, so if there is a standard or more
> accepted method of determining FV capability that libvirt or some of the other
> virt tools use, it seems to me that we should call that instead of coding a
> unique to v7 solution; especially if it can work on RHEL6 as well.

Yeah, after looking at it, the right thing to do here is to ask libvirt for this information.  If you do:

# virsh capabilities

on the dom0, then you will get back a piece of XML describing the types of guests this hypervisor supports.  Within that XML you will see various <guest> stanzas, and within those guest stanzas you will see <os_type> elements that either say "xen" or "hvm".  An os_type of "xen" means paravirt guests and an os_type of "hvm" means a fully-virt guest.  The good news is that KVM also uses an os_type of "hvm" to represent the guests it supports, so something like the following python code should work for both:

import xml.dom.minidom
import sys

cappath = sys.argv[1]

parser = xml.dom.minidom.parse(cappath)
elems = parser.getElementsByTagName("guest")
found_hvm = False
for elem in elems:
    os_type = elem.getElementsByTagName("os_type")[0].childNodes[0]
    if os_type.data == "hvm":
        found_hvm = True
        break

if found_hvm:
    print "HVM supported"
else:
    print "HVM not supported"

Chris Lalancette

Using virsh to detect whether or not a system is HVM capable means v7 will have to require the libvirt rpm be installed on all certified systems  Is this acceptable?

If not, this will do:

if grep -q hvm /sys/hypervisor/properties/capabilities; then
  echo 'HVM supported (xen)'
elif grep -qe vmx -e svm /proc/cpuinfo; then
  echo 'HVM supported (KVM)'
else
  echo 'HVM not supported'
fi

Created attachment 414530 [details]
fvtest.py patch changing planning and modes for FV tests

This patch changes FV test planning to use "virsh capabilities" to determine if FV tests are planned.   if it returns the os_type as "HVM" the FV tests will be planned for non-RT kernels, RHEL 5 beyond.

When the FV tests run, if /sys/hypervisor/properties/capabilities has HVM, then xen virtualization is tested.   Otherwise, kvm virtualization is assumed.

The FV tests will log diagnostic information about whether or not the system has virtualization cpuflags, but they are no longer used to test planning or execution.

The patch will fix manual planning of FV tests.

The patch assumes libvirt is installed.

Created attachment 414531 [details]
tags.py patch adding os_type

Created attachment 414708 [details]
fvtest.py patch changing planning and modes for FV tests

This patch changes the logic of FV testing:

1) Test planning
Three methods are used to detect if the system is capable of FV
- if libvirt is installed, it will use "virsh capabilities"
- /sys/hypervisor/properties/capabilities is checked
- cpu flags are checked.

If any of the above methods detect FV support, the FV tests are planned.
This logic eliminates the need to libvirt installation prior to v7 planning,
while allowing it to be used if helpful to testing.   

Note that if FV tests are planned with kvm virtualization, v7 will automate the installation of libvirt and several other rpms to support kvm testing at the end
of the planning process.

2) Test execution
The tests default to kvm virtualization.   if a xen kernel is installed,
xen virtualization is tested.   

This logic supports manual planning for both kvm (as a default), and xen,
if a xen kernel is installed, independent of the planning logic in 1).

Verified this bug PASS with v7-1.2-2.el5 on rhel55-s-x86_64-xen system

scenario 1 - without libvirt package installed, v7 can detect fv and yum install libvirt package.
1. # v7 plan
Tested OS: Red Hat Enterprise Linux Server 5 (Tikanga)
Kernel RPM: kernel-xen-2.6.18-194.el5
v7 version 1.2, release 2
/bin/sh: virsh: command not found
/bin/sh: virsh: command not found
/bin/sh: virsh: command not found
/bin/sh: virsh: command not found
Hardware: Dell Inc. unknown OptiPlex 780
OS: Tikanga 5.5
saved configuration to /var/v7/results.xml
Created a new plan with 15 tests on 116 devices
Checking for additional required packages based on the test plan:
dvd requires dvd+rw-tools, mkisofs, cdrecord
audio requires sox, system-config-soundcard
cdrom requires dvd+rw-tools, mkisofs, cdrecord
profiler requires oprofile
fv_storage requires libvirt, libvirt-python, python-virtinst
fv_network requires libvirt, libvirt-python, python-virtinst
fv_memory requires libvirt, libvirt-python, python-virtinst
video requires xorg-x11-apps, system-config-display
fv_core requires libvirt, libvirt-python, python-virtinst
info requires kernel-xen-devel
The following rpms are required for testing:
 libvirt
Would you like to install them now? (y|n) y
response: y
Running yum: --------------------------------
This system is not registered with RHN.
RHN support will be disabled.
Error: Could not install rpm
"yum install -y  libvirt" has output on stderr
Rechecking required rpms
All required rpms installed
saved test plan to /var/v7/results.xml

2. # rpm -qa|grep libvirt
libvirt-cim-0.5.8-3.el5
libvirt-0.6.3-33.el5
libvirt-python-0.6.3-33.el5

scenario 2 - with libvirt package installed, v7 can detect fv.
1. # v7 plan
Tested OS: Red Hat Enterprise Linux Server 5 (Tikanga)
Kernel RPM: kernel-xen-2.6.18-194.el5
v7 version 1.2, release 2
Hardware: Dell Inc. unknown OptiPlex 780
OS: Tikanga 5.5
saved configuration to /var/v7/results.xml
Created a new plan with 15 tests on 116 devices
Checking for additional required packages based on the test plan:
dvd requires dvd+rw-tools, mkisofs, cdrecord
audio requires sox, system-config-soundcard
cdrom requires dvd+rw-tools, mkisofs, cdrecord
profiler requires oprofile
fv_storage requires libvirt, libvirt-python, python-virtinst
fv_network requires libvirt, libvirt-python, python-virtinst
fv_memory requires libvirt, libvirt-python, python-virtinst
video requires xorg-x11-apps, system-config-display
fv_core requires libvirt, libvirt-python, python-virtinst
info requires kernel-xen-devel
All required rpms installed
saved test plan to /var/v7/results.xml

These two look like cosmetic bugs that should be reisited later:


<snip>
Kernel RPM: kernel-xen-2.6.18-194.el5
v7 version 1.2, release 2
/bin/sh: virsh: command not found
/bin/sh: virsh: command not found
/bin/sh: virsh: command not found
/bin/sh: virsh: command not found
Hardware: Dell Inc. unknown OptiPlex 780
OS: Tikanga 5.5
saved configuration to /var/v7/results.xml
</snip>

<snip>
Would you like to install them now? (y|n) y
response: y
Running yum: --------------------------------
This system is not registered with RHN.
RHN support will be disabled.
Error: Could not install rpm
"yum install -y  libvirt" has output on stderr
Rechecking required rpms
All required rpms installed
</snip>

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0463.html