Bug 854143 - Failed to get the Linux guest information from host with hypervkvpd running
Failed to get the Linux guest information from host with hypervkvpd running
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: hypervkvpd (Show other bugs)
5.9
Unspecified Unspecified
urgent Severity urgent
: rc
: ---
Assigned To: Tomáš Hozza
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-04 03:40 EDT by Shengnan Wang
Modified: 2013-01-08 02:49 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-08 02:49:12 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
windows guest information in GuestIntrinsicExchangeItems (304.25 KB, image/jpeg)
2012-09-04 03:42 EDT, Shengnan Wang
no flags Details
Linux guest information in GuestIntrinsicExchangeItems (179.46 KB, image/jpeg)
2012-09-04 03:43 EDT, Shengnan Wang
no flags Details
windows_guest_information.jpg (84.71 KB, image/jpeg)
2012-09-04 03:44 EDT, Shengnan Wang
no flags Details
the WS2008 guest information & Linux5.9 x86_64 guest information.jpg (158.88 KB, image/jpeg)
2012-09-07 03:15 EDT, Shengnan Wang
no flags Details
KVP daemon code (20.41 KB, application/octet-stream)
2012-09-07 10:02 EDT, K. Y. Srinivasan
no flags Details
RHEL5.9 guest version information.jpg (171.00 KB, image/jpeg)
2012-09-12 06:05 EDT, Shengnan Wang
no flags Details

  None (edit)
Description Shengnan Wang 2012-09-04 03:40:31 EDT
Description of problem:

GuestIntrinsicExchangeItems stores kvp data which is automatically sent from the guest operating system to the parent partition. For the Windows guest,the guest information can be seen form GuestIntrinsicExchangeItems which contents the guest IP, the FQDN, OS name, and OS version and so on. However, fail to get the Linux guest information from GuestIntrinsicExchangeItems on host. Test on both i386 guest and x86_64 guest. Install the hypervkvpd package on the guest and the hypervkvpd service is running. 

[root@localhost ~]# service hypervkvpd status
hv_kvp_daemon (pid 6876) is running...


Version-Release number of selected component (if applicable):
Host: Windows 2008 R2 Hyper-V Server Core 
Hyper-V Version: 6.1.7600.16385
Guest: Both RHEL5.9 i386 & RHEL5.9 x86_64 (2.6.18-339.el5)
Hypervkvpd package Name: hypervkvpd-0-0.5.el5
Pv drivers: 
[root@localhost ~]# lsmod |grep hv_*
hv_netvsc              25665  0 
hid_base_hv            68177  1 hid_hyperv
hv_utils               12001  0 
hv_storvsc             17601  2 
hv_vmbus               30265  4 hv_netvsc,hid_hyperv,hv_utils,hv_storvsc


How reproducible:
100%

Steps to Reproduce:

1. Install the hypervkvpd package and start the hypervkvpd service on the hyper-v Linux guest.For Windows guest, please check weather the service 'Hyper-V Data Exchange Service' is running. It can be found in the "Component Services >> Services >> Hyper-V Data Exchange Service".
2. Login the Hyper-V host. Use powershell via "c:\Windows\System32\powershell" .
3. Get the virtual machine object $vm via powershell. 
   # $Vm = gwmi -namespace root\virtualization -query "Select * From Msvm_ComputerSystem Where ElementName='{$Guest_name}'"
4. Get the kvp object $kvp via powershell.
   # $Kvp = gwmi -namespace root\virtualization -query "Associators of {$Vm} Where AssocClass=Msvm_SystemDevice ResultClass=Msvm_KvpExchangeComponent"
5. There is guest information in the output "GuestIntrinsicExchangeItems". And you can use the script from page http://blogs.msdn.com/b/virtual_pc_guy/archive/2008/11/18/hyper-v-script-looking-at-kvp-guestintrinsicexchangeitems.aspx?Redirected=true to display the results clearly from the xml blob.


Actual results:
For the Linux guest,
At step 4, there is no value in the GuestIntrinsicExchangeItems line from the kvp output table. Details please see the attached file Linux_guest_information_xml.jpg .
At step 5, execute the script without output. 

For the Windows guest:
At step 4, there is guest information in the GuestIntrinsicExchangeItems line from the kvp output table. Details please see the attached file Windows_guest_information_xml.jpg .
At step 5, the guest information is displayed clearly. Details please see the attached file Windows_guest_information.jpg .

Expected results:

For the Linux guest,
At step 4, there is guest information in the GuestIntrinsicExchangeItems line from the kvp output table. The guest information contents the guest IP, the FQDN, OS name, and OS version and so on.
At step 5, the guest information is displayed clearly.

Additional info:
1. Both x86_64 guest and the i386 guest failed to get the information from host with hypervkvpd running.
Comment 1 Shengnan Wang 2012-09-04 03:42:14 EDT
Created attachment 609570 [details]
windows guest information in GuestIntrinsicExchangeItems
Comment 2 Shengnan Wang 2012-09-04 03:43:20 EDT
Created attachment 609572 [details]
Linux guest information in GuestIntrinsicExchangeItems
Comment 3 Shengnan Wang 2012-09-04 03:44:56 EDT
Created attachment 609573 [details]
windows_guest_information.jpg
Comment 4 RHEL Product and Program Management 2012-09-04 04:09:50 EDT
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.
Comment 5 jason wang 2012-09-04 05:26:43 EDT
cc K.Y. for help
Comment 6 K. Y. Srinivasan 2012-09-04 10:28:09 EDT
The KVP functionality is functional in RHEL 5.8 and RHEL 6.x that we currently support. So, I am not sure why it does not seem to work for you guys. Is this code currently in the alpha release that we MSFT are testing. If it is we can debug this issue. If not, could you make the necessary packages available to Hashir for testing and debugging this problem.
Comment 7 Tomáš Hozza 2012-09-05 03:38:16 EDT
(In reply to comment #6)
> The KVP functionality is functional in RHEL 5.8 and RHEL 6.x that we
> currently support. So, I am not sure why it does not seem to work for you
> guys. Is this code currently in the alpha release that we MSFT are testing.
> If it is we can debug this issue. If not, could you make the necessary
> packages available to Hashir for testing and debugging this problem.

Maybe the problem is, that in hypervkvpd package, the daemon source is probably not the same version as you are using. I assume that, because I packaged the source I was given and it was some time ago. Can you please tell me which version of hv_kvp_daemon.c are you using for testing the daemon on RHEL5.8?

Also here you can find the hypervkvpd package we are using for testing:
https://brewweb.devel.redhat.com/buildinfo?buildID=231904
Comment 8 K. Y. Srinivasan 2012-09-05 19:56:52 EDT
Tomas,

It looks like the daemon code you packaged was not right. We have locally verified that with the correct daemon code, we do not see this issue on RHEL 5.9.

A while ago you had indicated that you might be able to pickup latest KVP bits that support IP injection as well (a feature that will permit VM replication on WS 2012). As you know all of the kernel side paches have been applied upstream for this feature. However, I still have about 6 patches that are currently in Greg's queue that need to be applied against the user level daemon to implement this IP injection functionality. I am hoping Greg will apply these soon. Now as you look at having to package the KVP daemon with the correct daemon code, would you be interested in picking up the bits that would give you IP injection as well.
Comment 9 Tomáš Hozza 2012-09-06 02:59:24 EDT
(In reply to comment #8)
> Tomas,
> 
> It looks like the daemon code you packaged was not right. We have locally
> verified that with the correct daemon code, we do not see this issue on RHEL
> 5.9.

Can you please send me the code you used on RHEL5.9, or link (commit) to the kernel upstream git(web) repo that is the same as the one you are using?

> A while ago you had indicated that you might be able to pickup latest KVP
> bits that support IP injection as well (a feature that will permit VM
> replication on WS 2012). As you know all of the kernel side paches have been
> applied upstream for this feature. However, I still have about 6 patches
> that are currently in Greg's queue that need to be applied against the user
> level daemon to implement this IP injection functionality. I am hoping Greg
> will apply these soon. Now as you look at having to package the KVP daemon
> with the correct daemon code, would you be interested in picking up the bits
> that would give you IP injection as well.

It is possible, but there is not much time (if any) to do so. The problem might be, that probably not all of the patches accepted upstream (especially patches for hyperv.h) have been backported to RHEL5.9.
Comment 10 Shengnan Wang 2012-09-07 03:13:07 EDT
Hi Tomas Hozza,

Testing the new hypervkvpd package(hypervkvpd-0-0.7), the results are as below (attachment "the WS2008 guest information & Linux5.9 x86_64 guest information.jpg"):
1. For the RHEL5.9 i386 guest, can't get its information from host.

2. For the RHEL5.9 x86_64 guest, we can get some information.
2.1 The version information is not right. For Linux5.9 guest(2.6.18-339.el5), in my opinion, the key value pair should be: 
    OSMajorVersion=2
    OSMinorVersion=6 
    OSBuildNumber=18 
    IntegrationServicesVersion=2.6.18-339

2.2 The value of the WS2008 guest is number "9" (which means x64 as from the url: http://msdn.microsoft.com/en-us/library/windows/apps/windows.system.processorarchitecture), while the value of the Linux5.9 x86_64 guest is a string "x86_64", do you think we need to make the output the same as MSFT?

2.3 Failed to get the FQDN with the error "getaddrinfo failed...". It should be "dhcp-66-73-99.englab.nay.redhat.com" as checked with "#hostname" in the guest.
Comment 11 Shengnan Wang 2012-09-07 03:15:15 EDT
Created attachment 610643 [details]
the WS2008 guest information & Linux5.9 x86_64 guest information.jpg
Comment 12 K. Y. Srinivasan 2012-09-07 10:01:02 EDT
I am attaching the daemon code that we are currently shipping for RHEL 5.8 and this seems to be functional (in our testing) on RHEL 5.9 as well. On a different note, the tarball I initially posted of our RHEL 5.8 drivers that was the starting point for the RHEL 5.9 Hyper-V work also had the daemon code that would have served our needs here.
Comment 13 K. Y. Srinivasan 2012-09-07 10:02:21 EDT
Created attachment 610738 [details]
KVP daemon code
Comment 14 Tomáš Hozza 2012-09-07 10:33:20 EDT
(In reply to comment #12)
> I am attaching the daemon code that we are currently shipping for RHEL 5.8
> and this seems to be functional (in our testing) on RHEL 5.9 as well. On a
> different note, the tarball I initially posted of our RHEL 5.8 drivers that
> was the starting point for the RHEL 5.9 Hyper-V work also had the daemon
> code that would have served our needs here.

I looked at the code you have attached and it is almost the same as I packed in the testing package hypervkvpd-0-0.7 (I got the source from Jason Wang). It was tested, but with the results stated in comment #10. You can find those packages here:

https://brewweb.devel.redhat.com/taskinfo?taskID=4838287

Can you please help us with the problem?

Thanks.
Comment 15 K. Y. Srinivasan 2012-09-07 11:37:15 EDT
We will take a look. With regards to comment #10:

1. Looks like for i386 you are getting back no information from the guest. We will try to reproduce the problem here and debug it. Can you verify if the daemon is running on the 32 bit VM (386 vm).

2. (under 2.1): We get all the OS info from parsing appropriate files under /etc and getting info from the system (uname()).

3. (under 2.2): It is ok return the arch value as we are doing here.

4. (under 2.3): I am curious why getaddrinfo() is failing? Can you look into this.
Comment 16 K. Y. Srinivasan 2012-09-07 14:55:44 EDT
We just completed testing locally here and everything works as expected on a RHEL5.9 VM (both 32 bit and 64 bit vms). Thomas, we tried downloading the packages that you have posted under comment #14 but we are not able to - it just hangs.  In our tests, we ran the RHEL 5.8 KVP daemon on the RHEL 5.9 system and it worked as expected.
Comment 17 Shengnan Wang 2012-09-10 01:46:29 EDT
(In reply to comment #15)
> We will take a look. With regards to comment #10:
> 
> 1. Looks like for i386 you are getting back no information from the guest.
> We will try to reproduce the problem here and debug it. Can you verify if
> the daemon is running on the 32 bit VM (386 vm).
> 

Hi  K. Y. Srinivasan,
Thank you for your quick reply. Retest the i386 guest via hypervkvpd-0-0.7.el5. Fail to get the i386 Linux guest information from GuestIntrinsicExchangeItems on host with the guest hypervkvpd service running.

[root@localhost ~]# rpm -q hypervkvpd
hypervkvpd-0-0.7.el5
[root@localhost ~]# service hypervkvpd status
hv_kvp_daemon (pid 8206) is running...
[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.18-339.el5 #1 SMP Mon Aug 27 15:42:35 EDT 2012 i686 athlon i386 GNU/Linux
Comment 18 K. Y. Srinivasan 2012-09-10 20:22:19 EDT
We tested with the latest bits Tomas posted (KVP daemon). Everything works as expected on a WS2008R2 host both on 32 bit as well as 64 bit RHEL 5.9 VMs. Given that you are not getting any output from a 32 bit VM, I am suspecting either that KVP functionality is not turned on in the settings page or perhaps the shell script run on the host may not be pointing to the correct VM.
Comment 19 Tomáš Hozza 2012-09-11 03:36:44 EDT
(In reply to comment #8)
> A while ago you had indicated that you might be able to pickup latest KVP
> bits that support IP injection as well (a feature that will permit VM
> replication on WS 2012).

As for the IP injection functionality, I don't think we are going to include it right now, because it is too late for it and when I tried to compile the newest daemon source for RHEL5.9 there were some missing declarations in kernel headers. This is most probably because some newer patches to headers (hyperv.h) are not backported to RHEL5.9 kernel yet and I don't think they will be in this release. But you can request it for the next release or RHEL6.4.
Comment 20 Shengnan Wang 2012-09-11 03:55:27 EDT
(In reply to comment #18)
> We tested with the latest bits Tomas posted (KVP daemon). Everything works
> as expected on a WS2008R2 host both on 32 bit as well as 64 bit RHEL 5.9
> VMs. Given that you are not getting any output from a 32 bit VM, I am
> suspecting either that KVP functionality is not turned on in the settings
> page or perhaps the shell script run on the host may not be pointing to the
> correct VM.

Hi  K. Y. Srinivasan,

Find the reason about the issue that 32bit RHEL guest meet. If remove the hv_utils driver and load it again in the guest, we can't get the guest info even restart the hypervkvpd service. In fact, if the hv_utils driver is reloaded, the heartbeat and shutdown from the host are not available too. After reboot the guest, the hv_utils driver will work well. And the kvp deamon, shutdown from host and heartbeat will work well. Is it another bug about hv_utils?
Comment 21 Shengnan Wang 2012-09-12 06:03:52 EDT
Hi bsarathy,
Host gets our RHEL5.9 guest version information via kvp daemon. For Linux5.9 guest(2.6.18-339.el5), in my opinion, the key value pair should be: 
    OSMajorVersion=2
    OSMinorVersion=6 
    OSBuildNumber=18 
    OSversion=2.6.18
    IntegrationServicesVersion=2.6.18-339
Now the version information we get via kvp daemon is as below (attachment "RHEL5.9 guest version information.jpg"):
    OSMajorVersion=
    OSMinorVersion= 
    OSBuildNumber= 2.6.18
    OSversion= 2.6.18
    IntegrationServicesVersion= 3.1

Is it acceptable or not?
Comment 22 Shengnan Wang 2012-09-12 06:05:13 EDT
Created attachment 612041 [details]
RHEL5.9 guest version information.jpg
Comment 23 K. Y. Srinivasan 2012-09-12 21:06:15 EDT
Comment 20:

It looks like this is a bug on the host side (ws2k8 r2). This problem is not there on the ws2012 hosts. I have notified the Windows team this issue.
Comment 24 K. Y. Srinivasan 2012-09-12 21:14:18 EDT
Comment 21:

This is what we have been displaying for all Linux guests. The LIS version was meaningful when the LIS drivers were distinct from the kernel. With these drivers being part of the kernel, I am not sure of the utility of the LIS version information. You probably want to set this string to be more meaningful in your environment. The current string is defined in linux/hyperv.h (HV_DRV_VERSION).
Comment 25 Shengnan Wang 2012-09-13 03:53:58 EDT
(In reply to comment #24)
> Comment 21:
> 
> This is what we have been displaying for all Linux guests. The LIS version
> was meaningful when the LIS drivers were distinct from the kernel. With
> these drivers being part of the kernel, I am not sure of the utility of the
> LIS version information. You probably want to set this string to be more
> meaningful in your environment. The current string is defined in
> linux/hyperv.h (HV_DRV_VERSION).

Tomas,
Would you please help to set the string of HV_DRV_VERSION in linux/hyperv.h as K.Y. said? 

Tomas and K.Y. ,
For the FQDN issue, the test result shows that fail to the FQDN with the error "getaddrinfo failed..." when the hostname is not the same with that in the file /etc/hosts. The detail result is as bellow:
+-----------------+----------------------+------------------------+---------------------------------------------------+
|   hostname      |                      |                        |                                                   | 
|(terminal output)|    /etc/hosts        | /etc/sysconfig/network | Test Results                                      |
+-----------------+----------------------+------------------------+---------------------------------------------------+
|test.test        | 127.0.0.1 test.test  | HOSTNAME=test.test     | get FQDN successfully (test.test)                 |
+-----------------+----------------------+------------------------+---------------------------------------------------+
|test1.test1      | 127.0.0.1 test.test  | HOSTNAME=test.test     | failed with the error "getaddrinfo failed..."     |
+-----------------+----------------------+------------------------+---------------------------------------------------+
|test.test        | 127.0.0.1 test1.test1| HOSTNAME=test.test     | failed with the error "getaddrinfo failed..."     |
+-----------------+----------------------+------------------------+---------------------------------------------------+
|test.test        | 127.0.0.1 test.test  | HOSTNAME=test1.test1   | get FQDN successfully (test.test)                 |
+-----------------+----------------------+------------------------+---------------------------------------------------+

Suggest that to get the FQDN from the hostname(terminal output) only, or it will be confused for customer using. We can leave the consistency of hostname and /etc/hosts to customer.If the hostname is not the same with that in the /etc/hosts, customers should modify their /etc/hosts file by themselves.
Comment 26 Tomáš Hozza 2012-09-13 06:01:37 EDT
(In reply to comment #25)
> Tomas,
> Would you please help to set the string of HV_DRV_VERSION in linux/hyperv.h
> as K.Y. said?

The hyperv.h header is part of the kernel-devel package. You have to ask Jason Wang to do it. You should file a bug on the kernel to resolve this problem.

> Tomas and K.Y. ,
> For the FQDN issue, the test result shows that fail to the FQDN with the
> error "getaddrinfo failed..." when the hostname is not the same with that in
> the file /etc/hosts.
> ...
> Suggest that to get the FQDN from the hostname(terminal output) only, or it
> will be confused for customer using. We can leave the consistency of
> hostname and /etc/hosts to customer.If the hostname is not the same with
> that in the /etc/hosts, customers should modify their /etc/hosts file by
> themselves.

The FQDN in the daemon is from calling getaddrinfo() function. The configuration you are describing shouldn't affect it. K.Y. can you please help with this problem?

I'm going to commit and build the daemon with the source used in scratch build package hypervkvpd-0-0.7 since K.Y. stated in comment #18 that everything worked fine when they tested it. If there will be any need for patching the daemon regarding the FQDN issue, please open a new bug for it.
Comment 28 Shengnan Wang 2012-09-13 07:22:23 EDT
(In reply to comment #23)
> Comment 20:
> 
> It looks like this is a bug on the host side (ws2k8 r2). This problem is not
> there on the ws2012 hosts. I have notified the Windows team this issue.

File a new bug to track the issue.
https://bugzilla.redhat.com/show_bug.cgi?id=856947
Comment 33 jason wang 2012-09-17 02:19:49 EDT
(In reply to comment #26)
> (In reply to comment #25)
> > Tomas,
> > Would you please help to set the string of HV_DRV_VERSION in linux/hyperv.h
> > as K.Y. said?
> 
> The hyperv.h header is part of the kernel-devel package. You have to ask
> Jason Wang to do it. You should file a bug on the kernel to resolve this
> problem.

It looks to me that current value "3.1" is OK. Since the driver were backported from upstream and we provide the same function as upstream also there's no hyperv driver in 2.6.18 period.
Comment 34 Shengnan Wang 2012-09-29 03:20:32 EDT
Test the issue on both i386 and x86_64 guest with hypervkvpd-0-0.7 package. we can get the Linux guest information from GuestIntrinsicExchangeItems on host. So change the status to verified.
Comment 36 errata-xmlrpc 2013-01-08 02:49:12 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-0111.html

Note You need to log in before you can comment on or make changes to this bug.