Hide Forgot
Description of problem: The virt-who can't report the guest's uuid if the guest's status is stop. Version-Release number of selected component (if applicable): virt-who-0.5-2.el5 subscription-manager-0.98.8-1.el5 python-rhsm-0.98.7-1.el5 vdsm22-4.5-67.2.el5_7 vdsm22-cli-4.5-67.2.el5_7 vdsm22-reg-4.5-67.2.el5_7 How reproducible: 100% Steps to Reproduce: 1. Install RHEV-H, and register to RHEV-M and register to candlepin. 2. Configure the virt-who. # vi /etc/sysconfig/virt-who VIRTWHO_BACKGROUND=1 (enable debugging) VIRTWHO_DEBUG=1 (run in background) VIRTWHO_VDSM=1 3. Add a new guest to RHEV-H(A),and then shutdown the guest 4. Restart the virt-who service in terminal1 in RHEV-H # service virt-who restart [root@wanghui02 ~]# service virt-who restart Stopping virt-who: [ OK ] Starting virt-who: Listening for events is not available in VDSM or ESX mode Virt-who is running in vdsm mode Starting infinite loop with 3600 seconds interval and event handling [ OK ] [root@wanghui02 ~]# Unable to obtain status from server, UEPConnection is likely not usable: Traceback (most recent call last): File "/usr/lib64/python2.4/logging/__init__.py", line 731, in emit File "/usr/lib64/python2.4/logging/__init__.py", line 617, in format File "/usr/lib64/python2.4/logging/__init__.py", line 405, in format File "/usr/lib64/python2.4/logging/__init__.py", line 272, in getMessage AttributeError: RemoteServerException instance has no attribute 'args' Traceback (most recent call last): File "/usr/lib64/python2.4/logging/handlers.py", line 71, in emit File "/usr/lib64/python2.4/logging/handlers.py", line 149, in shouldRollover File "/usr/lib64/python2.4/logging/__init__.py", line 617, in format File "/usr/lib64/python2.4/logging/__init__.py", line 405, in format File "/usr/lib64/python2.4/logging/__init__.py", line 272, in getMessage AttributeError: RemoteServerException instance has no attribute 'args' Sending update to updateConsumer: [] ^^^^Can't report the uuid of the guest that the status is stop Actual results: Can't report the guest's UUID if the guest's status is stop Expected results: Can report the guest's UUID if the guest exists. Additional info:
This can be also same bug as Bug 769177. Can you retest this with virt-who-0.5-3.el5? Thank you.
(In reply to comment #1) > This can be also same bug as Bug 769177. Can you retest this with > virt-who-0.5-3.el5? Thank you. There is no new rhev-h build.And the rhev-h is read-only,so can't install the virt-who-0.5-3.el5 on rhev-h. I will verify the issue on the next rhev-h build.
(In reply to comment #1) > This can be also same bug as Bug 769177. Can you retest this with > virt-who-0.5-3.el5? Thank you. Hi,Radek The issue testing is blocked.I will retest it in the next build. I have tried to test the issue by the step. 1.Download the virt-who-0.5-3.el5 from brew 2.Install the virt-who-0.5-3.el5 on rhev-h #mount / -o remount,rw #rpm -Uvh virt-who-0.5-3.el5.noarch.rpm 3.Register the rhev-h to the rhev-m and candlepin 4.Add storage from nfs server to rhev-h ^^^^^^^^^^^^Can't add storage to rhev-h,because the rhev-h is mounted in step 2 5.Reboot the rhev-h ^^^^^^^^^^^^The virt-who-0.5-3 became virt-who-0.5-2
I verified the issue. The result is failed. verify step: 1. Install RHEV-H, and register to RHEV-M and register to SAM. 2. Configure the virt-who. # vi /etc/sysconfig/virt-who VIRTWHO_BACKGROUND=1 (enable debugging) VIRTWHO_DEBUG=1 (run in background) VIRTWHO_VDSM=1 VIRTINEVAL=60 3. Add two new guest to RHEV-H,and keep them running 4. Restart the virt-who service in terminal1 in RHEV-H 5. Then shutdown the one of guests 6. Check the update info [root@wanghui02 rhsm]# service virt-who restart Stopping virt-who: [ OK ] Starting virt-who: Listening for events is not available in VDSM or ESX mode Virt-who is running in vdsm mode Starting infinite loop with 1 seconds interval and event handling [ OK ] [root@wanghui02 rhsm]# Sending update to updateConsumer: ['9c7fc937-ae4f-4564-8c46-f099bab61bab', 'c3a7ea3e-ffb3-4443-bf85-0c4d37436c22'] Sending update to updateConsumer: ['9c7fc937-ae4f-4564-8c46-f099bab61bab', 'c3a7ea3e-ffb3-4443-bf85-0c4d37436c22'] ^^^^^^^^^^^^^^^^Before shut down any guest Sending update to updateConsumer: ['c3a7ea3e-ffb3-4443-bf85-0c4d37436c22'] ^^^^^^^^^^^^^^^^After shut down one of guest
Version of verified in comment4 as following: rhev-h-5.8snapshot3.0 virt-who-0.5-3.el5 subscription-manager-0.98.10-1.el5 python-rhsm-0.98.9-1.el5
The same issue will compose if the guest's status changed to suspend.
This is not a bug. Domains in the RHEV are transient, they no longer exists on the host when suspended or stopped. When the guest is suspended it doesn't appear in the list of guests on given host. Stopped guest can be started on other host so it doesn't make sense to report it. This behaviour has been confirmed with RHEV guys, so I'm closing this as NOTABUG.
(In reply to comment #7) > This is not a bug. Domains in the RHEV are transient, they no longer exists on > the host when suspended or stopped. When the guest is suspended it doesn't > appear in the list of guests on given host. Stopped guest can be started on > other host so it doesn't make sense to report it. > > This behaviour has been confirmed with RHEV guys, so I'm closing this as > NOTABUG. The guest belongs an unique host that assigned to guest when the guest is installing . Pls see the attachments.
Created attachment 552273 [details] The guest belongs to rhev01 when runnning
Created attachment 552274 [details] The guest belongs to rhev01 when the stustus is stop
Created attachment 552275 [details] Assigned one host to the guest
(In reply to comment #7) > This is not a bug. Domains in the RHEV are transient, they no longer exists on > the host when suspended or stopped. When the guest is suspended it doesn't > appear in the list of guests on given host. Stopped guest can be started on > other host so it doesn't make sense to report it. Radek, The problem here is when a guest is stopped who consumes a bonus subscription derived from the host, if virt-who stops reporting the uuid of the stopped guest to the entitlement server, then the guest's entitlement cert will be revoked and the guest will becomes uncompliant when it resumes or boots again. This is not what we want for a customer I think. Reopening this bug. cc'ing Bryan in case he has more comments. Thanks, Keqin > > This behaviour has been confirmed with RHEV guys, so I'm closing this as > NOTABUG.
I don't think there is simple solution for this issue. VDSM daemon on host system doesn't know about stopped guests (if the guest is not associated with one certain host). There can be some stopped/suspended guest that consumes subscription but is not assigned to any host.
(In reply to comment #13) > I don't think there is simple solution for this issue. VDSM daemon on host > system doesn't know about stopped guests (if the guest is not associated with > one certain host). There can be some stopped/suspended guest that consumes Radek, The guest was infact associated with one specified host, not auto. pls see the screenshots attached. > subscription but is not assigned to any host.
Bryan, If domains are transient as per comment 15 and comment 16, then we will get guest entitlement cert revoked if it is stopped/suspended then restarted. This is not acceptable by customers I think. Can we fix this by adding autosubscribe immediately after entitlement cert becomes revoked? Pls let us know how this will be fixed. We would appreciate it if this bug could be fixed before RHEV-H snapshot-5, so that we could push out RHEA-2011:12093-07 - new package: virt-who (https://errata.devel.redhat.com/errata/stateview/12093 ) on Jan 21 due to holidays (Jan 23 ~ Jan 27) in China. Currently this bug is not added to above virt-who erratum, but I think this bug should block the erratum. Thanks, Keqin
Can we step back to look at the requirements that are driving this? Maybe there is a simpler solution. * If a new guest is started on a host, it should consume an entitlement associated with that specific host * If that guest is migrated to a new host, the entitlement should change so that it consumes the entitlement for the new host and releases the entitlement for the old host * If a guest is suspended or powered off, it should continue to consume an entitlement for the last host that it was running on * If a guest is undefined in RHEVM (not virsh undefine, but deleting the guest from RHEVM entirely) then the entitlement for the guest needs to be relinquished Are these requirements accurate, or am I missing something? If they are correct, then it seems like: * virt-who when it detects a guest running sends the UUID to mgmt server so it can associate an entitlement with that new guest * If virt-who stops reporting a UUID for a guest, nothing changes. The Guest continues to consume that same entitlement, until: * virt-who on another host detects that same guest running, and sends UUID to mgmt server where the mgmt server now moves the entitlement from one host to another * RHEVM undefines the VM which means either * RHEVM needs to instruct SAM to remove the entilement * User needs to manually go into SAM to do that Thoughts?
Perry that is a correct understanding and is in line with what Chris suggested. However, given that the guest detection is done via connection to RHEV-H and not RHEV-M the last bullet (undefned) can not be determined. I would propose we do that in a Z Stream or future release.
(In reply to comment #30) > Perry that is a correct understanding and is in line with what Chris suggested. > However, given that the guest detection is done via connection to RHEV-H and vdsm, not RHEV-H. Again, this whole situation is not specific to RHEV-H as it occurs on RHEL + vdsm systems as well. > not RHEV-M the last bullet (undefned) can not be determined. I would propose we > do that in a Z Stream or future release. Which bullet, the one that says 'User needs to go manually into SAM to do that'?
I added the migration requirements and design to the Virtualization Entitlement requirements document. https://engineering.redhat.com/trac/Entitlement/wiki/Virtualization62 Requirements: (agree on what Perry outlined) 1. If a new guest is started on a host, it should consume an entitlement associated with that specific host 2. If that guest is migrated to a new host, the entitlement should change so that it consumes the entitlement for the new host and releases the entitlement for the old host 3. If a guest is suspended or powered off, it should continue to consume an entitlement for the last host that it was running on 4. (deferred to RHEL7) If a guest is undefined in RHEVM (not virsh undefine, but deleting the guest from RHEVM entirely) then the entitlement for the guest needs to be relinquished The key difference in migration design between what Perry outlined and what I added to the doc is that the virt-who and RHSM need to access SAM in a serialized fashion. * Migration Design: intended to function on both RHEL/KVM and RHEV 1. When virt-who detects a guest running sends the UUID to SAM so it can associate an entitlement with that new guest 2. If virt-who stops reporting a UUID for a guest, nothing changes. The Guest continues to consume that same entitlement, until: 3. virt-who must contact SAM, before RHSM contacts SAM, otherwise SAM cannot resolve host to guest mapping 4. guest starts on a different host 5. virt-who on another host detects that same guest running and sends UUID to SAM 6. RHSM on guest contacts SAM 7. SAM issues new entitlement to the guest running on the new host Comments?
Bhavna, For requirement 1 in the first half of the comment, do you mean the guest should automatically consume an entitlement? I don't think that will work since the system will be autosubscribed, and then resubscribed when the user registers the system. We can perform requirement 2 since the system was already registered, but note that the entitlement may be different than what was on the system before the migration (this would occur if the new hypervisor has no free entitlements, or if it was subscribed to something different than the old hypervisor). Requirement 3 is straightforward, I am working on implementing that now. For the second half of #32, the fix we are implementing now should satisfy those criteria.
Referring to requirement #3, it not clear what occurs during guest migration for RHEL on RHEL and for RHEL on RHEV cases. Here is my understanding. * in RHEL on RHEL case, virt-who receives definite information from libvirt whether the guest is being migrated or destroyed. During migration, SAM will continue to associate guest with the old host, until virt-who on the new host provides updated information. However, if a guest is destroyed, virt-who stops reporting the guest UUID and SAM revokes guest's entitlement. * in RHEL on RHEVH case, virt-who and vdsm cannot identify whether guest is being migrated or destroyed, and thus we get the root cause of this bug. Option 1. Current implementation, where a guest stops consuming an entitlement. Current proposal is described in comments 23 and 25, which is not acceptable due to 4 hour delay. Option 2. Let guest continue consuming an entitlement. This way, removed guest entitlements will accumulate. It is a distinct disadvantage for RHEV/RHEVH compared to RHEV/RHEL-KVM.
(In reply to comment #34) > * in RHEL on RHEL case, virt-who receives definite information from libvirt > whether the guest is being migrated or destroyed. During migration, SAM will > continue to associate guest with the old host, until virt-who on the new host > provides updated information. > > However, if a guest is destroyed, virt-who stops reporting the guest UUID and > SAM revokes guest's entitlement. > > * in RHEL on RHEVH case, virt-who and vdsm cannot identify whether guest is > being migrated or destroyed, and thus we get the root cause of this bug. Just to be pedantic, what you meant to say there was: "in RHEL on RHEVM controlled Hypervisor case (which includes both RHEVH and RHEL + vdsm)". This issue isn't RHEVH specific, it's more accurately vdsm specific And I'm not sure why you say that "vdsm cannot identify whether a guest is being migrated or destroyed". vdsm is the component that issues the migrate/destroy command, so it is the canonical place to ask for that information > Option 1. Current implementation, where a guest stops consuming an entitlement. > Current proposal is described in comments 23 and 25, which is not acceptable > due to 4 hour delay. > > Option 2. Let guest continue consuming an entitlement. This way, removed guest > entitlements will accumulate. It is a distinct disadvantage for RHEV/RHEVH > compared to RHEV/RHEL-KVM. Restating for clarity: It is a distinct disadvantage for RHEV/vdsm as compared to RHEL/libvirt.
Perry, thanks for clarifications. It seems that the solution has to have two distinct parts: 1. Improvement of vdsm <-> virt-who interaction to obtain accurate information about guest UUIDs as guests are suspended/powered off/destroyed/migrated. 2. Corrections to candlepin behavior upon receiving updated information from virt-who. Is this accurate?
(In reply to comment #36) > Perry, thanks for clarifications. > > It seems that the solution has to have two distinct parts: > > 1. Improvement of vdsm <-> virt-who interaction to obtain accurate information > about guest UUIDs as guests are suspended/powered off/destroyed/migrated. > > 2. Corrections to candlepin behavior upon receiving updated information from > virt-who. > > Is this accurate? Sounds accurate to me. Basically vdsm needs to proactively tell virt-who what is going on via some events in the same way that you're saying libvirt does? To be honest I have no idea how virt-who is getting event notification for things like migrate vs. suspend vs. destroy from libvirt. (dallan, do you know how this is currently working today)?
We had an email thread about this last November, and I recommended that the correct way to learn about changes to VM state was through events. I don't know if that suggestion was adopted though.
A blocker for this bug has been closed as "NOTABUG". See comment 36/37 which identified this blocker. Comment 38 won't work since libvirt with vdsm22 is not available in RHEL5. In this light, the only option that I see available is the following solution: 1. candlepin never removed entitlement certificates for guest UUIDs reported from RHEL5 hosts.
Mike, I just wrapped up a patch that implements what you mention in Comment 40, but also performs an autosubscribe when a guest is migrated (assuming it is reported first by host #2 before host #1 reports it as gone; otherwise we maintain current behavior). Should I leave in the autosubscribe, or back it out? Please set bz to ASSIGNED if you want it back out and I'll revert that portion of the patch. d81639b master 0.5.13+
After conversation with mkhusid, I am altering the fix so that entitlements for guests are not revoked when they are paused. Everything else will work as-is. This should resolve the originally reported issue.
Hui, Candlepin 0.5.14 has the fix for this, and should be available for you. Let me know if you are not able to get it from the sam repo. 868eb08 master 0.5.14+
Hui, I am looking into the migration issue now. Is it ok to mark this bug as VERIFIED, since the migration bug is a new bz?
Verified as comment 45,marking as verified.