Bug 1168530

Summary: [RFE] qemu-ga cannot be used after hotplug it but can be used after restart libvirtd
Product: Red Hat Enterprise Linux 7 Reporter: Luyao Huang <lhuang>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.2CC: dyuan, lmiksik, mzhan, pkrempa, rbalakri, shyu, zhwang
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.2.15-1.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-19 05:56:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luyao Huang 2014-11-27 09:01:18 UTC
description of problem:
qemu-ga cannot be used after hot-plug it but can be used after restart libvirtd

Version-Release number of selected component (if applicable):
libvirt-1.2.8-9.el7.x86_64


How reproducible:
100%

Steps to Reproduce:

1.prepare a guest install agent in guest but no agent settings in xml

# virsh dumpxml test3 --inactive

no agent settings

2.start guest
# virsh start test3
Domain test3 started

3.prepare a agent xml and use update-device to update
# cat agent.xml
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/test3.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

# virsh attach-device test3 agent.xml
Device attached successfully

4.use dumpxml check agent state and found :
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/test3.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

5.but cannot use agent with error:
# virsh domtime test3
error: argument unsupported: QEMU guest agent is not configured

6.
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

7.
start qemu-ga service in guest and guest agent can be used now

# virsh domtime test3
Time: 1416966559


Actual results:
qemu-ga cannot be used after hotplug it but can be used after restart libvirtd

Expected results:

Can be used after hotplug it
Or cannot be used after restart libvirtd

Additional info:

Because libvirt do not check the if the channel have been hotplug is a agent
(which will use a special virtio-channel).So cannot use it when hotplug it.

If libvirtd restart libvirt will call qemuProcessReconnect, and call qemuConnectAgent to check if guest have a agent, if it has will connect it.

Comment 2 Peter Krempa 2015-04-26 15:25:11 UTC
Fixed upstream:

commit a03e2d3a9927eb2d207dc83e3e45c2e403732330
Author: Peter Krempa <pkrempa>
Date:   Fri Apr 24 16:48:26 2015 +0200

    qemu: Connect to guest agent after channel hotplug
    
    If a user hot-attaches the guest agent channel libvirt would ignore it
    until the restart of libvirtd or shutdown/destroy and start of the VM
    itself.
    
    This patch adds code that opens or closes the guest agent connection
    according to the state of the guest agent channel according to
    connect/disconnect events.
    
    To allow opening the channel from the event handler qemuConnectAgent
    needed to be exported.

commit e1c04108d7602133d74c815929df2c81178d8dde
Author: Peter Krempa <pkrempa>
Date:   Fri Apr 24 16:43:38 2015 +0200

    qemu: agent: Differentiate errors when the agent channel was hotplugged
    
    When the guest agent channel gets hotplugged to a VM, libvirt would
    still report that "QEMU guest agent is not configured" rather than
    stating that the connection was not established yet.
    
    Currently the code won't be able to connect to the agent after hotplug
    but that will change in a later patch.
    
    As the qemuFindAgentConfig() helper is quite helpful in this case move
    it to a more usable place and export it.

v1.2.14-312-ga03e2d3

Comment 4 zhenfeng wang 2015-05-26 11:15:40 UTC
Hi Peter
Verify this bug with libvirt-1.2.15-2.el7.x86_64, all steps in comment0 could get the expect result. However, met other issues while do more testing with the following steps, please help check it thanks

1.Prepare a guest install agent in guest but no agent settings in xml
# virsh dumpxml 7.0 --inactive

no agent settings

2.Prepare a agent xml and use update-device to update
# cat agent.xml
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/7.0.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

# virsh attach-device 7.0 agent.xml --config
Device attached successfully


3.Start the guest, after guest start successfully, do S3 with the guest
# virsh dompmsuspend 7.0 --target mem
Domain 7.0 successfully suspended

# virsh list
 Id    Name                           State
----------------------------------------------------
 26    7.0                            pmsuspended

4.Detach the guest agent from the guest, then wakeup the guest
# virsh detach-device 7.0 agent.xml 
Device detached successfully

# virsh dompmwakeup 7.0
Domain 7.0 successfully woken up

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 26    7.0                            running

5.Re-attach the guest agent to the guest, found the guest agent will always stay in 'disconnected' status, also will fail to operate the virsh command depends on guest agent
# virsh dumpxml 7.0|grep agent
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/7.0.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>

# virsh domtime 7.0
error: Guest agent is not responding: QEMU guest agent is not connected

6.Restart libvirtd service, still couldn't connected guest agent
#systemctl restart libvirtd

# virsh dumpxml 7.0 |grep agent
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/7.0.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>

# time virsh domtime 7.0
error: Guest agent is not responding: Guest agent not available for now


real	0m5.009s
user	0m0.006s
sys	0m0.003s

7.Login in the guest, find the qemu-guest-agent service was in running status.

Comment 5 zhenfeng wang 2015-06-15 08:10:09 UTC
Hi
Just retest the steps in comment 4 with libvirt-1.2.16-1.el7.x86_64, got a different result that will fail to hotplug the guest agent to guest, so i dount that was there anything change for this part, can you help check it ? thanks  

1.prepare a guest install agent in guest but no agent settings in xml

# virsh dumpxml vm1 --inactive

no agent settings

2.start guest
# virsh start vm1
Domain vm1 started

3.prepare a agent xml 
# cat agent.xml
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/vm1.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

4.Attach the guest agent to the guest, will fail to operate it and get the following error
# virsh attach-device vm1 agent.xml 
error: Failed to attach device from agent.xml
error: internal error: virtio serial device has invalid address type

5.Attach the guest agent with --config option, could attach it successfully
# virsh attach-device vm1 agent.xml  --config
Device attached successfully

Comment 6 Peter Krempa 2015-06-30 11:39:49 UTC
(In reply to zhenfeng wang from comment #5)
> Hi
> Just retest the steps in comment 4 with libvirt-1.2.16-1.el7.x86_64, got a
> different result that will fail to hotplug the guest agent to guest, so i
> dount that was there anything change for this part, can you help check it ?
> thanks  
> 

...

> 3.prepare a agent xml 
> # cat agent.xml
>     <channel type='unix'>
>       <source mode='bind' path='/var/lib/libvirt/qemu/vm1.agent'/>
>       <target type='virtio' name='org.qemu.guest_agent.0'/>
>       <address type='virtio-serial' controller='0' bus='0' port='1'/>
>     </channel>
> 
> 4.Attach the guest agent to the guest, will fail to operate it and get the
> following error
> # virsh attach-device vm1 agent.xml 
> error: Failed to attach device from agent.xml
> error: internal error: virtio serial device has invalid address type

Looks like you reproduced the issue tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1230039

Comment 7 zhenfeng wang 2015-07-03 03:18:18 UTC
Hi Peter
I could still hit the issue in comment4 with libvirt-1.2.17-1.el7.x86_64, can you help check it? thanks

Comment 8 Peter Krempa 2015-07-17 13:16:59 UTC
Would you mind filing a bug that guest agent hotplug does not work correctly with S3-suspended guests? That is a separate set of problems due to the state the guest is in.

Comment 9 zhenfeng wang 2015-07-20 05:58:11 UTC
Thanks peter's reply, have file a bug 1244564 to track the comment4's issue, the other verify steps for the original bug are like following  

1.prepare a guest install agent in guest but no agent settings in xml

# virsh dumpxml virt-tests-vm1 --inactive

no agent settings

2.start guest
# virsh start virt-tests-vm1
Domain virt-tests-vm1 started

3.prepare a agent xml and use update-device to update
# cat agent.xml
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/virt-tests-vm1.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

# virsh attach-device virt-tests-vm1 agent.xml
Device attached successfully

4.use dumpxml check agent state and found :
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/virt-tests-vm1.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

5. Run corresponding virsh cmd, could get expect result
# virsh domtime virt-tests-vm1 --pretty
Time: 2015-07-20 05:55:11

6.
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service

7.Restart libvirtd or qemu-ga could still get the expect result

# virsh domtime virt-tests-vm1 --pretty
Time: 2015-07-20 05:56:18

Accoring to the upper steps, mark this bug verifed

Comment 10 zhenfeng wang 2015-10-13 08:37:56 UTC
Hi Peter
I just try to hotplug guest agent to the windows guest,  however, found it didn't work, unless restart the qemu-guest-agent service inside the windows guest. Re-try it with RHEL guest, it works for rhel guest after hotplug the guest agent, any difference? can you help check it thanks. 


pkginfo
libvirt-1.2.17-13.el7.x86_64

steps
1.start a windows guest with guest agent installation
#virsh dumpxml win7
--
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-win7/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>


2.Prepare a xml for agent
#cat agent.xml
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-win7/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

3.Detach the agent from the windows guest
#virsh detach-device win7 agent.xml
Device detached successfully

# virsh domtime win7
error: argument unsupported: QEMU guest agent is not configured

4.Re-attach the agent to the windows guest, check the guest's xml, found the agent wasn't in "connected" status, also run command which depends on guest agent will get error

# virsh attach-device win7 agent.xml 
Device attached successfully

#virsh dumpxml win7
--
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-win7/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

# virsh domtime win7 --pretty
error: Guest agent is not responding: QEMU guest agent is not connected

5.Restart libvirtd service, still didn't work
# service libvirtd restart
Redirecting to /bin/systemctl restart  libvirtd.service
# virsh domtime win7 --pretty
error: Guest agent is not responding: Guest agent not available for now


6.Restart agent service inside windows guest, agent works
# virsh domtime win7 --pretty
Time: 2015-10-13 15:32:48

Comment 11 Peter Krempa 2015-10-13 09:29:32 UTC
(In reply to zhenfeng wang from comment #10)

...

> # virsh domtime win7 --pretty
> error: Guest agent is not responding: QEMU guest agent is not connected
> 
> 5.Restart libvirtd service, still didn't work
> # service libvirtd restart
> Redirecting to /bin/systemctl restart  libvirtd.service
> # virsh domtime win7 --pretty
> error: Guest agent is not responding: Guest agent not available for now
> 
> 
> 6.Restart agent service inside windows guest, agent works
> # virsh domtime win7 --pretty
> Time: 2015-10-13 15:32:48

Well, obviously the guest agent service in windows doesn't handle vanishing of the virtio channel and it's re-attachment gracefully. AFAIK in linux this is handled by systemd.

Comment 12 zhenfeng wang 2015-10-14 11:07:32 UTC
Hi peter
Thanks for your quick response, have another doubt that is it availabel to get qemu-guest-agent on windows to  handle the vanishing of the virtio channel? so that it's coincident with rhel guest's behavior which didn't need restart qemu-guest-agent service inside guest after hotplug

Comment 13 Peter Krempa 2015-10-14 11:54:25 UTC
I don't know if it would make sense on windows. That is something for the guest agent developers.

Comment 15 errata-xmlrpc 2015-11-19 05:56:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html