Bug 1115436

Summary: Refusing to open BOTH of two host connections
Product: [Fedora] Fedora Reporter: Dr. David Alan Gilbert <dgilbert>
Component: virt-managerAssignee: Cole Robinson <crobinso>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: berrange, crobinso, dgilbert, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-02 14:34:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
virt-manager log file none

Description Dr. David Alan Gilbert 2014-07-02 11:09:25 UTC
Created attachment 914110 [details]
virt-manager log file

Description of problem:
I have two remote hosts; vl402 and vl403 that I use for migration testing.  I've got them set up as host connections in virt-manager.
Clicking on either one of them is connecting and showing me the list of VMs on that host, however if I then click on the other host it closes the first connection, and the same in the opposite direction.

(F20 desktop, RHEL7 hosts)
The ssh connections to the two hosts are open OK.

(This used to work, not quite sure when it broke - I haven't used it for a week or two)

On the hosts I see:
Jul 02 06:45:03 virtlab403.virt.bos.redhat.com libvirtd[1588]: metadata not found: Requested metadata element is not present
Jul 02 06:47:41 virtlab403.virt.bos.redhat.com libvirtd[1588]: No response from client 0x7f8337e44e70 after 5 keepalive messages in 30 seconds
Jul 02 06:50:51 virtlab403.virt.bos.redhat.com libvirtd[1588]: No response from client 0x7f8337e45220 after 5 keepalive messages in 30 seconds


Version-Release number of selected component (if applicable):
virt-manager-1.0.1-3.fc20.noarch
libvirt-client-1.1.3.5-2.fc20.x86_64


How reproducible:
100% at the moment

Steps to Reproduce:
1. double click on host 1 and wait for it to connect
2. double click on host 2 and wait for it to connect

Actual results:
Host 1 disconnects after stage 2

Expected results:
Both hosts stay connected

Additional info:

Comment 1 Cole Robinson 2014-07-02 14:04:44 UTC
Are vl402 and and vl403 the same machine? The capabilities output shows the same UUID. Not sure if that would cause issues or not.

Can you try connecting to both hosts with virsh and issuing a few commands?

terminal #1: virsh --connect qemu+ssh://root@vl402/system
terminal #2: virsh --connect qemu+ssh://root@vl403/system

Then try calling nodeinfo and capabilities on both, see if anything hangs

Comment 2 Dr. David Alan Gilbert 2014-07-02 14:19:24 UTC
It does seem to have started working now (the only thing I did was remove some stale macvtab entries I'd manually added previously on those machinbes - not sure if it's related or not);
anyway the nodeinfo/capabilities - as collected now; note tehre is a 1 bit difference in the UUID:

    <uuid>44454c4c-3100-104e-8034-c6c04f465831</uuid>
    <uuid>44454c4c-3100-104e-8035-c6c04f465831</uuid>
                                ^
they're identical twins of machines.

[dgilbert@dgilbert-t530 ~]$ virsh --connect qemu+ssh://root@vl402/system
Welcome to virsh, the virtualisation interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # nodeinfo
CPU model:           x86_64
CPU(s):              8
CPU frequency:       1199 MHz
CPU socket(s):       1
Core(s) per socket:  4
Thread(s) per core:  1
NUMA cell(s):        2
Memory size:         24476780 KiB

virsh # capabilities
<capabilities>

  <host>
    <uuid>44454c4c-3100-104e-8035-c6c04f465831</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>SandyBridge</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='1'/>
      <feature name='pdpe1gb'/>
      <feature name='osxsave'/>
      <feature name='dca'/>
      <feature name='pcid'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>                                                                                           
      <feature name='ht'/>                                                                                           
      <feature name='ss'/>                                                                                           
      <feature name='acpi'/>                                                                                         
      <feature name='ds'/>                                                                                           
      <feature name='vme'/>                                                                                          
    </cpu>                                                                                                           
    <power_management>                                                                                               
      <suspend_disk/>                                                                                                
      <suspend_hybrid/>                                                                                              
    </power_management>                                                                                              
    <migration_features>                                                                                             
      <live/>                                                                                                        
      <uri_transports>                                                                                               
        <uri_transport>tcp</uri_transport>                                                                           
      </uri_transports>                                                                                              
    </migration_features>                                                                                            
    <topology>                                                                                                       
      <cells num='2'>                                                                                                
        <cell id='0'>                                                                                                
          <memory unit='KiB'>12536364</memory>                                                                       
          <cpus num='4'>                                                                                             
            <cpu id='0' socket_id='0' core_id='0' siblings='0'/>                                                     
            <cpu id='2' socket_id='0' core_id='1' siblings='2'/>                                                     
            <cpu id='4' socket_id='0' core_id='2' siblings='4'/>
            <cpu id='6' socket_id='0' core_id='3' siblings='6'/>
          </cpus>
        </cell>
        <cell id='1'>
          <memory unit='KiB'>12582912</memory>
          <cpus num='4'>
            <cpu id='1' socket_id='1' core_id='0' siblings='1'/>
            <cpu id='3' socket_id='1' core_id='1' siblings='3'/>
            <cpu id='5' socket_id='1' core_id='2' siblings='5'/>
            <cpu id='7' socket_id='1' core_id='3' siblings='7'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine canonical='pc-i440fx-rhel7.0.0' maxCpus='255'>pc</machine>
      <machine canonical='pc-q35-rhel7.0.0' maxCpus='255'>q35</machine>
      <machine maxCpus='255'>rhel6.0.0</machine>
      <machine maxCpus='255'>rhel6.1.0</machine>
      <machine maxCpus='255'>rhel6.2.0</machine>
      <machine maxCpus='255'>rhel6.3.0</machine>
      <machine maxCpus='255'>rhel6.4.0</machine>
      <machine maxCpus='255'>rhel6.5.0</machine>
      <machine maxCpus='1'>none</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine canonical='pc-i440fx-rhel7.0.0' maxCpus='255'>pc</machine>
      <machine canonical='pc-q35-rhel7.0.0' maxCpus='255'>q35</machine>
      <machine maxCpus='255'>rhel6.0.0</machine>
      <machine maxCpus='255'>rhel6.1.0</machine>
      <machine maxCpus='255'>rhel6.2.0</machine>
      <machine maxCpus='255'>rhel6.3.0</machine>
      <machine maxCpus='255'>rhel6.4.0</machine>
      <machine maxCpus='255'>rhel6.5.0</machine>
      <machine maxCpus='1'>none</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>

[dgilbert@dgilbert-t530 ~]$ virsh --connect qemu+ssh://root@vl403/system   
Welcome to virsh, the virtualisation interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # nodeinfo
CPU model:           x86_64
CPU(s):              8
CPU frequency:       1199 MHz
CPU socket(s):       1
Core(s) per socket:  4
Thread(s) per core:  1
NUMA cell(s):        2
Memory size:         24474896 KiB

virsh # capabilities
<capabilities>

  <host>
    <uuid>44454c4c-3100-104e-8034-c6c04f465831</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>SandyBridge</model>
      <vendor>Intel</vendor>
      <topology sockets='1' cores='4' threads='1'/>
      <feature name='pdpe1gb'/>
      <feature name='osxsave'/>
      <feature name='dca'/>
      <feature name='pcid'/>
      <feature name='pdcm'/>
      <feature name='xtpr'/>
      <feature name='tm2'/>
      <feature name='est'/>
      <feature name='smx'/>
      <feature name='vmx'/>
      <feature name='ds_cpl'/>
      <feature name='monitor'/>
      <feature name='dtes64'/>
      <feature name='pbe'/>
      <feature name='tm'/>
      <feature name='ht'/>
      <feature name='ss'/>
      <feature name='acpi'/>
      <feature name='ds'/>
      <feature name='vme'/>
    </cpu>
    <power_management>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='2'>
        <cell id='0'>
          <memory unit='KiB'>12536364</memory>
          <cpus num='4'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0'/>
            <cpu id='2' socket_id='0' core_id='1' siblings='2'/>
            <cpu id='4' socket_id='0' core_id='2' siblings='4'/>
            <cpu id='6' socket_id='0' core_id='3' siblings='6'/>
          </cpus>
        </cell>
        <cell id='1'>
          <memory unit='KiB'>12582912</memory>
          <cpus num='4'>
            <cpu id='1' socket_id='1' core_id='0' siblings='1'/>
            <cpu id='3' socket_id='1' core_id='1' siblings='3'/>
            <cpu id='5' socket_id='1' core_id='2' siblings='5'/>
            <cpu id='7' socket_id='1' core_id='3' siblings='7'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine canonical='pc-i440fx-rhel7.0.0' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>rhel6.4.0</machine>
      <machine canonical='pc-q35-rhel7.0.0' maxCpus='255'>q35</machine>
      <machine maxCpus='255'>rhel6.2.0</machine>
      <machine maxCpus='1'>none</machine>
      <machine maxCpus='255'>rhel6.1.0</machine>
      <machine maxCpus='255'>rhel6.5.0</machine>
      <machine maxCpus='255'>rhel6.0.0</machine>
      <machine maxCpus='255'>rhel6.3.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine canonical='pc-i440fx-rhel7.0.0' maxCpus='255'>pc</machine>
      <machine maxCpus='255'>rhel6.4.0</machine>
      <machine canonical='pc-q35-rhel7.0.0' maxCpus='255'>q35</machine>
      <machine maxCpus='255'>rhel6.2.0</machine>
      <machine maxCpus='1'>none</machine>
      <machine maxCpus='255'>rhel6.1.0</machine>
      <machine maxCpus='255'>rhel6.5.0</machine>
      <machine maxCpus='255'>rhel6.0.0</machine>
      <machine maxCpus='255'>rhel6.3.0</machine>
      <domain type='qemu'>
      </domain>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>

Comment 3 Cole Robinson 2014-07-02 14:34:46 UTC
Okay, if it isn't reproducible now, then just closing it. I can really tell what virt-manager could be doing wrong here that would make a libvirt API call hang like that. If you reproduce in the future, please reopen.