Bug 1326388

Summary: virt-who is not reporting ESX hosts above 100
Product: Red Hat Enterprise Linux 7 Reporter: Rick Dixon <rdixon>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED NOTABUG QA Contact: Li Bin Liu <liliu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: hsun, ovasik, sgao, shihliu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-13 20:34:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Rick Dixon 2016-04-12 14:58:17 UTC
Description of problem:

I am seeing that there are missing ESX servers that are not getting reported to the satellite server.  It has scanned in 529 ESX hosts, but for systems that are named tdc-tst-srv-host-101 and higher, it's not reporting them.  It accounts for 01 - 100, but stops there.

Version-Release number of selected component (if applicable):

virt-who-0.14-9

How reproducible:

Always

Steps to Reproduce:
1. Create 110 ESX hosts, all with the same naming convention: tdc-tst-srv-host-001 through tdc-tst-srv-host-109

2. Install virt-who on a separate VM, per the installation guide, and configure as below:

~~~
[root@example]# cat /etc/sysconfig/virt-who
# Enviromental variables for virt-who service can be specified here.
#
# These enviromental variables are only used when starting virt-who as service,
# otherwise you must specify them manually.
#
# Note that if some value contains special character, it must be escapted
# or the value must be quoted - for example ampersand in the password.

# Start virt-who on background, perform doublefork and monitor for virtual guest
# events (if possible). It is NOT recommended to turn off this option for
# starting virt-who as service.
VIRTWHO_BACKGROUND=1

# Enable debugging output.
VIRTWHO_DEBUG=1

# Send the list of guest IDs and exit immediately.
#VIRTWHO_ONE_SHOT=0

# Acquire and send list of virtual guest each N seconds, 0 means default
# configuration.
#VIRTWHO_INTERVAL=0

# Virt-who subscription manager backend. Enable only one option from the following:
# Report to Subscription Asset Manager (SAM) or the Red Hat Customer Portal
# VIRTWHO_SAM=1

# Report to Sattellite version 6
VIRTWHO_SATELLITE6=1

# Report to Satellite verision 5
# VIRTWHO_SATELLITE5=0
# Report to Satellite [Legacy]
# VIRTWHO_SATELLITE=0


# Following configuration is now deprecated. Please use configuration file
# in /etc/virt-who.d/. See virt-who-config(5) for details how to configure it.

# virt-who mode, enable only one option from following 5:
# Use libvirt to list virtual guests [default]
#VIRTWHO_LIBVIRT=1
# Use vdsm to list virtual guests
#VIRTWHO_VDSM=0

# Register ESX machines using vCenter
# VIRTWHO_ESX=1

# Register guests using RHEV-M
#VIRTWHO_RHEVM=0
# Register guest using Hyper-V
#VIRTWHO_HYPERV=0

# Options for Libvirt mode
#VIRTWHO_LIBVIRT_OWNER=
#VIRTWHO_LIBVIRT_ENV=
#VIRTWHO_LIBVIRT_SERVER=
#VIRTWHO_LIBVIRT_USERNAME=
#VIRTWHO_LIBVIRT_PASSWORD=

# Options for ESX mode
#VIRTWHO_ESX_OWNER=
#VIRTWHO_ESX_ENV=
#VIRTWHO_ESX_SERVER=
#VIRTWHO_ESX_USERNAME=
#VIRTWHO_ESX_PASSWORD=

# Options for RHEV-M mode
#VIRTWHO_RHEVM_OWNER=
#VIRTWHO_RHEVM_ENV=
#VIRTWHO_RHEVM_SERVER=
#VIRTWHO_RHEVM_USERNAME=
#VIRTWHO_RHEVM_PASSWORD=

# Options for HYPER-V mode
#VIRTWHO_HYPERV_OWNER=
#VIRTWHO_HYPERV_ENV=
#VIRTWHO_HYPERV_SERVER=
#VIRTWHO_HYPERV_USERNAME=
#VIRTWHO_HYPERV_PASSWORD=

# Options for Satellite 5 backend
#VIRTWHO_SATELLITE_SERVER=
#VIRTWHO_SATELLITE_USERNAME=
#VIRTWHO_SATELLITE_PASSWORD=
###########################################
###########################################


[root@example]# cat /etc/virt-who.d/vcenter05
# Virt-who config

[vcenter05]

# type=esx,rhevm,kvm
type=esx

# server=vcenter OR individual ESXi host
server=vcenter.example.com

# username syntax for Active Directory users is either
# UPN based=user
# NETBIOS based domain\\user
# Note the double slashes above.
username=<REDACTED>

# Get encrypted password from virt-who-password
encrypted_password=<REDACTED>

# Organization Label in Satellite 6 where the hypervisors
# should be reported to.
owner=OWNER

# Note, do NOT use any env other than Library or virt-who will fail
# (bz1283740)
env=Library

# Report hypervisors with hostname and not UUID because UUIDs are ugly.
hypervisor_id=hostname
~~~

Actual results: 

After polling the environment, virt-who stops reporting after tdc-tst-srv-host-001. From rhsm.log:
~~~
[...SNIP...]
    "tdc-tst-srv-host-100.example.com": [
        {
            "attributes": {
                "active": 1, 
                "virtWhoType": "esx"
            }, 
            "guestId": "123-456-789-etc-etc-etc", 
            "state": 1
        } 
[...SNIP...]        
    ]
}
[DEBUG]  @subscriptionmanager.py:111 - Authenticating with certificate: /etc/pki/consumer/cert.pem
[INFO]  @virtwho.py:147 - virt-who host/guest association update successful
~~~

Expected results:

All 110 hosts should be reported by virt-who.

Additional info:

This is a somewhat large envrionment:

     grep 'guestId' rhsm.log | sort -u | wc -l
     7456