Bug 1259550

I have similar issues with registration of VMWare hosts using virt-who.  My findings / assumptions are as follows based upon a decent amount of digging:

1.  You run virt-who against your virtual environment - it pulls a list of VMs ... good so far.
2.  virt-who reports this big list to SAM on the satellite 6.1 master - It's a big list so the SSL connection from virt-who to Satellite times out.  That's OK because a bunch of data is in Katello's processing queue for creation of new VM hosts and updates to the mappings of guest to host relationships ... it'll just keep processing - not ideal but still OK.
3.  Here's the problem ... even with oneshot mode, virt-who in it's infinite wisdom reconnects to Satellite and RESENDS the list of changes
4.  it seems as though Katello receives this as "OK, here's more to process" and starts a PARALLEL thread processing the "new" data
5. Here's the stupid bit ... in the foreman database, there's a katello-systems table.  This has an ID, the UUID and the hostname info in it.  Two major design issues here ... First, there is no unique constraint in the table enforcing that the hostname / UUID combos have to be unique (so you can have duplicates per the database) and apparently Katello doesn't have great error checkin in this case because it goes ahead and creates duplicate records.

From here, in the GUI, you can see 2 entries for some of the content hosts created after virt-who.  You select to delete them in a clean up effort and Katello deletes all references to them EXCEPT THE SECOND RECORD IN THE KATELLO-SYSTEMS table.  This orphan record then wreaks havoc in that it breaks your ability to get content-host lists, among other things because you keep throwing "409 Gone" errors. 

I found a work-around .... but this is NOT something I got from Red Hat so it's probably not a supported action (though appears to work on my 6.1.1 system with no ill effects):
1. identify duplicate records in the katello_systems table
2. delete second record by ID, NOT BY UUID OR HOSTNAME since these are now not unique
3. after cleaning out the table's duplicates, run "foreman-rake katello:reindex"

For convenience here are the queries you'll need to delete duplicate records (I had 446 duplicates...)

delete collections FK if needed :
delete from katello_system_host_collections where system_id in (select id from (select id,row_number() over (partition by uuid,name order by id) as rnum from katello_systems) t where t.rnum > 1);

delete hypervisor from systems (it keeps the one with the lowest ID) :
delete from katello_systems where id in (select id from (select id,row_number() over (partition by uuid,name order by id) as rnum from katello_systems) t where t.rnum > 1);

As stated by Sean, this is not supported by Red Hat.

We have often seen issues where different UUIDs for a single host are returned depending on how they are queried. For this reason we recommend using one and only one virt-who backend-type & connection type for querying a given hypervisor. When using libvirt for example, depending on the version of libvirt being used & whether you query using the local interface or remote interface a different UUID will be returned. 

Have you tried this interaction with RHEV using only libvirt-remote mode for all the virt-who connections? Do you still end up with duplicate records if remote mode is used for all virt-who connections to libvirt? 

If you are using the same backend for querying from virt-who and virt-who is not reporting different UUIDs then would you consider this issue a duplicate of BZ 1365248?