Collaborator note
#######################
Imported customer db and checked table

db query:
vmdb_production=# select provider_id,parent_ems_id,tenant_id,name,realm,last_refresh_error from ext_management_systems ;
 provider_id | parent_ems_id |   tenant_id    |   name   | realm |               last_refresh_error               
-------------+---------------+----------------+----------+-------+------------------------------------------------
             |               | 10000000000001 | CTC DMZ  |       | 
             |               | 10000000000001 | CTC_Core |       | Validation failed: Name has already been taken


evm err msg: 
MIQ(ManageIQ::Providers::OpenshiftEnterprise::ContainerManager::Refresher#refresh) EMS: [CTC Core], id: [10000000000001] Refresh failed

Seems like tenant_id is conflict or missing provider_id.
My understanding, tenant_id could be same as long as under different provider_id.

Need update from engineering.

Problem in container_label_tag_mapping breaking refresh.
Seems to be a different error from bug 1358257

I need dumps of container_label_tag_mappings and tags

Possible work around(did not test this): 
0. validate refresh broken
1. dump container_label_tag_mappings table
2. note: auto tagging would not work
3. test if refresh still breaks

Note: besides handling this specific flow we might want to suppress and log exceptions (ERROR)  during refresh. That would allow it to complete regardless of
label to tag mapping

Federico what is your opinion on this?

(In reply to Mooli Tayer from comment #7)
> Note: besides handling this specific flow we might want to suppress and log
> exceptions (ERROR)  during refresh. That would allow it to complete
> regardless of
> label to tag mapping
> 
> Federico what is your opinion on this?

OK to continue refresh but do not suppress the errors too much (e.g. stacktrace etc.) because we want to be able to debug the issues.

Errors must be extremely evident.

Created attachment 1205334 [details]
evm.log

This log has the debug patch in it.

Created attachment 1205335 [details]
production.log

The production.log has rails debug logging on if that is needed.

Short update: in support call last week we "fixed" refresh for the customer by deleting a few mapping lines — but the bugs are still there.

I believe this was caused by bug 1382347.
bug 1382361 could affect this customer too.
We're working on fixing all that + adding UI for auto-tagging (work in progress https://github.com/ManageIQ/manageiq/pull/11591).

Let's keep development updates to those BZs / github PRs;
this BZ will remain focused on the specific customer's refresh issue, in case it re-surfaces.

[----] E, [2016-10-20T08:21:53.931633 #24787:cc5998] ERROR -- :
MIQ(ManageIQ::Providers::OpenshiftEnterprise::ContainerM
anager::Refresher#refresh) EMS: [Pathfinder], id: [1000000000001] Refresh
failed
[----] E, [2016-10-20T08:21:53.931948 #24787:cc5998] ERROR -- :
[ActiveRecord::RecordInvalid]: Validation failed: Name h
as already been taken  Method:[rescue in block in refresh]
[----] E, [2016-10-20T08:21:53.932055 #24787:cc5998] ERROR -- :
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active
_record/validations.rb:78:in `raise_validation_error'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/validations.rb:50:in
`save!'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/attribute_methods/dirty.rb:30:in
`save!'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/transactions.rb:324:in
`block in save!'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/transactions.rb:395:in
`block in with_transaction_return
ing_status'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:
in `block in transaction'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/connection_adapters/abstract/transaction.rb:189:in
`with
in_new_transaction'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:
in `transaction'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/transactions.rb:211:in
`transaction'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/transactions.rb:392:in
`with_transaction_returning_statu
s'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/transactions.rb:324:in
`save!'
/opt/rh/cfme-gemset/gems/activerecord-5.0.0.1/lib/active_record/suppressor.rb:45:in
`save!'

Saif can you elaborate on the assistance you need?

Federico _ that most recent error the customer provided.

the customer attached the database dump to the case but its too large to added the bugzilla.

Short update on underlying work:

Refresh & auto-tagging breakages - bug 1382347, bug 1382361 - PR merged (master, 5.7 and 5.6), on QE.
We implemented UI for auto-tagging - bug 1383405 - PR merged (master, 5.7).
Also, enabled auto-tagging tags in reports - bug 1388659 - PR merged (master, 5.7, 5.6).

Of course QE testing of all that may uncover additional problems.

There is also ongoing work to support direct use of labels in reports (without mapping them to tags) - bug 1382720.

*** Bug 1373712 has been marked as a duplicate of this bug. ***

removing "Blocks:" 4 unrelated bugs, I don't know why bugzilla decided I added them.