Bug 2067331

Description of problem:

Upgrade to Satellite 6.9 and 6.10 fails with error "ActiveRecord::RecordInvalid: Validation failed: Remote execution features is invalid" during db:seed stage

The issue is only reproducible in the infra of the users who had ran into the problem. 


Version-Release number of selected component (if applicable):

Satellite 6.9 ( and one case of Satellite 6.10 )

How reproducible:

In the customer's infra


Steps to Reproduce:
1. Install and configure Satellite 6.8 and Upgrade to 6.9 
2. Or else Install and configure Satellite 6.9, configure the insights stuff and inventory sync etc and then upgrade to 6.10


Actual results:

db:seed fails:

2021-10-19 10:55:40 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Starting to evaluate the resource (1750 of 2365)
2021-10-19 10:55:40 [DEBUG ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: '/usr/sbin/foreman-rake db:seed' won't be executed because of failed check 'refreshonly'
2021-10-19 10:55:40 [DEBUG ] [configure] Exec[foreman-rake-db:seed](provider=posix): Executing '/usr/sbin/foreman-rake db:seed'
2021-10-19 10:55:40 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:seed'
2021-10-19 10:56:07 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
2021-10-19 10:56:07 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted!
2021-10-19 10:56:07 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: ActiveRecord::RecordInvalid: Validation failed: Remote execution features is invalid



Expected results:

No such issues


Additional info:

For whichever customer this issue had happened, the only root cause was that, there was a rh_cloud_remediate_hosts entry duplicated in remote_execution_features table and they never had any job_template_id assigned to them either. And only solution was to get rid of those entries and re-run the db:seed followed by satellite-installer.  

# su - postgres -c "echo \"select id,label,name,job_template_id from remote_execution_features order by id;\" | psql foreman"
id |               label               |                   name                   | job_template_id
----+-----------------------------------+------------------------------------------+-----------------

--- 8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< ----- 

18 | rh_cloud_remediate_hosts          | Apply Insights recommendations           |                    
19 | rh_cloud_remediate_hosts          | Apply Insights recommendations           |                    

--- 8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< -----  8< ----- 
(22 rows)

And we have no idea how these duplications are getting created.