Bug 1427465

Summary: upgrade w/ customer db failed at step: import_puppet_modules
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: UpgradesAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WORKSFORME QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.2.8CC: bbuckingham, egolov, inecas, jcallaha, jsherril, mbacovsk, sghai
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-02 08:49:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1410795    

Description Sachin Ghai 2017-02-28 09:42:55 UTC
Description of problem:

Customer db upgrade from 6.1 -> 6.2.8 snap4 failed at below step:


[SERVER] out: Upgrade Step: import_puppet_modules (this may take a while) ...
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
[SERVER] out: rake aborted!
[SERVER] out: RestClient::ResourceNotFound: 404 Resource Not Found
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in `block in get_response'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `call'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `process_result'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:50:in `block in transmit'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:44:in `transmit'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:78:in `get_response'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:66:in `call'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/resources/repository.rb:138:in `unit_search'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/services/katello/pulp/pulp_content_unit.rb:63:in `ids_for_repository'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:99:in `block in import_all'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:98:in `import_all'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/2.4/import_puppet_modules.rake:9:in `block (4 levels) in <top (required)>'
[SERVER] out: Tasks: TOP => katello:upgrades:2.4:import_puppet_modules
[SERVER] out: (See full trace by running task with --trace)
[SERVER] out: Importing Puppet Modules
[SERVER] out: 
[SERVER] out: Upgrade step import_puppet_modules failed. Check logs for more information.

Version-Release number of selected component (if applicable):
upgrade sat6.1 -> 6.2.8 snap4

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Sachin Ghai 2017-02-28 09:47:12 UTC
~]# foreman-rake katello:upgrades:2.4:import_puppet_modules --trace
** Invoke katello:upgrades:2.4:import_puppet_modules (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
** Execute katello:upgrades:2.4:import_puppet_modules
Importing Puppet Modules
rake aborted!
RestClient::ResourceNotFound: 404 Resource Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:44:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:78:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/resources/repository.rb:138:in `unit_search'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/services/katello/pulp/pulp_content_unit.rb:63:in `ids_for_repository'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:99:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:98:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/2.4/import_puppet_modules.rake:9:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
/opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'
Tasks: TOP => katello:upgrades:2.4:import_puppet_module

Comment 3 Brad Buckingham 2017-03-01 12:17:16 UTC
The behavior above looks like it could be a duplicate of bug 1418292 opened by another user; therefore, it was likely not introduced by 6.2.8.

From a quick look, it seems that the database is missing data in the backend service (e.g. pulp in this case).  It is likely that running rake tasks from bug 1375075 may resolve this (e.g. correct_puppet_environments/correct_repositories); however, that bugzilla is not currently included in 6.2.8.

Note: we have another bugzilla to introduce executing these rake tasks as part of the --upgrade.  That effort is tracked with bug 1425437

Comment 4 Brad Buckingham 2017-03-01 12:20:00 UTC
Did upgrades work with this database/backup on previous 6.2.z?  (e.g. 6.2.7)

Comment 5 Sachin Ghai 2017-03-01 12:28:11 UTC
@Brad: yeah.. I didn't see this issue w/ 6.2.7 as well as w/ last three snaps of 6.2.8. It appears only w/ snap4.

Comment 6 Justin Sherrill 2017-03-01 13:04:34 UTC
Sachin, 

Did you test upgrades using to 6.2.7 this same customer db?  This error seems like a data integrity error related to the data that exists prior to the upgrade, and not really a bug.

Comment 8 Sachin Ghai 2017-03-02 08:35:38 UTC
Okay.. I re-tried the upgrade w/ same customer db and this time upgrade went fine. I didn't see the reported issue..

logs snippet from CI job:

--
[SERVER] out: 
[SERVER] out: Upgrade Step: import_puppet_modules (this may take a while) ...
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
[SERVER] out: Importing Puppet Modules
[SERVER] out: 
[SERVER] out: Upgrade Step: import_subscriptions (this may take a while) ...
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
[SERVER]
--

removing the blocker flag.

Comment 9 Sachin Ghai 2017-03-02 08:49:04 UTC
Closing this as its no longer reproducible.