Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1427465 - upgrade w/ customer db failed at step: import_puppet_modules
Summary: upgrade w/ customer db failed at step: import_puppet_modules
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.2.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: Sat6_Upgrades
TreeView+ depends on / blocked
 
Reported: 2017-02-28 09:42 UTC by Sachin Ghai
Modified: 2017-03-06 12:23 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-02 08:49:04 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Sachin Ghai 2017-02-28 09:42:55 UTC 
Description of problem:

Customer db upgrade from 6.1 -> 6.2.8 snap4 failed at below step:


[SERVER] out: Upgrade Step: import_puppet_modules (this may take a while) ...
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
[SERVER] out: rake aborted!
[SERVER] out: RestClient::ResourceNotFound: 404 Resource Not Found
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in `block in get_response'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `call'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `process_result'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:50:in `block in transmit'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:44:in `transmit'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:78:in `get_response'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:66:in `call'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/resources/repository.rb:138:in `unit_search'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/services/katello/pulp/pulp_content_unit.rb:63:in `ids_for_repository'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:99:in `block in import_all'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
[SERVER] out: /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:98:in `import_all'
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/2.4/import_puppet_modules.rake:9:in `block (4 levels) in <top (required)>'
[SERVER] out: Tasks: TOP => katello:upgrades:2.4:import_puppet_modules
[SERVER] out: (See full trace by running task with --trace)
[SERVER] out: Importing Puppet Modules
[SERVER] out: 
[SERVER] out: Upgrade step import_puppet_modules failed. Check logs for more information.

Version-Release number of selected component (if applicable):
upgrade sat6.1 -> 6.2.8 snap4

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Sachin Ghai 2017-02-28 09:47:12 UTC 
~]# foreman-rake katello:upgrades:2.4:import_puppet_modules --trace
** Invoke katello:upgrades:2.4:import_puppet_modules (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
** Execute katello:upgrades:2.4:import_puppet_modules
Importing Puppet Modules
rake aborted!
RestClient::ResourceNotFound: 404 Resource Not Found
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:50:in `block in transmit'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:853:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.38/lib/restclient_ext/request.rb:44:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/resource.rb:67:in `post'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:78:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:66:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/resources/repository.rb:138:in `unit_search'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/services/katello/pulp/pulp_content_unit.rb:63:in `ids_for_repository'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:99:in `block in import_all'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block (2 levels) in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:52:in `block in find_each'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:125:in `find_in_batches'
/opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/relation/batches.rb:51:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/content_view_puppet_environment.rb:98:in `import_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/lib/katello/tasks/upgrades/2.4/import_puppet_modules.rake:9:in `block (4 levels) in <top (required)>'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/ruby/monitor.rb:211:in `mon_synchronize'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
/opt/rh/rh-ruby22/root/usr/share/gems/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
/opt/rh/rh-ruby22/root/usr/bin/rake:33:in `<main>'
Tasks: TOP => katello:upgrades:2.4:import_puppet_module
Comment 3 Brad Buckingham 2017-03-01 12:17:16 UTC 
The behavior above looks like it could be a duplicate of bug 1418292 opened by another user; therefore, it was likely not introduced by 6.2.8.

From a quick look, it seems that the database is missing data in the backend service (e.g. pulp in this case).  It is likely that running rake tasks from bug 1375075 may resolve this (e.g. correct_puppet_environments/correct_repositories); however, that bugzilla is not currently included in 6.2.8.

Note: we have another bugzilla to introduce executing these rake tasks as part of the --upgrade.  That effort is tracked with bug 1425437
Comment 4 Brad Buckingham 2017-03-01 12:20:00 UTC 
Did upgrades work with this database/backup on previous 6.2.z?  (e.g. 6.2.7)
Comment 5 Sachin Ghai 2017-03-01 12:28:11 UTC 
@Brad: yeah.. I didn't see this issue w/ 6.2.7 as well as w/ last three snaps of 6.2.8. It appears only w/ snap4.
Comment 6 Justin Sherrill 2017-03-01 13:04:34 UTC 
Sachin, 

Did you test upgrades using to 6.2.7 this same customer db?  This error seems like a data integrity error related to the data that exists prior to the upgrade, and not really a bug.
Comment 8 Sachin Ghai 2017-03-02 08:35:38 UTC 
Okay.. I re-tried the upgrade w/ same customer db and this time upgrade went fine. I didn't see the reported issue..

logs snippet from CI job:

--
[SERVER] out: 
[SERVER] out: Upgrade Step: import_puppet_modules (this may take a while) ...
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.105/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
[SERVER] out: Importing Puppet Modules
[SERVER] out: 
[SERVER] out: Upgrade Step: import_subscriptions (this may take a while) ...
[SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.41/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
[SERVER]
--

removing the blocker flag.
Comment 9 Sachin Ghai 2017-03-02 08:49:04 UTC 
Closing this as its no longer reproducible.
Note You need to log in before you can comment on or make changes to this bug.