Bug 1339901

Summary: Upgrade[619 -> 6.2 snap13.1] failed with error: Report processor failed: Could not send report to Foreman at https://<sat-server>/api/reports: SSL_connect returned=1
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: UpgradesAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, lpramuk, mmccune
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: katello-installer-base-3.0.0.37-1,satellite-installer-6.2.0.5-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:31:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1335807    
Attachments:
Description Flags
foreman-debug none

Description Sachin Ghai 2016-05-26 06:13:36 UTC
Description of problem:
Installer raises error while upgrading sat6.1.9 -> sat6.2 snap13.1

[ WARN 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns:   20150622090115 ChangeReportedAt
[ WARN 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns:   20150705131449 AddTypeToReports
[ WARN 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns:   20150721131324 ChangeBookmarkReportController
[ WARN 2016-05-26 01:35:38 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns:   20150728122736 ChangeReportPermissions
[DEBUG 2016-05-26 01:36:18 main]  Processing report from qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com with processor Puppet::Reports::Foreman
[ERROR 2016-05-26 01:36:18 main]  Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
  20150622090115 ChangeReportedAt
  20150705131449 AddTypeToReports
  20150721131324 ChangeBookmarkReportController
  20150728122736 ChangeReportPermissions
[ERROR 2016-05-26 01:37:31 main]  Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
  20150622090115 ChangeReportedAt
  20150705131449 AddTypeToReports
  20150721131324 ChangeBookmarkReportController
  20150728122736 ChangeReportPermissions


Version-Release number of selected component (if applicable):
upgrade sat6.1.9 -> sat6.2 snap13.1

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
Installer should succeed without any error

Additional info:

[root@qe-sat6-upgrade-rhel7 ~]# satellite-installer --scenario satellite --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
Upgrade Step: migrate_gutterball...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
Installing             Done                                               [100%] [..................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
[r

Comment 1 Sachin Ghai 2016-05-26 06:14:11 UTC
Error: Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
["/usr/share/ruby/net/http.rb:918:in `connect'", "/usr/share/ruby/net/http.rb:918:in `block in connect'", "/usr/share/ruby/timeout.rb:52:in `timeout'", "/usr/share/ruby/net/http.rb:918:in `connect'", "/usr/share/ruby/net/http.rb:862:in `do_start'", "/usr/share/ruby/net/http.rb:851:in `start'", "/usr/share/ruby/net/http.rb:1367:in `request'", "/usr/share/ruby/vendor_ruby/puppet/reports/foreman.rb:62:in `process'", "/usr/share/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'", "/usr/share/ruby/vendor_ruby/puppet/indirector/report/processor.rb:53:in `block in processors'", "/usr/share/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `each'", "/usr/share/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/usr/share/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/usr/share/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'", "/usr/share/ruby/vendor_ruby/puppet/indirector/indirection.rb:283:in `save'", "/usr/share/ruby/vendor_ruby/puppet/configurer.rb:250:in `send_report'", "/usr/share/ruby/vendor_ruby/puppet/configurer.rb:242:in `run_internal'", "/usr/share/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'", "/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'", "/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'", "/usr/share/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'", "/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'", "/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'", "/usr/share/ruby/vendor_ruby/puppet/context.rb:64:in `override'", "/usr/share/ruby/vendor_ruby/puppet.rb:246:in `override'", "/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'", "/usr/share/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'", "/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'", "/usr/share/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'", "/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'", "/usr/share/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'", "/usr/share/ruby/vendor_ruby/puppet/application.rb:381:in `run'", "/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'", "/usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'", "/usr/bin/puppet:8:in `<main>'"]

Comment 2 Sachin Ghai 2016-05-26 06:16:20 UTC
Pending Migrations:

[ERROR 2016-05-26 01:37:31 main]  Report processor failed: Could not send report to Foreman at https://qe-sat6-upgrade-rhel7.satqe.lab.eng.rdu2.redhat.com/api/reports: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert unknown ca
[ERROR 2016-05-26 01:37:31 main] Run `rake db:migrate` to update your database then try again.
You have 24 pending migrations:
  20150612135546 CreateHostStatus
  20150614171717 RenamePuppetclassCountersForLkSti
  20150618093433 RemoveUnusedFieldsFromHosts
  20150622090115 ChangeReportedAt
  20150705131449 AddTypeToReports
  20150708070742 AddFullNameToSetting
  20150714132601 RemoveIsParam
  20150714140850 RemoveNewFromComputeAttributes
  20150714151223 RemoveChefProxy
  20150721131324 ChangeBookmarkReportController
  20150728122736 ChangeReportPermissions
  20150811170401 AddMergeDefaultToLookupKey
  20150819105725 AddLookupValueMatchToHostAndHostgroup
  20150917155300 UpdateHostStatusStatusFieldInt
  20151009084350 DropPtables
  20151019174035 RenameDomainHostCount
  20151025120534 AddHiddenValueToLookupKey
  20151104100257 AddHostsCountToHostgroup
  20151109152507 AddHostStatusHostIdIndex
  20151120153254 DeleteBootableInterface
  20151220093801 RemoveSpacesFromSmartVariableKey
  20160201131211 AddExpiredLogsToSmartProxy
  20160215143900 AddSubnetDomainRelationConstraints
  20160404074723 DowncaseDisplayTypes

Comment 3 Sachin Ghai 2016-05-26 07:37:35 UTC
Created attachment 1161771 [details]
foreman-debug

Comment 5 Brad Buckingham 2016-05-26 12:24:08 UTC
Eric,  the error looks very much like the error we saw yesterday with the CRL revocation list in /etc/httpd/conf.d/05-foreman-ssl.conf.  I am assigning it to you to confirm; however, we can re-assign if it is different.

We do not currently have this machine available; however, it is being spun up.

Comment 6 Eric Helms 2016-05-26 17:03:10 UTC
Created redmine issue http://projects.theforeman.org/issues/15203 from this bug

Comment 7 Bryan Kearney 2016-05-26 22:18:21 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/15203 has been closed

Comment 8 Tomer Brisker 2016-05-31 13:13:39 UTC
*** Bug 1340548 has been marked as a duplicate of this bug. ***

Comment 9 Sachin Ghai 2016-06-02 09:56:05 UTC
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap14 on rhel7.

The reported issue related to "Report processor failed" is no longer reproducible with upgrade performed with above
builds. I didn't see similar exception neither at "yum update -y" step nor while running installer. thank you for the fix.

Comment 10 Bryan Kearney 2016-07-27 11:31:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501