Bug 1239020

Summary: satellite upgrade is failing from sat 6.0.8 to 6.1.1 GA snap 11
Product: Red Hat Satellite Reporter: Jitendra Yejare <jyejare>
Component: UpgradesAssignee: Stephen Benjamin <stbenjam>
Status: CLOSED CURRENTRELEASE QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: bkearney, mmccune, sghai, stbenjam
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-12 16:05:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1195450    
Attachments:
Description Flags
Foreman Debug Logs none

Description Jitendra Yejare 2015-07-03 09:31:50 UTC
Created attachment 1045799 [details]
Foreman Debug Logs

Description of problem:
Satellite upgrade is failing with latest GA snap 11.
From Sat 6.0.8 -> sat 6.1 GA snap 11 

Version-Release number of selected component (if applicable):
Sat6.1.0-GA-SNAP11

How reproducible:


Steps to Reproduce:
1. Upgrade from sat 6.0.8 to latest sat 6.1 GA snap 11.
2.
3.

Actual results:
The upgrade output logs are throwing NameError logs.

Output logs:
[root@sat6-qe-rhel7 ~]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing            
Done                                               [100%]
[...............................................................................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
/usr/share/katello-installer/hooks/post/30-upgrade.rb:3:in `block (4
levels) in load': undefined local variable or method `noop' for
#<Kafo::HookContext:0x0000000325bc70> (NameError)
    from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in
`instance_eval'
    from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:32:in
`block (4 levels) in load'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in
`instance_exec'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `execute'
    from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:49:in
`block in execute'
    from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `each'
    from /usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in
`execute'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:339:in
`block in run_installation'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/exit_handler.rb:24:in `call'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/exit_handler.rb:24:in `exit'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:122:in `exit'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:338:in
`run_installation'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:109:in
`execute'
    from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
    from /usr/share/gems/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
    from
/usr/share/gems/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:116:in `run'
    from /usr/sbin/katello-installer:46:in `<main>

Expected results:
No failure logs.

Additional info:

hammer ping output after upgrade failure;
Hammer ping:
[root@sat6-qe-rhel7 ~]# hammer ping
[Foreman] Username: admin
[Foreman] Password for admin:
candlepin:     
    Status:          ok
    Server Response: Duration: 619ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 71ms
pulp:          
    Status:          ok
    Server Response: Duration: 116ms
pulp_auth:     
    Status:          ok
    Server Response: Duration: 55ms
elasticsearch: 
    Status:          ok
    Server Response: Duration: 115ms
foreman_tasks: 
    Status:          ok
    Server Response: Duration: 1ms

Comment 2 Stephen Benjamin 2015-07-07 16:56:14 UTC
Upstream issue is  http://projects.theforeman.org/issues/10986

Comment 3 Stephen Benjamin 2015-07-09 13:13:30 UTC
katello-installer|9d83961717287d6542770d208cc21414170e078f

Comment 6 Sachin Ghai 2015-07-16 07:34:15 UTC
Verified with sat6.1.1 GA snap13. The reported issue has been fixed. I was able to upgrade sat6.0.8 to sta6.1.1 snap13 without reported issue.

[root@cloud-qe-21 ~]# katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: start_httpd...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
Upgrade Step: migrate_pulp...
Upgrade Step: Restarting services...
Upgrade Step: db:seed...
Upgrade Step: Running errata import task (this may take a while)...
Upgrade Step: Update gpg key urls to support capsule isolation (this may take a while)...
Upgrade Step: Update repositories to specify metadata_expire (this may take a while)...
Katello upgrade completed!
[root@cloud-qe-21 ~]#

Comment 7 Sachin Ghai 2015-07-16 11:08:44 UTC
Assigning back.

Issue has been fixed in rhel71. but its still reproducible on rhel66.

--
[root@cloud-qe-6 ~]# katello-installer --upgrade


Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
/usr/share/katello-installer/hooks/post/30-upgrade.rb:3:in `load': undefined local variable or method `noop' for #<Kafo::HookContext:0x7ffa02c16958> (NameError)
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `instance_exec'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hook_context.rb:13:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:49:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `each'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/hooking.rb:47:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:339:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/exit_handler.rb:24:in `call'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/exit_handler.rb:24:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:122:in `exit'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:338:in `run_installation'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:109:in `execute'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:67:in `run'
	from /usr/lib/ruby/gems/1.8/gems/clamp-0.6.2/lib/clamp/command.rb:125:in `run'
	from /usr/lib/ruby/gems/1.8/gems/kafo-0.6.5.9/lib/kafo/kafo_configure.rb:116:in `run'
	from /usr/sbin/katello-installer:46

Comment 10 Sachin Ghai 2015-07-21 09:05:52 UTC
Verified with sat6.1.1 GA snap13 compose2.

The reported issue is not reproducible. However we found another issue related to httpd service and for that we filed a new bz: https://bugzilla.redhat.com/show_bug.cgi?id=1244808

if we run-run the upgrade, then installer succeed:



[root@cloud-qe-3 ~]#  katello-installer --upgrade
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_mongo...
Upgrade Step: start_httpd...
Upgrade Step: migrate_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: migrate_foreman...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/katello-installer/katello-installer.log
Upgrade Step: migrate_pulp...
Upgrade Step: Restarting services...
Upgrade Step: db:seed...
Upgrade Step: Running errata import task (this may take a while)...
Upgrade Step: Update gpg key urls to support capsule isolation (this may take a while)...
Upgrade Step: Update repositories to specify metadata_expire (this may take a while)...
Katello upgrade completed!
[root@cloud-qe-3 ~]# rpm -qa | grep katello-installer-base
katello-installer-base-2.3.14-1.el6_6sat.noarch


So moving this bz to verified.

Comment 11 Sachin Ghai 2015-07-21 10:29:43 UTC
(In reply to Sachin Ghai from comment #10)
> Verified with sat6.1.1 GA snap13 compose2.
> 
> The reported issue is not reproducible. However we found another issue
> related to httpd service and for that we filed a new bz:
> https://bugzilla.redhat.com/show_bug.cgi?id=1244808
> 
> if we run-run the upgrade, then installer succeed:
> 
Rephrasing..

if we re-run the upgrade, then installer succeed:

Comment 12 Bryan Kearney 2015-08-12 16:05:17 UTC
This bug was fixed in Satellite 6.1.1 which was delivered on 12 August, 2015.