Bug 1342449

Summary: Capsule upgrade failed with error: must be present at /usr/share/katello-installer-base/modules/capsule/manifests/init.pp:88 on node <capsule_url>
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: UpgradesAssignee: Brad Buckingham <bbuckingham>
Status: CLOSED ERRATA QA Contact: Sachin Ghai <sghai>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, ehelms, pgervase, pmoravec
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: katello-installer-base-3.0.0.43-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-27 11:16:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1343084, 1343438    
Bug Blocks: 1335807    
Attachments:
Description Flags
foreman-debug from capsule none

Description Sachin Ghai 2016-06-03 09:49:20 UTC
Description of problem:
I was trying to perform capsule upgrade from sat6.1.9 -> sat6.2 snap14.1 on rhel7. Upgrade failed at below step:

]# satellite-installer --scenario capsule --upgrade --certs-tar /root/qe-capsule-rhel7.satqe.lab.eng.rdu2.redhat.com-certs.tar 
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: remove_nodes_importers...
Upgrade Step: Running installer...
Preparing installation Done                                              
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/capsule.log
Upgrade Step: restart_services...


[DEBUG 2016-06-03 05:43:37 main] Hook /usr/share/katello-installer-base/hooks/post/30-upgrade.rb returned nil
[ INFO 2016-06-03 05:43:37 main] All hooks in group post finished
[DEBUG 2016-06-03 05:43:37 main] Exit with status code: 1 (signal was 1)
[ERROR 2016-06-03 05:43:37 main] Errors encountered during run:
[ERROR 2016-06-03 05:43:37 main]  must be present at /usr/share/katello-installer-base/modules/capsule/manifests/init.pp:88 on node qe-capsule-rhel7.satqe.lab.eng.rdu2.redhat.com
[ERROR 2016-06-03 05:43:37 main]  must be present at /usr/share/katello-installer-base/modules/capsule/manifests/init.pp:88 on node qe-capsule-rhel7.satqe.lab.eng.rdu2.redhat.com


Line 88 in /usr/share/katello-installer-base/modules/capsule/manifests/init.pp points:
validate_present($pulp_oauth_secret)


Version-Release number of selected component (if applicable):
capsule upgrade from sat6.1.9 -> sat6.2 snap14.1

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
upgrade should be successful.

Additional info:

Comment 1 Sachin Ghai 2016-06-03 09:54:13 UTC
Steps to reproduce:

0) re-generated capsule certs on satellite for capsule and copied over capsule node
1) re-register the 6.1.9 capsule with upgraded 6.2 Satellite ( due to bz 1341712)
2) set the capsule 6.2 repo
3) yum update -y
4) satellite-installer --scenario capsule --upgrade --certs-tar /root/qe-capsule-rhel7.satqe.lab.eng.rdu2.redhat.com-certs.tar

Comment 2 Sachin Ghai 2016-06-03 09:56:30 UTC
Created attachment 1164427 [details]
foreman-debug from capsule

Comment 3 Sachin Ghai 2016-06-03 10:09:22 UTC
I'm facing this issue even If I perform upgrade without generating certs.. Not sure if its required while performing upgrade from 6.1.9 -> 6.2 GA snaps. So tried both ways.

Comment 5 Brad Buckingham 2016-06-03 11:53:15 UTC
In order to work around this error include the --pulp-oauth-secret when issuing the upgrade command.  E.g.

satellite-installer --scenario capsule --upgrade\
                    --certs-tar ~/capsule61-rhel7.example.com-certs.tar\
                    --certs-update-all --regenerate --deploy\
                    --pulp-oauth-secret "cMKZjEc9amv5nmEhReGu3N9RoCByszkD"\
                    --register-in-foreman false

The lack of pulp oauth secret may be a result of bug 1341756; however, it is being investigated.

Comment 9 Brad Buckingham 2016-06-07 14:16:22 UTC
The changes coming with bug 1343084 and bug 1343438 will address this; however, rather than close this as a duplicate, I'd like it to go through verification to ensure it is fixed.

Comment 10 Sachin Ghai 2016-06-10 08:02:09 UTC
Verified with upgrade from sat6.1.9 -> sat6.2 GA snap15.1 on rhel7. The
reported issue is fixed now. 

capsule upgrade is completed successfully without explicitly passing --pulp-oauth-secret parameter. However "--register-in-foreman" param is still required for successful capsule upgrade. Will file another issue for this.

~]# satellite-installer --scenario capsule --upgrade --bmc true --certs-tar=/root/qe-capsule-rhel7.satqe.lab.eng.rdu2.redhat.com-certs.tar --register-in-foreman false

Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: remove_nodes_importers...
Upgrade Step: Running installer...
Installing             Done                                              
[100%] [............................]
  The full log is at /var/log/foreman-installer/capsule.log
Upgrade Step: restart_services...
Katello upgrade completed!

Comment 11 Bryan Kearney 2016-07-27 11:16:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501