Bug 1366029

Summary: [RFE] satellite installer doesn't allow for upgrading puppet
Product: Red Hat Satellite Reporter: Bryan Kearney <bkearney>
Component: PuppetAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.0CC: bbuckingham, cduryee, dcaplan, gkonda, lpramuk
Target Milestone: UnspecifiedKeywords: FutureFeature, Reopened, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 12:35:52 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bryan Kearney 2016-08-10 19:47:31 UTC
Currently, an existing Katello installation with Puppet 3 requires stepping through manual steps to upgrade to Puppet 4.

Ideally, this would be done via foreman-installer, possibly via new hooks.

Comment 1 Bryan Kearney 2016-08-10 19:47:35 UTC
Created from redmine issue http://projects.theforeman.org/issues/16053

Comment 6 Lukas Pramuk 2016-10-05 10:38:51 UTC
This wasnt delivered with rebase.

Comment 7 Lukas Pramuk 2016-10-18 12:51:38 UTC
Actually it was, lets test it.

Comment 8 Lukas Pramuk 2016-10-18 12:56:10 UTC
# rpm -ivh https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
Retrieving https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
warning: /var/tmp/rpm-tmp.mAm7Qr: Header V4 RSA/SHA512 Signature, key ID 4bd6ec30: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:puppetlabs-release-pc1-1.1.0-2.el################################# [100%]

# satellite-installer --upgrade-puppet --disable-system-checks
Upgrading puppet...
...

Upgrade Step: upgrade_puppet_package...
...

Upgrade Step: stop_services...
...

Upgrade Step: copy_data...



Puppet 3 to 4 upgrade initialization complete, continuing with installation
backtrace:
/opt/puppetlabs/puppet/lib/ruby/2.1.0/json/common.rb:155:in `parse'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/json/common.rb:155:in `parse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/facts/facter.rb:35:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/facts/facter.rb:35:in `to_hash'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/facts/facter.rb:35:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/node.rb:117:in `fact_merge'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/plain.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:208:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:540:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
/opt/puppetlabs/bin/puppet:5:in `<main>'
Installing             Done                                               [100%] [.....................................]
  Success!
  * Katello is running at https://<FQDN>
  * To install additional capsule on separate machine continue by running:

      capsule-certs-generate --capsule-fqdn "$CAPSULE" --certs-tar "~/$CAPSULE-certs.tar"

  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
...

Puppet upgrade completed!

# puppet -V
-bash: puppet: command not found

# exec bash

# puppet -V
4.7.0

Comment 9 Lukas Pramuk 2016-10-18 13:06:12 UTC
FailedQA.

@Sat6.3.0-Snap4
katello-installer-base-3.2.0-2.rc2.el7.noarch

with these issues:

1. upstream puppet repo still needs to be pre-installed

2. consider upgrade steps verbosity (entire yum cmds output) - minor

3. `parse' error occured

Comment 10 Lukas Pramuk 2016-10-18 13:11:43 UTC
This sound familiar (mongodb_is_master) ...

[ERROR 2016-10-18 08:31:32 main]  Facter: error while resolving custom fact "mongodb_is_master": 757: unexpected token at '2016-10-18T08:31:32.074-0400 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
[ERROR 2016-10-18 08:31:32 main] 2016-10-18T08:31:32.075-0400 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146'
[ERROR 2016-10-18 08:31:32 main] backtrace:
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/2.1.0/json/common.rb:155:in `parse'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/2.1.0/json/common.rb:155:in `parse'
[ERROR 2016-10-18 08:31:32 main] /usr/share/katello-installer-base/modules/mongodb/lib/facter/is_master.rb:8:in `block (2 levels) in <top (required)>'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/facts/facter.rb:35:in `call'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/facts/facter.rb:35:in `to_hash'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/facts/facter.rb:35:in `find'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/node.rb:117:in `fact_merge'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/plain.rb:17:in `find'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:208:in `main'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `block in run'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:540:in `exit_on_fail'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:344:in `run'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
[ERROR 2016-10-18 08:31:32 main] /opt/puppetlabs/bin/puppet:5:in `<main>'
[DEBUG 2016-10-18 08:31:32 main]  Facter: fact "mongodb_is_master" resolved to null and will not be added.

Comment 11 Bryan Kearney 2016-10-18 14:14:53 UTC
Upstream bug component is Puppet

Comment 12 Lukas Pramuk 2016-10-19 13:50:09 UTC
Also consider one more issue:

4. Running satellite-installer on puppet4 with --upgrade-puppet should stop immediately saying "You are running already on upgraded puppet"
(instead of running installer needlessly)

Comment 13 Lukas Pramuk 2016-10-24 11:08:53 UTC
Another and major issue is:

5) by using --upgrade-puppet option the locations of puppet ssl ca/cert are not updated to new puppet4 locations in /etc/

Capsule - tab 'Puppet CA':
Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient...)

Capsule - tab 'Logs'
ERROR    Disabling all modules in the group ['puppet_proxy_puppet_api', 'puppet'] due to a failure in one of them: File at '/var/lib/puppet/ssl/certs/ca.pem' defined in 'puppet_ssl_ca' cannot be found.

Workaround is to run installer once again to adjust certs locations:
# satellite-installer \
 --foreman-proxy-puppet-ssl-ca /etc/puppetlabs/puppet/ssl/certs/ca.pem
 --foreman-proxy-puppet-ssl-cert /etc/puppetlabs/puppet/ssl/certs/`hostname`.pem

We need to have installer done it all in once just by --upgrade-puppet

Comment 14 Chris Duryee 2016-10-24 14:47:53 UTC
To summarize, there are five issues:

1. upstream puppet repo still needs to be pre-installed (will use this BZ to track this issue)

2. consider upgrade steps verbosity (entire yum cmds output) https://bugzilla.redhat.com/show_bug.cgi?id=1388133

3. `parse' error occured https://bugzilla.redhat.com/show_bug.cgi?id=1381081

4. Running satellite-installer on puppet4 with --upgrade-puppet should stop immediately saying "You are running already on upgraded puppet"
(instead of running installer needlessly) https://bugzilla.redhat.com/show_bug.cgi?id=1388134

5) by using --upgrade-puppet option the locations of puppet ssl ca/cert are not updated to new puppet4 locations in /etc/ (comment 13) https://bugzilla.redhat.com/show_bug.cgi?id=1388136

Comment 15 Chris Duryee 2016-10-27 14:54:44 UTC
Can you try using the internal repo instead of PC1 for P4 packages? these packages are the same as what will eventually be in the 6.3.0 snaps. They tested OK for us, but if you could try them out as well, we can make sure that it won't block testing when we add them in later.

Comment 20 Satellite Program 2017-05-25 16:12:03 UTC
Upstream bug assigned to cduryee

Comment 21 Satellite Program 2017-05-25 18:11:58 UTC
Upstream bug assigned to ehelms

Comment 23 Lukas Pramuk 2017-08-31 12:50:15 UTC
VERIFIED.

@satellite-6.3.0-16.0.beta.el7sat.noarch
puppet-agent-1.8.2-2.el7sat.x86_64
puppet-agent-oauth-0.5.1-2.el7sat.noarch
puppetserver-2.7.2-2.el7sat.noarch

Currently there is candidate downstream repo and by enabling the repo the installer is able to proceed with clean puppet4 installation or to upgrade puppet3 installation using --upgrade-puppet option

Comment 26 errata-xmlrpc 2018-02-21 12:35:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336