Bug 826426

Summary: Puppet can't connect to puppetserver (certificate failure)
Product: [Fedora] Fedora Reporter: Pablo Iranzo Gómez <pablo.iranzo>
Component: puppetAssignee: Jeroen van Meeuwen <vanmeeuwen+fedora>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 17CC: dFyD0d4hFT0P, john, k.georgiou, smithj4, tmz, trailtotale, vanmeeuwen+fedora
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-03-19 21:30:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pablo Iranzo Gómez 2012-05-30 07:43:39 UTC
Description of problem:
Hi,
After upgrade to F17, puppet is no longer able to connect to puppetmaster but, other puppet clients (running F16, EL6, etc can)

Version-Release number of selected component (if applicable):

puppet-2.7.13-1.fc17.noarch

How reproducible:
puppetd -vt  --server=yourpuppetserver


Steps to Reproduce:
1.puppetd -vt  --server=yourpuppetserver
2.puppet will fail
3.
  
Actual results:

on Client:
/usr/share/rubygems/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
err: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
Exiting; failed to retrieve certificate and waitforcert is disabled


On server:
[2012-05-30 09:42:30] ERROR OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen'
        /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
        /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
        /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
        /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:127:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:142:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:114:in `main'
        /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:46:in `run_command'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305:in `run'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:420:in `hook'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305:in `run'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:411:in `exit_on_fail'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:305:in `run'
        /usr/sbin/puppetmasterd:4


Expected results:
Puppet should work :)

Comment 1 Pablo Iranzo Gómez 2012-05-30 07:47:35 UTC
Of course, I've tried deleting and recreating certificates on both client and server and it failed.

I've also tried to setup puppet on a new machine (fully wiped) and same problem.

On F16, EL6 or EL5 it works as expected even when recreating certificates for both server and client

Comment 2 Clayton Louden 2012-05-30 10:49:04 UTC
Same here :(

Comment 3 Todd Zullinger 2012-05-30 17:49:45 UTC
This is a known issue.  You cannot generally use a newer puppet client to talk to an older puppet master.  This is an unfortunate compatibility issue upstream.

Our hands were tied with Fedora 17 as it moved to ruby-1.9 and puppet-2.6 has far too many failures with ruby-1.9 to be usable.  Moving all the other releases to puppet-2.7 also has significant costs and would burden folks who are not running Fedora 17.  So it's a bit of a lose/lose situation for us as maintainers.

See bug #822014 for a request to update EPEL for puppet-2.7.

Comment 4 Pablo Iranzo Gómez 2012-05-30 18:31:33 UTC
Todd, I knew of that problem, but I even tried upgrading my server puppet to the one provided by the yum repo at puppetlabs which his 'newer' than the one provided by Fedora

puppet-2.7.14-1.el6.noarch
puppet-server-2.7.14-1.el6.noarch
at baseurl=http://yum.puppetlabs.com/el/$releasever/products/$basearch/


And had that problem too happens:

master:
==> /var/log/puppet/masterhttp.log <==
[2012-05-30 20:24:44] ERROR OpenSSL::SSL::SSLError: SSL_accept returned=1 errno=0 state=SSLv3 read client certificate A: tlsv1 alert unknown ca
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `accept'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:44:in `listen'
        /usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
        /usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
        /usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
        /usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
        /usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'
        /usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
        /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'
        /usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:144:in `run_command'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'
        /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
        /usr/sbin/puppetmasterd:4



And client (F17):
[root@x201 ~]# puppetd -vt --server=merak.no-ip.org
/usr/share/rubygems/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
err: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
Exiting; failed to retrieve certificate and waitforcert is disabled


If its all because of ruby, this is a serious problem... and as F17 had to move to ruby 1.9.. it's a bigger one.

I remember having several versions of 'unison$VERSIONUMBER" because of some incompatibilities between them... ¿would it be possible to have a ruby1.X required by puppet26" or something like that?

Thanks
Pablo

Comment 5 Clayton Louden 2012-05-31 11:34:37 UTC
It seem like ruby 1.9 is completely unsopported by puppet:

http://docs.puppetlabs.com/guides/faq.html#which-versions-of-ruby-does-puppet-support

Comment 6 Clayton Louden 2012-05-31 14:59:36 UTC
Just fyi, 

F16 client to F17 master

[root@PuppetClient ~]# puppet agent --server=PuppetMaster.example.com --no-daemonize --verbose
info: Creating a new SSL key for puppetclient.example.com
warning: peer certificate won't be verified in this SSL session
info: Caching certificate for ca
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for puppetclient.example.com
info: Certificate Request fingerprint (md5): 6E:9D:6A:44:BD:E0:FF:3F:FB:BC:4D:4F:AF:94:A0:10
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session

After signing the certificate on the master:

[root@PuppetClient ~]# puppet agent --server=PuppetMaster.example.com --no-daemonize --verbose
warning: peer certificate won't be verified in this SSL session
info: Caching certificate for puppetclient.example.com
notice: Starting Puppet client version 2.6.16
info: Caching certificate_revocation_list for ca
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
notice: Using cached catalog
err: Could not retrieve catalog; skipping run



F17 client to F16 master and F17 client to F17 master

[root@PuppetClient ~]# puppet agent --server=PuppetMaster.example.com --no-daemonize --verbose
/usr/share/rubygems/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
info: Creating a new SSL key for puppetclient.example.com
err: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client

Certificate request never shows up on the master (puppet cert --list is empty)


version being 2.6.16 for F16 and 2.7.13 for F17 respectively.

Comment 7 Clayton Louden 2012-06-04 08:49:48 UTC
Based on http://projects.puppetlabs.com/issues/9084, here's a workaround:

- Copy /var/lib/puppet/ssl/certs/ca.pem from master to client.
- Create the following symlink on the client:

ln -s /var/lib/puppet/ssl/certs/ca.pem $(openssl version -d|cut -d\" -f2)/certs/$(openssl x509 -hash -noout -in /var/lib/puppet/ssl/certs/ca.pem).0

Comment 8 John Florian 2012-08-11 20:00:20 UTC
(In reply to comment #7)
> Based on http://projects.puppetlabs.com/issues/9084, here's a workaround:
> 
> - Copy /var/lib/puppet/ssl/certs/ca.pem from master to client.
> - Create the following symlink on the client:
> 
> ln -s /var/lib/puppet/ssl/certs/ca.pem $(openssl version -d|cut -d\"
> -f2)/certs/$(openssl x509 -hash -noout -in
> /var/lib/puppet/ssl/certs/ca.pem).0


This work around was very helpful in the early days of Fedora 17, but it no longer seems necessary.  I'm not sure what exactly changed, but I just successfully brought up a new F17 node (against an F17 master) and the process was same as the old days once again.  Yeah!

Comment 9 Clayton Louden 2012-09-03 00:59:50 UTC
(In reply to comment #8)
> This work around was very helpful in the early days of Fedora 17, but it no
> longer seems necessary.  I'm not sure what exactly changed, but I just
> successfully brought up a new F17 node (against an F17 master) and the
> process was same as the old days once again.  Yeah!

Here too! Working fine again with the latest puppet version from updates repo.