Bug 1986934 - agent fails all run attempts
Summary: agent fails all run attempts
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: puppet
Version: 34
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Breno
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1999941 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-28 14:45 UTC by John Florian
Modified: 2021-09-16 19:13 UTC (History)
6 users (show)

Fixed In Version: puppet-7.9.0-1.fc34
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-16 19:13:53 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description John Florian 2021-07-28 14:45:23 UTC
Description of problem:
Agent dumps a stack trace no matter how I attempt to use it.

Version-Release number of selected component (if applicable):
puppet-5.5.20-2.fc33.noarch
puppet-headless-5.5.20-2.fc33.noarch

How reproducible:
always

Steps to Reproduce:
1. dnf install puppet
2. puppet --help

Actual results:
$ puppet --help
/usr/share/ruby/vendor_ruby/puppet/util.rb:461:in `uri_encode': undefined method `escape' for URI:Module (NoMethodError)
        from /usr/share/ruby/vendor_ruby/puppet/util.rb:337:in `path_to_uri'
        from /usr/share/ruby/vendor_ruby/puppet/pops/model/ast.rb:4863:in `register_pcore_types'
        from /usr/share/ruby/vendor_ruby/puppet/pops.rb:119:in `<module:Puppet>'
        from /usr/share/ruby/vendor_ruby/puppet/pops.rb:1:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/share/ruby/vendor_ruby/puppet/parser/compiler.rb:8:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/share/ruby/vendor_ruby/puppet/parser.rb:6:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/share/ruby/vendor_ruby/puppet.rb:302:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/share/ruby/vendor_ruby/puppet/util/command_line.rb:12:in `<top (required)>'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/bin/puppet:4:in `<main>'


Expected results:
I was hoping it worked at least as good as with F33 which wasn't exactly an awesome experience.  In F33 there was lots of spurious warnings about the undefined method "escape" for URI:Module, but it otherwise worked.  On F34, that error is shown once but then the agent crashes before doing anything useful.

Additional info:

Comment 1 Breno 2021-07-28 21:11:10 UTC
Thanks for filing the ticket, John. We recently pushed puppet 7 to rawhide.
We'll be pushing it to f34 as well.

Comment 2 John Florian 2021-07-29 13:30:33 UTC
That's great news Breno!  I had been getting concerned about the age of the agent that Fedora was providing.  Thanks to you and all who are helping to make this happen.

Comment 3 Breno 2021-08-03 16:05:22 UTC
You're very welcome John. We appreciate it. 
It was a big team effort.

Comment 4 Breno 2021-08-16 17:26:34 UTC
Hey John, if you don't mind giving it a test here https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec4a4b2634 and a thumbs up, to help with the testing cycle.

Comment 5 Fedora Update System 2021-08-16 18:38:32 UTC
FEDORA-2021-ec4a4b2634 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec4a4b2634

Comment 6 Fedora Update System 2021-08-17 01:09:01 UTC
FEDORA-2021-ec4a4b2634 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ec4a4b2634`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec4a4b2634

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Trever Adams 2021-08-18 00:53:33 UTC
This does NOT work. It appears the configuration layout has changed. /etc/puppet is now /etc/puppetlabs/puppet.

Additionally:

Error: Cannot create /opt/puppetlabs/puppet/public; parent directory /opt/puppetlabs/puppet does not exist
Error: /File[/opt/puppetlabs/puppet/public]/ensure: change from 'absent' to 'directory' failed: Cannot create /opt/puppetlabs/puppet/public; parent directory /opt/puppetlabs/puppet does not exist
Debug: Finishing transaction 10900
Error: Could not prepare for execution: Got 1 failure(s) while initializing: File[/opt/puppetlabs/puppet/public]: change from 'absent' to 'directory' failed: Cannot create /opt/puppetlabs/puppet/public; parent directory /opt/puppetlabs/puppet does not exist

This regardless of changing varlib or any other variable in the configuration.

Comment 8 Ewoud Kohl van Wijngaarden 2021-08-18 10:48:04 UTC
The config location change was intentional, though I personally am still not very happy with that. It was changed because the EPEL8 packaging already used that. I personally still like for it to remain /etc/puppet.

As for the /opt directory: I noted the same thing in https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec4a4b2634 and opened https://src.fedoraproject.org/rpms/puppet/pull-request/14 which I think should address it.

In the future, providing info in Bodhi helps. Negative karma prevents it from propagating. Positive karma if it does work is also good.

Comment 9 Trever Adams 2021-08-18 11:18:04 UTC
Ewoud, I am quite unhappy about it myself. As for Bodhi, I don't believe I have access. I will look into it.

Comment 10 John Florian 2021-08-18 15:23:43 UTC
Breno, thanks for the heads up.  I'm now using this successfully on several Fedora 34 hosts.  I'd give karma on bodhi but I get a server error when I try to login.

Comment 11 John Florian 2021-08-19 12:31:36 UTC
One thing I ran into on one host was an error trying to apply the catalog.  I don't have the exact error in front of me now, but basically it was failing because the directory /opt/puppetlabs/puppet/public didn't exist where puppet is trying to store a last_run_summary.yaml file.

It hadn't been a problem I'd noticed on other hosts because at one time they had puppet7-release and puppet-agent-7 installed from the puppetlabs repo.  That packaging created this directory, and also left it behind when removed them to try this test build out.  The one host that gave me problems never had that setup.  It went straight from Fedora 33 with puppet-5.5 to Fedora 34 with the broken build to this test build.  I manually created the directory and all went well afterwards.

I'm not sure if the package sb creating this directory or if the default config should have this file going somewhere like /var/lib/puppet.

Comment 12 Trever Adams 2021-08-19 13:54:12 UTC
server:
Notice: Starting Puppet master version 5.5.20
Info: Bad Request: An environment parameter must be specified
Info: Bad Request: An environment parameter must be specified

client:
Error: Could not download CA certificate: Bad Request
Error: Could not run: Could not download CA certificate: Bad Request

If I manually copy over the ca and crl stuff:
Error: undefined method `message' for #<Puppet::HTTP::ResponseNetHTTP:0x000055d72166abb8 @url=#<URI::HTTPS https://puppetmaster:8140/puppet-ca/v1/certificate/puppetclient>, @code=400, @reason="Bad Request", @nethttp=#<Net::HTTPBadRequest 400 Bad Request readbody=true>>
Error: Could not run: undefined method `message' for #<Puppet::HTTP::ResponseNetHTTP:0x000055d72166abb8 @url=#<URI::HTTPS https://puppetmaster:8140/puppet-ca/v1/certificate/puppetclient>, @code=400, @reason="Bad Request", @nethttp=#<Net::HTTPBadRequest 400 Bad Request readbody=true>>

I am not sure what is missing or broken here. F32/F33 machines work fine. F34 and new do not.

Is it possible that a dependency didn't get updated and/or required with the new package?

Comment 13 Trever Adams 2021-08-19 22:59:40 UTC
Is server functionality permanently gone then?

Comment 14 Ewoud Kohl van Wijngaarden 2021-08-20 13:15:03 UTC
Puppet 6 removed the Ruby-based server functionality (AKA puppetmaster). You now must use the JVM-based server (AKA puppetserver). This is not packaged (at least for now).

Comment 15 Trever Adams 2021-08-21 20:44:44 UTC
What is necessary for it to be packaged?

Comment 16 Breno 2021-08-24 00:47:06 UTC
Trever,

It is necessary to compile puppetserver.jar.
I have a working spec file here [1], but it does not compile puppetserver.jar and therefore is not in compliance with Fedora's policies.

To compile it, it's used clojure via leiningen [2]. So we also need to package leiningen, and make sure it will compile everything 100% offline.

1 https://copr.fedorainfracloud.org/coprs/brandfbb/puppetserver/
2 https://leiningen.org/

Comment 17 Fedora Update System 2021-08-24 09:25:33 UTC
FEDORA-2021-ec4a4b2634 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec4a4b2634

Comment 18 Fedora Update System 2021-08-25 20:41:59 UTC
FEDORA-2021-ec4a4b2634 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ec4a4b2634`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ec4a4b2634

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Ewoud Kohl van Wijngaarden 2021-09-01 12:41:52 UTC
*** Bug 1999941 has been marked as a duplicate of this bug. ***

Comment 20 Ewoud Kohl van Wijngaarden 2021-09-02 12:02:13 UTC
The Bodhi can be promoted to stable. John and Trever: you've tested the previous update (7.7.0) and found issues. Have you had a look a the new build (7.9.0)? I'd feel more confident with promoting if I wasn't the only person who tested.

Comment 21 John Florian 2021-09-07 14:50:23 UTC
Ewoud, I have upgraded a half dozen hosts from the 7.7.0 build to the 7.9.0 one.  I needed to move /etc/puppetlabs/ssl to /etc/puppet or to recreate/clean/sign the key to accommodate the alt location that was used temporarily with the 7.7.0 build.  It looked like the install might of handled the same with puppet.conf on its own.  In any case, things look "Fedora traditional" now and all hosts are working fine from what I've seen thus far.

FWIW, only two hosts could get the upgrade with `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ec4a4b2634`.  Even if I added `--refresh` or did `dnf clean all` prior I couldn't get dnf to discover the newer build in updates-testing.  With the other hosts, I ultimately had to install directly from the Koji URL.

Comment 22 Ewoud Kohl van Wijngaarden 2021-09-07 16:38:02 UTC
I appreciate the testing. I wonder if there may have been issues with changing the Bodhi version from 7.7.0 to 7.9.0 and that something (dnf?) decides it already has all the info it needs.

Unless anyone objects, I'll look at promoting it later this week.

Comment 23 Fedora Update System 2021-09-16 19:13:53 UTC
FEDORA-2021-ec4a4b2634 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.