Created attachment 694362 [details] patch against fedora distgit Hello, a race condition in Puppet has been finally fixed upstream (pending merge). http://projects.puppetlabs.com/issues/7165 Since our (Katello, Aeolus, Foreman) installers are based on puppet and we support RHEL6/EPEL6, we need to have this working in the EPEL6 Puppet version. I have backported the patch, here is the testing build: http://koji.fedoraproject.org/koji/taskinfo?taskID=4935701 I am attaching the patch against EPEL fedora distgit.
Looks good. One minor thing is that we should wait for the upstream patch to get merged before submitting this because that patch may change before final merge which will cause divergent trees.
Indeed. And the patch was accepted upstream! Great.
The patch was applied to the unreleased 3.2 with a few changes to fix subsequent issues found, it looks like. 9d59f20 ((#7165) Properties can invalidate scheduled refreshes on change) is the second commit that was made. I didn't look closely at whether there were any changes made to the first commit or not. Can someone attach a newly backported patch for 2.6? I'll see about getting it included. Unfortunately, I don't really have the bandwidth to dig through the 3.2 code and properly backport this myself. Thanks.
Created attachment 712560 [details] patch #2 against fedora distgit I've rebased lzap's patch against the package, and updated the patch within to the merged version.
puppet-2.6.18-2.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/puppet-2.6.18-2.el6
puppet-2.6.18-2.el5 has been submitted as an update for Fedora EPEL 5. https://admin.fedoraproject.org/updates/puppet-2.6.18-2.el5
Thanks for all the help with this one Lukas and Dominic!
Package puppet-2.6.18-2.el6: * should fix your issue, * was pushed to the Fedora EPEL 6 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=epel-testing puppet-2.6.18-2.el6' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-EPEL-2013-0720/puppet-2.6.18-2.el6 then log in and leave karma (feedback).
puppet-2.6.18-2.el5 has been pushed to the Fedora EPEL 5 stable repository. If problems still persist, please make note of it in this bug report.
puppet-2.6.18-2.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.
This patch causes an error: err: /Stage[main]/Katello::Service/Service[katello-jobs]: Could not evaluate: un defined method `[]=' for nil:NilClass Reproducer: yum install katello-all katello-configure After some debugging, here is the line, that raises the issue: /usr/lib/ruby/site_ruby/1.8/puppet/transaction/event_manager.rb def dequeue_events_for_resource(target, callback) target.info "Unscheduling #{callback} on #{target}" --> @event_queues[target][callback] = {} end In this case, @event_queues[target] is nil, therefore it fails.
Coincidentally, I ran into this yesterday as well. In my case, I had a service set to ensure => running. When it was stopped, this error would be printed. Hopefully Dominic or Lukas will have some ideas for correcting it.
I've submitted a fix upstream and will provide a patch here once merged: http://newprojects.puppetlabs.com/issues/20128 https://github.com/puppetlabs/puppet/pull/1585
Created attachment 733042 [details] #20128 patch against dist-git It's been merged upstream as 3e5cea2, attaching a patch for the package. https://github.com/puppetlabs/puppet/commit/3e5cea2 Thanks for the diagnosis as well Ivan!
*** Bug 949549 has been marked as a duplicate of this bug. ***
*** Bug 950066 has been marked as a duplicate of this bug. ***
*** Bug 951015 has been marked as a duplicate of this bug. ***
puppet-2.6.18-3.el5 has been submitted as an update for Fedora EPEL 5. https://admin.fedoraproject.org/updates/puppet-2.6.18-3.el5
puppet-2.6.18-3.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/puppet-2.6.18-3.el6
Thanks again Dominic. Hopefully this update will get some testing and karma so that it can be pushed to stable without having to wait for the 2 week timeout. I have run it for a day on el5 and el6 hosts where I saw the issue and it seems to be fixed.
(In reply to comment #20) > Thanks again Dominic. Hopefully this update will get some testing and karma > so that it can be pushed to stable without having to wait for the 2 week > timeout. > > I have run it for a day on el5 and el6 hosts where I saw the issue and it > seems to be fixed. Excellent, thanks. I'll prod the team here to try it and give karma once it's pushed to updates-testing.
Package puppet-2.6.18-3.el6: * should fix your issue, * was pushed to the Fedora EPEL 6 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=epel-testing puppet-2.6.18-3.el6' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-EPEL-2013-5424/puppet-2.6.18-3.el6 then log in and leave karma (feedback).
I've just tested with the new puppet-2.6.18-3.el6, and am still having a major problem. The error message is gone, but it still is not starting some services. I don't understand why it is "unscheduling" the refresh in the first place. notice: /Stage[main]/Rpm::Mcelog::Mcelogd/Service[mcelogd]/ensure: ensure changed 'stopped' to 'running' info: /Stage[main]/Rpm::Mcelog::Mcelogd/Service[mcelogd]: Unscheduling refresh on Service[mcelogd] After puppet runs, the mcelogd service is still not running.
Sorry, the previous comment was incomplete. On one server, the mcelogd service is still not running, but that seems to be a bug in the mcelogd init script. On another server, the mcelogd service DOES start, but when I run puppet again it still thinks it's not running. I saw the same behavior from puppet on this server, and assumed the mcelogd service did not start. This could very well be another bug in the mcelogd init script, as this is the only service I'm seeing this behavior from.
(In reply to comment #23) > I've just tested with the new puppet-2.6.18-3.el6, and am still having a > major problem. The error message is gone, but it still is not starting some > services. I don't understand why it is "unscheduling" the refresh in the > first place. > > notice: /Stage[main]/Rpm::Mcelog::Mcelogd/Service[mcelogd]/ensure: ensure > changed 'stopped' to 'running' > info: /Stage[main]/Rpm::Mcelog::Mcelogd/Service[mcelogd]: Unscheduling > refresh on Service[mcelogd] "Unscheduling" refers to the fact the service started "cancelling out" any existing notifies from say, files that notify the service, so it won't subsequently restart mcelogd. (In reply to comment #24) > On one server, the mcelogd service is still not running, but that seems to > be a bug in the mcelogd init script. Indeed, or mcelogd exiting after being started. > On another server, the mcelogd service DOES start, but when I run puppet > again it still thinks it's not running. I saw the same behavior from puppet > on this server, and assumed the mcelogd service did not start. This could > very well be another bug in the mcelogd init script, as this is the only > service I'm seeing this behavior from. Ditto, possibly an issue with the status command not providing LSB compatible exit codes if the daemon was in fact running. Can't see that the issue is relevant to this bug.
puppet-2.6.18-3.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.
puppet-2.6.18-3.el5 has been pushed to the Fedora EPEL 5 stable repository. If problems still persist, please make note of it in this bug report.