Red Hat Bugzilla – Bug 1012001
all openstack services: unable to connect to AMQP server (after upgrading to qpid 0.24)
Last modified: 2015-06-21 20:08:26 EDT
Description of problem:
landed in Fedora 19.
It now places qpidd.conf under /etc/qpid/qpidd.conf instead of directly in /etc.
Qpid places it's qpidd.conf directly in /etc.
This results in qpid expecting authentication.
This is where the Qpid configuration files will live going forward, in /etc/qpid.
Darryl, there is a compatibility cost of course with moving configs around.
Have you considered that and a possible transition period/process to support for example /etc/qpidd.conf as a symlink into /etc/qpid/ or something?
(In reply to Pádraig Brady from comment #2)
> Darryl, there is a compatibility cost of course with moving configs around.
> Have you considered that and a possible transition period/process to support
> for example /etc/qpidd.conf as a symlink into /etc/qpid/ or something?
No, that wasn't considered. Having a symlink from /etc/qpid/qpidd.conf to /etc/qpidd.conf wouldn't work since qpidd will look for the configuration in /etc/qpidd.
I discussed this with other package maintainers. The general consensus was that, since the upgrade process would leave the file /etc/qpidd.conf.rpmsave behind after an upgrade, along with the release notes for the server, the sysadmin would take the appropriate steps to migrate their existing configuration to the new file location.
(In reply to Darryl L. Pierce from comment #3)
> (In reply to Pádraig Brady from comment #2)
> > Darryl, there is a compatibility cost of course with moving configs around.
> > Have you considered that and a possible transition period/process to support
> > for example /etc/qpidd.conf as a symlink into /etc/qpid/ or something?
> No, that wasn't considered. Having a symlink from /etc/qpid/qpidd.conf to
> /etc/qpidd.conf wouldn't work since qpidd will look for the configuration in
What I mean here is that qpid could use the new /etc/qpid/qpidd.conf
while other tools like puppet modules etc. could continue to use
/etc/qpidd.conf (symlink) until updated to the new location.
> I discussed this with other package maintainers. The general consensus was
> that, since the upgrade process would leave the file /etc/qpidd.conf.rpmsave
> behind after an upgrade, along with the release notes for the server, the
> sysadmin would take the appropriate steps to migrate their existing
> configuration to the new file location.
I don't think this is acceptable during a release TBH.
I.E. the system shouldn't break on (auto) update.
IMHO this fits here: http://fedoraproject.org/wiki/Updates_Policy#All_other_updates
I even disagree with the request to support both config options (/etc/qpidd.conf) and /etc/qpid/qpidd.conf), because qpid brings it's new, own config to /etc/qpid/qpidd.conf). Given some script like puppet places a new qpid.conf in /etc, which config should have precedence? I see arguments in both directions.
So this upgrade in a released version is currently more than just unfortunate.
(In reply to Matthias Runge from comment #5)
> IMHO this fits here:
> I even disagree with the request to support both config options
> (/etc/qpidd.conf) and /etc/qpid/qpidd.conf), because qpid brings it's new,
> own config to /etc/qpid/qpidd.conf). Given some script like puppet places a
> new qpid.conf in /etc, which config should have precedence? I see arguments
> in both directions.
> So this upgrade in a released version is currently more than just
I'm willing, for the 0.24 release, to modify the spec to create a symlink from /etc/qpid/qpidd.conf to /etc/qpid.conf if that will smooth the transition. However, anybody who has modified their configuration will still need to migrate their configuration to the new file. Will that provide a sufficient transition?
qpid-cpp-0.24-4.fc20.1 has been submitted as an update for Fedora 20.
qpid-cpp-0.24-3.fc19.1 has been submitted as an update for Fedora 19.
qpid-cpp-0.24-3.fc18.1 has been submitted as an update for Fedora 18.
This won't help in case of puppet-qpid module, but I guess we would have to fix that in the module.
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qpid-cpp-0.24-4.fc20.1'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
qpid-cpp-0.24-3.fc19.1 has been pushed to the Fedora 19 stable repository. If problems still persist, please make note of it in this bug report.
qpid-cpp-0.24-3.fc18.1 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
> Package qpid-cpp-0.24-4.fc20.1 * should fix your issue
No it doesn't
[boris@localhost ~]$ rpm -iv qpid-cpp-0.24-4.fc21.1.src.rpm
[boris@localhost ~]$ rpmbuild -bb qpid-cpp.spec
[boris@localhost ~]$ cat rpmbuild/RPMS/x*/inst.sh
sudo yum install python-qpid-qmf-0.24-4.fc19.1.x86_64.rpm \
[boris@localhost x86_64]$ ./inst.sh
[sudo] password for boris:
Loaded plugins: langpacks, refresh-packagekit
Examining python-qpid-qmf-0.24-4.fc19.1.x86_64.rpm: python-qpid-qmf-0.24-4.fc19.1.x86_64
Marking python-qpid-qmf-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-cpp-client-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-client-0.24-4.fc19.1.x86_64
Marking qpid-cpp-client-0.24-4.fc19.1.x86_64.rpm as an update to qpid-cpp-client-0.24-3.fc19.1.x86_64
Examining qpid-cpp-client-devel-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-client-devel-0.24-4.fc19.1.x86_64
Marking qpid-cpp-client-devel-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-cpp-client-rdma-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-client-rdma-0.24-4.fc19.1.x86_64
Marking qpid-cpp-client-rdma-0.24-4.fc19.1.x86_64.rpm as an update to qpid-cpp-client-rdma-0.24-3.fc19.1.x86_64
Examining qpid-cpp-debuginfo-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-debuginfo-0.24-4.fc19.1.x86_64
Marking qpid-cpp-debuginfo-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-cpp-server-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-server-0.24-4.fc19.1.x86_64
Marking qpid-cpp-server-0.24-4.fc19.1.x86_64.rpm as an update to qpid-cpp-server-0.24-3.fc19.1.x86_64
Examining qpid-cpp-server-ha-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-server-ha-0.24-4.fc19.1.x86_64
Marking qpid-cpp-server-ha-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-cpp-server-rdma-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-server-rdma-0.24-4.fc19.1.x86_64
Marking qpid-cpp-server-rdma-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-cpp-server-store-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-server-store-0.24-4.fc19.1.x86_64
Marking qpid-cpp-server-store-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-cpp-server-xml-0.24-4.fc19.1.x86_64.rpm: qpid-cpp-server-xml-0.24-4.fc19.1.x86_64
Marking qpid-cpp-server-xml-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-qmf-0.24-4.fc19.1.x86_64.rpm: qpid-qmf-0.24-4.fc19.1.x86_64
Marking qpid-qmf-0.24-4.fc19.1.x86_64.rpm to be installed
Examining qpid-qmf-devel-0.24-4.fc19.1.x86_64.rpm: qpid-qmf-devel-0.24-4.fc19.1.x86_64
Marking qpid-qmf-devel-0.24-4.fc19.1.x86_64.rpm to be installed
Examining ruby-qpid-qmf-0.24-4.fc19.1.x86_64.rpm: ruby-qpid-qmf-0.24-4.fc19.1.x86_64
Marking ruby-qpid-qmf-0.24-4.fc19.1.x86_64.rpm to be installed
--> Running transaction check
---> Package python-qpid-qmf.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-cpp-client.x86_64 0:0.24-3.fc19.1 will be updated
---> Package qpid-cpp-client.x86_64 0:0.24-4.fc19.1 will be an update
---> Package qpid-cpp-client-devel.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-cpp-client-rdma.x86_64 0:0.24-3.fc19.1 will be updated
---> Package qpid-cpp-client-rdma.x86_64 0:0.24-4.fc19.1 will be an update
---> Package qpid-cpp-debuginfo.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-cpp-server.x86_64 0:0.24-3.fc19.1 will be updated
---> Package qpid-cpp-server.x86_64 0:0.24-4.fc19.1 will be an update
---> Package qpid-cpp-server-ha.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-cpp-server-rdma.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-cpp-server-store.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-cpp-server-xml.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-qmf.x86_64 0:0.24-4.fc19.1 will be installed
---> Package qpid-qmf-devel.x86_64 0:0.24-4.fc19.1 will be installed
---> Package ruby-qpid-qmf.x86_64 0:0.24-4.fc19.1 will be installed
--> Finished Dependency Resolution
http://repos.fedorapeople.org/repos/openstack/openstack-grizzly/fedora-19/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
Package Arch Version Repository Size
python-qpid-qmf x86_64 0.24-4.fc19.1 /python-qpid-qmf-0.24-4.fc19.1.x86_64 7.1 M
qpid-cpp-client-devel x86_64 0.24-4.fc19.1 /qpid-cpp-client-devel-0.24-4.fc19.1.x86_64 2.7 M
qpid-cpp-debuginfo x86_64 0.24-4.fc19.1 /qpid-cpp-debuginfo-0.24-4.fc19.1.x86_64 140 M
qpid-cpp-server-ha x86_64 0.24-4.fc19.1 /qpid-cpp-server-ha-0.24-4.fc19.1.x86_64 652 k
qpid-cpp-server-rdma x86_64 0.24-4.fc19.1 /qpid-cpp-server-rdma-0.24-4.fc19.1.x86_64 78 k
qpid-cpp-server-store x86_64 0.24-4.fc19.1 /qpid-cpp-server-store-0.24-4.fc19.1.x86_64 0.0
qpid-cpp-server-xml x86_64 0.24-4.fc19.1 /qpid-cpp-server-xml-0.24-4.fc19.1.x86_64 107 k
qpid-qmf x86_64 0.24-4.fc19.1 /qpid-qmf-0.24-4.fc19.1.x86_64 1.4 M
qpid-qmf-devel x86_64 0.24-4.fc19.1 /qpid-qmf-devel-0.24-4.fc19.1.x86_64 225 k
ruby-qpid-qmf x86_64 0.24-4.fc19.1 /ruby-qpid-qmf-0.24-4.fc19.1.x86_64 705 k
qpid-cpp-client x86_64 0.24-4.fc19.1 /qpid-cpp-client-0.24-4.fc19.1.x86_64 4.4 M
qpid-cpp-client-rdma x86_64 0.24-4.fc19.1 /qpid-cpp-client-rdma-0.24-4.fc19.1.x86_64 264 k
qpid-cpp-server x86_64 0.24-4.fc19.1 /qpid-cpp-server-0.24-4.fc19.1.x86_64 6.3 M
Install 10 Packages
Upgrade 3 Packages
Total size: 163 M
Is this ok [y/d/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Updating : qpid-cpp-client-0.24-4.fc19.1.x86_64 1/16
Updating : qpid-cpp-client-rdma-0.24-4.fc19.1.x86_64 2/16
Updating : qpid-cpp-server-0.24-4.fc19.1.x86_64 3/16
Installing : qpid-qmf-0.24-4.fc19.1.x86_64 4/16
Installing : qpid-cpp-client-devel-0.24-4.fc19.1.x86_64 5/16
Installing : qpid-qmf-devel-0.24-4.fc19.1.x86_64 6/16
Installing : qpid-cpp-server-ha-0.24-4.fc19.1.x86_64 7/16
Installing : python-qpid-qmf-0.24-4.fc19.1.x86_64 8/16
Installing : ruby-qpid-qmf-0.24-4.fc19.1.x86_64 9/16
Installing : qpid-cpp-server-xml-0.24-4.fc19.1.x86_64 10/16
Installing : qpid-cpp-server-rdma-0.24-4.fc19.1.x86_64 11/16
Installing : qpid-cpp-server-store-0.24-4.fc19.1.x86_64 12/16
Installing : qpid-cpp-debuginfo-0.24-4.fc19.1.x86_64 13/16
Cleanup : qpid-cpp-server-0.24-3.fc19.1.x86_64 14/16
Cleanup : qpid-cpp-client-0.24-3.fc19.1.x86_64 15/16
Cleanup : qpid-cpp-client-rdma-0.24-3.fc19.1.x86_64 16/16
Verifying : qpid-cpp-server-ha-0.24-4.fc19.1.x86_64 1/16
Verifying : qpid-cpp-server-xml-0.24-4.fc19.1.x86_64 2/16
Verifying : qpid-cpp-server-rdma-0.24-4.fc19.1.x86_64 3/16
Verifying : python-qpid-qmf-0.24-4.fc19.1.x86_64 4/16
Verifying : qpid-cpp-server-0.24-4.fc19.1.x86_64 5/16
Verifying : qpid-qmf-devel-0.24-4.fc19.1.x86_64 6/16
Verifying : qpid-qmf-0.24-4.fc19.1.x86_64 7/16
Verifying : qpid-cpp-client-rdma-0.24-4.fc19.1.x86_64 8/16
Verifying : qpid-cpp-debuginfo-0.24-4.fc19.1.x86_64 9/16
Verifying : ruby-qpid-qmf-0.24-4.fc19.1.x86_64 10/16
Verifying : qpid-cpp-client-0.24-4.fc19.1.x86_64 11/16
Verifying : qpid-cpp-server-store-0.24-4.fc19.1.x86_64 12/16
Verifying : qpid-cpp-client-devel-0.24-4.fc19.1.x86_64 13/16
Verifying : qpid-cpp-server-0.24-3.fc19.1.x86_64 14/16
Verifying : qpid-cpp-client-rdma-0.24-3.fc19.1.x86_64 15/16
Verifying : qpid-cpp-client-0.24-3.fc19.1.x86_64 16/16
python-qpid-qmf.x86_64 0:0.24-4.fc19.1 qpid-cpp-client-devel.x86_64 0:0.24-4.fc19.1
qpid-cpp-debuginfo.x86_64 0:0.24-4.fc19.1 qpid-cpp-server-ha.x86_64 0:0.24-4.fc19.1
qpid-cpp-server-rdma.x86_64 0:0.24-4.fc19.1 qpid-cpp-server-store.x86_64 0:0.24-4.fc19.1
qpid-cpp-server-xml.x86_64 0:0.24-4.fc19.1 qpid-qmf.x86_64 0:0.24-4.fc19.1
qpid-qmf-devel.x86_64 0:0.24-4.fc19.1 ruby-qpid-qmf.x86_64 0:0.24-4.fc19.1
qpid-cpp-client.x86_64 0:0.24-4.fc19.1 qpid-cpp-client-rdma.x86_64 0:0.24-4.fc19.1 qpid-cpp-server.x86_64 0:0.24-4.fc19.1
[boris@localhost ~]$ sudo packstack --answer-file=/home/boris/packstack-answers-20131008-220433.txt
192.168.1.127_swift.pp : [ DONE ]
[ ERROR ]
ERROR : Error during puppet run : Error: Could not prefetch quantum_network provider 'quantum': Execution of '/usr/bin/quantum net-list --format=csv --column=id --quote=none' returned 1: [Errno 111] Connection refused
Please check log file /var/tmp/packstack/20131008-232557-5lcymK/openstack-setup.log for more information
please verify, this "Connection refused" really comes from qpid, and not from quantum. This one looks suspicious for me, since OpenStack Havana doesn't contain quantum at all. (FYI, quantum was renamed to neutron back in April).
It comes from /var/log/quantum/server.log or /var/log/neutron/server.log. I've also tried Havana&Neutron.
2013-10-09 09:34:54 ERROR [quantum.openstack.common.rpc.impl_qpid] Unable to connect to AMQP server: connection-forced: Authentication failed(320). Sleeping 60 seconds
Created attachment 809672 [details]
Thank you Boris. Sounds like a valid qpid issue.
qpid-cpp-0.24-4.fc20.1 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
(In reply to Matthias Runge from comment #15)
> please verify, this "Connection refused" really comes from qpid, and not
> from quantum. This one looks suspicious for me, since OpenStack Havana
> doesn't contain quantum at all. (FYI, quantum was renamed to neutron back in
I did a little googling on the error message and found this:
Are you guys sure this a bug in qpid and not in how it's being configured? Is your auth properly setup, etc?
since the issue went away, after adding auth=no to qpidd.conf the issue went away.
Since packstack (the deployment tool) places the config in /etc/qpidd.conf and not at /etc/qpid/qpidd.conf, it's still a config placement issue. IMHO one can verify by downgrading qpid.
(In reply to Matthias Runge from comment #21)
> since the issue went away, after adding auth=no to qpidd.conf the issue went
> Since packstack (the deployment tool) places the config in /etc/qpidd.conf
> and not at /etc/qpid/qpidd.conf, it's still a config placement issue. IMHO
> one can verify by downgrading qpid.
Can't you update packstack to now put the qpidd.conf file in /etc/qpid/ where it will live going forward?
Yes we can, that's the reason why bug #1012814 was opened.
Being able to workaround this is not the point.
One can always workaround stuff, but with various costs.
There were two issues here.
1. Upstream didn't consider backwards compat when moving configs
2. The incompat change was pushed to F19
1. s bad, but 2. compounded the issue.
Please consider backwards compatibility in future.