Bug 1055660

Can you share the contents of your /etc/qpid/qpidd.conf file? Also any changes the configuration for the service that have been made.

The contents of /etc/qpid/qpidd.conf are:

    log-enable=info+

I have made no changes to the service.

Whether or not this problems happens seems to be a bit of a crapshoot.  Sometimes I reboot to find qpid listening on both IPv4 and IPv6 sockets, sometimes just IPv6.

With info+ logging enabled, when qpid listens on both interfaces it logs:

Jan 20 21:38:03 lars-devstack.os1 qpidd[666]: 2014-01-20 21:38:03 [Network] info Listening to: 0.0.0.0:5672
Jan 20 21:38:03 lars-devstack.os1 qpidd[666]: 2014-01-20 21:38:03 [Network] info Listening to: [::]:5672
Jan 20 21:38:03 lars-devstack.os1 qpidd[666]: 2014-01-20 21:38:03 [Network] notice Listening on TCP/TCP6 port 5672


While when it only listens on the IPv6 socket it only logs:

Jan 20 21:32:39 lars-devstack.os1 qpidd[359]: 2014-01-20 21:32:39 [Network] info Listening to: [::]:5672
Jan 20 21:32:39 lars-devstack.os1 qpidd[359]: 2014-01-20 21:32:39 [Network] notice Listening on TCP/TCP6 port 5672

I suspect this is a function of parallel startup:

By default qpidd will actively look for all the active interfaces when it starts. It is very possible that when qpidd is started at boot the network inerfaces aren't all up yet.

It should be possible to make qpidd startup dependent on the appropriate interfaces receiving an IP address and only then being started.

So in order to work on simple systems you might need to wait for IPv6/IPv4 addresses to be available on the loopback interface and IPv4/IPv6 addresses to be available on the em1 interface.

To clarify a little, qpidd doesn't bind to the catch all address and so will not notice if new network interfaces arrive whilst it is running.

It *appears* to be binding to the catch all address.  Or do you mean some other than `0.0.0.0`?  I added an `ExecStartPre` to the `qpidd.service` file to capture the network state when the service starts, and there did not appear to be any correlation between the state of `eth0` and successfully binding to the IPv4 socket.

That is, if the `ExecStartPre` script ran `ip addr`, on a *successful* startup (qpidd bound to both the IPv4 and IPv6 sockets), the output of `ip addr` was:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

That is, `eth0` didn't even show up in the output.  At other times, the output look like this:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether fa:16:3e:41:e7:92 brd ff:ff:ff:ff:ff:ff

But I haven't been able to spot a correlation between either state and the problem.  It certainly looks like a race condition *somewhere*.

(In reply to Andrew Stitcher from comment #3)
> I suspect this is a function of parallel startup:
> 
> By default qpidd will actively look for all the active interfaces when it
> starts. It is very possible that when qpidd is started at boot the network
> inerfaces aren't all up yet.
> 
> It should be possible to make qpidd startup dependent on the appropriate
> interfaces receiving an IP address and only then being started.
> 
> So in order to work on simple systems you might need to wait for IPv6/IPv4
> addresses to be available on the loopback interface and IPv4/IPv6 addresses
> to be available on the em1 interface.

Hrm, that shouldn't be the case. The systemd service file for qpidd has:

  After=network.target

so it shouldn't be starting in parallel to but after the network service has started.

Actually, the service file for qpid-cpp-server-0.24-8.fc20.x86_64 has:

[Unit]
Description=An AMQP message broker daemon.
Documentation=man:qpidd(1) http://qpid.apache.org/
Requires=network.target

That is, it has a Requires dependency, but not an After dependency.  Let me add that and see what happens.

(In reply to Lars Kellogg-Stedman from comment #7)
> Actually, the service file for qpid-cpp-server-0.24-8.fc20.x86_64 has:
> 
> [Unit]
> Description=An AMQP message broker daemon.
> Documentation=man:qpidd(1) http://qpid.apache.org/
> Requires=network.target
> 
> That is, it has a Requires dependency, but not an After dependency.  Let me
> add that and see what happens.

CORRECTION: This change is in the 0.26 update that will go out. I'm going to go ahead an push an updated 0.24 with the fixed services file since 0.26 isn't up to a vote yet for release.

That should do it.  Empirical evidence suggests that adding the After constraint fixes the problem.

(In reply to Lars Kellogg-Stedman from comment #5)
> It *appears* to be binding to the catch all address.  Or do you mean some
> other than `0.0.0.0`? 

I misspoke - If you specify an interface to bind to it will enumerate the interfaces and only bind to addresses in can see on interfaces. Here where you are not specifying any interface it *will* bind to the catchall addresses that it finds. However if networking is not fully up yet then it won't necessarily find all the catch all addresses.

qpid-cpp-0.24-9.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/qpid-cpp-0.24-9.fc20

qpid-cpp-0.24-6.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/qpid-cpp-0.24-6.fc19

Package qpid-cpp-0.24-9.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing qpid-cpp-0.24-9.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-1366/qpid-cpp-0.24-9.fc20
then log in and leave karma (feedback).

qpid-cpp-0.24-9.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

qpid-cpp-0.24-6.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

*** Bug 1049488 has been marked as a duplicate of this bug. ***