Bug 1296377

I was a little confused when I entered this bug. I did see some errors connecting to 127.0.0.1, but I think those cleared up after I set net.ipv6=True. The error in the message above is still happening, but I'm not sure why it can't connect to the replicaset.

When trying to build MongoDB replicasets, Puppet is using mongo_node_ips parameter to configure replicasets.
Can you provide this value to make sure that's an IPv6?

I think I've found the issue in puppet-mongodb. The Puppet provider is using the following line to evaluate commands using the mongo client:

        if host
          out = mongo([db, '--quiet', '--host', host, '--eval', cmd])
        else
          out = mongo([db, '--quiet', '--host', get_conn_string, '--eval', cmd])

(from https://github.com/puppetlabs/puppetlabs-mongodb/blob/56c0014aee8356caec60596f32bf03776f548fbe/lib/puppet/provider/mongodb.rb#L124-L127)

According to the mongo(1) man page:

       --ipv6 Enables  IPv6  support, which allows mongo to connect to the MongoDB instance using an IPv6 network. All MongoDB programs and processes,  including  mongo, disable IPv6 support by default.

So we would need to add --ipv6 as a parameter to the mongo command, also in https://github.com/puppetlabs/puppetlabs-mongodb/blob/56c0014aee8356caec60596f32bf03776f548fbe/lib/puppet/provider/mongodb.rb#L83 . Without it, the mongo client will never try to use IPv6 addreses.

I have tested the command-line adding --ipv6 in an IPv4-only environment, and it does not do any harm, so it looks safe to always enable IPv6.

https://github.com/puppetlabs/puppetlabs-mongodb/pull/254 contains a proposed fix to puppetlabs-mongodb

Thanks a lot Javier for finding the root cause! Sofer and I are working on a patch that would not be too intrusive for IPv4 deployments, even it's not a risk to enable ipv6 flag on ipv4 deployments, I think it's worth to enable it only when needed.

https://github.com/puppetlabs/puppetlabs-mongodb/pull/255

openstack-puppet-modules-2015.1.8-41.el7ost.noarch

Deployment failed with:

Error: Unable to connect to mongodb server! (127.0.0.1:27017)
Error: /Stage[main]/Main/Mongodb_conn_validator[fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017]/ensure: change from absent to present failed: Unable to connect to mongodb server! (127.0.0.1:27017)
Error: Could not prefetch mongodb_replset provider 'mongo': Execution of '/usr/bin/mongo --quiet --host fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017 --eval printjson(rs.conf())' returned 1: 2016-01-19T07:13:58.544-0500 getaddrinfo("fd00:fd00:fd00:2000:f816:3eff:fedf:aed5") failed: Address family for hostname not supported
2016-01-19T07:13:58.545-0500 Error: couldn't connect to server fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017 (0.0.0.0), address resolved to 0.0.0.0 at src/mongo/shell/mongo.js:148
[2016-01-19 07:14:04,847] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/a8d40e7d-6e5a-47c8-bb62-39af769b98e6.pp. [6]

Attaching the os-collect-config log.

Hey Marius,

Can you please attach os-collect-config & all mongodb configuration files?
I'm investigating.

Thanks

Regarding your logs, I think we're missing brackets somewhere in the config, so I'm curious to see if you have them and if not if you actually provide it.
Otherwise, I'll continue my patch in puppetlabs/mongodb.

Created attachment 1116197 [details]
os-collect-config

Sorry, I forgot to attach the os-collect-config log. Attaching it now.

This is the mongod.conf:

[root@overcloud-controller-0 heat-admin]# cat /etc/mongod.conf 
# mongo.conf - generated from Puppet


#where to log
logpath=/var/log/mongodb/mongodb.log
logappend=true
ipv6=true
# Set this option to configure the mongod or mongos process to bind to and
# listen for connections from applications on this address.
# You may concatenate a list of comma separated values to bind mongod to multiple IP addresses.
bind_ip = fd00:fd00:fd00:2000:f816:3eff:fe63:289e
# fork and run in background
fork=true
dbpath=/var/lib/mongodb
# location of pidfile
pidfilepath=/var/run/mongodb/mongod.pid
# Enables journaling
journal = true
# Turn on/off security.  Off is currently the default
noauth=true
# Configure ReplicaSet membership
replSet = tripleo

It confirms we really need to have brackets, in the config, and when running the Puppet provider.
I'm working on a fix, following-up my pull-request.

[root@overcloud-controller-0 ~]# mongo --ipv6 --host fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017  <<<'rs.status()'
MongoDB shell version: 2.6.9
connecting to: fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017/test
{
	"set" : "tripleo",
	"date" : ISODate("2016-01-21T11:38:50Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 1111,
			"optime" : Timestamp(1453376225, 2),
			"optimeDate" : ISODate("2016-01-21T11:37:05Z"),
			"electionTime" : Timestamp(1453375227, 1),
			"electionDate" : ISODate("2016-01-21T11:20:27Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "fd00:fd00:fd00:2000:f816:3eff:fe8c:f2:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 1109,
			"optime" : Timestamp(1453376225, 2),
			"optimeDate" : ISODate("2016-01-21T11:37:05Z"),
			"lastHeartbeat" : ISODate("2016-01-21T11:38:49Z"),
			"lastHeartbeatRecv" : ISODate("2016-01-21T11:38:49Z"),
			"pingMs" : 0,
			"syncingTo" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017"
		},
		{
			"_id" : 2,
			"name" : "fd00:fd00:fd00:2000:f816:3eff:fedd:b44:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 1109,
			"optime" : Timestamp(1453376225, 2),
			"optimeDate" : ISODate("2016-01-21T11:37:05Z"),
			"lastHeartbeat" : ISODate("2016-01-21T11:38:49Z"),
			"lastHeartbeatRecv" : ISODate("2016-01-21T11:38:49Z"),
			"pingMs" : 0,
			"syncingTo" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017"
		}
	],
	"ok" : 1
}
bye

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0265.html