Bug 1296377 - mongodb cannot connect to replicaset when IPv6 enabled
Summary: mongodb cannot connect to replicaset when IPv6 enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: z4
: 7.0 (Kilo)
Assignee: Emilien Macchi
QA Contact: Marius Cornea
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-07 01:48 UTC by Dan Sneddon
Modified: 2016-10-11 13:30 UTC (History)
12 users (show)

Fixed In Version: openstack-puppet-modules-2015.1.8-42.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-02-18 16:43:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
os-collect-config (322.21 KB, text/plain)
2016-01-19 14:40 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0265 0 normal SHIPPED_LIVE openstack-packstack and openstack-puppet-modules bug fix advisory 2016-02-18 21:41:18 UTC

Description Dan Sneddon 2016-01-07 01:48:56 UTC
Description of problem:
When using IPv6 TripleO Heat templates, Mongo DB cannot connect to the replicaset. It attempts to connect to 127.0.0.1:27017, but this is not reachable.

Version-Release number of selected component (if applicable):
OSP 8 beta
mongodb 2.6.11-1.el7
mongodb-server 2.6.11-1.el7
python-pymongo.x86_64        3.0.3-1.el7

How reproducible:
100%

Steps to Reproduce:
1. Deploy with IPv6 TripleO Heat templates
2.
3.

Actual results:
When net.ipv6 is set to true in the mongod.conf, it will respect the IPv6 bind_ip address and bind to that IPv6 address. OpenStack will still try to connect to MongoDB on 127.0.0.1:27017, and this fails when the service is listening on the IPv6 address.

Expected results:
OpenStack should be trying to reach mongo on the actual IP and port where it is listening.

Additional info: 

Netstat showing that mongo is listening on port 27017 on an IPv6 address. 

[root@overcloud-controller-0 etc]# netstat -anp | grep mongo
tcp6       0      0 fd00:fd00:fd00:40:27017 :::*                    LISTEN      23564/mongod

I can connect to mongo directly via the IP:

[root@overcloud-controller-0 etc]# telnet fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1 27017
Trying fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1...
Connected to fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1.
Escape character is '^]'.

Error message from failed deployment.

Error: Could not prefetch mongodb_replset provider 'mongo': Could not evalute MongoDB shell command: printjson(rs.conf())\u001b[0m\n\u001b[1;31mWarning: Can't connect to replicaset member fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1:27017.\u001b[0m\n\u001b[1;31mError: /Stage[main]/Main/Mongodb_replset[tripleo]: Could not evaluate: Can't connect to any member of replicaset tripleo.

Comment 3 Dan Sneddon 2016-01-08 19:19:05 UTC
I was a little confused when I entered this bug. I did see some errors connecting to 127.0.0.1, but I think those cleared up after I set net.ipv6=True. The error in the message above is still happening, but I'm not sure why it can't connect to the replicaset.

Comment 4 Emilien Macchi 2016-01-13 15:30:08 UTC
When trying to build MongoDB replicasets, Puppet is using mongo_node_ips parameter to configure replicasets.
Can you provide this value to make sure that's an IPv6?

Comment 5 Javier Peña 2016-01-16 10:58:45 UTC
I think I've found the issue in puppet-mongodb. The Puppet provider is using the following line to evaluate commands using the mongo client:

        if host
          out = mongo([db, '--quiet', '--host', host, '--eval', cmd])
        else
          out = mongo([db, '--quiet', '--host', get_conn_string, '--eval', cmd])

(from https://github.com/puppetlabs/puppetlabs-mongodb/blob/56c0014aee8356caec60596f32bf03776f548fbe/lib/puppet/provider/mongodb.rb#L124-L127)

According to the mongo(1) man page:

       --ipv6 Enables  IPv6  support, which allows mongo to connect to the MongoDB instance using an IPv6 network. All MongoDB programs and processes,  including  mongo, disable IPv6 support by default.

So we would need to add --ipv6 as a parameter to the mongo command, also in https://github.com/puppetlabs/puppetlabs-mongodb/blob/56c0014aee8356caec60596f32bf03776f548fbe/lib/puppet/provider/mongodb.rb#L83 . Without it, the mongo client will never try to use IPv6 addreses.

I have tested the command-line adding --ipv6 in an IPv4-only environment, and it does not do any harm, so it looks safe to always enable IPv6.

Comment 6 Javier Peña 2016-01-16 11:44:35 UTC
https://github.com/puppetlabs/puppetlabs-mongodb/pull/254 contains a proposed fix to puppetlabs-mongodb

Comment 7 Emilien Macchi 2016-01-16 12:48:01 UTC
Thanks a lot Javier for finding the root cause! Sofer and I are working on a patch that would not be too intrusive for IPv4 deployments, even it's not a risk to enable ipv6 flag on ipv4 deployments, I think it's worth to enable it only when needed.

Comment 10 Marius Cornea 2016-01-19 12:43:20 UTC
openstack-puppet-modules-2015.1.8-41.el7ost.noarch

Deployment failed with:

Error: Unable to connect to mongodb server! (127.0.0.1:27017)
Error: /Stage[main]/Main/Mongodb_conn_validator[fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017]/ensure: change from absent to present failed: Unable to connect to mongodb server! (127.0.0.1:27017)
Error: Could not prefetch mongodb_replset provider 'mongo': Execution of '/usr/bin/mongo --quiet --host fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017 --eval printjson(rs.conf())' returned 1: 2016-01-19T07:13:58.544-0500 getaddrinfo("fd00:fd00:fd00:2000:f816:3eff:fedf:aed5") failed: Address family for hostname not supported
2016-01-19T07:13:58.545-0500 Error: couldn't connect to server fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017 (0.0.0.0), address resolved to 0.0.0.0 at src/mongo/shell/mongo.js:148
[2016-01-19 07:14:04,847] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/a8d40e7d-6e5a-47c8-bb62-39af769b98e6.pp. [6]

Attaching the os-collect-config log.

Comment 11 Emilien Macchi 2016-01-19 13:48:34 UTC
Hey Marius,

Can you please attach os-collect-config & all mongodb configuration files?
I'm investigating.

Thanks

Comment 12 Emilien Macchi 2016-01-19 13:52:14 UTC
Regarding your logs, I think we're missing brackets somewhere in the config, so I'm curious to see if you have them and if not if you actually provide it.
Otherwise, I'll continue my patch in puppetlabs/mongodb.

Comment 13 Marius Cornea 2016-01-19 14:40:06 UTC
Created attachment 1116197 [details]
os-collect-config

Sorry, I forgot to attach the os-collect-config log. Attaching it now.

Comment 14 Marius Cornea 2016-01-19 14:40:43 UTC
This is the mongod.conf:

[root@overcloud-controller-0 heat-admin]# cat /etc/mongod.conf 
# mongo.conf - generated from Puppet


#where to log
logpath=/var/log/mongodb/mongodb.log
logappend=true
ipv6=true
# Set this option to configure the mongod or mongos process to bind to and
# listen for connections from applications on this address.
# You may concatenate a list of comma separated values to bind mongod to multiple IP addresses.
bind_ip = fd00:fd00:fd00:2000:f816:3eff:fe63:289e
# fork and run in background
fork=true
dbpath=/var/lib/mongodb
# location of pidfile
pidfilepath=/var/run/mongodb/mongod.pid
# Enables journaling
journal = true
# Turn on/off security.  Off is currently the default
noauth=true
# Configure ReplicaSet membership
replSet = tripleo

Comment 15 Emilien Macchi 2016-01-19 14:48:57 UTC
It confirms we really need to have brackets, in the config, and when running the Puppet provider.
I'm working on a fix, following-up my pull-request.

Comment 17 Marius Cornea 2016-01-21 11:39:19 UTC
[root@overcloud-controller-0 ~]# mongo --ipv6 --host fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017  <<<'rs.status()'
MongoDB shell version: 2.6.9
connecting to: fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017/test
{
	"set" : "tripleo",
	"date" : ISODate("2016-01-21T11:38:50Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 1111,
			"optime" : Timestamp(1453376225, 2),
			"optimeDate" : ISODate("2016-01-21T11:37:05Z"),
			"electionTime" : Timestamp(1453375227, 1),
			"electionDate" : ISODate("2016-01-21T11:20:27Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "fd00:fd00:fd00:2000:f816:3eff:fe8c:f2:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 1109,
			"optime" : Timestamp(1453376225, 2),
			"optimeDate" : ISODate("2016-01-21T11:37:05Z"),
			"lastHeartbeat" : ISODate("2016-01-21T11:38:49Z"),
			"lastHeartbeatRecv" : ISODate("2016-01-21T11:38:49Z"),
			"pingMs" : 0,
			"syncingTo" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017"
		},
		{
			"_id" : 2,
			"name" : "fd00:fd00:fd00:2000:f816:3eff:fedd:b44:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 1109,
			"optime" : Timestamp(1453376225, 2),
			"optimeDate" : ISODate("2016-01-21T11:37:05Z"),
			"lastHeartbeat" : ISODate("2016-01-21T11:38:49Z"),
			"lastHeartbeatRecv" : ISODate("2016-01-21T11:38:49Z"),
			"pingMs" : 0,
			"syncingTo" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017"
		}
	],
	"ok" : 1
}
bye

Comment 21 errata-xmlrpc 2016-02-18 16:43:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0265.html


Note You need to log in before you can comment on or make changes to this bug.