Red Hat Bugzilla – Bug 1296377
mongodb cannot connect to replicaset when IPv6 enabled
Last modified: 2016-10-11 09:30:47 EDT
Description of problem: When using IPv6 TripleO Heat templates, Mongo DB cannot connect to the replicaset. It attempts to connect to 127.0.0.1:27017, but this is not reachable. Version-Release number of selected component (if applicable): OSP 8 beta mongodb 2.6.11-1.el7 mongodb-server 2.6.11-1.el7 python-pymongo.x86_64 3.0.3-1.el7 How reproducible: 100% Steps to Reproduce: 1. Deploy with IPv6 TripleO Heat templates 2. 3. Actual results: When net.ipv6 is set to true in the mongod.conf, it will respect the IPv6 bind_ip address and bind to that IPv6 address. OpenStack will still try to connect to MongoDB on 127.0.0.1:27017, and this fails when the service is listening on the IPv6 address. Expected results: OpenStack should be trying to reach mongo on the actual IP and port where it is listening. Additional info: Netstat showing that mongo is listening on port 27017 on an IPv6 address. [root@overcloud-controller-0 etc]# netstat -anp | grep mongo tcp6 0 0 fd00:fd00:fd00:40:27017 :::* LISTEN 23564/mongod I can connect to mongo directly via the IP: [root@overcloud-controller-0 etc]# telnet fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1 27017 Trying fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1... Connected to fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1. Escape character is '^]'. Error message from failed deployment. Error: Could not prefetch mongodb_replset provider 'mongo': Could not evalute MongoDB shell command: printjson(rs.conf())\u001b[0m\n\u001b[1;31mWarning: Can't connect to replicaset member fd00:fd00:fd00:4000:f816:3eff:fec4:8bf1:27017.\u001b[0m\n\u001b[1;31mError: /Stage[main]/Main/Mongodb_replset[tripleo]: Could not evaluate: Can't connect to any member of replicaset tripleo.
I was a little confused when I entered this bug. I did see some errors connecting to 127.0.0.1, but I think those cleared up after I set net.ipv6=True. The error in the message above is still happening, but I'm not sure why it can't connect to the replicaset.
When trying to build MongoDB replicasets, Puppet is using mongo_node_ips parameter to configure replicasets. Can you provide this value to make sure that's an IPv6?
I think I've found the issue in puppet-mongodb. The Puppet provider is using the following line to evaluate commands using the mongo client: if host out = mongo([db, '--quiet', '--host', host, '--eval', cmd]) else out = mongo([db, '--quiet', '--host', get_conn_string, '--eval', cmd]) (from https://github.com/puppetlabs/puppetlabs-mongodb/blob/56c0014aee8356caec60596f32bf03776f548fbe/lib/puppet/provider/mongodb.rb#L124-L127) According to the mongo(1) man page: --ipv6 Enables IPv6 support, which allows mongo to connect to the MongoDB instance using an IPv6 network. All MongoDB programs and processes, including mongo, disable IPv6 support by default. So we would need to add --ipv6 as a parameter to the mongo command, also in https://github.com/puppetlabs/puppetlabs-mongodb/blob/56c0014aee8356caec60596f32bf03776f548fbe/lib/puppet/provider/mongodb.rb#L83 . Without it, the mongo client will never try to use IPv6 addreses. I have tested the command-line adding --ipv6 in an IPv4-only environment, and it does not do any harm, so it looks safe to always enable IPv6.
https://github.com/puppetlabs/puppetlabs-mongodb/pull/254 contains a proposed fix to puppetlabs-mongodb
Thanks a lot Javier for finding the root cause! Sofer and I are working on a patch that would not be too intrusive for IPv4 deployments, even it's not a risk to enable ipv6 flag on ipv4 deployments, I think it's worth to enable it only when needed.
https://github.com/puppetlabs/puppetlabs-mongodb/pull/255
openstack-puppet-modules-2015.1.8-41.el7ost.noarch Deployment failed with: Error: Unable to connect to mongodb server! (127.0.0.1:27017) Error: /Stage[main]/Main/Mongodb_conn_validator[fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017]/ensure: change from absent to present failed: Unable to connect to mongodb server! (127.0.0.1:27017) Error: Could not prefetch mongodb_replset provider 'mongo': Execution of '/usr/bin/mongo --quiet --host fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017 --eval printjson(rs.conf())' returned 1: 2016-01-19T07:13:58.544-0500 getaddrinfo("fd00:fd00:fd00:2000:f816:3eff:fedf:aed5") failed: Address family for hostname not supported 2016-01-19T07:13:58.545-0500 Error: couldn't connect to server fd00:fd00:fd00:2000:f816:3eff:fedf:aed5:27017 (0.0.0.0), address resolved to 0.0.0.0 at src/mongo/shell/mongo.js:148 [2016-01-19 07:14:04,847] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/a8d40e7d-6e5a-47c8-bb62-39af769b98e6.pp. [6] Attaching the os-collect-config log.
Hey Marius, Can you please attach os-collect-config & all mongodb configuration files? I'm investigating. Thanks
Regarding your logs, I think we're missing brackets somewhere in the config, so I'm curious to see if you have them and if not if you actually provide it. Otherwise, I'll continue my patch in puppetlabs/mongodb.
Created attachment 1116197 [details] os-collect-config Sorry, I forgot to attach the os-collect-config log. Attaching it now.
This is the mongod.conf: [root@overcloud-controller-0 heat-admin]# cat /etc/mongod.conf # mongo.conf - generated from Puppet #where to log logpath=/var/log/mongodb/mongodb.log logappend=true ipv6=true # Set this option to configure the mongod or mongos process to bind to and # listen for connections from applications on this address. # You may concatenate a list of comma separated values to bind mongod to multiple IP addresses. bind_ip = fd00:fd00:fd00:2000:f816:3eff:fe63:289e # fork and run in background fork=true dbpath=/var/lib/mongodb # location of pidfile pidfilepath=/var/run/mongodb/mongod.pid # Enables journaling journal = true # Turn on/off security. Off is currently the default noauth=true # Configure ReplicaSet membership replSet = tripleo
It confirms we really need to have brackets, in the config, and when running the Puppet provider. I'm working on a fix, following-up my pull-request.
[root@overcloud-controller-0 ~]# mongo --ipv6 --host fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017 <<<'rs.status()' MongoDB shell version: 2.6.9 connecting to: fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017/test { "set" : "tripleo", "date" : ISODate("2016-01-21T11:38:50Z"), "myState" : 1, "members" : [ { "_id" : 0, "name" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 1111, "optime" : Timestamp(1453376225, 2), "optimeDate" : ISODate("2016-01-21T11:37:05Z"), "electionTime" : Timestamp(1453375227, 1), "electionDate" : ISODate("2016-01-21T11:20:27Z"), "self" : true }, { "_id" : 1, "name" : "fd00:fd00:fd00:2000:f816:3eff:fe8c:f2:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1109, "optime" : Timestamp(1453376225, 2), "optimeDate" : ISODate("2016-01-21T11:37:05Z"), "lastHeartbeat" : ISODate("2016-01-21T11:38:49Z"), "lastHeartbeatRecv" : ISODate("2016-01-21T11:38:49Z"), "pingMs" : 0, "syncingTo" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017" }, { "_id" : 2, "name" : "fd00:fd00:fd00:2000:f816:3eff:fedd:b44:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 1109, "optime" : Timestamp(1453376225, 2), "optimeDate" : ISODate("2016-01-21T11:37:05Z"), "lastHeartbeat" : ISODate("2016-01-21T11:38:49Z"), "lastHeartbeatRecv" : ISODate("2016-01-21T11:38:49Z"), "pingMs" : 0, "syncingTo" : "fd00:fd00:fd00:2000:f816:3eff:fe06:16b:27017" } ], "ok" : 1 } bye
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0265.html