Documentation: Controller replacement guide, Delete the failed node from MongoDB section: need to add a note about finding the IP to connect to. Currently it says: Delete the failed node from MongoDB. First, connect to MongoDB on any of remaining nodes: [heat-admin@overcloud-controller-0 ~]$ mongo --host 192.168.0.23 MongoDB shell version: 2.6.9 connecting to: 192.168.0.23:27017/test Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see http://docs.mongodb.org/ Questions? Try the support group http://groups.google.com/group/mongodb-user tripleo:SECONDARY> So the first option a user would try - is to connect to the IP as seen in the output from 'nova list', which would fail. I see that the IP is on InternalApi on my setup. But maybe it's best to looking the IP where port 27017 is open.
Two things in "Delete the failed node from MongoDB" section: 1) The IPs (shown in the output and used for connection) in the guide differ: sudo netstat -tulnp | grep 27017 tcp 0 0 192.168.201.47:27017 0.0.0.0:* LISTEN 13415/mongod [heat-admin@overcloud-controller-0 ~]$ mongo --host 192.168.0.47 2) I was unable to delete the node when connected to secondary: { "_id" : 1, "name" : "10.19.94.13:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : Timestamp(1462580916, 5), "optimeDate" : ISODate("2016-05-07T00:28:36Z"), "lastHeartbeat" : ISODate("2016-05-07T03:55:16Z"), "lastHeartbeatRecv" : ISODate("2016-05-07T00:33:26Z"), "pingMs" : 0, "syncingTo" : "10.19.94.16:27017" }, { "_id" : 2, "name" : "10.19.94.14:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18603, "optime" : Timestamp(1462593277, 6), "optimeDate" : ISODate("2016-05-07T03:54:37Z"), "self" : true } ], "ok" : 1 } tripleo:SECONDARY> rs.remove('10.19.94.13:27017') { "ok" : 0, "errmsg" : "replSetReconfig command must be sent to the current replica set primary." } tripleo:SECONDARY> Note: I was able to delete it (although with errors shown) when I connected to primary: { "_id" : 1, "name" : "10.19.94.13:27017", "health" : 0, "state" : 8, "stateStr" : "(not reachable/healthy)", "uptime" : 0, "optime" : Timestamp(1462580916, 5), "optimeDate" : ISODate("2016-05-07T00:28:36Z"), "lastHeartbeat" : ISODate("2016-05-07T03:57:01Z"), "lastHeartbeatRecv" : ISODate("2016-05-07T00:33:26Z"), "pingMs" : 0, "syncingTo" : "10.19.94.16:27017" }, { "_id" : 2, "name" : "10.19.94.14:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 18687, "optime" : Timestamp(1462593277, 6), "optimeDate" : ISODate("2016-05-07T03:54:37Z"), "lastHeartbeat" : ISODate("2016-05-07T03:57:06Z"), "lastHeartbeatRecv" : ISODate("2016-05-07T03:57:06Z"), "pingMs" : 0, "syncingTo" : "10.19.94.16:27017" } ], "ok" : 1 } tripleo:PRIMARY> rs.remove('10.19.94.13:27017') 2016-05-07T03:57:19.541+0000 DBClientCursor::init call() failed 2016-05-07T03:57:19.543+0000 Error: error doing query: failed at src/mongo/shell/query.js:81 2016-05-07T03:57:19.545+0000 trying reconnect to 10.19.94.16:27017 (10.19.94.16) failed 2016-05-07T03:57:19.547+0000 reconnect 10.19.94.16:27017 (10.19.94.16) ok
(In reply to Alexander Chuzhoy from comment #3) > Two things in "Delete the failed node from MongoDB" section: > 1) The IPs (shown in the output and used for connection) in the guide differ: > sudo netstat -tulnp | grep 27017 > tcp 0 0 192.168.201.47:27017 0.0.0.0:* LISTEN > 13415/mongod > [heat-admin@overcloud-controller-0 ~]$ mongo --host 192.168.0.47 > > Added fix for this. > tripleo:PRIMARY> rs.remove('10.19.94.13:27017') > 2016-05-07T03:57:19.541+0000 DBClientCursor::init call() failed > 2016-05-07T03:57:19.543+0000 Error: error doing query: failed at > src/mongo/shell/query.js:81 > 2016-05-07T03:57:19.545+0000 trying reconnect to 10.19.94.16:27017 > (10.19.94.16) failed > 2016-05-07T03:57:19.547+0000 reconnect 10.19.94.16:27017 (10.19.94.16) ok This error should be normal and should be what's expected. Might be a good idea to add a note on this item. As for the PRIMARY vs SECONDARY issue, I'll add a note for that too. Something along the lings of: ===IMPORTANT=== You must run the command against the PRIMARY replica set. If you see the following message: "replSetReconfig command must be sent to the current replica set primary." Relog into MongoDB on the node designated as PRIMARY.
Verified. This section of the doc looks good.
Changes now live on the customer portal.