Description of problem: Storage cluster did not 'notice' new node added, thus throughput was not changed Version-Release number of selected component (if applicable): JON 3.3 ER05 How reproducible: reproduced twice (2 trays) Steps to Reproduce: 1. server with 2 storage nodes installed and running 2. stop both nodes 3. restart both nodes 4. add new node (3rd node) Actual results: After step3. throughput is set to 30K After step4. throughput is set to 60K Expected results: After step3. throughput must be 60K After step4. throughput must be 90K Additional info: server.log attached
Created attachment 951076 [details] new_server.log
As that setup doesn't exists anymore, I'll add this as a comment only for now. When the third node was added, the DataStax driver still reported only 2 nodes up (as seen from the metrics), so I'm not sure if the third node was really joined at that point. We use the amount of nodes connected as reported by the driver, so the 60k was correct behaviour on our side.
I am looked at the logs some. I am not entirely clear what happened with deploying/undeploying nodes. There were a lot of errors though. The throttling changes after steps 3 and 4 however, look correct to me since the environment reports that there storage nodes in the cluster. If there were three nodes in the cluster, then I would expect that the request limit would be 90k after step 4. Armine, can you try to reproduce without all the of the (un)deploy errors?
After a long discussion with Armine, I do not think there is any issue here. We have not been able to reproduce the problems reported. I am going to close this. We can open a different BZs if need be for the other things discussed.