+++ This bug was initially created as a clone of Bug #1779742 +++

Description of problem:

Testing the test suite on newer Centos 8 builder, i see that the aformentioned test fail on them. Since that's freshly installed system, this step is blocking:

ok  21 [     36/    821] < 156> 'gluster-mountbroker status'

ps show :
 6781 pts/0    S+     0:00                          \_ timeout --foreground -k 5 500 prove -vmfe /bin/bash ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
 6782 pts/0    S+     0:00                              \_ /usr/bin/perl -w /bin/prove -vmfe /bin/bash ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
 6783 pts/0    S+     0:00                                  \_ /bin/bash ./tests/00-geo-rep/00-georep-verify-non-root-setup.t
 7890 ?        Ss     0:00                                      \_ /bin/sh /bin/ssh-copy-id -i /root/.ssh/id_rsa.pub nroot.0.1
 7916 ?        S      0:00                                          \_ /bin/sh /bin/ssh-copy-id -i /root/.ssh/id_rsa.pub nroot.0.1
 7917 ?        S      0:36                                              \_ ssh -v -o PreferredAuthentications=, -o ControlPath=none nroot.0.1
 7918 ?        R      0:05                                              \_ sed -ne s/.*remote software version //p


that's because the ssh key need to be accepted automatically.

--- Additional comment from sankarshan on 2019-12-04 16:09:50 UTC ---

You might want to change the severity of this RHBZ if this is blocking the CentOS 8 builder upgrade tests.

--- Additional comment from Deepshikha khandelwal on 2019-12-10 06:15:56 UTC ---

Kotresh, can you please take a look at the failing test?

--- Additional comment from Sunny Kumar on 2019-12-10 09:52:49 UTC ---

Hi Deepshikha,

Looking at it.

/sunny

--- Additional comment from Sunny Kumar on 2019-12-16 09:43:33 UTC ---

Hi,

Can you provide machine for further analysis, it will help to resolve this faster.

/sunny

--- Additional comment from M. Scherer on 2019-12-16 12:33:33 UTC ---



--- Additional comment from M. Scherer on 2019-12-16 12:36:50 UTC ---

There isn't a need for much analysis, the problem is that on a new system, the remote key need to be accepted by ssh-copy-id, that's why it is blocked on the command: "ssh -v -o PreferredAuthentications=, -o ControlPath=none nroot.0.1".

I do not know how that was supposed to work in the first place, but I suspect this did worked by accident, because we do not reinstall the builder on each run. The attached patch should work, and to test it, no need for anything special, just use a fresh VM (either vagrant, virt-builder, anything) and test. just make sure the know_hosts file of root is empty, since that's the state of a freshly install VM.

--- Additional comment from Sunny Kumar on 2019-12-16 12:58:47 UTC ---

(In reply to M. Scherer from comment #6)
> There isn't a need for much analysis, the problem is that on a new system,
> the remote key need to be accepted by ssh-copy-id, that's why it is blocked
> on the command: "ssh -v -o PreferredAuthentications=, -o ControlPath=none
> nroot.0.1".
> 
Thanks for analysis and patch.

> I do not know how that was supposed to work in the first place, but I
> suspect this did worked by accident, because we do not reinstall the builder
> on each run.
Yes, I suppose it worked because builders were not freshly installed.

>The attached patch should work, and to test it, no need for
> anything special, just use a fresh VM (either vagrant, virt-builder,
> anything) and test. just make sure the know_hosts file of root is empty,
> since that's the state of a freshly install VM.

--- Additional comment from Worker Ant on 2019-12-16 13:00:44 UTC ---

REVIEW: https://review.gluster.org/23887 (test: fix non-root test case for geo-rep) posted (#1) for review on master by Sunny Kumar

--- Additional comment from Sunny Kumar on 2019-12-16 14:03:00 UTC ---

The option for maintaining order for authentication method: 

"PreferredAuthentications: publickey" -- fails on regression as well on local system.

I have changed its order to password.

--- Additional comment from Sunny Kumar on 2019-12-16 14:59:45 UTC ---

Update:

Can be reproduced in system by removing/making_empty "~/.ssh/known_hosts" file for root.

Works with patch:
https://review.gluster.org/#/c/glusterfs/+/23887/2/


/sunny

--- Additional comment from Worker Ant on 2019-12-18 08:11:04 UTC ---

REVIEW: https://review.gluster.org/23887 (test: fix non-root test case for geo-rep) merged (#3) on master by Amar Tumballi