Bug 1480507 - tests: Pre-requisite setup to run geo-rep test case on regression machines.
tests: Pre-requisite setup to run geo-rep test case on regression machines.
Status: NEW
Product: GlusterFS
Classification: Community
Component: project-infrastructure (Show other bugs)
mainline
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: bugs@gluster.org
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-11 05:39 EDT by Kotresh HR
Modified: 2017-08-17 09:56 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Kotresh HR 2017-08-11 05:39:55 EDT
Description of problem:

Geo-replication test cases were disabled in master. I have sent a patch [1]
to re-enable geo-replication test cases. But it will fail as few pre-requisite steps are required.

Pre-requisite:
1. Setup passwordless SSH in all regression test machines for root. Please add it to the script which will spawn new regression machines as it will avoid doing it again on witnessing failures.

[1]: https://review.gluster.org/#/c/18024/1

I remember we did do few other changes as the path to install gluster binaries is different in regression machines. But Let's give it a run after having password less SSH. We can get through this step by step. 

Let me know if any doubts.

Version-Release number of selected component (if applicable):
mainline

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 M. Scherer 2017-08-11 08:00:20 EDT
We already do have authentication setup:

https://github.com/gluster/gluster.org_ansible_configuration/blob/master/roles/jenkins_builder/tasks/authroot_georep.yml

Can you detail what kind of failure you are seeing now ?
Comment 2 Kotresh HR 2017-08-11 08:43:26 EDT
(In reply to M. Scherer from comment #1)
> We already do have authentication setup:
> 
> https://github.com/gluster/gluster.org_ansible_configuration/blob/master/
> roles/jenkins_builder/tasks/authroot_georep.yml
> 
> Can you detail what kind of failure you are seeing now ?

Cool, I will trigger run and try it out. I thought, with new regression machines, it is no longer there.

Thanks,
Kotresh HR
Comment 3 Kotresh HR 2017-08-12 04:30:16 EDT
(In reply to Kotresh HR from comment #2)
> (In reply to M. Scherer from comment #1)
> > We already do have authentication setup:
> > 
> > https://github.com/gluster/gluster.org_ansible_configuration/blob/master/
> > roles/jenkins_builder/tasks/authroot_georep.yml
> > 
> > Can you detail what kind of failure you are seeing now ?
> 
> Cool, I will trigger run and try it out. I thought, with new regression
> machines, it is no longer there.
> 
> Thanks,
> Kotresh HR

But the regression [1] is still failing with no password less SSH for root as below.

13:10:11 [13:10:11] Running tests in file ./tests/geo-rep/georep-basic-dr-rsync.t
13:10:24 Passwordless ssh login has not been setup with slave32.cloud.gluster.org for user root.
13:11:25 Geo-replication session between master and slave32.cloud.gluster.org::slave does not exist.
13:11:26 Geo-replication session between master and slave32.cloud.gluster.org::slave does not exist.

[1] https://build.gluster.org/job/centos6-regression/5975/consoleFull

May be it's not copied to authorized_keys properly ?
Comment 4 Aravinda VK 2017-08-16 02:18:16 EDT
Kotresh,

I suggest skip gsec_create and push-pem steps from the geo-rep tests.

- Run gsec_create command as part of setup and store all the files outside the build setup(May be in /root/data/georep_keys/)
- Add the common_secret.pem.pub content to same node authorized_keys file.

Every Geo-rep test will copy secret.*.pem files to $BUILD/var/lib/glusterd/geo-replication/ and create session without `push-pem`.
Comment 5 Kotresh HR 2017-08-16 05:42:27 EDT
(In reply to Aravinda VK from comment #4)
> Kotresh,
> 
> I suggest skip gsec_create and push-pem steps from the geo-rep tests.
> 
> - Run gsec_create command as part of setup and store all the files outside
> the build setup(May be in /root/data/georep_keys/)
> - Add the common_secret.pem.pub content to same node authorized_keys file.
> 
> Every Geo-rep test will copy secret.*.pem files to
> $BUILD/var/lib/glusterd/geo-replication/ and create session without
> `push-pem`.

Yeah, seems to be better approach. 

@mscherer, can we get that done ?
Comment 6 M. Scherer 2017-08-16 07:59:32 EDT
No, I rather try to figure what was wrong, and why it was working before.
And if that was never working, why it wasn't detected sooner.

So far, I did found some quoting issue (and pushed a fix), but this was likely present since more than 1 year. 

Were the test disabled since that time ?

Also, the key to connect is /root/.ssh/id_georep , you can use it for the test. I did make sure this is properly limited to localhost, to avoid various security issue. 

And the reason this is done like this is because the tests were not cleaning up  the old root key when they broke (or when it did block the system), which in turn did cause trouble to connect as root after a while, IIRC (cause there is a limit, if only computational to the number of key you can place in authorized_keys).

We did manage last time to avoid breakage since we were using salt, but now we use ansible, any sshd breakage would be much more annoying to fix.
Comment 7 Nigel Babu 2017-08-16 23:58:02 EDT
(In reply to M. Scherer from comment #6)
> No, I rather try to figure what was wrong, and why it was working before.
> And if that was never working, why it wasn't detected sooner.

Agreed. Kotresh and Aravinda, please do not fix this from the tests, but let us fix this from the infra end.
Comment 8 Kotresh HR 2017-08-17 09:56:00 EDT
Hi M. Scherer/Nigel,

I see that, it is creating separate ssh key pair for geo-rep( /root/.ssh/id_georep). This will not work as geo-rep won't use ssh with -i option.
It requires passwordless for root with default ssh key pair (/root/.ssh/id_rsa)

Testcase:
ssh root@<local-hostname>

(with out -i flag, should login with out password on all regression machines)

I think previously, they had setup this and it was working on few and was failing on which it was not setup.

Thanks,
Kotresh HR

Note You need to log in before you can comment on or make changes to this bug.