Bug 1813917 - [RHEL-8.2]-Geo-rep fails to start on a default setup , where selinux is enabled
Summary: [RHEL-8.2]-Geo-rep fails to start on a default setup , where selinux is enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: selinux
Version: rhgs-3.5
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: RHGS 3.5.z Batch Update 2
Assignee: Sunny Kumar
QA Contact: Arun Kumar
URL:
Whiteboard:
Depends On: 1821759 1824842 1825177
Blocks: 1786127
TreeView+ depends on / blocked
 
Reported: 2020-03-16 13:26 UTC by Upasana
Modified: 2020-06-16 06:19 UTC (History)
14 users (show)

Fixed In Version: glusterfs-6.0-33
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1816663 (view as bug list)
Environment:
Last Closed: 2020-06-16 06:19:39 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2572 0 None None None 2020-06-16 06:19:56 UTC

Description Upasana 2020-03-16 13:26:49 UTC
Description of problem:
=======================
While creating a geo-rep session , 'gluster system:: execute gsec_create' fails to create the required permissions/files required hence geo-rep fails to start
on a SELINUX enabled setup (which is by default enabled)

Logs - 
WITH SELINUX
=============
[root@dhcp47-160 geo-replication]# gluster system:: execute gsec_create
Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub
[root@dhcp47-160 geo-replication]# ls
common_secret.pem.pub  gsyncd_template.conf

WITH SELINUX SET TO PERMISSIVE
===============================
[root@dhcp47-160 geo-replication]# gluster system:: execute gsec_create
Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub
[root@dhcp47-160 geo-replication]# ls
common_secret.pem.pub  gsyncd_template.conf


Version-Release number of selected component (if applicable):
=============================================================
glusterfs-6.0-30.el8rhgs.x86_64

How reproducible:
=================
2/2

Steps to Reproduce:
1.Create a master and a slave setup ,enable shared storage on master
2.disable performance.quick-read on SLAVE volume
3.set password less ssh from primary master node to primary slave node 
4.gluster system:: execute gsec_create

Actual results:
===============
when selinux is enabled it fails to create some files/permissions

[root@dhcp46-200 geo-replication]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31


[root@dhcp47-160 geo-replication]#  gluster system:: execute gsec_create
Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub

[root@dhcp47-160 geo-replication]# ls
common_secret.pem.pub  gsyncd_template.conf  
[root@dhcp47-160 geo-replication]# 


because of which when i start geo-rep it is going into faulty state

[root@dhcp47-160 geo-replication]# gluster volume geo-replication replica-vol 10.70.47.89::replica-vol status
 
MASTER NODE                          MASTER VOL     MASTER BRICK           SLAVE USER    SLAVE                       SLAVE NODE    STATUS     CRAWL STATUS    LAST_SYNCED          
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dhcp47-160.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v1/rep1    root          10.70.47.89::replica-vol    N/A           Faulty     N/A             N/A                  
dhcp46-200.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v2/rep2    root          10.70.47.89::replica-vol    N/A           Created    N/A             N/A                  
dhcp47-29.lab.eng.blr.redhat.com     replica-vol    /mnt/bricks/v2/rep3    root          10.70.47.89::replica-vol    N/A           Created    N/A             N/A   


Expected results:
=================
when selinux is disabled this is working fine

On Master
========
[root@dhcp47-160 geo-replication]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31

[root@dhcp47-160 geo-replication]#  gluster system:: execute gsec_create
Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub


[root@dhcp47-160 geo-replication]# ls
common_secret.pem.pub  gsyncd_template.conf  secret.pem  secret.pem.pub  tar_ssh.pem  tar_ssh.pem.pub

[root@dhcp47-160 geo-replication]# gluster volume geo-replication replica-vol 10.70.47.89::replica-vol status
 
MASTER NODE                          MASTER VOL     MASTER BRICK           SLAVE USER    SLAVE                       SLAVE NODE                           STATUS     CRAWL STATUS       LAST_SYNCED                  
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dhcp47-160.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v1/rep1    root          10.70.47.89::replica-vol    dhcp47-175.lab.eng.blr.redhat.com    Active     Changelog Crawl    2020-03-16 18:28:58          
dhcp47-29.lab.eng.blr.redhat.com     replica-vol    /mnt/bricks/v2/rep3    root          10.70.47.89::replica-vol    dhcp47-175.lab.eng.blr.redhat.com    Passive    N/A                N/A                          
dhcp46-200.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v2/rep2    root          10.70.47.89::replica-vol    dhcp47-175.lab.eng.blr.redhat.com    Passive    N/A                N/A   



Additional info:
================
[root@dhcp47-160 geo-replication]# gluster v status
Status of volume: gluster_shared_storage
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick dhcp47-29.lab.eng.blr.redhat.com:/var
/lib/glusterd/ss_brick                      49154     0          Y       29648
Brick dhcp46-200.lab.eng.blr.redhat.com:/va
r/lib/glusterd/ss_brick                     49154     0          Y       11582
Brick dhcp47-160.lab.eng.blr.redhat.com:/va
r/lib/glusterd/ss_brick                     49153     0          Y       29344
Self-heal Daemon on localhost               N/A       N/A        Y       30450
Self-heal Daemon on dhcp47-29.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       30324
Self-heal Daemon on dhcp46-200.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       12244
 
Task Status of Volume gluster_shared_storage
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: replica-vol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick dhcp47-160.lab.eng.blr.redhat.com:/mn
t/bricks/v1/rep1                            49152     0          Y       29164
Brick dhcp46-200.lab.eng.blr.redhat.com:/mn
t/bricks/v2/rep2                            49152     0          Y       11415
Brick dhcp47-29.lab.eng.blr.redhat.com:/mnt
/bricks/v2/rep3                             49152     0          Y       29481
Self-heal Daemon on localhost               N/A       N/A        Y       30450
Self-heal Daemon on dhcp46-200.lab.eng.blr.
redhat.com                                  N/A       N/A        Y       12244
Self-heal Daemon on dhcp47-29.lab.eng.blr.r
edhat.com                                   N/A       N/A        Y       30324
 
Task Status of Volume replica-vol
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp47-160 geo-replication]# gluster v info
 
Volume Name: gluster_shared_storage
Type: Replicate
Volume ID: eefc97be-793b-4fd6-9f33-6bfaa5f996a3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: dhcp47-29.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick
Brick2: dhcp46-200.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick
Brick3: dhcp47-160.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
cluster.enable-shared-storage: enable
 
Volume Name: replica-vol
Type: Replicate
Volume ID: f06998c0-30a6-4174-8fb7-d2faed1c62b0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: dhcp47-160.lab.eng.blr.redhat.com:/mnt/bricks/v1/rep1
Brick2: dhcp46-200.lab.eng.blr.redhat.com:/mnt/bricks/v2/rep2
Brick3: dhcp47-29.lab.eng.blr.redhat.com:/mnt/bricks/v2/rep3
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
cluster.enable-shared-storage: enable
[root@dhcp47-160 geo-replication]# 



Thanks a lot kotresh for debugging this.

Comment 1 Upasana 2020-03-16 13:33:36 UTC
(In reply to Upasana from comment #0)
> Description of problem:
> =======================
> While creating a geo-rep session , 'gluster system:: execute gsec_create'
> fails to create the required permissions/files required hence geo-rep fails
> to start
> on a SELINUX enabled setup (which is by default enabled)
> 
> Logs - 
> WITH SELINUX
> =============
> [root@dhcp47-160 geo-replication]# gluster system:: execute gsec_create
> Common secret pub file present at
> /var/lib/glusterd/geo-replication/common_secret.pem.pub
> [root@dhcp47-160 geo-replication]# ls
> common_secret.pem.pub  gsyncd_template.conf
> 
> WITH SELINUX SET TO PERMISSIVE
> ===============================
[root@dhcp47-160 geo-replication]#  gluster system:: execute gsec_create
Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub
[root@dhcp47-160 geo-replication]# ls
common_secret.pem.pub  gsyncd_template.conf  secret.pem  secret.pem.pub  tar_ssh.pem  tar_ssh.pem.pub

Correcting the logs here
> 
> 
> Version-Release number of selected component (if applicable):
> =============================================================
> glusterfs-6.0-30.el8rhgs.x86_64
> 
> How reproducible:
> =================
> 2/2
> 
> Steps to Reproduce:
> 1.Create a master and a slave setup ,enable shared storage on master
> 2.disable performance.quick-read on SLAVE volume
> 3.set password less ssh from primary master node to primary slave node 
> 4.gluster system:: execute gsec_create
> 
> Actual results:
> ===============
> when selinux is enabled it fails to create some files/permissions
> 
> [root@dhcp46-200 geo-replication]# sestatus 
> SELinux status:                 enabled
> SELinuxfs mount:                /sys/fs/selinux
> SELinux root directory:         /etc/selinux
> Loaded policy name:             targeted
> Current mode:                   enforcing
> Mode from config file:          enforcing
> Policy MLS status:              enabled
> Policy deny_unknown status:     allowed
> Memory protection checking:     actual (secure)
> Max kernel policy version:      31
> 
> 
> [root@dhcp47-160 geo-replication]#  gluster system:: execute gsec_create
> Common secret pub file present at
> /var/lib/glusterd/geo-replication/common_secret.pem.pub
> 
> [root@dhcp47-160 geo-replication]# ls
> common_secret.pem.pub  gsyncd_template.conf  
> [root@dhcp47-160 geo-replication]# 
> 
> 
> because of which when i start geo-rep it is going into faulty state
> 
> [root@dhcp47-160 geo-replication]# gluster volume geo-replication
> replica-vol 10.70.47.89::replica-vol status
>  
> MASTER NODE                          MASTER VOL     MASTER BRICK          
> SLAVE USER    SLAVE                       SLAVE NODE    STATUS     CRAWL
> STATUS    LAST_SYNCED          
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> --------------
> dhcp47-160.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v1/rep1   
> root          10.70.47.89::replica-vol    N/A           Faulty     N/A      
> N/A                  
> dhcp46-200.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v2/rep2   
> root          10.70.47.89::replica-vol    N/A           Created    N/A      
> N/A                  
> dhcp47-29.lab.eng.blr.redhat.com     replica-vol    /mnt/bricks/v2/rep3   
> root          10.70.47.89::replica-vol    N/A           Created    N/A      
> N/A   
> 
> 
> Expected results:
> =================
> when selinux is disabled this is working fine
> 
> On Master
> ========
> [root@dhcp47-160 geo-replication]# sestatus 
> SELinux status:                 enabled
> SELinuxfs mount:                /sys/fs/selinux
> SELinux root directory:         /etc/selinux
> Loaded policy name:             targeted
> Current mode:                   permissive
> Mode from config file:          permissive
> Policy MLS status:              enabled
> Policy deny_unknown status:     allowed
> Memory protection checking:     actual (secure)
> Max kernel policy version:      31
> 
> [root@dhcp47-160 geo-replication]#  gluster system:: execute gsec_create
> Common secret pub file present at
> /var/lib/glusterd/geo-replication/common_secret.pem.pub
> 
> 
> [root@dhcp47-160 geo-replication]# ls
> common_secret.pem.pub  gsyncd_template.conf  secret.pem  secret.pem.pub 
> tar_ssh.pem  tar_ssh.pem.pub
> 
> [root@dhcp47-160 geo-replication]# gluster volume geo-replication
> replica-vol 10.70.47.89::replica-vol status
>  
> MASTER NODE                          MASTER VOL     MASTER BRICK          
> SLAVE USER    SLAVE                       SLAVE NODE                        
> STATUS     CRAWL STATUS       LAST_SYNCED                  
> -----------------------------------------------------------------------------
> -----------------------------------------------------------------------------
> ------------------------------------------------
> dhcp47-160.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v1/rep1   
> root          10.70.47.89::replica-vol    dhcp47-175.lab.eng.blr.redhat.com 
> Active     Changelog Crawl    2020-03-16 18:28:58          
> dhcp47-29.lab.eng.blr.redhat.com     replica-vol    /mnt/bricks/v2/rep3   
> root          10.70.47.89::replica-vol    dhcp47-175.lab.eng.blr.redhat.com 
> Passive    N/A                N/A                          
> dhcp46-200.lab.eng.blr.redhat.com    replica-vol    /mnt/bricks/v2/rep2   
> root          10.70.47.89::replica-vol    dhcp47-175.lab.eng.blr.redhat.com 
> Passive    N/A                N/A   
> 
> 
> 
> Additional info:
> ================
> [root@dhcp47-160 geo-replication]# gluster v status
> Status of volume: gluster_shared_storage
> Gluster process                             TCP Port  RDMA Port  Online  Pid
> -----------------------------------------------------------------------------
> -
> Brick dhcp47-29.lab.eng.blr.redhat.com:/var
> /lib/glusterd/ss_brick                      49154     0          Y      
> 29648
> Brick dhcp46-200.lab.eng.blr.redhat.com:/va
> r/lib/glusterd/ss_brick                     49154     0          Y      
> 11582
> Brick dhcp47-160.lab.eng.blr.redhat.com:/va
> r/lib/glusterd/ss_brick                     49153     0          Y      
> 29344
> Self-heal Daemon on localhost               N/A       N/A        Y      
> 30450
> Self-heal Daemon on dhcp47-29.lab.eng.blr.r
> edhat.com                                   N/A       N/A        Y      
> 30324
> Self-heal Daemon on dhcp46-200.lab.eng.blr.
> redhat.com                                  N/A       N/A        Y      
> 12244
>  
> Task Status of Volume gluster_shared_storage
> -----------------------------------------------------------------------------
> -
> There are no active volume tasks
>  
> Status of volume: replica-vol
> Gluster process                             TCP Port  RDMA Port  Online  Pid
> -----------------------------------------------------------------------------
> -
> Brick dhcp47-160.lab.eng.blr.redhat.com:/mn
> t/bricks/v1/rep1                            49152     0          Y      
> 29164
> Brick dhcp46-200.lab.eng.blr.redhat.com:/mn
> t/bricks/v2/rep2                            49152     0          Y      
> 11415
> Brick dhcp47-29.lab.eng.blr.redhat.com:/mnt
> /bricks/v2/rep3                             49152     0          Y      
> 29481
> Self-heal Daemon on localhost               N/A       N/A        Y      
> 30450
> Self-heal Daemon on dhcp46-200.lab.eng.blr.
> redhat.com                                  N/A       N/A        Y      
> 12244
> Self-heal Daemon on dhcp47-29.lab.eng.blr.r
> edhat.com                                   N/A       N/A        Y      
> 30324
>  
> Task Status of Volume replica-vol
> -----------------------------------------------------------------------------
> -
> There are no active volume tasks
>  
> [root@dhcp47-160 geo-replication]# gluster v info
>  
> Volume Name: gluster_shared_storage
> Type: Replicate
> Volume ID: eefc97be-793b-4fd6-9f33-6bfaa5f996a3
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: dhcp47-29.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick
> Brick2: dhcp46-200.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick
> Brick3: dhcp47-160.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> cluster.enable-shared-storage: enable
>  
> Volume Name: replica-vol
> Type: Replicate
> Volume ID: f06998c0-30a6-4174-8fb7-d2faed1c62b0
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: dhcp47-160.lab.eng.blr.redhat.com:/mnt/bricks/v1/rep1
> Brick2: dhcp46-200.lab.eng.blr.redhat.com:/mnt/bricks/v2/rep2
> Brick3: dhcp47-29.lab.eng.blr.redhat.com:/mnt/bricks/v2/rep3
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> geo-replication.indexing: on
> geo-replication.ignore-pid-check: on
> changelog.changelog: on
> cluster.enable-shared-storage: enable
> [root@dhcp47-160 geo-replication]# 
> 
> 
> 
> Thanks a lot kotresh for debugging this.

Comment 40 Sunny Kumar 2020-05-11 09:19:30 UTC
Upstream Patch:

https://review.gluster.org/#/c/glusterfs/+/24433/

Comment 45 errata-xmlrpc 2020-06-16 06:19:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2572


Note You need to log in before you can comment on or make changes to this bug.