Bug 1228164 - [Snapshot] Python crashes with trace back notification when shared storage is unmount from Storage Node
Summary: [Snapshot] Python crashes with trace back notification when shared storage is...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: snapshot
Version: rhgs-3.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: RHGS 3.1.0
Assignee: Avra Sengupta
QA Contact: senaik
URL:
Whiteboard: Scheduler
Depends On:
Blocks: 1202842 1223636 1228613 1230167
TreeView+ depends on / blocked
 
Reported: 2015-06-04 10:33 UTC by Anil Shah
Modified: 2016-09-17 12:56 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.7.1-2
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1228613 (view as bug list)
Environment:
Last Closed: 2015-07-29 04:56:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1495 0 normal SHIPPED_LIVE Important: Red Hat Gluster Storage 3.1 update 2015-07-29 08:26:26 UTC

Description Anil Shah 2015-06-04 10:33:55 UTC
Description of problem:

Once you unmount shared storage volume from storage nodes, link file entry in /etc/cron.d/ folder checks for  /var/run/gluster/shared_storage/snaps/glusterfs_snap_cron_tasks file and keep sending notification to user whenever any command is executed through helper script

Version-Release number of selected component (if applicable):

[root@darkknight cron.d]# rpm -qa | grep glusterfs
glusterfs-3.7.0-3.el6rhs.x86_64
glusterfs-server-3.7.0-3.el6rhs.x86_64
glusterfs-api-3.7.0-3.el6rhs.x86_64
glusterfs-geo-replication-3.7.0-3.el6rhs.x86_64
glusterfs-libs-3.7.0-3.el6rhs.x86_64
glusterfs-fuse-3.7.0-3.el6rhs.x86_64
glusterfs-cli-3.7.0-3.el6rhs.x86_64
glusterfs-client-xlators-3.7.0-3.el6rhs.x86_64


How reproducible:

100%

Steps to Reproduce:
1. Create 2*2 distributed replicate volume.

2. Create  shared storage replicate volume on storage node which is not part of volume whose snapshot is scheduled. and mount on each storage node on path /var/run/gluster/shared_storage 
3. initialize scheduler on each storage node e.g run snap_scheduler.py init  command 
4. Enable scheduler on storage nodes e.g run snap_scheduler.py enable 
5. Unmount shared storage from all storage nodes.


Actual results:

[root@darkknight ~]# snap_scheduler.py list
snap_scheduler: Failed: Shared storage is not mounted at /var/run/gluster/shared_storage
You have new mail in /var/spool/mail/root


[root@darkknight ~]# snap_scheduler.py list
snap_scheduler: Failed: Shared storage is not mounted at /var/run/gluster/shared_storage
[root@darkknight ~]# snap_scheduler.py status
snap_scheduler: Failed: Shared storage is not mounted at /var/run/gluster/shared_storage
[root@darkknight ~]# snap_scheduler.py list
snap_scheduler: Failed: Shared storage is not mounted at /var/run/gluster/shared_storage
You have new mail in /var/spool/mail/root
[root@darkknight ~]# tailf /var/spool/mail/root 
Message-Id: <20150604102802.173A93CA0050@darkknight.localdomain>
Date: Thu,  4 Jun 2015 15:58:02 +0530 (IST)

Traceback (most recent call last):
  File "/usr/sbin/gcron.py", line 151, in <module>
    main()
  File "/usr/sbin/gcron.py", line 123, in main
    if os.lstat(GCRON_TASKS).st_mtime > \
OSError: [Errno 2] No such file or directory: '/var/run/gluster/shared_storage/snaps/glusterfs_snap_cron_tasks'


From root@darkknight.localdomain  Thu Jun  4 15:59:01 2015
Return-Path: <root@darkknight.localdomain>
X-Original-To: root
Delivered-To: root@darkknight.localdomain
Received: by darkknight.localdomain (Postfix, from userid 0)
	id 430EE3CA003F; Thu,  4 Jun 2015 15:59:01 +0530 (IST)
From: root@darkknight.localdomain (Cron Daemon)
To: root@darkknight.localdomain
Subject: Cron <root@darkknight> PATH=$PATH:/usr/local/sbin:/usr/sbin gcron.py --update
Content-Type: text/plain; charset=UTF-8
Auto-Submitted: auto-generated
X-Cron-Env: <LANG=en_US.UTF-8>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>
Message-Id: <20150604102901.430EE3CA003F@darkknight.localdomain>
Date: Thu,  4 Jun 2015 15:59:01 +0530 (IST)

Traceback (most recent call last):
  File "/usr/sbin/gcron.py", line 151, in <module>
    main()
  File "/usr/sbin/gcron.py", line 123, in main
    if os.lstat(GCRON_TASKS).st_mtime > \
OSError: [Errno 2] No such file or directory: '/var/run/gluster/shared_storage/snaps/glusterfs_snap_cron_tasks'



Expected results:

Proper clean should be done once you unmount shared storage

Additional info:

[root@darkknight ~]# gluster v info  vol0
 
Volume Name: vol0
Type: Distributed-Replicate
Volume ID: 5f93429a-056a-42d4-9d52-199df0ff1187
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.33.214:/rhs/brick1/b1
Brick2: 10.70.33.219:/rhs/brick1/b2
Brick3: 10.70.33.225:/rhs/brick1/b3
Brick4: 10.70.44.13:/rhs/brick1/b4
Options Reconfigured:
features.quota-deem-statfs: on
features.barrier: disable
features.inode-quota: on
features.quota: on
features.uss: enable
performance.readdir-ahead: on
snap-activate-on-create: enable

Comment 3 Anil Shah 2015-07-02 11:59:56 UTC
[root@darkknightrises cron.d]# snap_scheduler.py list
snap_scheduler: Failed: Shared storage is not mounted at /var/run/gluster/shared_storage
[root@darkknightrises cron.d]# 


Couldn't see any python crashes.
Bug verified on build glusterfs-3.7.1-6.el6rhs

Comment 4 errata-xmlrpc 2015-07-29 04:56:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1495.html


Note You need to log in before you can comment on or make changes to this bug.