Bug 1384604

Summary: cinder-backup process leaks file descriptors if it back up ceph volume
Product: Red Hat OpenStack Reporter: Eric Harney <eharney>
Component: python-os-brickAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: lkuchlan <lkuchlan>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: apevec, egafford, eharney, jobernar, jschluet, knoha, lhh, nlevinki, srevivo, tbarron, tshefi, ykawada
Target Milestone: rcKeywords: Regression, Triaged
Target Release: 10.0 (Newton)Flags: tshefi: automate_bug-
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: python-os-brick-1.6.1-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1372235 Environment:
Last Closed: 2016-12-14 16:19:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1372235, 1397167    
Bug Blocks:    

Description Eric Harney 2016-10-13 16:17:18 UTC 
+++ This bug was initially created as a clone of Bug #1372235 +++

1. Bug Overview:
a) Description of bug report: 
   [RHOSP 9.0 Bug]: cinder-backup process leaks file descriptors if it back up ceph volume

b) Bug Description: 

  Everytime 'cinder backup-create <ceph volume ID>' command is executed, cinder-backup process leaks file descriptor.
  As a result, cinder-backup process has reached the maximum number of open files and it cannot open new socket file.
  Therefore cinder-backup service can't create backup volumes anymore. 

  Version-Release number of selected component:
    The problem occurs in RHEL-OSP9
    openstack-cinder-8.1.0-1.el7ost.noarch.rpm
    python-os-brick-1.1.0-1.el7ost.noarch.rpm

2. Bug Details:

h) Steps to Reproduce: 

  1. Set up ceph as backend for cinder-volume service.
  2. Set up cinder-backup service.
  2. Create ceph volume.
  3. Execute 'cinder backup-create <ceph volume ID>' multiple times. 

i) Actual results: 

File descriptors remain open and the number of them grow each time the command is executed. 

# cinder list
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |     Name    | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| f416e991-e8e5-4a60-8043-38b7e97ed902 | available | ceph volume |  1   |     ceph    |  false   |             |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+

Create backup volume.

# cinder backup-create f416e991-e8e5-4a60-8043-38b7e97ed902
+-----------+--------------------------------------+
|  Property |                Value                 |
+-----------+--------------------------------------+
|     id    | 927d11da-3f7a-4e23-a705-840b3ca31345 |
|    name   |                 None                 |
| volume_id | f416e991-e8e5-4a60-8043-38b7e97ed902 |
+-----------+--------------------------------------+

# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
|                  ID                  |              Volume ID               |   Status  | Name | Size | Object Count |   Container   |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| 927d11da-3f7a-4e23-a705-840b3ca31345 | f416e991-e8e5-4a60-8043-38b7e97ed902 | available |  -   |  1   |      22      | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+

At the end of successful backups, cinder-backup still has be connecting to ceph-mon.(Port 6789)

cinder-backup host# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" 
cinder-backup  7935     cinder    8u     IPv4            5149737       0t0     TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <----

Furthermore when backup volumes is created multiple times, it increase in number.

# cinder backup-create f416e991-e8e5-4a60-8043-38b7e97ed902
+-----------+--------------------------------------+
|  Property |                Value                 |
+-----------+--------------------------------------+
|     id    | 71536727-cf6c-4fd9-9acf-dcbe3b56eac2 |
|    name   |                 None                 |
| volume_id | f416e991-e8e5-4a60-8043-38b7e97ed902 |
+-----------+--------------------------------------+

# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
|                  ID                  |              Volume ID               |   Status  | Name | Size | Object Count |   Container   |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| 71536727-cf6c-4fd9-9acf-dcbe3b56eac2 | f416e991-e8e5-4a60-8043-38b7e97ed902 | available |  -   |  1   |      22      | volumebackups |
| 927d11da-3f7a-4e23-a705-840b3ca31345 | f416e991-e8e5-4a60-8043-38b7e97ed902 | available |  -   |  1   |      22      | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+

# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" 
cinder-backup  7935     cinder    8u     IPv4            5149737       0t0     TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <---
cinder-backup  7935     cinder   11u     IPv4            5159395       0t0     TCP 172.31.132.50:35908->172.31.132.50:6789 (ESTABLISHED) <---

It seems that cinder-backup leaks socket file descriptors.

j) Expected results: 

Socket file descriptors should be closed.
Comment 4 lkuchlan 2016-10-31 15:41:27 UTC 
Tested using:
python-os-brick-1.6.1-2.el7ost.noarch

Verification flow:

[stack@instack ~]$ cinder backup-create 7d20ea3b-51c0-4962-805e-c0f26e1f71b1
+-----------+--------------------------------------+
| Property  | Value                                |
+-----------+--------------------------------------+
| id        | 299b0430-9901-4f3d-9f8f-0beac6c6e0d9 |
| name      | None                                 |
| volume_id | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 |
+-----------+--------------------------------------+

[root@overcloud-controller-1 ~]# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789"
cinder-backup   11548     cinder    4u     IPv4            1689298       0t0      TCP 10.35.140.45:54030->10.35.140.45:6789 (ESTABLISHED)
cinder-backup   11548     cinder   15u     IPv4            1690501       0t0      TCP 10.35.140.45:54168->10.35.140.45:6789 (ESTABLISHED)

[stack@instack ~]$ cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+-----------+
| ID                                   | Volume ID                            | Status    | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+-----------+
| 299b0430-9901-4f3d-9f8f-0beac6c6e0d9 | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 | available | -    | 1    | 0            | backups   |
| 43ad9844-ea57-4260-807f-22240f63ba45 | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | -    | 1    | 0            | backups   |
| 524d64f6-7527-4842-8268-e8acadca4d7b | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | -    | 1    | 0            | backups   |
| 65709f92-82a4-4651-921c-80b57ba15a71 | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 | available | -    | 1    | 0            | backups   |
| 901b902e-f507-4612-8edd-47e4e904806f | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | -    | 1    | 0            | backups   |
| a514745a-f233-4107-9169-5b68b6a3439d | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 | available | -    | 1    | 0            | backups   |
| c666c16c-49b2-4762-aaa9-07de00ceeb66 | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | -    | 1    | 0            | backups   |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+-----------+

# Cinder closes the connection to ceph cluster after the backup is finished.
[root@overcloud-controller-1 ~]# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789"
Comment 6 errata-xmlrpc 2016-12-14 16:19:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html