Bug 1397167 - [RHEL-OSP 9.0 Bug] cinder-backup process leaks file descriptors if it back up ceph volume
Summary: [RHEL-OSP 9.0 Bug] cinder-backup process leaks file descriptors if it back up...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-os-brick
Version: 9.0 (Mitaka)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 9.0 (Mitaka)
Assignee: Lon Hohberger
QA Contact: lkuchlan
URL:
Whiteboard:
Depends On: 1372235
Blocks: 1384604
TreeView+ depends on / blocked
 
Reported: 2016-11-21 18:42 UTC by Lon Hohberger
Modified: 2017-02-15 13:56 UTC (History)
14 users (show)

Fixed In Version: python-os-brick-1.1.0-4.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1372235
Environment:
Last Closed: 2017-02-15 13:56:45 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1628626 None None None 2016-11-21 18:42:10 UTC

Description Lon Hohberger 2016-11-21 18:42:11 UTC
+++ This bug was initially created as a clone of Bug #1372235 +++

1. Bug Overview:
a) Description of bug report: 
   cinder-backup process leaks file descriptors if it back up ceph volume

b) Bug Description: 

  Everytime 'cinder backup-create <ceph volume ID>' command is executed, cinder-backup process leaks file descriptor.
  As a result, cinder-backup process has reached the maximum number of open files and it cannot open new socket file.
  Therefore cinder-backup service can't create backup volumes anymore. 

  Version-Release number of selected component:
    The problem occurs in RHEL-OSP9
    openstack-cinder-8.1.0-1.el7ost.noarch.rpm
    python-os-brick-1.1.0-1.el7ost.noarch.rpm

c) How reproducible: always

d) Steps to Reproduce: 

  1. Set up ceph as backend for cinder-volume service.
  2. Set up cinder-backup service.
  2. Create ceph volume.
  3. Execute 'cinder backup-create <ceph volume ID>' multiple times. 

e) Actual results: 

File descriptors remain open and the number of them grow each time the command is executed. 

# cinder list
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |     Name    | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available | ceph volume |  1   |     ceph    |  false   |             |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+

Create backup volume.

# cinder backup-create 9a1dd000-d345-431c-a55d-68cf1cc1d4e0
+-----------+--------------------------------------+
|  Property |                Value                 |
+-----------+--------------------------------------+
|     id    | 927d11da-3f7a-4e23-a705-840b3ca31345 |
|    name   |                 None                 |
| volume_id | dced20de-03cc-4737-9a80-4d47d2a0c203 |
+-----------+--------------------------------------+

# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
|                  ID                  |              Volume ID               |   Status  | Name | Size | Object Count |   Container   |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| dced20de-03cc-4737-9a80-4d47d2a0c203 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available |  -   |  1   |      22      | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+

At the end of successful backups, cinder-backup still has be connecting to ceph-mon.(Port 6789)

cinder-backup host# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" 
cinder-backup  7935     cinder    8u     IPv4            5149737       0t0     TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <----

Furthermore when backup volumes is created multiple times, it increase in number.

# cinder backup-create 9a1dd000-d345-431c-a55d-68cf1cc1d4e0
+-----------+--------------------------------------+
|  Property |                Value                 |
+-----------+--------------------------------------+
|     id    | 220ce436-28b3-4420-9003-44fb4bb96dc4 |
|    name   |                 None                 |
| volume_id | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 |
+-----------+--------------------------------------+

# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
|                  ID                  |              Volume ID               |   Status  | Name | Size | Object Count |   Container   |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| 220ce436-28b3-4420-9003-44fb4bb96dc4 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available |  -   |  1   |      22      | volumebackups |
| dced20de-03cc-4737-9a80-4d47d2a0c203 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available |  -   |  1   |      22      | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+

# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" 
cinder-backup  7935     cinder    8u     IPv4            5149737       0t0     TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <---
cinder-backup  7935     cinder   11u     IPv4            5159395       0t0     TCP 172.31.132.50:35908->172.31.132.50:6789 (ESTABLISHED) <---

It seems that cinder-backup leaks socket file descriptors.

f) Expected results: 

Socket file descriptors should be closed.

Comment 3 Jon Schlueter 2017-01-03 13:16:19 UTC
According to our records, this should be resolved by python-os-brick-1.1.0-4.el7ost.  This build is available now.

Comment 4 nlevinki 2017-02-06 12:59:13 UTC
This bug was  created as a clone of Bug #1372235 and it was verified.
The code is the same and everything else


Note You need to log in before you can comment on or make changes to this bug.