Bug 1397167

+++ This bug was initially created as a clone of Bug #1372235 +++

1. Bug Overview:
a) Description of bug report: 
   cinder-backup process leaks file descriptors if it back up ceph volume

b) Bug Description: 

  Everytime 'cinder backup-create <ceph volume ID>' command is executed, cinder-backup process leaks file descriptor.
  As a result, cinder-backup process has reached the maximum number of open files and it cannot open new socket file.
  Therefore cinder-backup service can't create backup volumes anymore. 

  Version-Release number of selected component:
    The problem occurs in RHEL-OSP9
    openstack-cinder-8.1.0-1.el7ost.noarch.rpm
    python-os-brick-1.1.0-1.el7ost.noarch.rpm

c) How reproducible: always

d) Steps to Reproduce: 

  1. Set up ceph as backend for cinder-volume service.
  2. Set up cinder-backup service.
  2. Create ceph volume.
  3. Execute 'cinder backup-create <ceph volume ID>' multiple times. 

e) Actual results: 

File descriptors remain open and the number of them grow each time the command is executed. 

# cinder list
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
|                  ID                  |   Status  |     Name    | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available | ceph volume |  1   |     ceph    |  false   |             |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+

Create backup volume.

# cinder backup-create 9a1dd000-d345-431c-a55d-68cf1cc1d4e0
+-----------+--------------------------------------+
|  Property |                Value                 |
+-----------+--------------------------------------+
|     id    | 927d11da-3f7a-4e23-a705-840b3ca31345 |
|    name   |                 None                 |
| volume_id | dced20de-03cc-4737-9a80-4d47d2a0c203 |
+-----------+--------------------------------------+

# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
|                  ID                  |              Volume ID               |   Status  | Name | Size | Object Count |   Container   |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| dced20de-03cc-4737-9a80-4d47d2a0c203 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available |  -   |  1   |      22      | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+

At the end of successful backups, cinder-backup still has be connecting to ceph-mon.(Port 6789)

cinder-backup host# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" 
cinder-backup  7935     cinder    8u     IPv4            5149737       0t0     TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <----

Furthermore when backup volumes is created multiple times, it increase in number.

# cinder backup-create 9a1dd000-d345-431c-a55d-68cf1cc1d4e0
+-----------+--------------------------------------+
|  Property |                Value                 |
+-----------+--------------------------------------+
|     id    | 220ce436-28b3-4420-9003-44fb4bb96dc4 |
|    name   |                 None                 |
| volume_id | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 |
+-----------+--------------------------------------+

# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
|                  ID                  |              Volume ID               |   Status  | Name | Size | Object Count |   Container   |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| 220ce436-28b3-4420-9003-44fb4bb96dc4 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available |  -   |  1   |      22      | volumebackups |
| dced20de-03cc-4737-9a80-4d47d2a0c203 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available |  -   |  1   |      22      | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+

# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" 
cinder-backup  7935     cinder    8u     IPv4            5149737       0t0     TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <---
cinder-backup  7935     cinder   11u     IPv4            5159395       0t0     TCP 172.31.132.50:35908->172.31.132.50:6789 (ESTABLISHED) <---

It seems that cinder-backup leaks socket file descriptors.

f) Expected results: 

Socket file descriptors should be closed.