+++ This bug was initially created as a clone of Bug #1372235 +++
1. Bug Overview:
a) Description of bug report:
cinder-backup process leaks file descriptors if it back up ceph volume
b) Bug Description:
Everytime 'cinder backup-create <ceph volume ID>' command is executed, cinder-backup process leaks file descriptor.
As a result, cinder-backup process has reached the maximum number of open files and it cannot open new socket file.
Therefore cinder-backup service can't create backup volumes anymore.
Version-Release number of selected component:
The problem occurs in RHEL-OSP9
openstack-cinder-8.1.0-1.el7ost.noarch.rpm
python-os-brick-1.1.0-1.el7ost.noarch.rpm
c) How reproducible: always
d) Steps to Reproduce:
1. Set up ceph as backend for cinder-volume service.
2. Set up cinder-backup service.
2. Create ceph volume.
3. Execute 'cinder backup-create <ceph volume ID>' multiple times.
e) Actual results:
File descriptors remain open and the number of them grow each time the command is executed.
# cinder list
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
| 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available | ceph volume | 1 | ceph | false | |
+--------------------------------------+-----------+-------------+------+-------------+----------+-------------+
Create backup volume.
# cinder backup-create 9a1dd000-d345-431c-a55d-68cf1cc1d4e0
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | 927d11da-3f7a-4e23-a705-840b3ca31345 |
| name | None |
| volume_id | dced20de-03cc-4737-9a80-4d47d2a0c203 |
+-----------+--------------------------------------+
# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| dced20de-03cc-4737-9a80-4d47d2a0c203 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available | - | 1 | 22 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
At the end of successful backups, cinder-backup still has be connecting to ceph-mon.(Port 6789)
cinder-backup host# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789"
cinder-backup 7935 cinder 8u IPv4 5149737 0t0 TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <----
Furthermore when backup volumes is created multiple times, it increase in number.
# cinder backup-create 9a1dd000-d345-431c-a55d-68cf1cc1d4e0
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | 220ce436-28b3-4420-9003-44fb4bb96dc4 |
| name | None |
| volume_id | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 |
+-----------+--------------------------------------+
# cinder backup-list
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
| 220ce436-28b3-4420-9003-44fb4bb96dc4 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available | - | 1 | 22 | volumebackups |
| dced20de-03cc-4737-9a80-4d47d2a0c203 | 9a1dd000-d345-431c-a55d-68cf1cc1d4e0 | available | - | 1 | 22 | volumebackups |
+--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+
# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789"
cinder-backup 7935 cinder 8u IPv4 5149737 0t0 TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <---
cinder-backup 7935 cinder 11u IPv4 5159395 0t0 TCP 172.31.132.50:35908->172.31.132.50:6789 (ESTABLISHED) <---
It seems that cinder-backup leaks socket file descriptors.
f) Expected results:
Socket file descriptors should be closed.