+++ This bug was initially created as a clone of Bug #1372235 +++ 1. Bug Overview: a) Description of bug report: [RHOSP 9.0 Bug]: cinder-backup process leaks file descriptors if it back up ceph volume b) Bug Description: Everytime 'cinder backup-create <ceph volume ID>' command is executed, cinder-backup process leaks file descriptor. As a result, cinder-backup process has reached the maximum number of open files and it cannot open new socket file. Therefore cinder-backup service can't create backup volumes anymore. Version-Release number of selected component: The problem occurs in RHEL-OSP9 openstack-cinder-8.1.0-1.el7ost.noarch.rpm python-os-brick-1.1.0-1.el7ost.noarch.rpm 2. Bug Details: h) Steps to Reproduce: 1. Set up ceph as backend for cinder-volume service. 2. Set up cinder-backup service. 2. Create ceph volume. 3. Execute 'cinder backup-create <ceph volume ID>' multiple times. i) Actual results: File descriptors remain open and the number of them grow each time the command is executed. # cinder list +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ | f416e991-e8e5-4a60-8043-38b7e97ed902 | available | ceph volume | 1 | ceph | false | | +--------------------------------------+-----------+-------------+------+-------------+----------+-------------+ Create backup volume. # cinder backup-create f416e991-e8e5-4a60-8043-38b7e97ed902 +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 927d11da-3f7a-4e23-a705-840b3ca31345 | | name | None | | volume_id | f416e991-e8e5-4a60-8043-38b7e97ed902 | +-----------+--------------------------------------+ # cinder backup-list +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ | 927d11da-3f7a-4e23-a705-840b3ca31345 | f416e991-e8e5-4a60-8043-38b7e97ed902 | available | - | 1 | 22 | volumebackups | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ At the end of successful backups, cinder-backup still has be connecting to ceph-mon.(Port 6789) cinder-backup host# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" cinder-backup 7935 cinder 8u IPv4 5149737 0t0 TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <---- Furthermore when backup volumes is created multiple times, it increase in number. # cinder backup-create f416e991-e8e5-4a60-8043-38b7e97ed902 +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 71536727-cf6c-4fd9-9acf-dcbe3b56eac2 | | name | None | | volume_id | f416e991-e8e5-4a60-8043-38b7e97ed902 | +-----------+--------------------------------------+ # cinder backup-list +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ | 71536727-cf6c-4fd9-9acf-dcbe3b56eac2 | f416e991-e8e5-4a60-8043-38b7e97ed902 | available | - | 1 | 22 | volumebackups | | 927d11da-3f7a-4e23-a705-840b3ca31345 | f416e991-e8e5-4a60-8043-38b7e97ed902 | available | - | 1 | 22 | volumebackups | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+---------------+ # lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" cinder-backup 7935 cinder 8u IPv4 5149737 0t0 TCP 172.31.132.50:35882->172.31.132.50:6789 (ESTABLISHED) <--- cinder-backup 7935 cinder 11u IPv4 5159395 0t0 TCP 172.31.132.50:35908->172.31.132.50:6789 (ESTABLISHED) <--- It seems that cinder-backup leaks socket file descriptors. j) Expected results: Socket file descriptors should be closed.
Tested using: python-os-brick-1.6.1-2.el7ost.noarch Verification flow: [stack@instack ~]$ cinder backup-create 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 +-----------+--------------------------------------+ | Property | Value | +-----------+--------------------------------------+ | id | 299b0430-9901-4f3d-9f8f-0beac6c6e0d9 | | name | None | | volume_id | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 | +-----------+--------------------------------------+ [root@overcloud-controller-1 ~]# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789" cinder-backup 11548 cinder 4u IPv4 1689298 0t0 TCP 10.35.140.45:54030->10.35.140.45:6789 (ESTABLISHED) cinder-backup 11548 cinder 15u IPv4 1690501 0t0 TCP 10.35.140.45:54168->10.35.140.45:6789 (ESTABLISHED) [stack@instack ~]$ cinder backup-list +--------------------------------------+--------------------------------------+-----------+------+------+--------------+-----------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+-----------+ | 299b0430-9901-4f3d-9f8f-0beac6c6e0d9 | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 | available | - | 1 | 0 | backups | | 43ad9844-ea57-4260-807f-22240f63ba45 | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | - | 1 | 0 | backups | | 524d64f6-7527-4842-8268-e8acadca4d7b | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | - | 1 | 0 | backups | | 65709f92-82a4-4651-921c-80b57ba15a71 | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 | available | - | 1 | 0 | backups | | 901b902e-f507-4612-8edd-47e4e904806f | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | - | 1 | 0 | backups | | a514745a-f233-4107-9169-5b68b6a3439d | 7d20ea3b-51c0-4962-805e-c0f26e1f71b1 | available | - | 1 | 0 | backups | | c666c16c-49b2-4762-aaa9-07de00ceeb66 | 988b7bf0-6846-499a-8731-db0e9a88d342 | available | - | 1 | 0 | backups | +--------------------------------------+--------------------------------------+-----------+------+------+--------------+-----------+ # Cinder closes the connection to ceph cluster after the backup is finished. [root@overcloud-controller-1 ~]# lsof +c 15 -i -n -P -p $(systemctl show -p MainPID openstack-cinder-backup | cut -d'=' -f2) | grep "cinder-backup.*6789"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html