Description of problem: See Resource warning on various openstack commands: A couple seen on: - openstack overcloud node introspect --all-manageable --provide - openstack overcloud profiles list sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 57944)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 38530), raddr=('192.168.24.2', 13989)> Environment: python3-tripleoclient-11.5.1-0.20190719020420.bffda01.el8ost.noarch Steps to Reproduce: Error are being seeing on various openstack <cmds> Actual results: (undercloud) [stack@undercloud-0 ~]$ openstack overcloud node introspect --all-manageable --provide Waiting for introspection to finish... Waiting for messages on queue 'tripleo' with no timeout. Introspection of node e7474b96-eb17-4646-92aa-ff48f4cf992e completed. Status:SUCCESS. Errors:None Introspection of node 7ee2654f-ca4d-4443-9fe2-e6cec0a0c673 completed. Status:SUCCESS. Errors:None Introspection of node 234b2785-2b68-4063-9613-417c1eea65ba completed. Status:SUCCESS. Errors:None Introspection of node 2a582315-ef78-450b-94db-9be4ae46e8c7 completed. Status:SUCCESS. Errors:None Introspection of node 4519f131-98d3-44e2-9320-3f2b26f08dad completed. Status:SUCCESS. Errors:None Introspection of node 22fb2d83-33e0-4eeb-8ac3-4e9616e852a0 completed. Status:SUCCESS. Errors:None Introspection of node ff5feead-f1e8-4819-831b-850aeb064d46 completed. Status:SUCCESS. Errors:None Introspection of node 73e66ad8-80cb-474a-a3ba-ff1311cb5164 completed. Status:SUCCESS. Errors:None Introspection of node 54611f30-038c-4ec0-976a-47cf3b3406d2 completed. Status:SUCCESS. Errors:None Introspection of node 7ff47282-3f08-4cc5-93d3-14f6bce5aad6 completed. Status:SUCCESS. Errors:None Successfully introspected 10 node(s). Introspection completed. Waiting for messages on queue 'tripleo' with no timeout. 10 node(s) successfully moved to the "available" state. sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 57944)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 38530), raddr=('192.168.24.2', 13989)> Also see when running (undercloud) [stack@undercloud-0 ~]$ openstack overcloud profiles list +--------------------------------------+--------------+-----------------+-----------------+-------------------+ | Node UUID | Node Name | Provision State | Current Profile | Possible Profiles | +--------------------------------------+--------------+-----------------+-----------------+-------------------+ | 7ff47282-3f08-4cc5-93d3-14f6bce5aad6 | ceph-0 | available | None | | | 73e66ad8-80cb-474a-a3ba-ff1311cb5164 | ceph-1 | available | None | | | e7474b96-eb17-4646-92aa-ff48f4cf992e | ceph-2 | available | None | | | ff5feead-f1e8-4819-831b-850aeb064d46 | compute-0 | available | compute | | | 54611f30-038c-4ec0-976a-47cf3b3406d2 | compute-1 | available | compute | | | 7ee2654f-ca4d-4443-9fe2-e6cec0a0c673 | compute-2 | available | compute | | | 234b2785-2b68-4063-9613-417c1eea65ba | compute-3 | available | compute | | | 2a582315-ef78-450b-94db-9be4ae46e8c7 | controller-0 | available | control | | | 4519f131-98d3-44e2-9320-3f2b26f08dad | controller-1 | available | control | | | 22fb2d83-33e0-4eeb-8ac3-4e9616e852a0 | controller-2 | available | control | | +--------------------------------------+--------------+-----------------+-----------------+-------------------+ sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 57984), raddr=('192.168.24.2', 13000)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 60532), raddr=('192.168.24.2', 13774)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 36908), raddr=('192.168.24.2', 13385)> Expected results: No resource warning errors Additional info:
I also just saw it on this command in another environment same compost (undercloud) [stack@undercloud-0 images]$ openstack overcloud image upload --image-path /home/stack/images/ /usr/lib/python3.6/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) Image "overcloud-full-vmlinuz" was uploaded. +--------------------------------------+------------------------+-------------+---------+--------+ | ID | Name | Disk Format | Size | Status | +--------------------------------------+------------------------+-------------+---------+--------+ | 8f70ee92-d3fe-4d51-ba3e-d1266ab948f1 | overcloud-full-vmlinuz | aki | 7868768 | active | +--------------------------------------+------------------------+-------------+---------+--------+ /usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_image.py:384: ResourceWarning: unclosed file <_io.BufferedReader name='/home/stack/images/overcloud-full.vmlinuz'> parsed_args.image_path, oc_vmlinuz_file) Image "overcloud-full-initrd" was uploaded. +--------------------------------------+-----------------------+-------------+----------+--------+ | ID | Name | Disk Format | Size | Status | +--------------------------------------+-----------------------+-------------+----------+--------+ | 93ea66ca-0a5f-4556-a921-535f7f1295d0 | overcloud-full-initrd | ari | 63142898 | active | +--------------------------------------+-----------------------+-------------+----------+--------+ /usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_image.py:402: ResourceWarning: unclosed file <_io.BufferedReader name='/home/stack/images/overcloud-full.initrd'> parsed_args.image_path, oc_initrd_file) Image "overcloud-full" was uploaded. +--------------------------------------+----------------+-------------+-----------+--------+ | ID | Name | Disk Format | Size | Status | +--------------------------------------+----------------+-------------+-----------+--------+ | 3fa07b1d-ae05-4f10-929f-55c78f4ebe14 | overcloud-full | qcow2 | 973996032 | active | +--------------------------------------+----------------+-------------+-----------+--------+ /usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_image.py:423: ResourceWarning: unclosed file <_io.BufferedReader name='/home/stack/images/overcloud-full.qcow2'> parsed_args.image_path, oc_file) Image "bm-deploy-kernel" was uploaded. +--------------------------------------+------------------+-------------+---------+--------+ | ID | Name | Disk Format | Size | Status | +--------------------------------------+------------------+-------------+---------+--------+ | 3e8257a5-9b52-479e-af45-4da1abf20e47 | bm-deploy-kernel | aki | 7868768 | active | +--------------------------------------+------------------+-------------+---------+--------+ /usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_image.py:476: ResourceWarning: unclosed file <_io.BufferedReader name='/home/stack/images/ironic-python-agent.kernel'> deploy_kernel_file)) Image "bm-deploy-ramdisk" was uploaded. +--------------------------------------+-------------------+-------------+-----------+--------+ | ID | Name | Disk Format | Size | Status | +--------------------------------------+-------------------+-------------+-----------+--------+ | 552cf64f-bada-46da-b30c-4f53d5249039 | bm-deploy-ramdisk | ari | 473553284 | active | +--------------------------------------+-------------------+-------------+-----------+--------+ /usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_image.py:492: ResourceWarning: unclosed file <_io.BufferedReader name='/home/stack/images/ironic-python-agent.initramfs'> deploy_ramdisk_file)) Image file "/var/lib/ironic/httpboot/agent.kernel" is up-to-date, skipping. Image file "/var/lib/ironic/httpboot/agent.ramdisk" already exists and can be updated with --update-existing. sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 52056), raddr=('192.168.24.2', 13292)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 55800), raddr=('192.168.24.2', 13000)>
Does this only happen with 'overcloud' commands? The image one in comment 3 looks like https://bugs.launchpad.net/tripleo/+bug/1837393 upstream which has a backport on the way, though it looks like the same fix possibly needs to be replicated for the other commands showing the issue. Not sure if tripleo-common or python-tripleoclient is the best place, moving to python-tripleoclient for now since only 'overcloud' commands are mentioned in the description.
These are the commands I have seen the issue thus far. I will add more as I see it.
I am seeing this issue as well during an OC deployment. openstack server list = No resource warning When i issue overcloud commands i do see this error. (undercloud) [stack@core-undercloud-0 ~]$ cat core_puddle_version RHOS_TRUNK-15.0-RHEL-8-20190829.n.1 (undercloud) [stack@core-undercloud-0 ~]$ openstack overcloud status | grep overcloud | overcloud | 2019-08-30 17:51:36 | 2019-08-30 17:51:36 | DEPLOYING | sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.220.2', 58384), raddr=('192.168.220.2', 13000)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.220.2', 52938), raddr=('192.168.220.2', 13989)> (undercloud) [stack@core-undercloud-0 ~]$
I am seeing while upload overcloud images. (undercloud) [stack@rhosp-director images]$ openstack overcloud image upload --image-path /home/stack/images Image "overcloud-full-vmlinuz" is up-to-date, skipping. Image "overcloud-full-initrd" is up-to-date, skipping. Image "overcloud-full" is up-to-date, skipping. Image "bm-deploy-kernel" is up-to-date, skipping. Image "bm-deploy-ramdisk" is up-to-date, skipping. Image file "/var/lib/ironic/httpboot/agent.kernel" is up-to-date, skipping. Image file "/var/lib/ironic/httpboot/agent.ramdisk" is up-to-date, skipping. sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 43928), raddr=('192.168.24.2', 13292)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 34432), raddr=('192.168.24.2', 13000)>
This is a bug in the python3 version of the various clients. We have proposed patches upstream but they aren't currently making progress. We managed to land the keystoneauth fix but glanceclient and swiftclient still have outstanding patches. keystoneauth: https://review.opendev.org/#/c/674139/ glance: https://review.opendev.org/#/c/674133/ swift: https://review.opendev.org/#/c/674320/
The Glance patch has been merged upstream, indeed. Is this serious enough to justify a backport to OSP15?
It's pretty ugly UX but not sure OSP15 would be worth it. IMHO definitely OSP16
I agree with Alex that it is "pretty ugly UX", but I don't think it should be a blocker.
I agree with Cyril. This code was with us for years and nobody complained until automated tests found it. That tells me that if anyone runs out of descriptors, it's not a showstopper for them. Maybe the clients do not live long enough, or maybe they use the same auth token for many sessions anyway (in Swift it saves a lot of wall time). So it cannot be that big a deal.
Seeing the same errors on multiple 'openstack' commands on OSP15z1: (undercloud) [stack@osp15d ~]$ cat /etc/rhosp-release Red Hat OpenStack Platform release 15.0.1 (Stein) (undercloud) [stack@osp15d ~]$ openstack overcloud profiles list +--------------------------------------+-----------+-----------------+-----------------+-------------------+ | Node UUID | Node Name | Provision State | Current Profile | Possible Profiles | +--------------------------------------+-----------+-----------------+-----------------+-------------------+ | a6f61bb5-3e4d-4ba7-b395-323241787191 | bm-0 | available | ceph-storage | | | 1f547eb5-e1ac-45d4-8118-ade7402e6ac5 | bm-1 | available | ceph-storage | | | ad1cc5a9-3384-4779-8fa8-e888934dd5af | bm-2 | available | ceph-storage | | | 2f563b56-ddd5-4263-b0dc-17512b70758b | bm-3 | available | ceph-storage | | | 4640f772-17ce-4326-b388-f9fa941d2c3a | bm-4 | available | swift-storage | | | c4d22d66-7bbe-47c1-8308-7f0afa0e9a1b | bm-5 | available | swift-storage | | | 0016d4c1-fe91-494b-b3e2-1e4f639160db | bm-6 | available | swift-storage | | | 80b2c62d-308c-4dec-92b5-a87e91dfc78d | bm-7 | available | control | | | fdb89ff3-f817-47d1-acca-175395d0cbd1 | bm-8 | available | control | | | 37d78226-6c90-4f5e-b2ca-3e73452c5a04 | bm-9 | available | control | | | d9b1a52e-2c95-4e5f-ac87-1f8b02db9632 | bm-10 | available | compute | | | 7014d8f9-8f1f-44c8-b92c-79cc41818405 | bm-11 | available | compute | | | ac91c665-a324-4e70-aa22-b5677ff74ab9 | bm-12 | available | compute | | | fc83944b-2bb2-4369-bb28-0c6991109e3b | bm-13 | available | compute | | | aa209f12-404f-4c7a-89fb-70b54a6bf94f | bm-14 | available | compute | | | 8cb2e023-00df-493e-af58-90db4339e7f6 | bm-15 | available | compute | | +--------------------------------------+-----------+-----------------+-----------------+-------------------+ sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.20.0.3', 50600), raddr=('10.20.0.3', 13000)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.20.0.3', 53304), raddr=('10.20.0.3', 13774)> sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.20.0.3', 49760), raddr=('10.20.0.3', 13385)> This does not look benign to end-users. Is there a fix coming on OSP15? Thank you, Vincent
@Vincent: do you have actual users complaining about this? We would like to avoid an OSP15 backport, as it would imply backporting patches to a few different projects.
@Cyril: Nope, no actual customers complaining so far.
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0658