Bug 2081279

Summary: vdsm-network.service crashes during initial deployment on CentOS Stream 9 due to libnl3 segfault
Product: Red Hat Enterprise Linux 9 Reporter: Sandro Bonazzola <sbonazzo>
Component: libnl3Assignee: Thomas Haller <thaller>
Status: CLOSED ERRATA QA Contact: David Jaša <djasa>
Severity: high Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, bugs, djasa, jwboyer, thaller, vbenes
Target Milestone: rcKeywords: Regression, Reopened, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libnl3-3.6.0-2.el9 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-15 11:14:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1986335    
Attachments:
Description Flags
Backtrace none

Description Sandro Bonazzola 2022-05-03 09:24:17 UTC 
journal shows:

May 03 08:43:52 node45el9.lab systemd[1]: Starting Virtual Desktop Server Manager network restoration...
May 03 08:43:52 node45el9.lab systemd-coredump[46442]: [🡕] Process 46437 (vdsm-tool) of user 0 dumped core.
May 03 08:43:53 node45el9.lab systemd[1]: vdsm-network.service: Main process exited, code=dumped, status=11/SEGV
May 03 08:43:53 node45el9.lab systemd[1]: vdsm-network.service: Failed with result 'core-dump'.
May 03 08:43:53 node45el9.lab systemd[1]: Failed to start Virtual Desktop Server Manager network restoration.

Corresponding systemd reaction:

May 03 08:43:52 node45el9.lab kernel: vdsm-tool[46437]: segfault at 8 ip 00007fa6690b6629 sp 00007ffe1c7a10b8 error 6 in libnl-route-3.so.200.26.0[7fa66907f000+47000]
May 03 08:43:52 node45el9.lab kernel: Code: 90 5b 31 c0 5d 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa f6 47 31 80 74 16 48 8b 56 18 48 8b 46 20 83 af ac 00 00 00 01 <48> 89 42 08 48 89 10 c3 66 66 2e 0f 1f>
May 03 08:43:52 node45el9.lab systemd[1]: Created slice Slice /system/systemd-coredump.
May 03 08:43:52 node45el9.lab systemd[1]: Started Process Core Dump (PID 46441/UID 0).
May 03 08:43:52 node45el9.lab systemd-coredump[46442]: Resource limits disable core dumping for process 46437 (vdsm-tool).
May 03 08:43:52 node45el9.lab systemd-coredump[46442]: [🡕] Process 46437 (vdsm-tool) of user 0 dumped core.
May 03 08:43:52 node45el9.lab systemd[1]: systemd-coredump: Deactivated successfully.

Looking at the service, I executed manually:
/usr/bin/vdsm-tool dump-bonding-options 
and it passed.

then:
 /usr/bin/vdsm-tool restore-nets
Segmentation fault (core dumped)

Enabled coredumps following https://access.redhat.com/solutions/4896

Dump shows:

May 03 09:19:33 node45el9.lab systemd-coredump[55163]: [🡕] Process 55158 (vdsm-tool) of user 0 dumped core.
                                                       
                                                       Module linux-vdso.so.1 with build-id 9959990f6ea5b8ae0059869c33eb31619c881252
                                                       Module unicodedata.cpython-39-x86_64-linux-gnu.so with build-id 37545532244cf7a383b70a59fd37f6da6855f8a8
                                                       Module termios.cpython-39-x86_64-linux-gnu.so with build-id c77ee28cd01ed9ca0602e5a5f58e7075de524534
                                                       Module _rust.abi3.so with build-id 45dcfee7a2821f4996ee84bb593910d298db58e4
                                                       Module libnispor.so.1.2.3 with build-id da7689afe970bdfa31cd40b6a91358e6c34c5281
                                                       Module libyaml-0.so.2 with build-id b63afab9bfe8a249975629245ed1b47e4d425648
                                                       Module _yaml.cpython-39-x86_64-linux-gnu.so with build-id 014764b3eef2dbd986a00eefc1ccd33f6ee7ab4f
                                                       Module libbrotlicommon.so.1 with build-id 4484908208e1f3ccfc9b4fb0a7dccdb8ba31e461
                                                       Module libkeyutils.so.1 with build-id 56cff55f1f24fbca7c8e8d05b26e0421753720de
                                                       Module libkrb5support.so.0 with build-id 8b740a44df023287834d61d6c3a7510941fa706a
                                                       Module libbrotlidec.so.1 with build-id 753ad737d1a65860a1859053e42f74288043f9cc
                                                       Module liblber-2.4.so.2 with build-id c27ca8b37480fa0a52155d39e3a99acf403fb30a
                                                       Module libldap_r-2.4.so.2 with build-id f4803730497290d32e0900a2c36822ada44d6b5a
                                                       Module libpsl.so.5 with build-id 1f55e40a4e9c796b425010712b1ef5c517d5ea4b
                                                       Module libnghttp2.so.14 with build-id 7eadbd56a0e5bcd3d8a6b39b9bab2327e380283a
                                                       Module libresolv.so.2 with build-id 73ef9c186afbcd3dbe0ed18464ab5436a1eee902
                                                       Module libcom_err.so.2 with build-id bc21efc61df35f5394875858e5763a7bfa0c1563
                                                       Module libk5crypto.so.3 with build-id 695a3237d0d65a13db2dc8fafa6e5fb3eba4bb3c
                                                       Module libkrb5.so.3 with build-id 291bd22cd50cff11c334d702b3b49b517fa7435b
                                                       Module libcrypt.so.2 with build-id 6ce4e5eb200e61d07398af52f8bcb316cf8466e0
                                                       Module libgssapi_krb5.so.2 with build-id 3c75c2f61f08fe61a1809bb63843620febc47daa
                                                       Module libgmp.so.10 with build-id 754b87178e725c166a68353f043ba7638e0723e1
                                                       Module libhogweed.so.6 with build-id 21c32760f1f50b9d69b43e966b93fb3de8baa076
                                                       Module libnettle.so.8 with build-id 3eda844d5a5df2443a281158b0d7ca6e81483b59
                                                       Module libtasn1.so.6 with build-id 3d3a2f6f0d4a70919496afe25e329abd189b7882
                                                       Module libunistring.so.2 with build-id 15e34cdfafa3547f9c700489b842ceb86f6fb73e
                                                       Module libidn2.so.0 with build-id 958c50fc94ecb196b24f3619762e7ec3f28a5b40
                                                       Module libp11-kit.so.0 with build-id 5e20d86b92c9f913571338c18cb70f74da7d3c0e
                                                       Module libattr.so.1 with build-id e67f1fc89e8ac6a35f6fda914bcf6144b9ccb99c
                                                       Module libmount.so.1 with build-id 1077c04e3ad4a2c41847935efac0dedefe8dab78
                                                       Module libgmodule-2.0.so.0 with build-id b682793fe3c9b4927bc12d9908d749fe798a9c7b
                                                       Module libpcre.so.1 with build-id cffb947bcc416dca3cd249cdb0a1c6f614549c30
                                                       Module libcurl.so.4 with build-id 35e4925a0ee3773ff6ba41d190bed7de554dd3c9
                                                       Module libtirpc.so.3 with build-id fede4621a018d300b74d52cf18fd314727b5b12c
                                                       Module libsasl2.so.3 with build-id 78a77f792072693b0a303f6d8924eeee033fb498
                                                       Module libssh.so.4 with build-id b379b62a208293a838db07054df7f0bec4ab4de7
                                                       Module libyajl.so.2 with build-id 7566fc033870a3570562a94020172bd833e2c351
                                                       Module libnuma.so.1 with build-id a0c23354fdb39f51ec9e31aeae27a9f8ee034500
                                                       Module libgnutls.so.30 with build-id dc5d93a8db53b2246dd1b5b24bc56956f4c135b2
                                                       Module libcap-ng.so.0 with build-id fdca0a301667e15db99d726152b57feeb35e4dbe
                                                       Module libaudit.so.1 with build-id 9bbee8705c4e1f2855273b732bc6497fdebac344
                                                       Module libacl.so.1 with build-id afd5611fa888ba30a62a2109d13bd3903d84c61c
                                                       Module libgio-2.0.so.0 with build-id b9f2c135589b3605526b93c8e94780016fe5bbd1
                                                       Module libgobject-2.0.so.0 with build-id fcfcfa460c0d0ebc14b2806e4d5900a8dc8df25e
                                                       Module libglib-2.0.so.0 with build-id f556acca796d08b0639ca153c54713cfba852bf6
                                                       Module libvirt.so.0 with build-id 8816550c7b1d1c207fc397d8c01bfa97e9ec722d
                                                       Module libvirtmod.cpython-39-x86_64-linux-gnu.so with build-id 2a044a2b5a36f29c6b0f2557320b46c37c56a5d2
                                                       Module libsqlite3.so.0 with build-id 3693dc299e6d46d1879d24089bd9359a1d9bb62a
                                                       Module _sqlite3.cpython-39-x86_64-linux-gnu.so with build-id b42d0bf1ecb9f5f70ad7ecf5eca8c88783d10194
                                                       Module _posixshmem.cpython-39-x86_64-linux-gnu.so with build-id bd54790d3fac826513db6710d2bc7e2522fdfa55
                                                       Module mmap.cpython-39-x86_64-linux-gnu.so with build-id fbf62b8a4742465578b287c9e5a3b81af35610ca
                                                       Module _multiprocessing.cpython-39-x86_64-linux-gnu.so with build-id b2e64c681890755d582280ed83d4a0dec8834204
                                                       Module libgcc_s.so.1 with build-id 8d8738d89978e21955b975d68b9d03e32fd25528
                                                       Module libnl-route-3.so.200 with build-id f8480d89efc1c4fea4ad2346a6295609cf51a18f
                                                       Module libnl-3.so.200 with build-id e24955deb76e3c52bf70100e6aecf1b6c98205cb
                                                       Module _selinux.cpython-39-x86_64-linux-gnu.so with build-id 117744436008d14c6fa96fc28afb1c4e92e07655
                                                       Module libpcre2-8.so.0 with build-id a5dafbac0243bdc2e3b5ab9a07d61d4b32f86927
                                                       Module libselinux.so.1 with build-id c1d23d40bcf1fae271256fcbe90a46ff6dbcce20
                                                       Module libxml2.so.2 with build-id 93d1e0ee905e2be426312585ff9b3716e993aff9
                                                       Module libfa.so.1 with build-id 035505c5a149146f43654fa0a7c6081083244c64
                                                       Module libaugeas.so.0 with build-id e532419e74d3eda2436500cc7fbafd6316d711e4
                                                       Module _blake2.cpython-39-x86_64-linux-gnu.so with build-id 53f8d2cec92b67bee189600066c54210c46dcb88
                                                       Module _hashlib.cpython-39-x86_64-linux-gnu.so with build-id 44ffb14fc0434f17f7fb171aead1cccf09bc89aa
                                                       Module _random.cpython-39-x86_64-linux-gnu.so with build-id 563dacff0c963f508efec2f96820c2b7c6c826a3
                                                       Module liblzma.so.5 with build-id 66b7fca8b63a61d19c5ded440ee6a8bf05dcafa5
                                                       Module _lzma.cpython-39-x86_64-linux-gnu.so with build-id 4a2886b4d94659d67cecd843e45550e65b887604
                                                       Module libbz2.so.1 with build-id dc9cd83a2cf3038bd3f04cc111c32e2c5698b5d3
                                                       Module _bz2.cpython-39-x86_64-linux-gnu.so with build-id 87a9d22545a53c8ec3db89e319061ab731cde3e4
                                                       Module zlib.cpython-39-x86_64-linux-gnu.so with build-id 66c0919ec1a077716a7f01f8734febc3e4c10eac
                                                       Module fcntl.cpython-39-x86_64-linux-gnu.so with build-id 5d5ecd4d03829fc86914a5b791189c2830b5a263
                                                       Module libblkid.so.1 with build-id 25ffb35e40228ef3df3249c7f1d8cd6135cbcf08
                                                       Module libaio.so.1 with build-id 9b75d21282f8e17ddfa06aff78dae4f8dcce4106
                                                       Module libsanlock.so.1 with build-id 2beb8f3e4bc6746ef66d540bbb0eb94b9439f63e
                                                       Module sanlock.cpython-39-x86_64-linux-gnu.so with build-id 0efe6b72269e99f5d4b262e551ab160ef3cbe9db
                                                       Module binascii.cpython-39-x86_64-linux-gnu.so with build-id 484d27a87b85f5a9a2e7195a64b56454f4a71034
                                                       Module libz.so.1 with build-id 4804f60abf6d78303295fcb9474bed2941e2e7b1
                                                       Module libcrypto.so.3 with build-id 224fbc9ed8ab697662a9b34e608fdee959ab8198
                                                       Module libssl.so.3 with build-id 75928863711cdfc51360837cde7a12326fbbf2f8
                                                       Module _ssl.cpython-39-x86_64-linux-gnu.so with build-id 892e89330d3a4ee22796c97c8657581c0c485451
                                                       Module libpthread.so.0 with build-id ce0059677116ec09edb8daffac4344721e1a23dc
                                                       Module libffi.so.8 with build-id 48e3675db4765a2e42729140922e11a10016f7ab
                                                       Module _ctypes.cpython-39-x86_64-linux-gnu.so with build-id 3b9aa86605b1fa3f794cbe2fe2c4d591218ccde0
                                                       Module _opcode.cpython-39-x86_64-linux-gnu.so with build-id 057f49ddcd872f0909ca02e79d9aa7abf5ec2276
                                                       Module _bisect.cpython-39-x86_64-linux-gnu.so with build-id f802378779f7d27bc1e175d8707c591c508ca225
                                                       Module _queue.cpython-39-x86_64-linux-gnu.so with build-id 58f5fe7973a70238bc4cdbff4438ad23f024dbf8
                                                       Module _datetime.cpython-39-x86_64-linux-gnu.so with build-id 5fb3bb2e3397950bbe513ca14bc065c572baa0be
                                                       Module array.cpython-39-x86_64-linux-gnu.so with build-id 543bfa1a817657801f982b381e89372669c06055
                                                       Module _socket.cpython-39-x86_64-linux-gnu.so with build-id 2b94f1da7f18bdaa0b534153d57ef051efffd184
                                                       Module _pickle.cpython-39-x86_64-linux-gnu.so with build-id b36ef29100bb359b370031621f7fdd7851231c44
                                                       Module _struct.cpython-39-x86_64-linux-gnu.so with build-id c7dc4c8e8b880fcd769205ffb3f8c2184962ad91
                                                       Module _json.cpython-39-x86_64-linux-gnu.so with build-id 4e6ce531b346b35e339a610e3c3f33f6bd3f6547
                                                       Module libuuid.so.1 with build-id da656ae81596c151559892375d730bd35f76048c
                                                       Module libgmp.so.10 with build-id 754b87178e725c166a68353f043ba7638e0723e1
                                                       Module libhogweed.so.6 with build-id 21c32760f1f50b9d69b43e966b93fb3de8baa076
                                                       Module libnettle.so.8 with build-id 3eda844d5a5df2443a281158b0d7ca6e81483b59
                                                       Module libtasn1.so.6 with build-id 3d3a2f6f0d4a70919496afe25e329abd189b7882
                                                       Module libunistring.so.2 with build-id 15e34cdfafa3547f9c700489b842ceb86f6fb73e
                                                       Module libidn2.so.0 with build-id 958c50fc94ecb196b24f3619762e7ec3f28a5b40
                                                       Module libp11-kit.so.0 with build-id 5e20d86b92c9f913571338c18cb70f74da7d3c0e
                                                       Module libattr.so.1 with build-id e67f1fc89e8ac6a35f6fda914bcf6144b9ccb99c
                                                       Module libmount.so.1 with build-id 1077c04e3ad4a2c41847935efac0dedefe8dab78
                                                       Module libgmodule-2.0.so.0 with build-id b682793fe3c9b4927bc12d9908d749fe798a9c7b
                                                       Module libpcre.so.1 with build-id cffb947bcc416dca3cd249cdb0a1c6f614549c30
                                                       Module libcurl.so.4 with build-id 35e4925a0ee3773ff6ba41d190bed7de554dd3c9
                                                       Module libtirpc.so.3 with build-id fede4621a018d300b74d52cf18fd314727b5b12c
                                                       Module libsasl2.so.3 with build-id 78a77f792072693b0a303f6d8924eeee033fb498
                                                       Module libssh.so.4 with build-id b379b62a208293a838db07054df7f0bec4ab4de7
                                                       Module libyajl.so.2 with build-id 7566fc033870a3570562a94020172bd833e2c351
                                                       Module libnuma.so.1 with build-id a0c23354fdb39f51ec9e31aeae27a9f8ee034500
                                                       Module libgnutls.so.30 with build-id dc5d93a8db53b2246dd1b5b24bc56956f4c135b2
                                                       Module libcap-ng.so.0 with build-id fdca0a301667e15db99d726152b57feeb35e4dbe
                                                       Module libaudit.so.1 with build-id 9bbee8705c4e1f2855273b732bc6497fdebac344
                                                       Module libacl.so.1 with build-id afd5611fa888ba30a62a2109d13bd3903d84c61c
                                                       Module libgio-2.0.so.0 with build-id b9f2c135589b3605526b93c8e94780016fe5bbd1
                                                       Module libgobject-2.0.so.0 with build-id fcfcfa460c0d0ebc14b2806e4d5900a8dc8df25e
                                                       Module libglib-2.0.so.0 with build-id f556acca796d08b0639ca153c54713cfba852bf6
                                                       Module libvirt.so.0 with build-id 8816550c7b1d1c207fc397d8c01bfa97e9ec722d
                                                       Module libvirtmod.cpython-39-x86_64-linux-gnu.so with build-id 2a044a2b5a36f29c6b0f2557320b46c37c56a5d2
                                                       Module libsqlite3.so.0 with build-id 3693dc299e6d46d1879d24089bd9359a1d9bb62a
                                                       Module _sqlite3.cpython-39-x86_64-linux-gnu.so with build-id b42d0bf1ecb9f5f70ad7ecf5eca8c88783d10194
                                                       Module _posixshmem.cpython-39-x86_64-linux-gnu.so with build-id bd54790d3fac826513db6710d2bc7e2522fdfa55
                                                       Module mmap.cpython-39-x86_64-linux-gnu.so with build-id fbf62b8a4742465578b287c9e5a3b81af35610ca
                                                       Module _multiprocessing.cpython-39-x86_64-linux-gnu.so with build-id b2e64c681890755d582280ed83d4a0dec8834204
                                                       Module libgcc_s.so.1 with build-id 8d8738d89978e21955b975d68b9d03e32fd25528
                                                       Module libnl-route-3.so.200 with build-id f8480d89efc1c4fea4ad2346a6295609cf51a18f
                                                       Module libnl-3.so.200 with build-id e24955deb76e3c52bf70100e6aecf1b6c98205cb
                                                       Module _selinux.cpython-39-x86_64-linux-gnu.so with build-id 117744436008d14c6fa96fc28afb1c4e92e07655
                                                       Module libpcre2-8.so.0 with build-id a5dafbac0243bdc2e3b5ab9a07d61d4b32f86927
                                                       Module libselinux.so.1 with build-id c1d23d40bcf1fae271256fcbe90a46ff6dbcce20
                                                       Module libxml2.so.2 with build-id 93d1e0ee905e2be426312585ff9b3716e993aff9
                                                       Module libfa.so.1 with build-id 035505c5a149146f43654fa0a7c6081083244c64
                                                       Module libaugeas.so.0 with build-id e532419e74d3eda2436500cc7fbafd6316d711e4
                                                       Module _blake2.cpython-39-x86_64-linux-gnu.so with build-id 53f8d2cec92b67bee189600066c54210c46dcb88
                                                       Module _hashlib.cpython-39-x86_64-linux-gnu.so with build-id 44ffb14fc0434f17f7fb171aead1cccf09bc89aa
                                                       Module _random.cpython-39-x86_64-linux-gnu.so with build-id 563dacff0c963f508efec2f96820c2b7c6c826a3
                                                       Module liblzma.so.5 with build-id 66b7fca8b63a61d19c5ded440ee6a8bf05dcafa5
                                                       Module _lzma.cpython-39-x86_64-linux-gnu.so with build-id 4a2886b4d94659d67cecd843e45550e65b887604
                                                       Module libbz2.so.1 with build-id dc9cd83a2cf3038bd3f04cc111c32e2c5698b5d3
                                                       Module _bz2.cpython-39-x86_64-linux-gnu.so with build-id 87a9d22545a53c8ec3db89e319061ab731cde3e4
                                                       Module zlib.cpython-39-x86_64-linux-gnu.so with build-id 66c0919ec1a077716a7f01f8734febc3e4c10eac
                                                       Module fcntl.cpython-39-x86_64-linux-gnu.so with build-id 5d5ecd4d03829fc86914a5b791189c2830b5a263
                                                       Module libblkid.so.1 with build-id 25ffb35e40228ef3df3249c7f1d8cd6135cbcf08
                                                       Module libaio.so.1 with build-id 9b75d21282f8e17ddfa06aff78dae4f8dcce4106
                                                       Module libsanlock.so.1 with build-id 2beb8f3e4bc6746ef66d540bbb0eb94b9439f63e
                                                       Module sanlock.cpython-39-x86_64-linux-gnu.so with build-id 0efe6b72269e99f5d4b262e551ab160ef3cbe9db
                                                       Module binascii.cpython-39-x86_64-linux-gnu.so with build-id 484d27a87b85f5a9a2e7195a64b56454f4a71034
                                                       Module libz.so.1 with build-id 4804f60abf6d78303295fcb9474bed2941e2e7b1
                                                       Module libcrypto.so.3 with build-id 224fbc9ed8ab697662a9b34e608fdee959ab8198
                                                       Module libssl.so.3 with build-id 75928863711cdfc51360837cde7a12326fbbf2f8
                                                       Module _ssl.cpython-39-x86_64-linux-gnu.so with build-id 892e89330d3a4ee22796c97c8657581c0c485451
                                                       Module libpthread.so.0 with build-id ce0059677116ec09edb8daffac4344721e1a23dc
                                                       Module libffi.so.8 with build-id 48e3675db4765a2e42729140922e11a10016f7ab
                                                       Module _ctypes.cpython-39-x86_64-linux-gnu.so with build-id 3b9aa86605b1fa3f794cbe2fe2c4d591218ccde0
                                                       Module _opcode.cpython-39-x86_64-linux-gnu.so with build-id 057f49ddcd872f0909ca02e79d9aa7abf5ec2276
                                                       Module _bisect.cpython-39-x86_64-linux-gnu.so with build-id f802378779f7d27bc1e175d8707c591c508ca225
                                                       Module _queue.cpython-39-x86_64-linux-gnu.so with build-id 58f5fe7973a70238bc4cdbff4438ad23f024dbf8
                                                       Module _datetime.cpython-39-x86_64-linux-gnu.so with build-id 5fb3bb2e3397950bbe513ca14bc065c572baa0be
                                                       Module array.cpython-39-x86_64-linux-gnu.so with build-id 543bfa1a817657801f982b381e89372669c06055
                                                       Module _socket.cpython-39-x86_64-linux-gnu.so with build-id 2b94f1da7f18bdaa0b534153d57ef051efffd184
                                                       Module _pickle.cpython-39-x86_64-linux-gnu.so with build-id b36ef29100bb359b370031621f7fdd7851231c44
                                                       Module _struct.cpython-39-x86_64-linux-gnu.so with build-id c7dc4c8e8b880fcd769205ffb3f8c2184962ad91
                                                       Module _json.cpython-39-x86_64-linux-gnu.so with build-id 4e6ce531b346b35e339a610e3c3f33f6bd3f6547
                                                       Module libuuid.so.1 with build-id da656ae81596c151559892375d730bd35f76048c
                                                       Module _uuid.cpython-39-x86_64-linux-gnu.so with build-id 857d956c529169cc7cc70f58b5385bb44b55e074
                                                       Module math.cpython-39-x86_64-linux-gnu.so with build-id 2d037cc1b72b376c0641c1bac330aeecb6cfa2f3
                                                       Module select.cpython-39-x86_64-linux-gnu.so with build-id 1cc18313458e4f58889d1758cfd0413ffdcba3a2
                                                       Module _posixsubprocess.cpython-39-x86_64-linux-gnu.so with build-id 113cf29e53c355662eeb676e97d82f2016344366
                                                       Module grp.cpython-39-x86_64-linux-gnu.so with build-id 4a3ad9db14c4e2eb3583fc4fa703becbac255c1c
                                                       Module syslog.cpython-39-x86_64-linux-gnu.so with build-id 1c6465270f61cf4fbf5a2ccd141c638f2de19d81
                                                       Module _heapq.cpython-39-x86_64-linux-gnu.so with build-id 3596e567a0f20587fbc3dacebd5a7c60391d36c1
                                                       Module ld-linux-x86-64.so.2 with build-id 6ae37ed0ebb1900854f7aa8675edddbbf33e8d2a
                                                       Module libm.so.6 with build-id 06480257823b66b01706ba8224a9f24fc280a5e8
                                                       Module libc.so.6 with build-id 0167a9d1c100affa1d04d151bb69f88b19342196
                                                       Module libpython3.9.so.1.0 with build-id bbe909b82db5ae1835b0022275d690951734a378
                                                       Module python3.9 with build-id fb06d3eaa4a1db190270f3f01e29cea48c9901c1
                                                       Stack trace of thread 55158:
                                                       #0  0x00007ff083c47629 rtnl_route_remove_nexthop (libnl-route-3.so.200 + 0x54629)
                                                       #1  0x00007ff083c476ab route_free_data (libnl-route-3.so.200 + 0x546ab)
                                                       #2  0x00007ff083c937a7 nl_object_free (libnl-3.so.200 + 0x107a7)
                                                       #3  0x00007ff083c93490 nl_cache_remove (libnl-3.so.200 + 0x10490)
                                                       #4  0x00007ff083c9361d nl_cache_clear (libnl-3.so.200 + 0x1061d)
                                                       #5  0x00007ff083c94778 nl_cache_free (libnl-3.so.200 + 0x11778)
                                                       #6  0x00007ff0849228b6 ffi_call_unix64 (libffi.so.8 + 0x78b6)
                                                       #7  0x00007ff08491f556 ffi_call_int.lto_priv.0 (libffi.so.8 + 0x4556)
                                                       #8  0x00007ff084937e5d _ctypes_callproc.cold (_ctypes.cpython-39-x86_64-linux-gnu.so + 0x8e5d)
                                                       #9  0x00007ff084941183 PyCFuncPtr_call (_ctypes.cpython-39-x86_64-linux-gnu.so + 0x12183)
                                                       #10 0x00007ff0854236b4 _PyObject_MakeTpCall (libpython3.9.so.1.0 + 0x1186b4)
                                                       #11 0x00007ff08542012e _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11512e)
                                                       #12 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #13 0x00007ff085420079 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x115079)
                                                       #14 0x00007ff0854345e8 gen_send_ex (libpython3.9.so.1.0 + 0x1295e8)
                                                       #15 0x00007ff08547cf97 builtin_next (libpython3.9.so.1.0 + 0x171f97)
                                                       #16 0x00007ff085428180 cfunction_vectorcall_FASTCALL (libpython3.9.so.1.0 + 0x11d180)
                                                       #17 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #18 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #19 0x00007ff0854300b1 method_vectorcall (libpython3.9.so.1.0 + 0x1250b1)
                                                       #20 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #21 0x00007ff0854345e8 gen_send_ex (libpython3.9.so.1.0 + 0x1295e8)
                                                       #22 0x00007ff08541b853 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x110853)
                                                       #23 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #24 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #25 0x00007ff08541a01d _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10f01d)
                                                       #26 0x00007ff085427bfe _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11cbfe)
                                                       #27 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #28 0x00007ff08541a01d _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10f01d)
                                                       #29 0x00007ff085427bfe _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11cbfe)
                                                       #30 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #31 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #32 0x00007ff085420079 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x115079)
                                                       #33 0x00007ff08541a01d _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10f01d)
                                                       #34 0x00007ff085427bfe _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11cbfe)
                                                       #35 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #36 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #37 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #38 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #39 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #40 0x00007ff08541a01d _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10f01d)
                                                       #41 0x00007ff085427bfe _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11cbfe)
                                                       #42 0x00007ff08541e24c _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11324c)
                                                       #43 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #44 0x00007ff08541b39b _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11039b)
                                                       #45 0x00007ff08541a01d _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10f01d)
                                                       #46 0x00007ff085496fe5 _PyEval_EvalCodeWithName (libpython3.9.so.1.0 + 0x18bfe5)
                                                       #47 0x00007ff085496f7d PyEval_EvalCodeEx (libpython3.9.so.1.0 + 0x18bf7d)
                                                       #48 0x00007ff085496f2f PyEval_EvalCode (libpython3.9.so.1.0 + 0x18bf2f)
                                                       #49 0x00007ff0854c6284 run_eval_code_obj (libpython3.9.so.1.0 + 0x1bb284)
                                                       #50 0x00007ff0854c2136 run_mod (libpython3.9.so.1.0 + 0x1b7136)
                                                       #51 0x00007ff085395ded pyrun_file.cold (libpython3.9.so.1.0 + 0x8aded)
                                                       #52 0x00007ff0854bc063 PyRun_SimpleFileExFlags (libpython3.9.so.1.0 + 0x1b1063)
                                                       #53 0x00007ff0854b92c8 Py_RunMain (libpython3.9.so.1.0 + 0x1ae2c8)

                                                       Stack trace of thread 55161:
                                                       #0  0x00007ff0851a22ea __futex_abstimed_wait_common (libc.so.6 + 0xa12ea)
                                                       #1  0x00007ff0851ad668 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0xac668)
                                                       #2  0x00007ff085409dda PyThread_acquire_lock_timed (libpython3.9.so.1.0 + 0xfedda)
                                                       #3  0x00007ff08549c4e1 acquire_timed (libpython3.9.so.1.0 + 0x1914e1)
                                                       #4  0x00007ff08549c2ef lock_PyThread_acquire_lock (libpython3.9.so.1.0 + 0x1912ef)
                                                       #5  0x00007ff085428af1 method_vectorcall_VARARGS_KEYWORDS (libpython3.9.so.1.0 + 0x11daf1)
                                                       #6  0x00007ff08541b5fd _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1105fd)
                                                       #7  0x00007ff08541a01d _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10f01d)
                                                       #8  0x00007ff085427bfe _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11cbfe)
                                                       #9  0x00007ff08541b5fd _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1105fd)
                                                       #10 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #11 0x00007ff085430152 method_vectorcall (libpython3.9.so.1.0 + 0x125152)
                                                       #12 0x00007ff08541e24c _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11324c)
                                                       #13 0x00007ff08541a01d _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10f01d)
                                                       #14 0x00007ff085427bfe _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11cbfe)
                                                       #15 0x00007ff08541e24c _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11324c)
                                                       #16 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #17 0x00007ff08541b5fd _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1105fd)
                                                       #18 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #19 0x00007ff08541b5fd _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1105fd)
                                                       #20 0x00007ff085427ea3 function_code_fastcall (libpython3.9.so.1.0 + 0x11cea3)
                                                       #21 0x00007ff085430152 method_vectorcall (libpython3.9.so.1.0 + 0x125152)
                                                       #22 0x00007ff0854e3a1a t_bootstrap (libpython3.9.so.1.0 + 0x1d8a1a)
                                                       #23 0x00007ff0854e39c8 pythread_wrapper (libpython3.9.so.1.0 + 0x1d89c8)
                                                       #24 0x00007ff0851a56ca start_thread (libc.so.6 + 0xa46ca)
                                                       #25 0x00007ff0851453f0 __clone3 (libc.so.6 + 0x443f0)
                                                       ELF object binary architecture: AMD x86-64
Comment 1 Sandro Bonazzola 2022-05-03 09:33:14 UTC 
Downgrading to libnl3 3.5.0-10.el9.x86_64 avoids the crash so it seems like a regression in libnl3 ; moving to RHEL
Comment 2 Thomas Haller 2022-05-03 10:38:47 UTC 
It sounds like you can reproduce this?

Is it possible to get a backtrace with debug symbols installed?
In particular, with libnl3-debuginfo-3.6.0-1.el9.x86_64.rpm from https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1983834
Comment 3 Sandro Bonazzola 2022-05-03 11:22:51 UTC 
The funny thing is that I tried installing debuginfo with dnf debuginfo-install and despite debug packages are there ( http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/debug/tree/ ) they couldn't be installed. I installed them manually and trace in the description is what I got from systemd-coredump after the debuginfo were installed.

Is there a procedure I can follow to generate more complete traces on RHEL 9 / CentOS Stream 9?
Comment 4 Sandro Bonazzola 2022-05-03 12:00:55 UTC 
Nevermind used the old good way:

ulimit -c unlimited

echo "/tmp/core" > /proc/sys/kernel/core_pattern

 /usr/bin/vdsm-tool restore-nets
Segmentation fault (core dumped)

gdb /usr/bin/python3 core.46834

I'll paste the trace once I will be able to extract it in a readable way.
Comment 5 Sandro Bonazzola 2022-05-03 12:05:28 UTC 
#0  nl_list_del (obj=0x55f6d6882438) at ./include/netlink/list.h:47
#1  rtnl_route_remove_nexthop (nh=0x55f6d6882420, route=0x55f6d6762ef0) at lib/route/route_obj.c:879
#2  rtnl_route_remove_nexthop (route=route@entry=0x55f6d6762ef0, nh=nh@entry=0x55f6d6882420) at lib/route/route_obj.c:875
#3  0x00007f77ce8136ab in route_free_data (c=0x55f6d6762ef0) at lib/route/route_obj.c:90
#4  route_free_data (c=0x55f6d6762ef0) at lib/route/route_obj.c:77
#5  0x00007f77ce85f7a7 in nl_object_free (obj=obj@entry=0x55f6d6762ef0) at lib/object.c:185
#6  0x00007f77ce85fa42 in nl_object_put (obj=obj@entry=0x55f6d6762ef0) at lib/object.c:227
#7  0x00007f77ce85f490 in nl_cache_remove (obj=0x55f6d6762ef0) at lib/cache.c:563
#8  0x00007f77ce85f61d in nl_cache_clear (cache=cache@entry=0x55f6d670cda0) at lib/cache.c:368
#9  0x00007f77ce860778 in __nl_cache_free (cache=0x55f6d670cda0) at lib/cache.c:373
#10 nl_cache_free (cache=0x55f6d670cda0) at lib/cache.c:414
#11 0x00007f77cf4ee8b6 in ffi_call_unix64 () at ../src/x86/unix64.S:105
#12 0x00007f77cf4eb556 in ffi_call_int (cif=<optimized out>, fn=<optimized out>, rvalue=<optimized out>, avalue=<optimized out>, closure=<optimized out>) at ../src/x86/ffi64.c:672
#13 0x00007f77cf503e5d in _call_function_pointer (argtypecount=<optimized out>, argcount=1, resmem=0x7ffd80df9bb0, restype=<optimized out>, atypes=<optimized out>, avalues=<optimized out>, 
    pProc=0x7f77ce8606d0 <nl_cache_free>, flags=4353) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/_ctypes/callproc.c:920
#14 _ctypes_callproc (pProc=<optimized out>, argtuple=<optimized out>, flags=<optimized out>, argtypes=<optimized out>, restype=0x7f77d020c660 <_Py_NoneStruct>, checker=<optimized out>)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/_ctypes/callproc.c:1263
#15 0x00007f77cf50d183 in PyCFuncPtr_call (self=self@entry=0x7f77cc42c100, inargs=inargs@entry=0x7f77cc66ce20, kwds=kwds@entry=0x0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/_ctypes/_ctypes.c:4201
#16 0x00007f77cffef6b4 in _PyObject_MakeTpCall (tstate=0x55f6d6187640, callable=0x7f77cc42c100, args=0x55f6d67384a0, nargs=1, keywords=0x0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:191
#17 0x00007f77cffec12e in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d67384a0, callable=<optimized out>, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:116
#18 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d67384a0, callable=<optimized out>, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:103
#19 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d67384a0, callable=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#20 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#22 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x55f6d6738320, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#23 function_code_fastcall (tstate=0x55f6d6187640, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#24 0x00007f77cffec079 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc424d10, callable=0x7f77ce764940, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#25 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc424d10, callable=0x7f77ce764940) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#26 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3489
#28 0x00007f77d00005e8 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc424b80, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#29 gen_send_ex (gen=0x7f77cc477f20, arg=<optimized out>, exc=0, closing=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/genobject.c:215
#30 0x00007f77d0048f97 in gen_iternext (gen=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/genobject.c:544
#31 builtin_next (self=<optimized out>, args=args@entry=0x55f6d690f5e8, nargs=1) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/bltinmodule.c:1382
#32 0x00007f77cfff4180 in cfunction_vectorcall_FASTCALL (func=0x7f77cfb2c2c0, args=0x55f6d690f5e8, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/methodobject.c:430
#33 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d690f5e8, callable=0x7f77cfb2c2c0, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#34 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d690f5e8, callable=0x7f77cfb2c2c0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#35 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#36 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#37 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x55f6d690f450, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#38 function_code_fastcall (tstate=0x55f6d6187640, co=<optimized out>, args=<optimized out>, nargs=4, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#39 0x00007f77cfffc0b1 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=4, args=0x7f77cc42f7a0, callable=0x7f77cf9a21f0, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#40 method_vectorcall (method=<optimized out>, args=0x7f77cc42f7a8, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/classobject.c:53
#41 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc42f7a8, callable=0x7f77cc428980, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#42 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc42f7a8, callable=0x7f77cc428980) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#43 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#44 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#45 0x00007f77d00005e8 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc42f610, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#46 gen_send_ex (gen=0x7f77cc477eb0, arg=<optimized out>, exc=0, closing=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/genobject.c:215
#47 0x00007f77cffe7853 in _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3308
#48 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc424400, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#49 function_code_fastcall (tstate=0x55f6d6187640, co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#50 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc41da00, callable=0x7f77ccd3a790, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#51 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc41da00, callable=0x7f77ccd3a790) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#52 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#53 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#54 0x00007f77cffe601d in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc41d840, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#55 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f77cc42dbe8, 
    kwcount=0, kwstep=1, defs=0x7f77ccd293b8, defcount=1, kwdefs=0x0, closure=0x0, name=0x7f77cf63d4f0, qualname=0x7f77cf63d4f0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4329
#56 0x00007f77cfff3bfe in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:396
#57 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc42dbe0, callable=0x7f77ccd22c10, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#58 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc42dbe0, callable=0x7f77ccd22c10) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#59 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#60 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#61 0x00007f77cffe601d in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc42da60, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#62 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f77cc42f3f0, 
    kwcount=0, kwstep=1, defs=0x7f77ccd2ab58, defcount=2, kwdefs=0x0, closure=0x0, name=0x7f77cfb6adf0, qualname=0x7f77cfb6adf0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4329
#63 0x00007f77cfff3bfe in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:396
#64 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc42f3e0, callable=0x7f77ccd3b040, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#65 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc42f3e0, callable=0x7f77ccd3b040) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#66 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#67 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#68 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc42f230, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#69 function_code_fastcall (tstate=0x55f6d6187640, co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#70 0x00007f77cffec079 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d662eae8, callable=0x7f77ccd519d0, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#71 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d662eae8, callable=0x7f77ccd519d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#72 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#73 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3489
#74 0x00007f77cffe601d in _PyEval_EvalFrame (throwflag=0, f=0x55f6d662e900, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#75 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f77cc64fd30, 
    kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7f77cc88ca30, qualname=0x7f77cc88ca30) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4329
#76 0x00007f77cfff3bfe in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:396
#77 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc64fd28, callable=0x7f77cc893ee0, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#78 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc64fd28, callable=0x7f77cc893ee0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#79 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#80 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#81 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc64fb80, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#82 function_code_fastcall (tstate=0x55f6d6187640, co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#83 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc47cc98, callable=0x7f77cc893c10, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#84 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc47cc98, callable=0x7f77cc893c10) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#85 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#86 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#87 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc47cb20, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#88 function_code_fastcall (tstate=0x55f6d6187640, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#89 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d643bc50, callable=0x7f77cc896280, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#90 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d643bc50, callable=0x7f77cc896280) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#91 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#92 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#93 0x00007f77cffe601d in _PyEval_EvalFrame (throwflag=0, f=0x55f6d643bad0, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#94 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f77cf814930, 
    kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x7f77cc8724b0, qualname=0x7f77cc8724b0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4329
#95 0x00007f77cfff3bfe in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:396
#96 0x00007f77cffea24c in PyObject_Call (kwargs=0x0, args=0x7f77cf814910, callable=0x7f77cc86dc10) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:293
#97 do_call_core (kwdict=0x0, callargs=0x7f77cf814910, func=0x7f77cc86dc10, tstate=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5125
#98 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3582
#99 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x55f6d628dba0, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#100 function_code_fastcall (tstate=0x55f6d6187640, co=<optimized out>, args=<optimized out>, nargs=0, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#101 0x00007f77cffe739b in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d61e1ee0, callable=0x7f77cf827a60, tstate=0x55f6d6187640)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#102 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d61e1ee0, callable=0x7f77cf827a60) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#103 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#104 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3520
#105 0x00007f77cffe601d in _PyEval_EvalFrame (throwflag=0, f=0x55f6d61e1d60, tstate=0x55f6d6187640) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#106 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=0, 
    kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4329
#107 0x00007f77d0062fe5 in _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=0x7f77cface440, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>, kwargs=0x0, 
    kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4361
#108 0x00007f77d0062f7d in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0, 
    defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4377
#109 0x00007f77d0062f2f in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:828
#110 0x00007f77d0092284 in run_eval_code_obj (tstate=0x55f6d6187640, co=0x7f77cf9009d0, globals=0x7f77cface440, locals=0x7f77cface440) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/pythonrun.c:1221
#111 0x00007f77d008e136 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7f77cface440, locals=0x7f77cface440, flags=<optimized out>, arena=<optimized out>)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/pythonrun.c:1242
#112 0x00007f77cff61ded in pyrun_file (fp=<optimized out>, filename=<optimized out>, start=<optimized out>, globals=<optimized out>, locals=<optimized out>, closeit=<optimized out>, flags=0x7ffd80dfb928)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/pythonrun.c:1140
#113 0x00007f77d0088063 in pyrun_simple_file (flags=0x7ffd80dfb928, closeit=1, filename=0x7f77cfa00080, fp=0x55f6d6183340) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/pythonrun.c:450
#114 PyRun_SimpleFileExFlags (fp=0x55f6d6183340, filename=<optimized out>, closeit=1, flags=0x7ffd80dfb928) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/pythonrun.c:483
#115 0x00007f77d00852c8 in pymain_run_file (cf=0x7ffd80dfb928, config=0x55f6d6186040) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/main.c:373
#116 pymain_run_python (exitcode=0x7ffd80dfb920) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/main.c:598
#117 Py_RunMain () at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/main.c:677
#118 0x00007f77d0055b3d in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/main.c:731
#119 0x00007f77cfd11e50 in __libc_start_call_main (main=main@entry=0x55f6d5c59160 <main>, argc=argc@entry=3, argv=argv@entry=0x7ffd80dfbb78) at ../sysdeps/nptl/libc_start_call_main.h:58
#120 0x00007f77cfd11efc in __libc_start_main_impl (main=0x55f6d5c59160 <main>, argc=3, argv=0x7ffd80dfbb78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd80dfbb68)
    at ../csu/libc-start.c:409
#121 0x000055f6d5c59095 in _start ()


thread 2:


#0  __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f77c4001f60) at futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x7f77c4001f60, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>, cancel=cancel@entry=true)
    at futex-internal.c:87
#2  0x00007f77cfd6e34f in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f77c4001f60, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, 
    private=<optimized out>) at futex-internal.c:139
#3  0x00007f77cfd795cf in do_futex_wait (sem=sem@entry=0x7f77c4001f60, abstime=0x0, clockid=0) at /usr/src/debug/glibc-2.34-29.el9.x86_64/nptl/sem_waitcommon.c:112
#4  0x00007f77cfd79668 in __new_sem_wait_slow64 (sem=0x7f77c4001f60, abstime=0x0, clockid=0) at /usr/src/debug/glibc-2.34-29.el9.x86_64/nptl/sem_waitcommon.c:184
#5  0x00007f77cfd796e1 in __new_sem_wait (sem=<optimized out>) at sem_wait.c:43
#6  0x00007f77cffd5dda in PyThread_acquire_lock_timed (lock=0x7f77c4001f60, microseconds=-1000000, intr_flag=1) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/thread_pthread.h:483
#7  0x00007f77d00684e1 in acquire_timed (lock=0x7f77c4001f60, timeout=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/_threadmodule.c:63
#8  0x00007f77d00682ef in lock_PyThread_acquire_lock (self=0x7f77cc463180, args=<optimized out>, kwds=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/_threadmodule.c:146
#9  0x00007f77cfff4af1 in method_vectorcall_VARARGS_KEYWORDS (func=0x7f77cfb4c2c0, args=0x55f6d6908c58, nargsf=<optimized out>, kwnames=<optimized out>)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/descrobject.c:348
#10 0x00007f77cffe75fd in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d6908c58, callable=0x7f77cfb4c2c0, tstate=0x55f6d663b1d0)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#11 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x55f6d6908c58, callable=0x7f77cfb4c2c0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#12 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#13 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3506
#14 0x00007f77cffe601d in _PyEval_EvalFrame (throwflag=0, f=0x55f6d6908ac0, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#15 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f77cc4203a0, 
    kwcount=0, kwstep=1, defs=0x7f77cf877688, defcount=1, kwdefs=0x0, closure=0x0, name=0x7f77cfaf9070, qualname=0x7f77cf87a6b0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4329
#16 0x00007f77cfff3bfe in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:396
#17 0x00007f77cffe75fd in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc420398, callable=0x7f77cf887e50, tstate=0x55f6d663b1d0)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#18 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc420398, callable=0x7f77cf887e50) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#19 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#20 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3506
#21 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc420210, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#22 function_code_fastcall (tstate=0x55f6d663b1d0, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#23 0x00007f77cfffc152 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>, tstate=<optimized out>)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#24 method_vectorcall (method=<optimized out>, args=0x7f77cfb5e058, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/classobject.c:83
#25 0x00007f77cffea24c in PyObject_Call (kwargs=0x7f77cc41a6c0, args=0x7f77cfb5e040, callable=0x7f77cc41a900) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:293
#26 do_call_core (kwdict=0x7f77cc41a6c0, callargs=0x7f77cfb5e040, func=0x7f77cc41a900, tstate=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5125
#27 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3582
#28 0x00007f77cffe601d in _PyEval_EvalFrame (throwflag=0, f=0x7f77c4000fe0, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#29 _PyEval_EvalCode (tstate=<optimized out>, _co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f77cfb5e058, 
    kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x7f77cc470db0, name=0x7f77cfb1eab0, qualname=0x7f77cf549760) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:4329
#30 0x00007f77cfff3bfe in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:396
#31 0x00007f77cffea24c in PyObject_Call (kwargs=0x7f77cc41a180, args=0x7f77cfb5e040, callable=0x7f77cf827af0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:293
#32 do_call_core (kwdict=0x7f77cc41a180, callargs=0x7f77cfb5e040, func=0x7f77cf827af0, tstate=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5125
#33 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3582
#34 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77ce87fe40, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#35 function_code_fastcall (tstate=0x55f6d663b1d0, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#36 0x00007f77cffe75fd in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc454bb8, callable=0x7f77cf80e280, tstate=0x55f6d663b1d0)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#37 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cc454bb8, callable=0x7f77cf80e280) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#38 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#39 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3506
#40 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cc454a40, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#41 function_code_fastcall (tstate=0x55f6d663b1d0, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#42 0x00007f77cffe75fd in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cd136c38, callable=0x7f77cf80e550, tstate=0x55f6d663b1d0)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#43 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7f77cd136c38, callable=0x7f77cf80e550) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:127
#44 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:5077
#45 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>, throwflag=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/ceval.c:3506
#46 0x00007f77cfff3ea3 in _PyEval_EvalFrame (throwflag=0, f=0x7f77cd136ac0, tstate=0x55f6d663b1d0) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/internal/pycore_ceval.h:40
#47 function_code_fastcall (tstate=0x55f6d663b1d0, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/call.c:330
#48 0x00007f77cfffc152 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>, tstate=<optimized out>)
    at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Include/cpython/abstract.h:118
#49 method_vectorcall (method=<optimized out>, args=0x7f77cfb5e058, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Objects/classobject.c:83
#50 0x00007f77d00afa1a in t_bootstrap (boot_raw=0x7f77cc40fa80) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Modules/_threadmodule.c:1040
#51 0x00007f77d00af9c8 in pythread_wrapper (arg=<optimized out>) at /usr/src/debug/python3.9-3.9.10-2.el9.x86_64/Python/thread_pthread.h:245
#52 0x00007f77cfd716ca in start_thread (arg=<optimized out>) at pthread_create.c:443
#53 0x00007f77cfd113f0 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Comment 7 Thomas Haller 2022-05-03 12:35:23 UTC 
Created attachment 1876736 [details]
Backtrace

A backtrace. Thanks Sandro for providing access!
Comment 8 Thomas Haller 2022-05-03 19:02:52 UTC 
fixed upstream: https://github.com/thom311/libnl/commit/9b0493646c20f3eecdf664e7e00c6c9162e04fba
Comment 11 Sandro Bonazzola 2022-05-04 10:04:26 UTC 
I can confirm that  libnl3-3.6.0-2.el9 from https://kojihub.stream.centos.org/koji/buildinfo?buildID=18351 solves the issue for me.
Comment 12 Thomas Haller 2022-05-04 11:32:32 UTC 
this issue was easy to reproduce with Sandro's help. Also easy to very that the fix avoided the crash.
Together with comment 11, I set this as pre-verified (Tested). Thanks Sandro.
Comment 14 Sandro Bonazzola 2022-05-26 06:55:40 UTC 
closing current release as I can confirm it works on current CentOS Stream 9
Comment 17 Vladimir Benes 2022-07-14 08:48:35 UTC 
(In reply to Sandro Bonazzola from comment #14)
> closing current release as I can confirm it works on current CentOS Stream 9

Sandro, thank you for verification!
Comment 20 David Jaša 2022-08-03 14:38:48 UTC 
Marking as VERIFIED based on comment 14.
Comment 22 errata-xmlrpc 2022-11-15 11:14:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libnl3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8286