Bug 1786222
| Summary: | glusterfs APIs used in forked process for storage labelling are not async signal safe causing deadlocks. | |||
|---|---|---|---|---|
| Product: | [Community] Virtualization Tools | Reporter: | kylechan <kylechan> | |
| Component: | libvirt | Assignee: | Libvirt Maintainers <libvirt-maint> | |
| Status: | CLOSED DEFERRED | QA Contact: | ||
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | unspecified | CC: | berrange, kylechan, libvirt-maint, mprivozn | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | All | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1786301 (view as bug list) | Environment: | ||
| Last Closed: | 2024-12-17 12:32:45 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
Thanks for debugging this, it is a really serious problem. Unfortunately it looks like it will be pretty difficult/painful to solve too, but we definitely need to solve it :-( I agree that it is serious problem, but I don't think we have a way out here. Dan, do you have an idea? (In reply to Michal Privoznik from comment #2) > I agree that it is serious problem, but I don't think we have a way out > here. Dan, do you have an idea? I don't know enough about the glfs api, but perhaps it is enough to open the connection in the main process and let the open handle be used in the child which might then only require read/write calls. Thank you for reporting this issue to the libvirt project. Unfortunately we have been unable to resolve this issue due to insufficient maintainer capacity and it will now be closed. This is not a reflection on the possible validity of the issue, merely the lack of resources to investigate and address it, for which we apologise. If you none the less feel the issue is still important, you may choose to report it again at the new project issue tracker https://gitlab.com/libvirt/libvirt/-/issues The project also welcomes contribution from anyone who believes they can provide a solution. |
Description of problem: If multiple domains were being started up in parallel, libvirtd might have deadlocked. Because in the child process use unsafe function after fork. libvirt-3.9.0 function trace: qemuProcessStart |_qemuProcessPrepareHostStorage |_ ... |_virStorageFileGetMetadataRecurse |_virStorageFileInitAs |_virStorageFileBackendGlusterInit |_glfs_init (first called) |_qemuProcessLaunch |_ ... |_virProcessRunInMountNamespace (callback : async signal safe functions only) |_ virFork |_ virProcessNamespaceHelper (child) |_virSecurityDACTransactionRun |_virSecurityDACSetOwnershipInternal |_qemuSecurityChownCallback |_virStorageFileInit |_virStorageFileInitAs |_virStorageFileBackendGlusterInit |_glfs_init (bug here: this is not async signal safe function!! which cause deadlock when muti domains starting in parallel) |__virProcessWait（parent） review libvirt-5.10.0 code, seems this problem is not fixed yet: virProcessRunInMountNamespace (callback : async signal safe functions only) |_virProcessRunInFork |_ virFork() |_ virProcessRunInForkHelper (child) |_ virProcessNamespaceHelper |_ virSecurityDACTransactionRun |_ ... |_ qemuSecurityChownCallback |_virStorageFileInit |_virStorageFileInitAs |_virStorageFileBackendGlusterInit |_glfs_init (bug here: this is not async signal safe function!! which cause deadlock when muti domains starting in parallel) Version-Release number of selected component (if applicable): CentOS Linux release 7.5.1804 libvirt-3.9.0 (and libvirt-5.10.0) How reproducible: occasionally Steps to Reproduce: 1. disk use glusterfs 2. start up domains in parallel Actual results: deadlocked. Expected results: domains start up success. Additional info: [root@CNSZ22PL334 gfs]# pstack 91613 Thread 6 (Thread 0x7ff993ec3700 (LWP 91633)): #0 0x00007ffa9f30ad42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffa8c5a03c8 in syncenv_task (proc=proc@entry=0x7ff9fc01ae70) at syncop.c:603 #2 0x00007ffa8c5a12f0 in syncenv_processor (thdata=0x7ff9fc01ae70) at syncop.c:695 #3 0x00007ffa9f306e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffa9f030bad in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff9946c4700 (LWP 91635)): #0 0x00007ffa9f30ad42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ffa8c5a03c8 in syncenv_task (proc=proc@entry=0x7ff9fc01b230) at syncop.c:603 #2 0x00007ffa8c5a12f0 in syncenv_processor (thdata=0x7ff9fc01b230) at syncop.c:695 #3 0x00007ffa9f306e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffa9f030bad in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff994ec5700 (LWP 91637)): #0 0x00007ffa9f30df3d in nanosleep () from /lib64/libpthread.so.0 #1 0x00007ffa8c573ba6 in gf_timer_proc (data=0x7ff9fc070a40) at timer.c:164 #2 0x00007ffa9f306e25 in start_thread () from /lib64/libpthread.so.0 #3 0x00007ffa9f030bad in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff9956c6700 (LWP 91638)): #0 0x00007ffa9f307f97 in pthread_join () from /lib64/libpthread.so.0 #1 0x00007ffa8c5c13c0 in event_dispatch_epoll (event_pool=0x7ff9fc038bf0) at event-epoll.c:746 #2 0x00007ffa8cc74674 in glfs_poller (data=<optimized out>) at glfs.c:672 #3 0x00007ffa9f306e25 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ffa9f030bad in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff9962c7700 (LWP 91639)): #0 0x00007ffa9f031183 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ffa8c5c0ea2 in event_dispatch_epoll_worker (data=0x7ffa841ea520) at event-epoll.c:649 #2 0x00007ffa9f306e25 in start_thread () from /lib64/libpthread.so.0 #3 0x00007ffa9f030bad in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ffa92d6f700 (LWP 91613)): #0 0x00007ffa9f30d51d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007ffa9f308e1b in _L_lock_812 () from /lib64/libpthread.so.0 #2 0x00007ffa9f308ce8 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007ffa935743d8 in _nss_files_getservbyport_r () from /lib64/libnss_files.so.2 #4 0x00007ffa9f04e9f3 in getservbyport_r@@GLIBC_2.2.5 () from /lib64/libc.so.6 #5 0x00007ffa9f054711 in getnameinfo () from /lib64/libc.so.6 #6 0x00007ffa8c56c121 in gf_resolve_ip6 (hostname=hostname@entry=0x7ff9fc01f920 "10.203.192.226", port=<optimized out>, family=<optimized out>, dnscache=dnscache@entry=0x7ff9fc077438, addr_info=addr_info@entry=0x7ffa92d6ded8) at common-utils.c:332 #7 0x00007ffa3c5e8b22 in af_inet_client_get_remote_sockaddr (sockaddr_len=0x7ffa92d6df68, sockaddr=0x7ffa92d6df80, this=0x7ff9fc0773c0) at name.c:263 #8 socket_client_get_remote_sockaddr (this=this@entry=0x7ff9fc0773c0, sockaddr=sockaddr@entry=0x7ffa92d6df80, sockaddr_len=sockaddr_len@entry=0x7ffa92d6df68, sa_family=sa_family@entry=0x7ffa92d6df66) at name.c:535 #9 0x00007ffa3c5e5608 in socket_connect (this=0x7ff9fc0773c0, port=0) at socket.c:3148 #10 0x00007ffa8ca60de9 in rpc_clnt_reconnect (conn_ptr=0x7ff9fc077040) at rpc-clnt.c:407 #11 0x00007ffa8ca60ea9 in rpc_clnt_start (rpc=rpc@entry=0x7ff9fc077010) at rpc-clnt.c:1196 #12 0x00007ffa8cc78f4d in glfs_mgmt_init (fs=fs@entry=0x7ff9fc07cb10) at glfs-mgmt.c:1032 #13 0x00007ffa8cc748e0 in glfs_volumes_init (fs=fs@entry=0x7ff9fc07cb10) at glfs.c:270 #14 0x00007ffa8cc75c3f in glfs_init_common (fs=0x7ff9fc07cb10) at glfs.c:1039 #15 0x00007ffa8cc75d3f in pub_glfs_init (fs=0x7ff9fc07cb10) at glfs.c:1084 #16 0x00007ffa8ce987ed in virStorageFileBackendGlusterInit (src=0x7ff9fc07d370) at storage/storage_backend_gluster.c:660 #17 0x00007ffa8df08892 in virStorageFileInitAs (src=src@entry=0x7ff9fc07d370, uid=uid@entry=4294967295, gid=gid@entry=4294967295) at storage/storage_source.c:168 #18 0x00007ffa8df0890c in virStorageFileInit (src=src@entry=0x7ff9fc07d370) at storage/storage_source.c:188 #19 0x00007ffa8ba66d53 in qemuSecurityChownCallback (src=<optimized out>, uid=0, gid=0) at qemu/qemu_driver.c:353 #20 0x00007ffaa0c53d32 in virSecurityDACSetOwnershipInternal (priv=0x7ffa8404ec70, src=0x7ffa3000b9c0, path=0x0, uid=0, gid=0) at security/security_dac.c:569 #21 0x00007ffaa0c54cf1 in virSecurityDACTransactionRun (pid=pid@entry=91559, opaque=opaque@entry=0x7ff9fc07d920) at security/security_dac.c:203 #22 0x00007ffaa0aa1d7e in virProcessNamespaceHelper (opaque=0x7ff9fc07d920, cb=0x7ffaa0c54cb0 <virSecurityDACTransactionRun>, pid=91559, errfd=65) at util/virprocess.c:1102 #23 virProcessRunInMountNamespace (pid=pid@entry=91559, cb=cb@entry=0x7ffaa0c54cb0 <virSecurityDACTransactionRun>, opaque=opaque@entry=0x7ff9fc07d920) at util/virprocess.c:1143 #24 0x00007ffaa0c53696 in virSecurityDACTransactionCommit (mgr=<optimized out>, pid=91559) at security/security_dac.c:516 #25 0x00007ffaa0c563e6 in virSecurityManagerTransactionCommit (mgr=0x7ffa8404eca0, pid=pid@entry=91559) at security/security_manager.c:286 #26 0x00007ffaa0c52c43 in virSecurityStackTransactionCommit (mgr=<optimized out>, pid=91559) at security/security_stack.c:166 #27 0x00007ffaa0c563e6 in virSecurityManagerTransactionCommit (mgr=0x7ffa8404f0b0, pid=91559) at security/security_manager.c:286 #28 0x00007ffa8ba7b808 in qemuSecuritySetAllLabel (driver=driver@entry=0x7ffa84016ee0, vm=vm@entry=0x7ffa300028e0, stdin_path=stdin_path@entry=0x0) at qemu/qemu_security.c:54 #29 0x00007ffa8ba0dd13 in qemuProcessLaunch (conn=conn@entry=0x7ff9d8000b80, driver=driver@entry=0x7ffa84016ee0, vm=vm@entry=0x7ffa300028e0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_START, incoming=incoming@entry=0x0, snapshot=snapshot@entry=0x0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=flags@entry=17) at qemu/qemu_process.c:5866 #30 0x00007ffa8ba10737 in qemuProcessStart (conn=conn@entry=0x7ff9d8000b80, driver=driver@entry=0x7ffa84016ee0, vm=vm@entry=0x7ffa300028e0, updatedCPU=updatedCPU@entry=0x0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_START, migrateFrom=migrateFrom@entry=0x0, migrateFd=migrateFd@entry=-1, migratePath=migratePath@entry=0x0, snapshot=snapshot@entry=0x0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=17, flags@entry=1) at qemu/qemu_process.c:6135 #31 0x00007ffa8ba742d6 in qemuDomainObjStart (conn=0x7ff9d8000b80, driver=driver@entry=0x7ffa84016ee0, vm=0x7ffa300028e0, flags=flags@entry=0, asyncJob=QEMU_ASYNC_JOB_START) at qemu/qemu_driver.c:7283 #32 0x00007ffa8ba74a16 in qemuDomainCreateWithFlags (dom=0x7ff9fc000b00, flags=0) at qemu/qemu_driver.c:7337 #33 0x00007ffaa0b6e9ec in virDomainCreate (domain=domain@entry=0x7ff9fc000b00) at libvirt-domain.c:6531 #34 0x000055dc160fe273 in remoteDispatchDomainCreate (server=0x55dc16b2bf30, msg=0x55dc16b3baa0, args=<optimized out>, rerr=0x7ffa92d6ec50, client=0x55dc16b3c6d0) at remote_dispatch.h:4222 #35 remoteDispatchDomainCreateHelper (server=0x55dc16b2bf30, client=0x55dc16b3c6d0, msg=0x55dc16b3baa0, rerr=0x7ffa92d6ec50, args=<optimized out>, ret=0x7ff9fc000d70) at remote_dispatch.h:4198 #36 0x00007ffaa0bdc682 in virNetServerProgramDispatchCall (msg=0x55dc16b3baa0, client=0x55dc16b3c6d0, server=0x55dc16b2bf30, prog=0x55dc16b37bf0) at rpc/virnetserverprogram.c:437 #37 virNetServerProgramDispatch (prog=0x55dc16b37bf0, server=server@entry=0x55dc16b2bf30, client=0x55dc16b3c6d0, msg=0x55dc16b3baa0) at rpc/virnetserverprogram.c:307 #38 0x000055dc1610eb2d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x55dc16b2bf30) at rpc/virnetserver.c:148 #39 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x55dc16b2bf30) at rpc/virnetserver.c:169 #40 0x00007ffaa0ab9851 in virThreadPoolWorker (opaque=opaque@entry=0x55dc16b387e0) at util/virthreadpool.c:167 #41 0x00007ffaa0ab8bd8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #42 0x00007ffa9f306e25 in start_thread () from /lib64/libpthread.so.0 #43 0x00007ffa9f030bad in clone () from /lib64/libc.so.6