Bug 1138229
| Summary: | Disconnections from glusterfs through libgfapi | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Alvaro Flores <alvaroflores> | ||||
| Component: | glusterfs | Assignee: | sankarshan <sankarshan> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | storage-qa-internal <storage-qa-internal> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.6 | CC: | alvaroflores, bugs, ndevos, smohan | ||||
| Target Milestone: | pre-dev-freeze | ||||||
| Target Release: | 6.7 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-12-06 11:44:55 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Alvaro, Could you attach the libgfapi based script, that experiences the disconnections when the "Steps to Reproduce" are followed, to the bug? Alvaro, The version of glusterfs you are using is not the community version. Could you confirm if you are seeing this issue on a corresponding upstream version and share the corresponding logs? Hello Krishnan. I sended you by email the source code of the tool that we use (to the @redhat.com mail showed in your name). I think that the problem is the number of threads, collapsing the glusterd, but the machine have enough capability to be able to manage them (not reached the number of file descriptors, not reached the cpu limit, enough ram, etc). It will fail with only 8 threads, is not a big number, so we are getting the error too quickly. KP, did you follow up on this? Alvaro, is this still an issue with the latest 3.6.2 packages? (In reply to Niels de Vos from comment #4) > KP, did you follow up on this? > > Alvaro, is this still an issue with the latest 3.6.2 packages? Niels, I am not an expert on libgfapi. I tried helping Alvaro long back. I am assigning it to the default assignee for libgfapi. Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available. The official life cycle policy can be reviewed here: http://redhat.com/rhel/lifecycle This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL: https://access.redhat.com/ |
Created attachment 934366 [details] Logs of the three gluster servers Description of problem: Disconnections from glusterfs through libgfapi. Version-Release number of selected component (if applicable): glusterfs-3.6.0.22-1.el6rhs.x86_64 How reproducible: Open some connections to the glusterfs through libgfapi and write 50K files and read 100K files os 1 MB Steps to Reproduce: 1. Open multiple connections (i.e 64) to a gluster volume using libgfapi. 2. open-write-close files using different threads on the multiple connections 3. Client get disconnected from the gluster volume (open fails) Actual results: I am having an issue with glusterfs disconnections while creating and writing files. The environment is 3 RHEV hypervisors running each one 2 virtual machines, one client and one server, both of them with RHEL 6.5 I am running a script that open n threads from the clients to write or read files from the glusterfs volume who shares the three servers, using libgfapi. Before try to do nothing, the script removes the destination folders of the writers if it exists. It only makes a mount using nfs to launch a rm -rf of the destination folder. Here i have the first strange log: nfs.log:[2014-09-04 07:46:16.260087] W [dht-layout.c:180:dht_layout_search] 0-vol02-dht: no subvolume for hash (value) = 3004083931 It only occurs in one of the three servers, and to daze me it only occurs with exactly 1000 files, so as i have to remove 3 folders, i have exactly 3000 events like this in the logs. After that, the script begins to open connections to the gluster volume, it connects to the gluster: [2014-09-04 07:56:59.609786] I [server-handshake.c:578:server_setvolume] 0-vol02-server: accepted client from vxoa01.vna-2273-2014/09/04-07:56:53:865815-vol02-client-22-0-0 (version: 3.6.0.22) but after one or two minutes it begins: [2014-09-04 07:58:17.290542] I [server-helpers.c:291:do_fd_cleanup] 0-vol02-server: fd cleanup on /rtest_3/dir_026/dptest.26008 [2014-09-04 07:58:17.290796] E [client_t.c:384:gf_client_unref] (-->/usr/lib64/glusterfs/3.6.0.22/xlator/features/locks.so(pl_flush_cbk+0xb9) [0x7f0b1a546399] (-->/usr/lib64/libglusterfs.so.0(default_flush_cbk+0xb9) [0x3be322d919] (-->/usr/lib64/glusterfs/3.6.0.22/xlator/debug/io-stats.so(io_stats_flush_cbk+0xed) [0x7f0b198d952d]))) 0-client_t: client is NULL [2014-09-04 07:58:17.353584] I [server-resolve.c:519:server_resolve_fd] 0-: fd not found in context [2014-09-04 07:58:17.353679] E [server-rpc-fops.c:1336:server_flush_cbk] 0-vol02-server: 831: FLUSH 4 (d35cf248-fa0b-4b9d-b4ca-733c4d457ea2) ==> (Wrong file descriptor) In the next 2 seconds i have the same message for a lot of different files in differnt volumes and the sessions are disconnected. The gluster volume is online before and after, all in online state and no messages in /var/log except the glusterfs ones. The limit of number of files in the kernel is far (cat /proc/sys/fs/file-nr don't go further of 3500 files and ulimit -n show 32000). Expected results: Avoid the disconnections. Additional info: Logs of the three servers are attached to this bug.