Bug 1721105

+++ This bug was initially created as a clone of Bug #1716812 +++

Description of problem:

gluster volume create 11 transport tcp,rdma 193.168.141.101:/tmp/11 193.168.141.101:/tmp/12 force
volume create: 11: failed: Failed to create volume files


Version-Release number of selected component (if applicable):

# gluster --version
glusterfs 4.1.8
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:9c:8b:a9 brd ff:ff:ff:ff:ff:ff
    inet 193.168.141.101/16 brd 193.168.255.255 scope global dynamic ens192
       valid_lft 2591093sec preferred_lft 2591093sec
    inet6 fe80::250:56ff:fe9c:8ba9/64 scope link
       valid_lft forever preferred_lft forever
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:9c:53:58 brd ff:ff:ff:ff:ff:ff


How reproducible:


Steps to Reproduce:
1.rxe_cfg start
2.rxe_cfg add ens192
3.gluster volume create 11 transport tcp,rdma 193.168.141.101:/tmp/11 193.168.141.101:/tmp/12 force


Actual results:
volume create: 11: failed: Failed to create volume files


Expected results:
Success to create volume

Additional info:

[2019-06-04 07:36:45.966125] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-glusterd: Started running glusterd version 4.1.8 (args: glusterd --xlator-option *.upgrade=on -N)
[2019-06-04 07:36:45.970884] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536
[2019-06-04 07:36:45.970900] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory
[2019-06-04 07:36:45.970906] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory
[2019-06-04 07:36:45.973455] E [rpc-transport.c:284:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/4.1.8/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2019-06-04 07:36:45.973468] W [rpc-transport.c:288:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2019-06-04 07:36:45.973473] W [rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2019-06-04 07:36:45.973478] E [MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2019-06-04 07:36:45.976348] I [MSGID: 106513] [glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202
[2019-06-04 07:36:45.977372] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 79e7e129-d041-48b6-b1d0-746c55d148fc
[2019-06-04 07:36:45.989706] I [MSGID: 106194] [glusterd-store.c:3850:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.listen-backlog 10
  8:     option upgrade on
  9:     option event-threads 1
 10:     option ping-timeout 0
 11:     option transport.socket.read-fail-log off
 12:     option transport.socket.keepalive-interval 2
 13:     option transport.socket.keepalive-time 10
 14:     option transport-type rdma
 15:     option working-directory /var/lib/glusterd
 16: end-volume
 17:
+------------------------------------------------------------------------------+
[2019-06-04 07:36:46.005401] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2019-06-04 07:36:46.006879] W [glusterfsd.c:1514:cleanup_and_exit] (-->/usr/lib64/libpthread.so.0(+0x7dd5) [0x7f55547bbdd5] -->glusterd(glusterfs_sigwaiter+0xe5) [0x55c659e7dd65] -->glusterd(cleanup_and_exit+0x6b) [0x55c659e7db8b] ) 0-: received signum (15), shutting down
[2019-06-04 07:36:46.006997] E [rpcsvc.c:1536:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2019-06-04 07:36:46.007004] E [rpcsvc.c:1662:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2019-06-04 07:36:46.007008] E [rpcsvc.c:1708:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
[2019-06-04 07:36:46.007061] E [rpcsvc.c:1536:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2019-06-04 07:36:46.007066] E [rpcsvc.c:1662:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2019-06-04 07:36:46.007070] E [rpcsvc.c:1708:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
[2019-06-04 07:37:18.784525] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 4.1.8 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2019-06-04 07:37:18.787926] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536
[2019-06-04 07:37:18.787944] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory
[2019-06-04 07:37:18.787950] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory
[2019-06-04 07:37:18.814752] W [MSGID: 103071] [rdma.c:4629:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2019-06-04 07:37:18.814780] W [MSGID: 103055] [rdma.c:4938:init] 0-rdma.management: Failed to initialize IB Device
[2019-06-04 07:37:18.814786] W [rpc-transport.c:351:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2019-06-04 07:37:18.814844] W [rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2019-06-04 07:37:18.814852] E [MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2019-06-04 07:37:19.617049] I [MSGID: 106513] [glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31202
[2019-06-04 07:37:19.617342] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 79e7e129-d041-48b6-b1d0-746c55d148fc
[2019-06-04 07:37:19.626546] I [MSGID: 106194] [glusterd-store.c:3850:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.listen-backlog 10
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:
+------------------------------------------------------------------------------+
[2019-06-04 07:37:19.626791] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2019-06-04 07:37:20.874611] W [MSGID: 101095] [xlator.c:181:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/4.1.8/xlator/nfs/server.so: cannot open shared object file: No such file or directory
[2019-06-04 07:37:20.889571] E [MSGID: 106068] [glusterd-volgen.c:1034:volgen_write_volfile] 0-management: failed to create volfile
[2019-06-04 07:37:20.889588] E [glusterd-volgen.c:6727:glusterd_create_volfiles] 0-management: Could not generate gfproxy client volfiles
[2019-06-04 07:37:20.889601] E [MSGID: 106122] [glusterd-syncop.c:1482:gd_commit_op_phase] 0-management: Commit of operation 'Volume Create' failed on localhost : Failed to create volume files
[2019-06-04 07:38:49.194175] W [MSGID: 101095] [xlator.c:181:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/4.1.8/xlator/nfs/server.so: cannot open shared object file: No such file or directory
[2019-06-04 07:38:49.211380] E [MSGID: 106068] [glusterd-volgen.c:1034:volgen_write_volfile] 0-management: failed to create volfile
[2019-06-04 07:38:49.211407] E [glusterd-volgen.c:6727:glusterd_create_volfiles] 0-management: Could not generate gfproxy client volfiles
[2019-06-04 07:38:49.211433] E [MSGID: 106122] [glusterd-syncop.c:1482:gd_commit_op_phase] 0-management: Commit of operation 'Volume Create' failed on localhost : Failed to create volume files

--- Additional comment from guolei on 2019-06-04 07:58:44 UTC ---

Test is ok on glusterfs3.12.9 ,failed on glusterfs3.13.2 and later version.



generate_client_volfiles (glusterd_volinfo_t *volinfo,
                          glusterd_client_type_t client_type)
{
        int                i                  = 0;
        int                ret                = -1;
        char               filepath[PATH_MAX] = {0,};
        char               *types[]           = {NULL, NULL, NULL};
        dict_t             *dict              = NULL;
        xlator_t           *this              = NULL;
        gf_transport_type  type               = GF_TRANSPORT_TCP;

        this = THIS;

        enumerate_transport_reqs (volinfo->transport_type, types);
        dict = dict_new ();
        if (!dict)
                goto out;
        for (i = 0; types[i]; i++) {
                memset (filepath, 0, sizeof (filepath));
                ret = dict_set_str (dict, "client-transport-type", types[i]);
                if (ret)
                        goto out;
                type = transport_str_to_type (types[i]);

                ret = dict_set_uint32 (dict, "trusted-client", client_type);
                if (ret)
                        goto out;

                if (client_type == GF_CLIENT_TRUSTED) {
                        ret = glusterd_get_trusted_client_filepath (filepath,
                                                                    volinfo,
                                                                    type);
                } else if (client_type == GF_CLIENT_TRUSTED_PROXY) {
                        glusterd_get_gfproxy_client_volfile (volinfo,
                                                             filepath,
                                                             PATH_MAX);  <---------------------------- Maybe this is the problem?  transport type should be passed to glusterd_get_gfproxy_client_volfile .Or filepath is NULL.
                        ret = dict_set_str (dict, "gfproxy-client", "on");
                } else {
                        ret = glusterd_get_client_filepath (filepath,
                                                            volinfo,
                                                            type);
                }
                if (ret) {
                        gf_msg (this->name, GF_LOG_ERROR, EINVAL,
                                GD_MSG_INVALID_ENTRY,
                                "Received invalid transport-type");
                        goto out;
                }

*                ret = generate_single_transport_client_volfile (volinfo,
                                                                filepath,
                                                                dict);*
                if (ret)
                        goto out;
        }

        /* Generate volfile for rebalance process */
        glusterd_get_rebalance_volfile (volinfo, filepath, PATH_MAX);
        ret = build_rebalance_volfile (volinfo, filepath, dict);

        if (ret) {
                gf_msg (this->name, GF_LOG_ERROR, 0,
                        GD_MSG_VOLFILE_CREATE_FAIL,
                        "Failed to create rebalance volfile for %s",
                        volinfo->volname);
                goto out;
        }

out:
        if (dict)
                dict_unref (dict);

        gf_msg_trace ("glusterd", 0, "Returning %d", ret);
        return ret;
}

void
glusterd_get_gfproxy_client_volfile (glusterd_volinfo_t *volinfo,
                                        char *path, int path_len)
{
        char                    workdir[PATH_MAX]      = {0, };
        glusterd_conf_t        *priv                    = THIS->private;

        GLUSTERD_GET_VOLUME_DIR (workdir, volinfo, priv);

        switch (volinfo->transport_type) {
        case GF_TRANSPORT_TCP:
                snprintf (path, path_len,
                                "%s/trusted-%s.tcp-gfproxy-fuse.vol",
                                workdir, volinfo->volname);
                break;

        case GF_TRANSPORT_RDMA:
                snprintf (path, path_len,
                                "%s/trusted-%s.rdma-gfproxy-fuse.vol",
                                workdir, volinfo->volname);
                break;
        default:
                break;
        }
}

--- Additional comment from Atin Mukherjee on 2019-06-10 12:18:36 UTC ---

Since type GF_TRANSPORT_BOTH_TCP_RDMA isn't handled in the function.

Poornima - Was this intentionally done or a bug? I feel it's the latter. Looking at glusterd_get_dummy_client_filepath () we just need to club GF_TRANSPORT_TCP & GF_TRANSPORT_BOTH_TCP_RDMA in the same place. Please confirm.

--- Additional comment from Sanju on 2019-06-10 17:17:57 UTC ---

Looking at the code, I feel we missed handle GF_TRANSPORT_BOTH_TCP_RDMA. As we have provided choice to create volume using tcp,rdma we should handle GF_TRANSPORT_BOTH_TCP_RDMA in glusterd_get_gfproxy_client_volfile().

This issue exists in the latest master too.

Thanks,
Sanju

--- Additional comment from Worker Ant on 2019-06-11 04:25:25 UTC ---

REVIEW: https://review.gluster.org/22851 (glusterd: add GF_TRANSPORT_BOTH_TCP_RDMA in glusterd_get_gfproxy_client_volfile) posted (#1) for review on master by Atin Mukherjee

--- Additional comment from Worker Ant on 2019-06-17 10:31:00 UTC ---

REVIEW: https://review.gluster.org/22851 (glusterd: add GF_TRANSPORT_BOTH_TCP_RDMA in glusterd_get_gfproxy_client_volfile) merged (#5) on master by Amar Tumballi