Description of problem:
After snapshot of encrypted volume is mounted have failure to read content of snapshot files. 

Version-Release number of selected component (if applicable): GlusterFS 3.6.5


How reproducible: 100% reproducible


Steps to Reproduce:
1. Create volume with enabled encryption:

# gluster volume info

Volume Name: volume-0001
Type: Distribute
Volume ID: 7f2ce583-d87d-44ba-969e-a47b20573429
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.10.60.182:/exports/brick-0001/volume-0001
Options Reconfigured:
nfs.addr-namelookup: off
nfs.disable: on
performance.stat-prefetch: off
performance.quick-read: off
server.allow-insecure: on
features.encryption: on
performance.write-behind: off
performance.open-behind: off
encryption.master-key: /var/tmp/volume-0001.gmek
encryption.data-key-size: 256
encryption.block-size: 4096
features.barrier-timeout: 240
network.ping-timeout: 45
snap-max-soft-limit: 100
snap-activate-on-create: enable


2. Mount volume on the system:

127.0.0.1:/volume-0001 on /samba/volume-0001 type fuse.glusterfs (rw,allow_other,max_read=131072)


3. Create "test.txt" file on the volume:

# cat /samba/volume-0001/cifs_share/test.txt
Mon Jan 25 16:36:15 PST 2016


4. Create snapshot:

# gluster snapshot create ss1 volume-0001
snapshot create: success: Snap ss1 created successfully

# gluster snapshot info
Snapshot                  : ss1
Snap UUID                 : dea7b9f2-df0f-4090-abba-bb43ac6076d4
Created                   : 2016-01-25 16:43:48
Snap Volumes:

        Snap Volume Name          : 099554820e594220ac95ab0f88f30e0e
        Origin Volume name        : volume-0001
        Snaps taken for volume-0001      : 1
        Snaps available for volume-0001  : 62
        Status                    : Started


5. Mount snapshot:

# mount -t glusterfs 10.10.60.182:/snaps/ss1/volume-0001 /tmp/ss1_mount

6. Read file "test.txt":

Actual results:

# cat /tmp/ss1_mount/cifs_share/test.txt
cat: /tmp/ss1_mount/cifs_share/test.txt: Read-only file system


Expected results:

It has to be the same result as for volume:

# cat /samba/volume-0001/cifs_share/test.txt
Mon Jan 25 16:36:15 PST 2016
 

Additional info:

We were able to fix this problem by modifying snapshot trusted vol file by re-arranging order of translators.

The original vol file:

# cat /var/lib/glusterd/snaps/ss1/099554820e594220ac95ab0f88f30e0e/trusted-099554820e594220ac95ab0f88f30e0e.tcp-fuse.vol

volume volume-0001-client-0
    type protocol/client
    option send-gids true
    option password f3f25e66-de49-43d3-b209-8d110e2cf814
    option username 76b1b06f-0eda-4f0a-8733-0e994ed1a92f
    option transport-type tcp
    option remote-subvolume /var/run/gluster/snaps/099554820e594220ac95ab0f88f30e0e/brick1/volume-0001
    option remote-host 10.10.60.182
    option ping-timeout 45
end-volume

volume 099554820e594220ac95ab0f88f30e0e-dht
    type cluster/distribute
    subvolumes volume-0001-client-0
end-volume

volume 099554820e594220ac95ab0f88f30e0e-read-only
    type features/read-only
    subvolumes 099554820e594220ac95ab0f88f30e0e-dht
end-volume

volume 099554820e594220ac95ab0f88f30e0e-crypt
    type encryption/crypt
    option block-size 4096
    option data-key-size 256
    option master-key /var/tmp/volume-0001.gmek
    subvolumes 099554820e594220ac95ab0f88f30e0e-read-only
end-volume

volume 099554820e594220ac95ab0f88f30e0e-read-ahead
    type performance/read-ahead
    subvolumes 099554820e594220ac95ab0f88f30e0e-crypt
end-volume

volume 099554820e594220ac95ab0f88f30e0e-io-cache
    type performance/io-cache
    subvolumes 099554820e594220ac95ab0f88f30e0e-read-ahead
end-volume

volume 099554820e594220ac95ab0f88f30e0e
    type debug/io-stats
    option count-fop-hits off
    option latency-measurement off
    subvolumes 099554820e594220ac95ab0f88f30e0e-io-cache
end-volume

The modified vol file: 

# cat /var/lib/glusterd/snaps/ss1/099554820e594220ac95ab0f88f30e0e/trusted-099554820e594220ac95ab0f88f30e0e.tcp-fuse.vol
volume volume-0001-client-0
    type protocol/client
    option send-gids true
    option password f3f25e66-de49-43d3-b209-8d110e2cf814
    option username 76b1b06f-0eda-4f0a-8733-0e994ed1a92f
    option transport-type tcp
    option remote-subvolume /var/run/gluster/snaps/099554820e594220ac95ab0f88f30e0e/brick1/volume-0001
    option remote-host 10.10.60.182
    option ping-timeout 45
end-volume

volume 099554820e594220ac95ab0f88f30e0e-dht
    type cluster/distribute
    subvolumes volume-0001-client-0
end-volume

volume 099554820e594220ac95ab0f88f30e0e-crypt
    type encryption/crypt
    option block-size 4096
    option data-key-size 256
    option master-key /var/tmp/volume-0001.gmek
    subvolumes 099554820e594220ac95ab0f88f30e0e-dht
end-volume

volume 099554820e594220ac95ab0f88f30e0e-read-only
    type features/read-only
    subvolumes 099554820e594220ac95ab0f88f30e0e-crypt
end-volume

volume 099554820e594220ac95ab0f88f30e0e-read-ahead
    type performance/read-ahead
    subvolumes 099554820e594220ac95ab0f88f30e0e-read-only
end-volume

volume 099554820e594220ac95ab0f88f30e0e-io-cache
    type performance/io-cache
    subvolumes 099554820e594220ac95ab0f88f30e0e-read-ahead
end-volume

volume 099554820e594220ac95ab0f88f30e0e
    type debug/io-stats
    option count-fop-hits off
    option latency-measurement off
    subvolumes
    099554820e594220ac95ab0f88f30e0e-io-cache
end-volume

After glusterd was stopped and started was able to read snapshot files:

# cat /tmp/ss1_mount/cifs_share/test.txt
Mon Jan 25 16:36:15 PST 2016

But failed to modify snapshot:

# cp test2.txt /tmp/ss1_mount/cifs_share/
cp: cannot create regular file `/tmp/ss1_mount/cifs_share/test2.txt': Read-only file system

The failure is expected result.