Bug 996768
Summary: | glusterfs-fuse3.4.0 mount option for read-only not functional on rhel 5.9 | ||
---|---|---|---|
Product: | [Community] GlusterFS | Reporter: | Khoi Mai <khoi.mai2008> |
Component: | access-control | Assignee: | Niels de Vos <ndevos> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | 3.4.0 | CC: | gluster-bugs, ndevos |
Target Milestone: | --- | Keywords: | Reopened |
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | glusterfs-3.4.1 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2013-10-17 13:52:36 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Khoi Mai
2013-08-13 22:13:28 UTC
[2013-08-28 22:11:10.128106] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0 (/usr/sbin/glusterfs --read-only --fuse-mountopts=noexec --volfile-id=/teststatic --volfile-server=omhq1444 --fuse-mountopts=noexec /content) [2013-08-28 22:11:10.129128] E [mount.c:267:gf_fuse_mount] 0-glusterfs-fuse: cannot open /dev/fuse (No such file or directory) [2013-08-28 22:11:10.129172] E [xlator.c:390:xlator_init] 0-fuse: Initialization of volume 'fuse' failed, review your volfile again [2013-08-28 22:11:23.471611] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0 (/usr/sbin/glusterfs --read-only --fuse-mountopts=noexec --volfile-id=/teststatic --volfile-server=omhq1444 --fuse-mountopts=noexec /content) [2013-08-28 22:11:23.491884] I [mount.c:290:gf_fuse_mount] 0-glusterfs-fuse: direct mount failed (Invalid argument), retry to mount via fusermount [2013-08-28 22:11:23.518608] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-08-28 22:11:23.518691] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2013-08-28 22:11:23.528532] I [quota.c:3051:quota_parse_limits] 0-teststatic-quota: /:322122547200 [2013-08-28 22:11:23.528572] I [quota.c:3083:quota_parse_limits] 0-teststatic-quota: /:322122547200 [2013-08-28 22:11:23.534868] I [socket.c:3480:socket_init] 0-teststatic-client-1: SSL support is NOT enabled [2013-08-28 22:11:23.534922] I [socket.c:3495:socket_init] 0-teststatic-client-1: using system polling thread [2013-08-28 22:11:23.536786] I [socket.c:3480:socket_init] 0-teststatic-client-0: SSL support is NOT enabled [2013-08-28 22:11:23.536826] I [socket.c:3495:socket_init] 0-teststatic-client-0: using system polling thread [2013-08-28 22:11:23.536897] I [client.c:2154:notify] 0-teststatic-client-0: parent translators are ready, attempting connect on transport [2013-08-28 22:11:23.540689] I [client.c:2154:notify] 0-teststatic-client-1: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume teststatic-client-0 2: type protocol/client 3: option transport-type tcp 4: option remote-subvolume /static/content 5: option remote-host omhq1439 6: end-volume 7: 8: volume teststatic-client-1 9: type protocol/client 10: option transport-type tcp 11: option remote-subvolume /static/content 12: option remote-host omhq1444 13: end-volume 14: 15: volume teststatic-replicate-0 16: type cluster/replicate 17: subvolumes teststatic-client-0 teststatic-client-1 18: end-volume 19: 20: volume teststatic-dht 21: type cluster/distribute 22: subvolumes teststatic-replicate-0 23: end-volume 24: 25: volume teststatic-quota 26: type features/quota 27: option timeout 0 28: option limit-set /:300GB 29: subvolumes teststatic-dht 30: end-volume 31: 32: volume teststatic-write-behind 33: type performance/write-behind 34: subvolumes teststatic-quota 35: end-volume 36: 37: volume teststatic-read-ahead 38: type performance/read-ahead 39: subvolumes teststatic-write-behind 40: end-volume 41: 42: volume teststatic-io-cache 43: type performance/io-cache 44: subvolumes teststatic-read-ahead 45: end-volume 46: 47: volume teststatic-quick-read 48: type performance/quick-read 49: subvolumes teststatic-io-cache 50: end-volume 51: 52: volume teststatic-open-behind 53: type performance/open-behind 54: subvolumes teststatic-quick-read 55: end-volume 56: 57: volume teststatic-md-cache 58: type performance/md-cache 59: subvolumes teststatic-open-behind 60: end-volume 61: 62: volume teststatic 63: type debug/io-stats 64: option count-fop-hits off 65: option latency-measurement off 66: subvolumes teststatic-md-cache 67: end-volume +------------------------------------------------------------------------------+ [2013-08-28 22:11:23.550673] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-teststatic-client-0: changing port to 49153 (from 0) [2013-08-28 22:11:23.550787] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-teststatic-client-1: changing port to 49153 (from 0) [2013-08-28 22:11:23.550836] W [socket.c:514:__socket_rwv] 0-teststatic-client-1: readv failed (No data available) [2013-08-28 22:11:23.554627] W [socket.c:514:__socket_rwv] 0-teststatic-client-0: readv failed (No data available) [2013-08-28 22:11:23.558276] I [client-handshake.c:1658:select_server_supported_programs] 0-teststatic-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-28 22:11:23.558690] I [client-handshake.c:1456:client_setvolume_cbk] 0-teststatic-client-1: Connected to 167.132.88.215:49153, attached to remote volume '/static/content'. [2013-08-28 22:11:23.558723] I [client-handshake.c:1468:client_setvolume_cbk] 0-teststatic-client-1: Server and Client lk-version numbers are not same, reopening the fds [2013-08-28 22:11:23.558786] I [afr-common.c:3698:afr_notify] 0-teststatic-replicate-0: Subvolume 'teststatic-client-1' came back up; going online. [2013-08-28 22:11:23.559193] I [client-handshake.c:450:client_set_lk_version_cbk] 0-teststatic-client-1: Server lk version = 1 [2013-08-28 22:11:23.565143] I [client-handshake.c:1658:select_server_supported_programs] 0-teststatic-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-28 22:11:23.569577] I [client-handshake.c:1456:client_setvolume_cbk] 0-teststatic-client-0: Connected to 167.132.84.201:49153, attached to remote volume '/static/content'. [2013-08-28 22:11:23.569626] I [client-handshake.c:1468:client_setvolume_cbk] 0-teststatic-client-0: Server and Client lk-version numbers are not same, reopening the fds [2013-08-28 22:11:23.585272] I [fuse-bridge.c:4726:fuse_graph_setup] 0-fuse: switched to graph 0 [2013-08-28 22:11:23.585527] I [client-handshake.c:450:client_set_lk_version_cbk] 0-teststatic-client-0: Server lk version = 1 [2013-08-28 22:11:23.585720] I [fuse-bridge.c:3682:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10 [2013-08-28 22:11:23.586532] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-teststatic-replicate-0: added root inode *** This bug has been marked as a duplicate of bug 980770 *** Can you please tell me what release this is addressed in for rhel5 ? I have [root@vx1590 test]# rpm -qa --last|more glusterfs-fuse-3.4.0-8.el5 Fri 30 Aug 2013 03:45:04 PM CDT glusterfs-3.4.0-8.el5 Fri 30 Aug 2013 03:45:04 PM CDT glusterfs-libs-3.4.0-8.el5 Fri 30 Aug 2013 03:45:03 PM CDT installed on my client running 2.6.18-348.el5xen when i mount up the gluster vol. even though in 'mount -l' it says so STILL. [root@vx1590 glusterfs]# mount -t glusterfs -o ro omhq1439:/teststatic /mnt/test I am not seeing the expected results of my mount options being honored. [root@vx1590 glusterfs]# mount -l /dev/mapper/Vol00_vx1590-root on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) . . . . /dev/xvda1 on /boot type ext3 (rw) [/boot] tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) . . . glusterfs#omhq1444:/teststatic on /content type fuse (ro,noexec,nosuid,nodev,default_permissions,allow_other,max_read=131072) glusterfs#omhq1439:/teststatic on /mnt/test type fuse (ro,default_permissions,allow_other,max_read=131072) [root@vx1590 glusterfs]# cd /mnt/test [root@vx1590 test]# touch junk [root@vx1590 test]# rm -f junk [root@vx1590 test]# Please help me understand if I am procedurally doing something wrong. ------------------------------------------------------------ [root@vx1590 glusterfs]# cat mnt-test.log [2013-08-30 20:59:02.016742] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0 (/usr/sbin/glusterfs --read-only --volfile-id=/teststatic --volfile-server=omhq1444 /mnt/test) [2013-08-30 20:59:02.045388] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-08-30 20:59:02.045478] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2013-08-30 20:59:02.055369] I [quota.c:3051:quota_parse_limits] 0-teststatic-quota: /:322122547200 [2013-08-30 20:59:02.055407] I [quota.c:3083:quota_parse_limits] 0-teststatic-quota: /:322122547200 [2013-08-30 20:59:02.060918] I [socket.c:3480:socket_init] 0-teststatic-client-1: SSL support is NOT enabled [2013-08-30 20:59:02.060977] I [socket.c:3495:socket_init] 0-teststatic-client-1: using system polling thread [2013-08-30 20:59:02.062871] I [socket.c:3480:socket_init] 0-teststatic-client-0: SSL support is NOT enabled [2013-08-30 20:59:02.062925] I [socket.c:3495:socket_init] 0-teststatic-client-0: using system polling thread [2013-08-30 20:59:02.063008] I [client.c:2154:notify] 0-teststatic-client-0: parent translators are ready, attempting connect on transport [2013-08-30 20:59:02.066699] I [client.c:2154:notify] 0-teststatic-client-1: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume teststatic-client-0 2: type protocol/client 3: option transport-type tcp 4: option remote-subvolume /static/content 5: option remote-host omhq1439 6: end-volume 7: 8: volume teststatic-client-1 9: type protocol/client 10: option transport-type tcp 11: option remote-subvolume /static/content 12: option remote-host omhq1444 13: end-volume 14: 15: volume teststatic-replicate-0 16: type cluster/replicate 17: subvolumes teststatic-client-0 teststatic-client-1 18: end-volume 19: 20: volume teststatic-dht 21: type cluster/distribute 22: subvolumes teststatic-replicate-0 23: end-volume 24: 25: volume teststatic-quota 26: type features/quota 27: option timeout 0 28: option limit-set /:300GB 29: subvolumes teststatic-dht 30: end-volume 31: 32: volume teststatic-write-behind 33: type performance/write-behind 34: subvolumes teststatic-quota 35: end-volume 36: 37: volume teststatic-read-ahead 38: type performance/read-ahead 39: subvolumes teststatic-write-behind 40: end-volume 41: 42: volume teststatic-io-cache 43: type performance/io-cache 44: subvolumes teststatic-read-ahead 45: end-volume 46: 47: volume teststatic-quick-read 48: type performance/quick-read 49: subvolumes teststatic-io-cache 50: end-volume 51: 52: volume teststatic-open-behind 53: type performance/open-behind 54: subvolumes teststatic-quick-read 55: end-volume 56: 57: volume teststatic-md-cache 58: type performance/md-cache 59: subvolumes teststatic-open-behind 60: end-volume 61: 62: volume teststatic 63: type debug/io-stats 64: option count-fop-hits off 65: option latency-measurement off 66: subvolumes teststatic-md-cache 67: end-volume +------------------------------------------------------------------------------+ [2013-08-30 20:59:02.075714] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-teststatic-client-1: changing port to 49153 (from 0) [2013-08-30 20:59:02.075795] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-teststatic-client-0: changing port to 49153 (from 0) [2013-08-30 20:59:02.075841] W [socket.c:514:__socket_rwv] 0-teststatic-client-0: readv failed (No data available) [2013-08-30 20:59:02.079251] W [socket.c:514:__socket_rwv] 0-teststatic-client-1: readv failed (No data available) [2013-08-30 20:59:02.082755] I [client-handshake.c:1658:select_server_supported_programs] 0-teststatic-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-30 20:59:02.082956] I [client-handshake.c:1658:select_server_supported_programs] 0-teststatic-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-30 20:59:02.083164] I [client-handshake.c:1456:client_setvolume_cbk] 0-teststatic-client-0: Connected to 167.132.84.201:49153, attached to remote volume '/static/content'. [2013-08-30 20:59:02.083194] I [client-handshake.c:1468:client_setvolume_cbk] 0-teststatic-client-0: Server and Client lk-version numbers are not same, reopening the fds [2013-08-30 20:59:02.083256] I [afr-common.c:3698:afr_notify] 0-teststatic-replicate-0: Subvolume 'teststatic-client-0' came back up; going online. [2013-08-30 20:59:02.083325] I [client-handshake.c:1456:client_setvolume_cbk] 0-teststatic-client-1: Connected to 167.132.88.215:49153, attached to remote volume '/static/content'. [2013-08-30 20:59:02.083353] I [client-handshake.c:1468:client_setvolume_cbk] 0-teststatic-client-1: Server and Client lk-version numbers are not same, reopening the fds [2013-08-30 20:59:02.096557] I [fuse-bridge.c:4726:fuse_graph_setup] 0-fuse: switched to graph 0 [2013-08-30 20:59:02.096773] I [client-handshake.c:450:client_set_lk_version_cbk] 0-teststatic-client-0: Server lk version = 1 [2013-08-30 20:59:02.096819] I [client-handshake.c:450:client_set_lk_version_cbk] 0-teststatic-client-1: Server lk version = 1 [2013-08-30 20:59:02.097039] I [fuse-bridge.c:3682:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10 [2013-08-30 20:59:02.097757] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-teststatic-replicate-0: added root inode [2013-08-30 21:02:11.877215] I [fuse-bridge.c:4585:fuse_thread_proc] 0-fuse: unmounting /mnt/test [2013-08-30 21:02:11.879071] W [glusterfsd.c:1002:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3222ad503d] (-->/lib64/libpthread.so.0 [0x322360683d] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x138) [0x405328]))) 0-: received signum (15), shutting down [2013-08-30 21:02:11.879113] I [fuse-bridge.c:5217:fini] 0-fuse: Unmounting '/mnt/test'. [2013-08-30 21:02:30.128758] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0 (/usr/sbin/glusterfs --read-only --volfile-id=/teststatic --volfile-server=omhq1439 /mnt/test) [2013-08-30 21:02:30.158042] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-08-30 21:02:30.158132] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2013-08-30 21:02:30.208321] I [quota.c:3051:quota_parse_limits] 0-teststatic-quota: /:322122547200 [2013-08-30 21:02:30.208365] I [quota.c:3083:quota_parse_limits] 0-teststatic-quota: /:322122547200 [2013-08-30 21:02:30.214084] I [socket.c:3480:socket_init] 0-teststatic-client-1: SSL support is NOT enabled [2013-08-30 21:02:30.214134] I [socket.c:3495:socket_init] 0-teststatic-client-1: using system polling thread [2013-08-30 21:02:30.215762] I [socket.c:3480:socket_init] 0-teststatic-client-0: SSL support is NOT enabled [2013-08-30 21:02:30.215801] I [socket.c:3495:socket_init] 0-teststatic-client-0: using system polling thread [2013-08-30 21:02:30.215892] I [client.c:2154:notify] 0-teststatic-client-0: parent translators are ready, attempting connect on transport [2013-08-30 21:02:30.220173] I [client.c:2154:notify] 0-teststatic-client-1: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume teststatic-client-0 2: type protocol/client 3: option transport-type tcp 4: option remote-subvolume /static/content 5: option remote-host omhq1439 6: end-volume 7: 8: volume teststatic-client-1 9: type protocol/client 10: option transport-type tcp 11: option remote-subvolume /static/content 12: option remote-host omhq1444 13: end-volume 14: 15: volume teststatic-replicate-0 16: type cluster/replicate 17: subvolumes teststatic-client-0 teststatic-client-1 18: end-volume 19: 20: volume teststatic-dht 21: type cluster/distribute 22: subvolumes teststatic-replicate-0 23: end-volume 24: 25: volume teststatic-quota 26: type features/quota 27: option timeout 0 28: option limit-set /:300GB 29: subvolumes teststatic-dht 30: end-volume 31: 32: volume teststatic-write-behind 33: type performance/write-behind 34: subvolumes teststatic-quota 35: end-volume 36: 37: volume teststatic-read-ahead 38: type performance/read-ahead 39: subvolumes teststatic-write-behind 40: end-volume 41: 42: volume teststatic-io-cache 43: type performance/io-cache 44: subvolumes teststatic-read-ahead 45: end-volume 46: 47: volume teststatic-quick-read 48: type performance/quick-read 49: subvolumes teststatic-io-cache 50: end-volume 51: 52: volume teststatic-open-behind 53: type performance/open-behind 54: subvolumes teststatic-quick-read 55: end-volume 56: 57: volume teststatic-md-cache 58: type performance/md-cache 59: subvolumes teststatic-open-behind 60: end-volume 61: 62: volume teststatic 63: type debug/io-stats 64: option count-fop-hits off 65: option latency-measurement off 66: subvolumes teststatic-md-cache 67: end-volume +------------------------------------------------------------------------------+ [2013-08-30 21:02:30.229435] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-teststatic-client-1: changing port to 49153 (from 0) [2013-08-30 21:02:30.229503] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-teststatic-client-0: changing port to 49153 (from 0) [2013-08-30 21:02:30.229550] W [socket.c:514:__socket_rwv] 0-teststatic-client-0: readv failed (No data available) [2013-08-30 21:02:30.233343] W [socket.c:514:__socket_rwv] 0-teststatic-client-1: readv failed (No data available) [2013-08-30 21:02:30.236871] I [client-handshake.c:1658:select_server_supported_programs] 0-teststatic-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-30 21:02:30.237219] I [client-handshake.c:1658:select_server_supported_programs] 0-teststatic-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2013-08-30 21:02:30.237331] I [client-handshake.c:1456:client_setvolume_cbk] 0-teststatic-client-0: Connected to 167.132.84.201:49153, attached to remote volume '/static/content'. [2013-08-30 21:02:30.237362] I [client-handshake.c:1468:client_setvolume_cbk] 0-teststatic-client-0: Server and Client lk-version numbers are not same, reopening the fds [2013-08-30 21:02:30.237430] I [afr-common.c:3698:afr_notify] 0-teststatic-replicate-0: Subvolume 'teststatic-client-0' came back up; going online. [2013-08-30 21:02:30.237634] I [client-handshake.c:450:client_set_lk_version_cbk] 0-teststatic-client-0: Server lk version = 1 [2013-08-30 21:02:30.237678] I [client-handshake.c:1456:client_setvolume_cbk] 0-teststatic-client-1: Connected to 167.132.88.215:49153, attached to remote volume '/static/content'. [2013-08-30 21:02:30.237706] I [client-handshake.c:1468:client_setvolume_cbk] 0-teststatic-client-1: Server and Client lk-version numbers are not same, reopening the fds [2013-08-30 21:02:30.252247] I [fuse-bridge.c:4726:fuse_graph_setup] 0-fuse: switched to graph 0 [2013-08-30 21:02:30.252505] I [client-handshake.c:450:client_set_lk_version_cbk] 0-teststatic-client-1: Server lk version = 1 [2013-08-30 21:02:30.252766] I [fuse-bridge.c:3682:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10 [2013-08-30 21:02:30.253535] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-teststatic-replicate-0: added root inode [root@vx1590 glusterfs]# (In reply to Khoi Mai from comment #3) > Can you please tell me what release this is addressed in for rhel5 ? You are correct. The bugfix has been included in the upstream master branch, but not in release-3.4. If you compile the sources from the git repository, the read-only functionality should work on kernel 2.6.18 too. I'll post this patch for inclusion in a 3.4.x minor update to prevent further confusion. REVIEW: http://review.gluster.org/5752 (fuse: pass the mountflags to older (< 2.6.21) kernels) posted (#1) for review on release-3.4 by Niels de Vos (ndevos) COMMIT: http://review.gluster.org/5752 committed in release-3.4 by Vijay Bellur (vbellur) ------ commit 9032b1dfbbf8d0bbf8731959ad1048f7bbc530a3 Author: Niels de Vos <ndevos> Date: Mon Sep 2 09:48:51 2013 +0200 fuse: pass the mountflags to older (< 2.6.21) kernels The change for Bug 853895 fixed mounting volumes read-only for recent kernels. Older kernels fail the first mount() syscall, and the second mount() did not add the 'mountflags'. Full analysis and a description for reproducing is in the bugreport. The test included in http://review.gluster.org/4163 would have caught this problem when the tests are executed on RHEL-5 or similar systems. Change-Id: Ifa6cd6476bf1939e009142ed4ed1b53b9c3171bd Master-Branch-Change-ID: I440591344a6a5af7b2018e37a2a1fda9de8b5ab2 Bug: 996768 Signed-off-by: Niels de Vos <ndevos> Reviewed-on: http://review.gluster.org/5752 Reviewed-by: Shishir Gowda <sgowda> Reviewed-by: Raghavendra Bhat <raghavendra> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur> |