Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This BZ is to track updating nfs-ganesha to V5.7 in RHCS/IBM Ceph 7.1.
Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
NFS Ganesha rebased v5.7 is available in downstream builds. Basic sanity was passed on this build. Moving this BZ to verified state. [ceph: root@ceph-msaini-rqfpbb-node1-installer /]# rpm -qa | grep nfs libnfsidmap-2.5.4-20.el9.x86_64 nfs-utils-2.5.4-20.el9.x86_64 nfs-ganesha-selinux-5.7-1.el9cp.noarch nfs-ganesha-5.7-1.el9cp.x86_64 nfs-ganesha-rgw-5.7-1.el9cp.x86_64 nfs-ganesha-ceph-5.7-1.el9cp.x86_64 nfs-ganesha-rados-grace-5.7-1.el9cp.x86_64 nfs-ganesha-rados-urls-5.7-1.el9cp.x86_64 [ceph: root@ceph-msaini-rqfpbb-node1-installer /]# ceph --version ceph version 18.2.1-38.el9cp (c709c61e19be87249f04b05dec1e586c1f9dd7b0) reef (stable)
NFS build - 5.7-2 and libntirpc-5.8-1 is updated in latest container images [ceph: root@cali013 /]# rpm -qa | grep nfs libnfsidmap-2.5.4-20.el9.x86_64 nfs-utils-2.5.4-20.el9.x86_64 nfs-ganesha-selinux-5.7-2.el9cp.noarch nfs-ganesha-5.7-2.el9cp.x86_64 nfs-ganesha-rgw-5.7-2.el9cp.x86_64 nfs-ganesha-ceph-5.7-2.el9cp.x86_64 nfs-ganesha-rados-grace-5.7-2.el9cp.x86_64 nfs-ganesha-rados-urls-5.7-2.el9cp.x86_64 # rpm -qa | grep libnt libntirpc-5.8-1.el9cp.x86_64 # ceph --version ceph version 18.2.1-89.el9cp (926619fe7135cbd6d305b46782ee7ecc7be199a3) reef (stable) [ceph: root@cali013 /]# ceph nfs cluster info { "cephfs-nfs": { "backend": [ { "hostname": "cali015", "ip": "10.8.130.15", "port": 12049 }, { "hostname": "cali016", "ip": "10.8.130.16", "port": 12049 } ], "monitor_port": 9049, "port": 2049, "virtual_ip": "10.8.130.236" } } QE will continue running the regression tests on the latest build.Moving this BZ to verified state
Verified this BZ with # ceph --version ceph version 18.2.1-116.el9cp (7709f0c4c90984d791f2f37b5672d6be5a8a6986) reef (stable) # rpm -qa | grep nfs libnfsidmap-2.5.4-20.el9.x86_64 nfs-utils-2.5.4-20.el9.x86_64 nfs-ganesha-selinux-5.7-3.el9cp.noarch nfs-ganesha-5.7-3.el9cp.x86_64 nfs-ganesha-rgw-5.7-3.el9cp.x86_64 nfs-ganesha-ceph-5.7-3.el9cp.x86_64 nfs-ganesha-rados-grace-5.7-3.el9cp.x86_64 nfs-ganesha-rados-urls-5.7-3.el9cp.x86_64 Build - nfs-ganesha-5.7-3.el9cp is present in container image and sanity on the build is passed. Moving this BZ to verified state As mentioned in comment #9, metrics names has also changed with the latest build # HELP exposer_transferred_bytes_total Transferred bytes to metrics services # TYPE exposer_transferred_bytes_total counter exposer_transferred_bytes_total 48139 # HELP exposer_scrapes_total Number of times metrics were scraped # TYPE exposer_scrapes_total counter exposer_scrapes_total 98 # HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds # TYPE exposer_request_latencies summary exposer_request_latencies_count 98 exposer_request_latencies_sum 30939 exposer_request_latencies{quantile="0.5"} 299 exposer_request_latencies{quantile="0.9"} 328 exposer_request_latencies{quantile="0.99"} 328 # HELP nfs_ganesha_mdcache_cache_hits_total Counter for total cache hits in mdcache. # TYPE nfs_ganesha_mdcache_cache_hits_total counter nfs_ganesha_mdcache_cache_hits_total{operation="lookup"} 2 nfs_ganesha_mdcache_cache_hits_total{operation="getattr"} 13 # HELP nfs_ganesha_mdcache_cache_misses_total Counter for total cache misses in mdcache. # TYPE nfs_ganesha_mdcache_cache_misses_total counter nfs_ganesha_mdcache_cache_misses_total{operation="lookup"} 2 nfs_ganesha_mdcache_cache_misses_total{operation="getattr"} 523 # HELP nfs_ganesha_mdcache_cache_misses_by_export_total Counter for total cache misses in mdcache, by export. # TYPE nfs_ganesha_mdcache_cache_misses_by_export_total counter nfs_ganesha_mdcache_cache_misses_by_export_total{export="export_id=3",operation="getattr"} 7 nfs_ganesha_mdcache_cache_misses_by_export_total{export="export_id=2",operation="getattr"} 510 nfs_ganesha_mdcache_cache_misses_by_export_total{export="export_id=1",operation="getattr"} 3 # HELP nfs_ganesha_rpcs_received_total Counter for total RPCs received. # TYPE nfs_ganesha_rpcs_received_total counter nfs_ganesha_rpcs_received_total 423 # HELP nfs_ganesha_rpcs_completed_total Counter for total RPCs completed. # TYPE nfs_ganesha_rpcs_completed_total counter nfs_ganesha_rpcs_completed_total 422 # HELP nfs_ganesha_client_bytes_received_total Total request bytes by client. # TYPE nfs_ganesha_client_bytes_received_total counter nfs_ganesha_client_bytes_received_total{client="10.8.128.221",operation="write"} 0 # HELP nfs_ganesha_client_bytes_sent_total Total response bytes sent by client. # TYPE nfs_ganesha_client_bytes_sent_total counter nfs_ganesha_client_bytes_sent_total{client="10.8.128.221",operation="write"} 253755392 # HELP nfs_ganesha_nfs_bytes_received_total Total request bytes. # TYPE nfs_ganesha_nfs_bytes_received_total counter nfs_ganesha_nfs_bytes_received_total{operation="write"} 0 # HELP nfs_ganesha_nfs_bytes_sent_total Total response bytes. # TYPE nfs_ganesha_nfs_bytes_sent_total counter nfs_ganesha_nfs_bytes_sent_total{operation="write"} 253755392 # HELP nfs_ganesha_nfs_bytes_received_by_export_total Total request bytes by export. # TYPE nfs_ganesha_nfs_bytes_received_by_export_total counter nfs_ganesha_nfs_bytes_received_by_export_total{export="export_id=2",operation="write"} 0 # HELP nfs_ganesha_nfs_bytes_sent_by_export_total Total response bytes by export. # TYPE nfs_ganesha_nfs_bytes_sent_by_export_total counter nfs_ganesha_nfs_bytes_sent_by_export_total{export="export_id=2",operation="write"} 253755392 # HELP nfs_ganesha_rpcs_in_flight Number of NFS requests received or in flight. # TYPE nfs_ganesha_rpcs_in_flight gauge nfs_ganesha_rpcs_in_flight 2 # HELP nfs_ganesha_nfs_request_size_bytes Request size in bytes. # TYPE nfs_ganesha_nfs_request_size_bytes histogram nfs_ganesha_nfs_request_size_bytes_count{operation="write"} 242 nfs_ganesha_nfs_request_size_bytes_sum{operation="write"} 253755392 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="2"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="4"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="8"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="16"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="32"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="64"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="128"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="256"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="512"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="1024"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="2048"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="4096"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="8192"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="16384"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="32768"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="65536"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="131072"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="262144"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="524288"} 0 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="1048576"} 242 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="2097152"} 242 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="4194304"} 242 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="8388608"} 242 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="16777216"} 242 nfs_ganesha_nfs_request_size_bytes_bucket{operation="write",le="+Inf"} 242 # HELP nfs_ganesha_nfs_response_size_bytes Response size in bytes. # TYPE nfs_ganesha_nfs_response_size_bytes histogram nfs_ganesha_nfs_response_size_bytes_count{operation="write"} 242 nfs_ganesha_nfs_response_size_bytes_sum{operation="write"} 253755392 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="2"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="4"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="8"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="16"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="32"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="64"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="128"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="256"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="512"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="1024"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="2048"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="4096"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="8192"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="16384"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="32768"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="65536"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="131072"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="262144"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="524288"} 0 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="1048576"} 242 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="2097152"} 242 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="4194304"} 242 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="8388608"} 242 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="16777216"} 242 nfs_ganesha_nfs_response_size_bytes_bucket{operation="write",le="+Inf"} 242 # HELP nfs_ganesha_nfs_request_size_by_export_bytes Request size by export in bytes. # TYPE nfs_ganesha_nfs_request_size_by_export_bytes histogram nfs_ganesha_nfs_request_size_by_export_bytes_count{export="export_id=2",operation="write"} 242 nfs_ganesha_nfs_request_size_by_export_bytes_sum{export="export_id=2",operation="write"} 253755392 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="2"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="4"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="8"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="16"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="32"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="64"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="128"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="256"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="512"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="1024"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="2048"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="4096"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="8192"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="16384"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="32768"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="65536"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="131072"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="262144"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="524288"} 0 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="1048576"} 242 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="2097152"} 242 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="4194304"} 242 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="8388608"} 242 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="16777216"} 242 nfs_ganesha_nfs_request_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="+Inf"} 242 # HELP nfs_ganesha_nfs_response_size_by_export_bytes Response size by export in bytes. # TYPE nfs_ganesha_nfs_response_size_by_export_bytes histogram nfs_ganesha_nfs_response_size_by_export_bytes_count{export="export_id=2",operation="write"} 242 nfs_ganesha_nfs_response_size_by_export_bytes_sum{export="export_id=2",operation="write"} 253755392 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="2"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="4"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="8"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="16"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="32"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="64"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="128"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="256"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="512"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="1024"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="2048"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="4096"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="8192"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="16384"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="32768"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="65536"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="131072"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="262144"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="524288"} 0 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="1048576"} 242 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="2097152"} 242 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="4194304"} 242 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="8388608"} 242 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="16777216"} 242 nfs_ganesha_nfs_response_size_by_export_bytes_bucket{export="export_id=2",operation="write",le="+Inf"} 242 # HELP nfs_ganesha Information about Ganesha instance. # TYPE nfs_ganesha gauge nfs_ganesha_info{ganesha="V5.7-2"} 1
(In reply to Mohit Bisht from comment #18) > Hi All, > > We have completed SPECstorage performance runs with most recent versions of > Ceph (18.2.1-194) and NFS (5.7-5). > > Summary: > ========= > Average Latency: > ================= > In comparison to the previous version (NFS: 5.7-4), the latest NFS version > (5.7-5) exhibits higher latency across various workloads. > > Overall Throughput: > ==================== > Similarly, the overall throughput with NFS version 5.7-5 has marginally > decrease compared to version 5.7-4 across most workloads. > > > NFS-Performance Dashboard: Link > Guest Cred: guest/ceph-qe > Note: Red Hat VPN access need to access above Grafana dashboard link. > > > Can we move bug to verified? Yes please. Note: Matt thinks the increased latency is within the margin of error. And without paging back in a bunch of statistics to support it, I'm inclined to agree.
Also note that the one patch that V5.7-5 has that was not in V5.7.4 was in V5.6-4. As a downstream only patch it was mistakenly left out of downstream V5.7.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Critical: Red Hat Ceph Storage 7.1 security, enhancements, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:3925
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days