Bug 1362014 - 2.0: rbd bench-write: segmentation fault when value of "--io-size" is greater than or equal to image size
Summary: 2.0: rbd bench-write: segmentation fault when value of "--io-size" is greater...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD
Version: 2.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: 3.0
Assignee: Jason Dillaman
QA Contact: Vasishta
Bara Ancincova
URL:
Whiteboard:
Depends On:
Blocks: 1322504 1383917 1412948 1494421
TreeView+ depends on / blocked
 
Reported: 2016-08-01 07:54 UTC by Harish NV Rao
Modified: 2017-12-05 23:31 UTC (History)
7 users (show)

Fixed In Version: RHEL: ceph-12.1.2-1.el7cp Ubuntu: ceph_12.1.2-2redhat1xenial
Doc Type: Bug Fix
Doc Text:
.The `rbd bench write` command no longer fails when `--io-size` is equal to the image size Previously, the `rbd bench-write --io-size <size> <image>` command failed with a segmentation fault if the size specified by the `--io-size` option was greater than 4 GB. With this update, the option is restricted from being too large.
Clone Of:
Environment:
Last Closed: 2017-12-05 23:31:14 UTC


Attachments (Terms of Use)
bt (17.99 KB, text/plain)
2016-08-01 07:54 UTC, Harish NV Rao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3387 normal SHIPPED_LIVE Red Hat Ceph Storage 3.0 bug fix and enhancement update 2017-12-06 03:03:45 UTC
Ceph Project Bug Tracker 16967 None None None 2016-08-09 00:28:18 UTC
Ceph Project Bug Tracker 18422 None None None 2017-01-04 21:36:36 UTC

Description Harish NV Rao 2016-08-01 07:54:14 UTC
Created attachment 1186276 [details]
bt

Description of problem:

 rbd bench-write: segmentation fault when value of "--io-size" is greater than or equal to image size.

Version-Release number of selected component (if applicable): ceph version 10.2.2-26.el7cp 


How reproducible: always


Steps to Reproduce:
1. create an image of size 100M
  --cmd: rbd  create test_rbd/cephRBD --size 100M

2. start 'rbd bench-write' on this image with '--io-size' as 101M
  --cmd: rbd --cluster ceph  bench-write test_rbd/cephRBD --io-size 101M --io-threads 3 --io-total 1M --io-pattern rand
  --result: segmentation fault seen

3. create an image of size 4096M

4. start 'rbd bench-write' on this image with '--io-size' as 4096M
  --cmd: rbd --cluster ceph  bench-write -p test_rbd --image test0 --io-size 4096M --io-threads 3 --io-total 1M --io-pattern rand
  --result: segmentation fault seen

Actual results:
segmentation fault observed after executing step 2 and 4.

Expected results:


Additional info:
1) Please see the attachment with bt for both the cases.
2) Image sizes:
[root@magna105 ubuntu]# rbd info test_rbd/test0
rbd image 'test0':
	size 4096 MB in 1024 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.658322ae8944a
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags: 
[root@magna105 ubuntu]# rbd info test_rbd/cephRBD
rbd image 'cephRBD':
	size 102400 kB in 25 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.6bb7e238e1f29
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	flags:

Comment 4 Jason Dillaman 2016-08-12 12:29:30 UTC
Upstream pull request: https://github.com/ceph/ceph/pull/10708

Comment 9 Vasishta 2016-10-07 12:32:13 UTC
Still facing this issue whenever io size and image size are greater than or equal to 4096 and io size is lesser than or equal to chosen image size. 
( 4096 =< io_size =< image_size )

Regards,
Vasishta

Comment 10 Jason Dillaman 2016-10-07 12:50:53 UTC
Please provide the exact command you executed and the exact result you witnessed. What you are describing doesn't match the bug (which was a crash when io_size was *greater* than the image size).

Comment 11 Vasishta 2016-10-07 14:13:10 UTC
Hi Jason,

As it is mentioned in Comment 0 that
'segmentation fault when value of "--io-size" is greater than or equal to image size', both cases (1. greater than and 2.equal) have been tried.

1. When io_size is greater than image_size - 

Error message is getting displayed, saying this is not possible


2. When io_size is equal to image_size -

   a. If image_size & io_size is lesser than 4096


sudo rbd --cluster ceph  bench-write images/img2 --io-size 4096M --io-threads 3 --io-total 100M --io-pattern rand

Comment 12 Vasishta 2016-10-07 14:36:27 UTC
Sorry Jason, 

Please ignore Comment 11. It is incomplete.
I'll provide full info shortly.


Regards,
Vasishta

Comment 13 Jason Dillaman 2016-10-07 16:51:37 UTC
Ack -- so comment 9 should read something along the lines of the following:

rbd bench-write will still crash if the io_size is *equal to* the image size so long as the image_size/io_size is less than X.

The original wording stated that it was crashing as long as io_size was less than the image size, which is a major issue if true.

Given that this is a small corner case, I am going to move this to the next release.

Comment 15 Jason Dillaman 2017-01-04 21:38:20 UTC
The actual issue is that the "--io-size=4096M" (i.e. 4GB io size) is leading to memory corruption due to overflow. The "rbd bench-write" command actually works properly if "--io-size" is equal to the image size so long as it doesn't pass the 4GB size boundary.

Comment 17 Vasishta 2017-09-12 15:38:11 UTC
Hi all, 

Couldn't complete Comment 11, Sorry for that.

As Jason has mentioned in Comment 15, Seg Fault was occurring whenever io-size was >= 4096, if image-size was >= 4096.
i,e image_size >= io-size >= 4096 (After the first fix) 

Now io-size has been capped to 4095M, irrespective of image size. 

$ sudo rbd bench  data/im2 --io-type write --io-size 4096M --io-threads 3 --io-total 100M --io-pattern rand --cluster 12_luminous
rbd: io-size should be less than 4G
bench failed: (22) Invalid argument

$ sudo rbd bench  data/im2 --io-type write --io-size 10240M --io-threads 3 --io-total 100M --io-pattern rand --cluster 12_luminous
rbd: io-size should be less than 4G
bench failed: (22) Invalid argument

$ sudo rbd bench  data/im3 --io-type write --io-size 5000M --io-threads 3 --io-total 100M --io-pattern rand --cluster 12_luminous
rbd: io-size 5000 MB larger than image size 102400 kB
bench failed: (22) Invalid argument

Moving to VERIFIED state.

Regards,
Vasishta

Comment 24 errata-xmlrpc 2017-12-05 23:31:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3387


Note You need to log in before you can comment on or make changes to this bug.