Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1775675

Summary: Problem cloning a volume larger than 500GB
Product: Red Hat OpenStack Reporter: Eric Harney <eharney>
Component: openstack-cinderAssignee: Eric Harney <eharney>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: medium Docs Contact: Chuck Copello <ccopello>
Priority: medium    
Version: 15.0 (Stein)Keywords: Triaged, ZStream
Target Milestone: z2   
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-cinder-14.0.3-0.20191203220448.ea67131.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-05 11:54:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eric Harney 2019-11-22 14:53:48 UTC
This bug was initially created as a copy of Bug #1764545

Description of problem:

This happen when the source volume is big enough (more than 500GB). With smaller volumes (~100GB) there's no problem.

The full error and the logs I'll post in a new comment as private, since it contains customer sensitive information.

Version-Release number of selected component (if applicable):

openstack-cinder-12.0.6-3.el7ost.noarch                     Thu Jun 13 11:05:12 2019
openstack-nova-api-17.0.9-9.el7ost.noarch                   Thu Jun 13 11:05:36 2019
openstack-nova-common-17.0.9-9.el7ost.noarch                Thu Jun 13 11:03:27 2019
openstack-nova-compute-17.0.9-9.el7ost.noarch               Thu Jun 13 11:05:07 2019
openstack-nova-conductor-17.0.9-9.el7ost.noarch             Thu Jun 13 11:05:36 2019
openstack-nova-console-17.0.9-9.el7ost.noarch               Thu Jun 13 11:05:36 2019
openstack-nova-migration-17.0.9-9.el7ost.noarch             Thu Jun 13 11:05:07 2019
openstack-nova-novncproxy-17.0.9-9.el7ost.noarch            Thu Jun 13 11:05:36 2019
openstack-nova-placement-api-17.0.9-9.el7ost.noarch         Thu Jun 13 11:05:35 2019
openstack-nova-scheduler-17.0.9-9.el7ost.noarch             Thu Jun 13 11:05:36 2019
puppet-cinder-12.4.1-4.el7ost.noarch                        Thu Jun 13 11:02:56 2019
puppet-nova-12.4.0-17.el7ost.noarch                         Thu Jun 13 11:02:56 2019
python2-cinderclient-3.5.0-1.el7ost.noarch                  Tue Dec 11 17:34:09 2018
python2-novaclient-10.1.0-1.el7ost.noarch                   Tue Dec 11 17:34:09 2018
python-cinder-12.0.6-3.el7ost.noarch                        Thu Jun 13 11:03:41 2019
python-nova-17.0.9-9.el7ost.noarch                          Thu Jun 13 11:03:27 2019


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
This seems to be similar to this bug[1] that was opened for OSP9. They both have similar error:
~~~
2019-10-21 14:25:28.790 76 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
2019-10-21 14:25:28.790 76 ERROR cinder.volume.manager Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=8 -- env LC_ALL=C qemu-img info /var/lib/cinder/mnt/[OMITTED]
2019-10-21 14:25:28.790 76 ERROR cinder.volume.manager Exit code: -9
~~~

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1402594

Comment 2 Tzach Shefi 2020-02-04 06:16:04 UTC
Verified on:
openstack-cinder-14.0.3-0.20191214220454.1d8efc5.el8ost.noarch


Created a 3par iSCSI backed volume, don't have sufficient disk on hosts to test with Cinder over LVM to internal ceph. 


(overcloud) [stack@undercloud-0 ~]$ cinder show afc5ef38-26c1-4bb3-b68e-40e3ad1c5adc
Invalid -W option ignored: invalid action: '"ignore'                                                                                                                             
+--------------------------------+------------------------------------------+                                                                                                    
| Property                       | Value                                    |                                                                                                    
+--------------------------------+------------------------------------------+                                                                                                    
| attached_servers               | ['4e0cdfe4-5da9-4f22-973d-9e6116944bd0'] |                                                                                                    
| attachment_ids                 | ['a984fd17-be27-46dc-b80b-0771bcc07115'] |                                                                                                    
| availability_zone              | nova                                     |                                                                                                    
| bootable                       | false                                    |                                                                                                    
| consistencygroup_id            | None                                     |                                                                                                    
| created_at                     | 2020-02-03T10:04:30.000000               |                                                                                                    
| description                    | None                                     |                                                                                                    
| encrypted                      | False                                    |                                                                                                    
| id                             | afc5ef38-26c1-4bb3-b68e-40e3ad1c5adc     |                                                                                                    
| metadata                       |                                          |                                                                                                    
| migration_status               | None                                     |                                                                                                    
| multiattach                    | False                                    |                                                                                                    
| name                           | 500G_Source_vol                          |                                                                                                    
| os-vol-host-attr:host          | controller-1@3par#SSD_r5                 |                                                                                                    
| os-vol-mig-status-attr:migstat | None                                     |
| os-vol-mig-status-attr:name_id | None                                     |
| os-vol-tenant-attr:tenant_id   | 6bba374211d843ab882fa4866c9fb6a2         |
| replication_status             | None                                     |
| size                           | 500                                      |
| snapshot_id                    | None                                     |
| source_volid                   | None                                     |
| status                         | in-use                                   |
| updated_at                     | 2020-02-03T10:05:16.000000               |
| user_id                        | 2046808535754b3982ea8806432f2117         |
| volume_type                    | 3par                                     |
+--------------------------------+------------------------------------------+


Booted an instance, attached disk to instance.
Mounted disk created ext4 fs, filled it with random data:

# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     253:0    0    1G  0 disk 
|-vda1  253:1    0 1015M  0 part /
`-vda15 253:15   0    8M  0 part 
vdb     253:16   0    1G  0 disk 
|-vdb1  253:17   0   35M  0 part 
`-vdb15 253:31   0    8M  0 part 
vdc     253:32   0  500G  0 disk /root/500
# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev                    240.1M         0    240.1M   0% /dev
/dev/vda1               978.9M     23.9M    914.1M   3% /
tmpfs                   244.2M         0    244.2M   0% /dev/shm
tmpfs                   244.2M     88.0K    244.1M   0% /run
/dev/vdc                492.0G    492.0G         0 100% /root/500

Detach volume from instance and clone volume:
(overcloud) [stack@undercloud-0 ~]$ cinder create 550 --source-volid afc5ef38-26c1-4bb3-b68e-40e3ad1c5adc --volume-type 3par --name Cloned_500G_Vol
Invalid -W option ignored: invalid action: '"ignore'
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2020-02-04T04:13:33.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 70ef3a1d-1e94-4e9b-b95e-238a1aefd1f8 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | Cloned_500G_Vol                      |
| os-vol-host-attr:host          | controller-1@3par#SSD_r5             |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 6bba374211d843ab882fa4866c9fb6a2     |
| replication_status             | None                                 |
| size                           | 550                                  |
| snapshot_id                    | None                                 |
| source_volid                   | afc5ef38-26c1-4bb3-b68e-40e3ad1c5adc |
| status                         | creating                             |
| updated_at                     | 2020-02-04T04:13:34.000000           |
| user_id                        | 2046808535754b3982ea8806432f2117     |
| volume_type                    | 3par                                 |
+--------------------------------+--------------------------------------+

Volume was successfully cloned
(overcloud) [stack@undercloud-0 ~]$ cinder list
Invalid -W option ignored: invalid action: '"ignore'
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name            | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
| 70ef3a1d-1e94-4e9b-b95e-238a1aefd1f8 | available | Cloned_500G_Vol | 550  | 3par        | false    |                                      |
| afc5ef38-26c1-4bb3-b68e-40e3ad1c5adc | available | 500G_Source_vol | 500  | 3par        | false    |                                      |
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+

Attached volume to instance, checking cloned volume we have data. 
Good to verify.

Comment 5 errata-xmlrpc 2020-03-05 11:54:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0712