Bug 1305674

Summary: ospd-8 poodle 2016-02-04.1 .setUpClass (tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON) failing
Product: [Community] khaleesi Reporter: Steve Linabery <slinaber>
Component: rdo-managerAssignee: wes hayutin <whayutin>
Status: CLOSED WONTFIX QA Contact: Attila Darazs <adarazs>
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: slinaber, srevivo, whayutin
Target Milestone: ---Keywords: Automation, AutomationBlocker, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-16 16:30:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Steve Linabery 2016-02-08 21:00:56 UTC
Description of problem:
ospd-8 poodle 2016-02-04.1  .setUpClass (tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON) failing

Stacktrace

Traceback (most recent call last):
testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/stack/tempest/tempest/test.py", line 262, in setUpClass
    six.reraise(etype, value, trace)
  File "/home/stack/tempest/tempest/test.py", line 255, in setUpClass
    cls.resource_setup()
  File "/home/stack/tempest/tempest/api/volume/test_volumes_list.py", line 66, in resource_setup
    volume = cls.create_volume(metadata=cls.metadata)
  File "/home/stack/tempest/tempest/api/volume/base.py", line 116, in create_volume
    cls.volumes_client.wait_for_volume_status(volume['id'], 'available')
  File "/home/stack/tempest/tempest/services/volume/base/base_volumes_client.py", line 153, in wait_for_volume_status
    waiters.wait_for_volume_status(self, volume_id, status)
  File "/home/stack/tempest/tempest/common/waiters.py", line 174, in wait_for_volume_status
    raise exceptions.VolumeBuildErrorException(volume_id=volume_id)
tempest.exceptions.VolumeBuildErrorException: Volume 404705d6-fdd1-434b-ba50-a04b2e94c36d failed to build and is in ERROR status

Comment 1 Eric Harney 2016-02-08 21:02:56 UTC
Please provide log files from cinder api, scheduler, and volume services.

Comment 5 Eric Harney 2016-02-08 21:35:46 UTC
The cinder scheduler logs on controller0 show an out of space error, so the root issue here seems to be that the VG used by the Cinder LVM driver doesn't have enough space.

It is slightly strange that this failure happens in cinder-volume at lvcreate time rather than being rejected by the scheduler.  This implies that the LVM driver is reporting that it has at least 1 GiB of space free but in actuality it has slightly less.

Without debug-level logs I can't see what it's actually reporting.

But making the VG larger should get this working.



2016-02-08 12:38:56.540 22957 INFO cinder.volume.flows.manager.create_volume [req-84f998ae-62cb-4082-9980-061cb05607c8 52e330bb9ebd4c99858f018e279a141c ea8168a80be74165be08d64e8f891b4b - - -] Volume 404705d6-fdd1-434b-ba50-a04b2e94c36d: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-404705d6-fdd1-434b-ba50-a04b2e94c36d'}
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm [req-84f998ae-62cb-4082-9980-061cb05607c8 52e330bb9ebd4c99858f018e279a141c ea8168a80be74165be08d64e8f891b4b - - -] Error creating Volume
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Traceback (most recent call last):
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm   File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 546, in create_volume
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm     run_as_root=True)
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 155, in execute
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm     return processutils.execute(*cmd, **kwargs)
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm     cmd=sanitized_cmd)
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm ProcessExecutionError: Unexpected error while running command.
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-404705d6-fdd1-434b-ba50-a04b2e94c36d cinder-volumes -L 1g
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Exit code: 5
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Stdout: u''
2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Stderr: u'  Volume group "cinder-volumes" has insufficient free space (225 extents): 256 required.\n'

225 extents * 4 MB extent size = 900 MB free in the VG.

Comment 6 Eric Harney 2016-02-08 21:40:06 UTC
Since another volume was created right before this:

2016-02-08 12:38:54.253 22957 INFO cinder.volume.flows.manager.create_volume [req-15e3f4e3-cb69-4577-86a7-c359da855eae 52e330bb9ebd4c99858f018e279a141c ea8168a80be74165be08d64e8f891b4b - - -] Volume volume-2fd23202-025a-4848-9f0b-19aa1c0e6d71 (2fd23202-025a-4848-9f0b-19aa1c0e6d71): created successfully

It's also possible that the driver was reporting between 1 and 2 GB free, tried to handle multiple volume creations close enough together so that the scheduler wouldn't have updated the free space information yet, and the second volume just didn't fit.  This is somewhat expected and so maybe my comment about space reporting in the LVM driver above isn't applicable.

Suggest just increasing the LVM VG size.

Comment 7 wes hayutin 2016-02-09 01:26:59 UTC
Giving the following a whirl..
By default the overcloud nodes are created w/ 40GB of diskspace, bumping it to 50GB to see if that provides more room to create lvm's.


https://review.gerrithub.io/#/c/262493/

Comment 9 wes hayutin 2016-02-10 16:11:33 UTC
increased the size of cinder-volume pv, prior to the last run
[root@overcloud-controller-0 log]# pvscan
  PV /dev/loop2   VG cinder-volumes   lvm2 [10.00 GiB / 10.00 GiB free]
  Total: 1 [10.00 GiB] / in use: 1 [10.00 GiB] / in no VG: 0 [0   ]

Comment 10 wes hayutin 2016-02-10 23:38:24 UTC
https://review.gerrithub.io/#/c/262788/