Bug 1305674
Summary: | ospd-8 poodle 2016-02-04.1 .setUpClass (tempest.api.volume.test_volumes_list.VolumesV2ListTestJSON) failing | ||
---|---|---|---|
Product: | [Community] khaleesi | Reporter: | Steve Linabery <slinaber> |
Component: | rdo-manager | Assignee: | wes hayutin <whayutin> |
Status: | CLOSED WONTFIX | QA Contact: | Attila Darazs <adarazs> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | unspecified | CC: | slinaber, srevivo, whayutin |
Target Milestone: | --- | Keywords: | Automation, AutomationBlocker, ZStream |
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-04-16 16:30:41 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Steve Linabery
2016-02-08 21:00:56 UTC
Please provide log files from cinder api, scheduler, and volume services. The cinder scheduler logs on controller0 show an out of space error, so the root issue here seems to be that the VG used by the Cinder LVM driver doesn't have enough space. It is slightly strange that this failure happens in cinder-volume at lvcreate time rather than being rejected by the scheduler. This implies that the LVM driver is reporting that it has at least 1 GiB of space free but in actuality it has slightly less. Without debug-level logs I can't see what it's actually reporting. But making the VG larger should get this working. 2016-02-08 12:38:56.540 22957 INFO cinder.volume.flows.manager.create_volume [req-84f998ae-62cb-4082-9980-061cb05607c8 52e330bb9ebd4c99858f018e279a141c ea8168a80be74165be08d64e8f891b4b - - -] Volume 404705d6-fdd1-434b-ba50-a04b2e94c36d: being created as raw with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': u'volume-404705d6-fdd1-434b-ba50-a04b2e94c36d'} 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm [req-84f998ae-62cb-4082-9980-061cb05607c8 52e330bb9ebd4c99858f018e279a141c ea8168a80be74165be08d64e8f891b4b - - -] Error creating Volume 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Traceback (most recent call last): 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm File "/usr/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 546, in create_volume 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm run_as_root=True) 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 155, in execute 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm return processutils.execute(*cmd, **kwargs) 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm cmd=sanitized_cmd) 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm ProcessExecutionError: Unexpected error while running command. 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvcreate -n volume-404705d6-fdd1-434b-ba50-a04b2e94c36d cinder-volumes -L 1g 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Exit code: 5 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Stdout: u'' 2016-02-08 12:38:56.759 22957 ERROR cinder.brick.local_dev.lvm Stderr: u' Volume group "cinder-volumes" has insufficient free space (225 extents): 256 required.\n' 225 extents * 4 MB extent size = 900 MB free in the VG. Since another volume was created right before this: 2016-02-08 12:38:54.253 22957 INFO cinder.volume.flows.manager.create_volume [req-15e3f4e3-cb69-4577-86a7-c359da855eae 52e330bb9ebd4c99858f018e279a141c ea8168a80be74165be08d64e8f891b4b - - -] Volume volume-2fd23202-025a-4848-9f0b-19aa1c0e6d71 (2fd23202-025a-4848-9f0b-19aa1c0e6d71): created successfully It's also possible that the driver was reporting between 1 and 2 GB free, tried to handle multiple volume creations close enough together so that the scheduler wouldn't have updated the free space information yet, and the second volume just didn't fit. This is somewhat expected and so maybe my comment about space reporting in the LVM driver above isn't applicable. Suggest just increasing the LVM VG size. Giving the following a whirl.. By default the overcloud nodes are created w/ 40GB of diskspace, bumping it to 50GB to see if that provides more room to create lvm's. https://review.gerrithub.io/#/c/262493/ increased the size of cinder-volume pv, prior to the last run [root@overcloud-controller-0 log]# pvscan PV /dev/loop2 VG cinder-volumes lvm2 [10.00 GiB / 10.00 GiB free] Total: 1 [10.00 GiB] / in use: 1 [10.00 GiB] / in no VG: 0 [0 ] |