Hide Forgot
Created attachment 818439 [details] Error 500 During Volume Create Description of problem: While creating volumes that consisted of 8-servers with each server having 8-bricks, the following error occurred when the volume creation time reached 2 minutes: Error while executing action: A Request to the Server failed with the following Status Code: 500 Version-Release number of selected component (if applicable): Big Bend How reproducible: Steps to Reproduce: 1. Create volume with 8 servers each server having 8-bricks 2. 3. Actual results: Expected results: Additional info:
Note, when the Error-500 does occur, the volume appears to still be created successfully (volume is present with expected bricks).
From the logs, it looks like the webpage has timed out, though the commands at the backend executed successfully. The Create volume request has taken around 3 mins to complete 2013-11-01 09:59:59,736 INFO [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] Running command: CreateGlusterVolumeCommand internal: false. Entities affected : ID: 73074451-92fb-47aa-8ab2-4dd7280035c6 Type: VdsGroups 2013-11-01 09:59:59,737 INFO [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] START, CreateGlusterVolumeVDSCommand(HostName = 10.16.159.123, HostId = 1c8d46f8-804d-4fe4-b22c-40ff83428754), log id: 175e26a9 2013-11-01 10:00:39,298 INFO [org.ovirt.engine.core.vdsbroker.gluster.CreateGlusterVolumeVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] FINISH, CreateGlusterVolumeVDSCommand, return: org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3cd755f, log id: 175e26a9 2013-11-01 10:00:39,461 INFO [org.ovirt.engine.core.bll.gluster.SetGlusterVolumeOptionCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] Running command: SetGlusterVolumeOptionCommand internal: true. Entities affected : ID: 16eca387-1b44-42ae-8b00-1c1668436a0b Type: GlusterVolume 2013-11-01 10:00:39,461 INFO [org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] START, SetGlusterVolumeOptionVDSCommand(HostName = 10.16.159.212, HostId = 2d18df5f-0b67-40d1-9716-18ea71d792bf), log id: 67880aa0 2013-11-01 10:01:12,848 INFO [org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] FINISH, SetGlusterVolumeOptionVDSCommand, log id: 67880aa0 2013-11-01 10:01:12,897 INFO [org.ovirt.engine.core.bll.gluster.SetGlusterVolumeOptionCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] Running command: SetGlusterVolumeOptionCommand internal: true. Entities affected : ID: 16eca387-1b44-42ae-8b00-1c1668436a0b Type: GlusterVolume 2013-11-01 10:01:12,898 INFO [org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] START, SetGlusterVolumeOptionVDSCommand(HostName = 10.16.159.18, HostId = 2c8332bf-2840-43f3-b79d-b7ae0025b584), log id: 2220c459 2013-11-01 10:01:44,113 INFO [org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] FINISH, SetGlusterVolumeOptionVDSCommand, log id: 2220c459 2013-11-01 10:01:44,161 INFO [org.ovirt.engine.core.bll.gluster.SetGlusterVolumeOptionCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] Running command: SetGlusterVolumeOptionCommand internal: true. Entities affected : ID: 16eca387-1b44-42ae-8b00-1c1668436a0b Type: GlusterVolume 2013-11-01 10:01:44,162 INFO [org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] START, SetGlusterVolumeOptionVDSCommand(HostName = 10.16.159.74, HostId = 0d8c82c0-debf-4133-a52a-6d85cd00a3e5), log id: 7d159e2c 2013-11-01 10:02:14,922 INFO [org.ovirt.engine.core.vdsbroker.gluster.SetGlusterVolumeOptionVDSCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] FINISH, SetGlusterVolumeOptionVDSCommand, log id: 7d159e2c 2013-11-01 10:02:14,929 INFO [org.ovirt.engine.core.bll.gluster.CreateGlusterVolumeCommand] (ajp-/127.0.0.1:8702-11) [10388f6a] Lock freed to object EngineLock [exclusiveLocks= key: 73074451-92fb-47aa-8ab2-4dd7280035c6 value: GLUSTER , sharedLocks= ]
Matt, as discussed, pls. put all the relevant context ( the perf setup and how many nodes, etc ) to this bug, so that it would help root-cause the problem.
This bug is due to a timeout value specified in apache conf. From the apache error log: [Wed Nov 27 01:38:08 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header One way to fix this is to increase the time out in apache conf.
Verified cb12.
Please review the edited DocText and signoff.
Change suggested.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0208.html