Bug 1667530

Hi,

Can you tell me what filesystem the source image is on?
Unfortunately, I don't know openstack too well, so do the two paths you show (/var/lib/glance/images/9d3a37b7-2060-41fe-9fbe-063aae72d36e and /var/lib/cinder/conversion/tmpHSGOBO) point to the same image?  (So is one a link to the other?)

Also, of course the obligatory question, can you try with a newer version of qemu-img?  (And, related, does the issue appear if you invoke qemu-img manually?)

Max

Filesystem is xfs.
The content of /var/lib/glance/images/9d3a37b7-2060-41fe-9fbe-063aae72d36e and /var/lib/cinder/conversion/tmpHSGOBO are the same. Just focus on /var/lib/cinder/conversion/tmpHSGOBO. The other one is in glance and it is not the source of the qemu-img. Sorry for confusion. The image from glance is copied to the compute machine.

We have hit the same issue again.
We tried to reproduce the problem on the same machine when qemu-img was frozen and we run it again on the exact same source image, and as destination we generated a new file in the same folder. It took less than 2 minutes the conversion of a 8GB/50GB image (on_disk/virtual size). Meantime the frozen qemu-img process remain frozen.

See SF case 02304013 Comment: #29   for details see also the image details in attachment #22 [details]

I've tried to recreate a qcow2 image with a similar qemu-img map output to the one shown in attachment 22 [details] of the support case, and I couldn't reproduce the issue so far (but I'll continue trying).  I don't think the FS is the issue as the file isn't fragmented much and I don't know of any issues with xfs right now.  Maybe the strace log will help.

So as far as I understand the issue never appeared when running qemu-img directly?  That does not make things easier, unfortunately...

I don't know of any fixes related to this issue, but still, qemu 2.9.0 is 2 years old, so it may be possible that this issue has been fixed in the meantime.  7.6 has 2.12.0, at least (which less than one year old), so is there any possibility to at least test that?

Max

It's not a performance issue. It looks like deadlock. The qemu-img process is not using any I/O or CPU at all. The rest of the machine is idle too (both CPU and disk I/O).
The qemu-img process is waiting for changes on a couple of file descriptor using ppoll() syscall:
ppoll([{fd=4<anon_inode:[signalfd]>, events=POLLIN}, {fd=6<anon_inode:[eventfd]>, events=POLLIN}, {fd=8<anon_inode:[eventfd]>, events=POLLIN}, {fd=9<anon_inode:[eventfd]>, events=POLLIN}], 4, NULL, NULL, 8

Hi,

A core dump sounds interesting, can you tell me where to find it?  (I didn't see any such attachment to the support case, and I couldn't find any core dumps in the SOS report files).
Also, do you know when in the conversion process the hang occurs?  From the file with the image details it looks like it's at the end (because the raw image is shown to contain probably all of the converted data), but maybe the image in there is just the output from the manual conversion.

Thanks,

Max

Hi Max! You can find a core dump in SF case 02299700. Please let me know if you have access to it. I do not know when the hang occurs. It is not happening very often, but when it is happening the only workaround we found so far is to kill qemu-img process that is idle (buth CPU and I/O).

Hi,

Sorry, while I was able to download the core file, I did not really get any info from it.  Pairing it up with a freshly built version of 2.9.0-10 did not work, nor did taking a 2.9.0-16 binary.  Trying to manually dissect the core file did not go very far either...

How often does this issue occur?  I have seen a couple of support cases that go into that direction, although some then seemed to turn out to be something else.
I am asking because I evidently have a hard time reproding it, and it does not seem to be that critical...

Max

I just had a look into the strace provided in SF 02304013 – there are pwrite64() calls starting 5 seconds after invocation for the whole 1.5 minutes until the trace ends.  So that doesn't seem to be a trace from a run where it hung.

Max

The strace was made with similar parameters that the command that was hung, to see if I can reproduce the issue. I couldn't reproduce it. The issue is intermittent, and this is why I think it is a deadlock issue.

I’ve found BZ 1524770 today which looks very much related.  It also has a reproducer (BZ 1524770 comment 11) which however didn’t work for me so far (no hangs in more than 50k iterations).

It too has been reported for 2.9.0-10, and apparently there is a fix for that issue in 2.9.0-16.

Max

Is there anything preventing the customer from updating to the current release of 2.9.0, which is 2.9.0-16, where BZ 1524770 was fixed?

Max

I have finally been able to get an environment with 2.9.0-10 in it (and its debugging symbols), and from it I can see that there are in-flight requests to the target image:

(gdb) p $dst->filename
$18 = "/dev/dm-7", '\000' <repeats 4086 times>
(gdb) p $dst->drv
$19 = (BlockDriver *) 0x5622a47c8460 <bdrv_raw>

(gdb) p $dst->in_flight
$20 = 2
(gdb) p $dst->file->bs->in_flight
$21 = 1

BZ 1524770 comment 13 says the problem is that there is a request in the thread pool that is in THREAD_DONE state.  The same is true here:
(gdb) p $dst.aio_context.thread_pool.head.lh_first.state
$37 = THREAD_DONE

BZ 1508886 (the original BZ) explains further that the associated ThreadPoolCo object still has ret == -115, which stands for EINPROGRESS:
(gdb) p ((ThreadPoolCo *)$dst.aio_context.thread_pool.head.lh_first.common.opaque).ret
$38 = -115

And that the completion BH is not scheduled:
(gdb) p $dst.aio_context.thread_pool.completion_bh.scheduled
$35 = false

(And this is the only request in the thread pool, by the way:
(gdb) p $dst.aio_context.thread_pool.request_list.tqh_first 
$44 = (struct ThreadPoolElement *) 0x0
(gdb) p $dst.aio_context.thread_pool.head.lh_first.all.le_next 
$45 = (struct ThreadPoolElement *) 0x0
)


So it all looks exactly like BZ 1524770 and BZ 1508886 (BZ 1513362 for 2.9.0).  This problem was fixed in 2.9.0-16.el7_4.12, so I’ll close this BZ (as far as I understand, only 2.9.0-10 was reported broken). 

Please reopen this BZ if newer qemu-img versions still have this problem.

Max

*** This bug has been marked as a duplicate of bug 1513362 ***

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days