Bug 2080458

Summary: Podman volume plugin timeout should be configurable
Product: Red Hat Enterprise Linux 8 Reporter: mangesh.panche
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Yuhui Jiang <yujiang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.6CC: bbaude, dornelas, dwalsh, jaykim, jligon, jnovy, lsm5, mamccoma, mheon, mrussell, pthomas, rmanes, toneata, tsweeney, umohnani, ypu
Target Milestone: rcKeywords: FutureFeature, Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-4.3.1-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2124676 2124952 2132814 2132992 2132993 (view as bug list) Environment:
Last Closed: 2023-05-16 08:18:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2124676, 2124952, 2132814, 2132992, 2132993    

Description mangesh.panche 2022-04-29 17:00:26 UTC
Description of problem:
"podman volume create" command times-out with following error, if the volume driver takes more than 5 seconds to create the volume.
 
Error: creating volume "myvol" in plugin myplugin: error sending request to volume plugin myplugin endpoint /VolumeDriver.Create: Post "http://plugin/VolumeDriver.Create": context deadline exceeded (Client.Timeout exceeded while awaiting headers)


Although the volume driver creates the volume successfully, the "podman volume ls" does not list the volume.


Version-Release number of selected component (if applicable):
RHEL 8.6 (Beta)
4.0.0.3

How reproducible:
100%

Steps to Reproduce:
1. Develop a custom volume driver plugin
2. Create volume using the custom driver plugin
# podman volume create --driver myplugin myvol
Error: creating volume "myvol" in plugin myplugin: error sending request to volume plugin myvol endpoint /VolumeDriver.Create: Post "http://plugin/VolumeDriver.Create": context deadline exceeded (Client.Timeout exceeded while awaiting headers)


Actual results:
# podman volume create --driver myplugin myvol
Error: creating volume "myvol" in plugin myplugin: error sending request to volume plugin myvol endpoint /VolumeDriver.Create: Post "http://plugin/VolumeDriver.Create": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Expected results:
# podman volume create --driver myplugin myvol
myvol

Additional info:
The timeout for volume plugin should be made configurable
https://github.com/containers/podman/blob/v4.0.0/libpod/plugin/volume_api.go#L158

Comment 2 Daniel Walsh 2022-05-02 20:36:23 UTC
Charlie could you take a look at this one.

Comment 3 Charlie Doern 2022-05-10 01:28:47 UTC
yes I will look at this today

Comment 4 Charlie Doern 2022-06-01 01:52:29 UTC
sorry, getting to this in the bug fixing sprint this week.

Comment 5 Charlie Doern 2022-06-01 20:12:46 UTC
PR in flight https://github.com/containers/podman/pull/14449

Comment 6 mangesh.panche 2022-07-26 15:59:41 UTC
Will this be available in next release of RHEL 8.6 which is planned for early August?

Comment 7 Tom Sweeney 2022-07-26 19:47:41 UTC
@mangesh.panche as this was a low level RFE, the fix has only been targeted towards the next release RHEL 8.7 and 9.1.  If this needs to be backported to RHEL 8.6, we'll need a business justification.

Setting this PR to Post and Assigning to Jindrich for any further packaging or BZ needs.

Comment 9 mangesh.panche 2022-08-01 19:57:20 UTC
The same issue is seen while starting the container which bind mounts the volume from plugin inside the container. 

ERROR: for myapp Cannot start service myapp: error mounting volume myvol for container c795384ce6822b2589e1e1727b093b699c7250d9b281f0428c4e19b7515f75d3: error sending request to volume plugin veritas endpoint /VolumeDriver.Mount: Post "http://plugin/VolumeDriver.Mount": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Comment 11 mangesh.panche 2022-08-16 21:24:34 UTC
Is there any update on this? 

The specifying timeout during volume create is not sufficient. It currently affects create and mount and may affect other operations as well. There should be specifying the timeout globally through the config file.

Comment 12 Tom Sweeney 2022-08-17 12:30:27 UTC
@mangesh.panche the current status is this is fixed in Podman v4.2 which will be part of RHEL 8.7 and 9.1.  Given the level of the reported bug, we have not planned to backport this fix.  If that is not sufficient, please elevate this bug and contact @dornelas

Comment 13 mangesh.panche 2022-08-17 17:48:06 UTC
@tsweeney  Apart from elevating the priority, the solution of providing timeout during the 'podman volume create' will not help. It affects other APIs as well. So, we need a different solution.

Comment 19 Tom Sweeney 2022-08-18 20:53:56 UTC
Thanks all for the discussion on this.  Given that, I'm going to move this BZ back to "Assigned" and have assigned it to Matt Heon.  I have also removed the RFE tag from the subject to clarify it is not an RFE.

Comment 40 mangesh.panche 2022-10-05 21:35:00 UTC
@tsweeney @mheon Is there any update on this?

Comment 47 Oneata Mircea Teodor 2022-10-07 09:46:28 UTC
*** Bug 2132814 has been marked as a duplicate of this bug. ***

Comment 54 errata-xmlrpc 2023-05-16 08:18:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758

Comment 55 errata-xmlrpc 2023-05-16 08:38:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2758

Comment 56 Red Hat Bugzilla 2023-09-18 04:36:24 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days