RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1328588 - docker-lvm-plugin never work
Summary: docker-lvm-plugin never work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker-latest
Version: 7.2
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-19 19:10 UTC by Qian Cai
Modified: 2016-05-12 14:55 UTC (History)
5 users (show)

Fixed In Version: docker-latest-1.10.3-22.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-12 14:55:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:1057 0 normal SHIPPED_LIVE new packages: docker-latest 2016-05-12 18:51:24 UTC

Description Qian Cai 2016-04-19 19:10:23 UTC
Description of problem:
Even after fixed the issue in,
https://bugzilla.redhat.com/show_bug.cgi?id=1326374#c11

# /usr/lib/docker-latest/docker-latest-lvm-plugin
FATA[0000] listen unix /run/docker/plugins/lvm.sock: bind: no such file or directory

One workaround is,
# mkdir /run/docker/plugins

Then, according to,
# /usr/lib/systemd/system/docker-latest-lvm-plugin.service
...
EnvironmentFile=-/etc/docker-latest/docker-latest-lvm-plugin
...

# cat /etc/docker-latest/docker-latest-lvm-plugin
vgcreate volume_group_one /dev/vda

# docker-latest volume create -d lvm --name foobar --opt size=0.2G
Error response from daemon: create foobar: VolumeDriver.Create: open /etc/docker/docker-lvm-plugin: no such file or directory

Seems like the configuration file is hard-cored.

Workaround it by using,

# ln -s /etc/docker-latest/docker-latest-lvm-plugin /etc/docker/docker-lvm-plugin

Then, we got this error,
# docker-latest volume create -d lvm --name foobar --opt size=0.2G
Error response from daemon: create foobar: VolumeDriver.Create: Volume group name must be provided for volume creation. Please update the config file /etc/docker/docker-lvm-plugin with volume group name.

Version-Release number of selected component (if applicable):
docker-latest-lvm-plugin-1.10.3-10.el7.x86_64

How reproducible:
always

Comment 2 Alex Jia 2016-04-28 14:16:34 UTC
Lokesh, Daniel, which patch is available for fixing the bug?

Comment 3 Lokesh Mandvekar 2016-04-28 14:58:41 UTC
still to be fixed, let me work on this and update the rpm


btw, caiqian


# cat /etc/docker-latest/docker-latest-lvm-plugin
vgcreate volume_group_one /dev/vda

Is this something that you created? I don't see this installed by default

Comment 4 Qian Cai 2016-04-28 17:15:15 UTC
(In reply to Lokesh Mandvekar from comment #3)
> still to be fixed, let me work on this and update the rpm
> 
> 
> btw, caiqian
> 
> 
> # cat /etc/docker-latest/docker-latest-lvm-plugin
> vgcreate volume_group_one /dev/vda
> 
> Is this something that you created? I don't see this installed by default
Yes.

Comment 5 Antonio Murdaca 2016-05-02 17:33:27 UTC
The binary location is wrong - it should be under /usr/libexec/docker as the other plugins (rhel-push-plugin for instance) 
I also suspect the unit file is wrong in substituting docker-latest (see my patches for rhel-push-plugin) 

Another substitution issue is that the conf file should not be docker-latest prefixed (nor the /etc/docker directory as far as I can tell) 

The plugin should also be running with systemd so the workaround to create the directory where the socket lives is correct and that's the current behavior (since the plugins run with systemd and are socket  actihated) 

Lokesh let me know if you need something else. I'll ping you on IRC and test a patch later today.

Comment 6 Lokesh Mandvekar 2016-05-02 17:42:33 UTC
Yup, all those fixes have been included in docker-latest-1.10.3-19.el7. Moving this to MODIFIED

Comment 7 Qian Cai 2016-05-02 17:57:24 UTC
I am afraid, this is still NOT working.

# rpm -qa | grep docker
docker-forward-journald-1.9.1-38.el7.x86_64
docker-selinux-1.9.1-38.el7.x86_64
docker-common-1.9.1-38.el7.x86_64
docker-rhel-push-plugin-1.10.3-19.el7.x86_64
docker-latest-1.10.3-19.el7.x86_64
docker-lvm-plugin-1.10.3-19.el7.x86_64
docker-1.9.1-38.el7.x86_64
docker-novolume-plugin-1.10.3-19.el7.x86_64

# systemctl start docker-lvm-plugin
# systemctl is-active docker-lvm-plugin
active

# systemctl restart docker-latest
# systemctl is-active docker-latest
active

# ls -l /run/docker/plugins/
total 0
srw-rw-rw-. 1 root root 0 May  2 10:59 docker-novolume-plugin.sock
srw-rw----. 1 root root 0 May  2 13:49 lvm.sock
srw-rw-rw-. 1 root root 0 May  2 10:27 rhel-push-plugin.sock

# cat /etc/docker/docker-lvm-plugin
vgcreate volume_group_one /dev/vdb

# docker-latest volume create -d lvm --name foobar --opt size=0.2G
Error response from daemon: create foobar: VolumeDriver.Create: Volume group name must be provided for volume creation. Please update the config file /etc/docker/docker-lvm-plugin with volume group name.

Comment 8 smahajan@redhat.com 2016-05-02 18:22:06 UTC
the config file `/etc/docker/docker-lvm-plugin` needs to be updated manually,
Did you update the config file with the volume group name ?

Shishir

Comment 9 Qian Cai 2016-05-02 18:31:57 UTC
Yes, I did

# vgcreate volume_group_one /dev/vdb
  Physical volume "/dev/vdb" successfully created
  Volume group "volume_group_one" successfully created

# cat /etc/docker/docker-lvm-plugin
volume_group_one

# docker-latest volume create -d lvm --name foobar --opt size=0.2G
Error response from daemon: create foobar: VolumeDriver.Create: Volume group name must be provided for volume creation. Please update the config file /etc/docker/docker-lvm-plugin with volume group name.

Comment 10 Lokesh Mandvekar 2016-05-02 18:51:57 UTC
i'll be shipping thie config file https://github.com/projectatomic/docker-lvm-plugin/blob/master/etc/docker/docker-lvm-plugin

Comment 11 Lokesh Mandvekar 2016-05-02 18:52:21 UTC
upcoming fix..

Comment 12 smahajan@redhat.com 2016-05-02 18:57:08 UTC
CAI,

I think you are creating and updating the config (/etc/docker/docker-lvm-plugin) file manually.
Which is breaking the format of how the value should be read.

format: VOLUME_GROUP=volume_group_one.
You can change the format, and it should work for now.

But I would suggest, wait for lsm5 fix and use the config file provided in the package.

Shishir

Comment 13 Qian Cai 2016-05-02 20:50:27 UTC
You are right. It is working like a charm now with the new format.

Comment 15 Alex Jia 2016-05-03 11:13:57 UTC
When the storage size is too small(less than 4096 blocks), it should be failed to create volume, but in fact, the volume is created on system.

# rpm -q docker-latest docker-lvm-plugin
docker-latest-1.10.3-21.el7.x86_64
docker-lvm-plugin-1.10.3-21.el7.x86_64

# docker-latest volume create -d lvm --name foobar --opt size=0.01G
Error response from daemon: create foobar: VolumeDriver.Create: agsize (3072 blocks) too small, need at least 4096 blocks
Usage: mkfs.xfs
/* blocksize */		[-b log=n|size=num]
/* metadata */		[-m crc=0|1,finobt=0|1]
/* data subvol */	[-d agcount=n,agsize=n,file,name=xxx,size=num,
			    (sunit=value,swidth=value|su=num,sw=num|noalign),
			    sectlog=n|sectsize=num
/* force overwrite */	[-f]
/* inode size */	[-i log=n|perblock=n|size=num,maxpct=n,attr=0|1|2,
			    projid32bit=0|1]
/* no discard */	[-K]
/* log subvol */	[-l agnum=n,internal,size=num,logdev=xxx,version=n
			    sunit=value|su=num,sectlog=n|sectsize=num,
			    lazy-count=0|1]
/* label */		[-L label (maximum 12 characters)]
/* naming */		[-n log=n|size=num,version=2|ci,ftype=0|1]
/* no-op info only */	[-N]
/* prototype file */	[-p fname]
/* quiet */		[-q]
/* realtime subvol */	[-r extsize=num,size=num,rtdev=xxx]
/* sectorsize */	[-s log=n|size=num]
/* version */		[-V]
			devicename
<devicename> is required unless -d name=xxx is given.
<num> is xxx (bytes), xxxs (sectors), xxxb (fs blocks), xxxk (xxx KiB),
      xxxm (xxx MiB), xxxg (xxx GiB), xxxt (xxx TiB) or xxxp (xxx PiB).
<value> is xxx (512 byte blocks).
[root@devs-1 ~]# lvs
  LV     VG               Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home   rhel             -wi-ao---- 47.45g                                                    
  root   rhel             -wi-ao---- 50.00g                                                    
  swap   rhel             -wi-ao----  2.00g                                                    
  foobar volume_group_one -wi-a----- 12.00m                                                    

# docker-latest volume create -d lvm --name foobar --opt size=0.2G
Error response from daemon: create foobar: VolumeDriver.Create:   Rounding up size to full physical extent 208.00 MiB
  Logical volume "foobar" already exists in volume group "volume_group_one"

# lvremove foobar volume_group_one
  Volume group "foobar" not found
  Cannot process volume group foobar
Do you really want to remove active logical volume foobar? [y/n]: y
  Logical volume "foobar" successfully removed


# docker-latest volume create -d lvm --name foobar --opt size=0.2G
foobar

# lvs
  LV     VG               Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home   rhel             -wi-ao----  47.45g                                                    
  root   rhel             -wi-ao----  50.00g                                                    
  swap   rhel             -wi-ao----   2.00g                                                    
  foobar volume_group_one -wi-a----- 208.00m

Comment 16 Antonio Murdaca 2016-05-03 11:16:39 UTC
That help message there is awful and it's exposing the lower interface. I'll take care of fixing it.

Comment 17 Antonio Murdaca 2016-05-03 13:13:50 UTC
that error is raised because there's no check in the code for the size argument. Going to add one, lvcreate isn't erroring out at create time.

Comment 18 Antonio Murdaca 2016-05-03 13:55:02 UTC
Fix https://github.com/projectatomic/docker-lvm-plugin/pull/6

Comment 19 Antonio Murdaca 2016-05-03 15:05:07 UTC
The PR got merged. We can rebuild.

Comment 20 Lokesh Mandvekar 2016-05-03 15:46:29 UTC
upcoming fix ... docker-latest-1.10.3-22.el7

Comment 22 Luwen Su 2016-05-05 02:36:28 UTC
The error message handling looks good to me:
docker-latest-1.10.3-22.el7.x86_64

#docker-latest volume create -d lvm --name foobar --opt size=0.01G
Error response from daemon: create foobar: create foobar: Error looking up volume plugin lvm: plugin not found

#lvs 
shows nothing

move to verified

Comment 23 Antonio Murdaca 2016-05-05 07:33:58 UTC
Sorry Luwen but the error message is saying 'plugin not found'. The error message you should see should be 'error partitioning volume'

Comment 24 smahajan@redhat.com 2016-05-05 15:00:25 UTC
Luwen, 

docker is throwing this error, as it is not able to find the volume driver plugin associated with the given name (lvm). Can you tell me the steps you took to get this error ?


Also, is there a machine I can SSH into to take a look into this ?

Shishir

Comment 26 Luwen Su 2016-05-10 04:46:09 UTC
(In reply to Antonio Murdaca from comment #23)
> Sorry Luwen but the error message is saying 'plugin not found'. The error
> message you should see should be 'error partitioning volume'

Yeah, sorry for that, i used the wrong machine (there are 4 vms in my host for testing) which the docker-volume-plugin is not started.

# vgdisplay 
  --- Volume group ---
  VG Name               docker-latest
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               8.00 GiB
  PE Size               4.00 MiB
  Total PE              2047
  Alloc PE / Size       823 / 3.21 GiB
  Free  PE / Size       1224 / 4.78 GiB
  VG UUID               BdH5DF-51Xh-t5h4-sfmb-9tXT-N35k-nSfSJS

# service docker-lvm-plugin status
Redirecting to /bin/systemctl status  docker-lvm-plugin.service
● docker-lvm-plugin.service - Docker Volume Driver for lvm volumes
   Loaded: loaded (/usr/lib/systemd/system/docker-lvm-plugin.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2016-05-10 12:42:46 CST; 2s ago

# docker-latest volume create -d lvm --name docker-latest --opt size=0.01G
Error response from daemon: create docker-latest: VolumeDriver.Create: error partitioning volume


I think i could verified this now with the steps above.

Comment 28 Luwen Su 2016-05-10 05:11:18 UTC
Make up more information/steps:

error trigger:
# docker-latest volume create -d lvm --name foobar --opt size=0.01G
Error response from daemon: create foobar: VolumeDriver.Create: error partitioning volume


Lvm create:
# docker-latest volume create -d lvm --name foobar --opt size=0.02G
foobar
# docker-latest volume create -d lvm --name docker-latest --opt size=0.02G
docker-latest

# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/docker-latest/foobar
  LV Name                foobar
  VG Name                docker-latest
  LV UUID                Poovyb-1wOK-j0hY-7T4U-PpnO-RYZh-y5QSdN
  LV Write Access        read/write
  LV Creation host, time myregistrydomain.com, 2016-05-10 13:09:31 +0800
  LV Status              available
  # open                 0
  LV Size                24.00 MiB
  Current LE             6
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           252:3
   
  --- Logical volume ---
  LV Path                /dev/docker-latest/docker-latest
  LV Name                docker-latest
  VG Name                docker-latest
  LV UUID                TDhgVX-oojR-TP2e-2fe7-tK1P-BP1o-xPLWXV
  LV Write Access        read/write
  LV Creation host, time myregistrydomain.com, 2016-05-10 13:09:45 +0800
  LV Status              available
  # open                 0
  LV Size                24.00 MiB
  Current LE             6
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           252:4

Comment 30 errata-xmlrpc 2016-05-12 14:55:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1057.html


Note You need to log in before you can comment on or make changes to this bug.