Bug 867376 - Provide criteria-based LVM reporting
Provide criteria-based LVM reporting
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
rawhide
All Linux
low Severity low
: ---
: ---
Assigned To: Peter Rajnoha
Fedora Extras Quality Assurance
: FutureFeature
Depends On:
Blocks: 1103923 1103924
  Show dependency treegraph
 
Reported: 2012-10-17 07:48 EDT by Peter Rajnoha
Modified: 2014-06-24 08:01 EDT (History)
12 users (show)

See Also:
Fixed In Version: lvm2-2.02.107-1.fc21
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 1103923 1112551 (view as bug list)
Environment:
Last Closed: 2014-06-24 05:55:31 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Peter Rajnoha 2012-10-17 07:48:42 EDT
I'm opening this to track and gather any ideas for this issue...

Sometimes we can see (in the scripts most notably) that the output needs to be laboriously parsed by external tools to filter out some of the output that is not actually needed or is even undesirable. Also, sometimes we need to run specific commands changing metadata or changing the activation status based on some other criteria than just PV/VG/LV name.

Examples of such criteria-based selection include reporting/displaying VGs that are clustered or local, or any other type-based criteria like mirror volumes, snapshot volumes, thin volumes, RAID volumes etc. Other criteria might be the size of the volume or whatever we report today in LVM display commands in general...

I think this would improve the way the scripts work with LVM commands. Also, the same filtering logic should probably be available via liblvm interface.

This is definitely not a high priority feature request, but just something that would be nice to have.
Comment 1 Fedora End Of Life 2013-04-03 14:56:31 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 2 David Teigland 2014-04-11 12:24:41 EDT
It may make sense to borrow the command line "selection" syntax for bug 1070381.

We want to be able to select vgs from the command line by: name, uuid, tags.
If a uuid or tag is used, it requires reading all vg's to match against the
uuid/tag args.  If a name is used, only the vg's with the given names are
read.  So a command with a uuid/tag arg can be much more expensive.  People
can intentionally select vgs only by name to avoid this expense.

The problem is that people use uuids as names, and they are indistinct as
command line args: we do not know if the arg is meant as one or the other.
When a uuid arg is seen, the process_each_vg code has to read all vgs in
case the uuid arg will match a vg's uuid, not a name.

This means a person with uuids as names, who is selecting the vgs by name on
the command line would unavoidably pay the expense of reading all vgs.

A solution would be to know from the command line syntax if the arg is meant to
match a vg's name or uuid.  If we knew this up front, we could avoid
unnecessarily reading all vgs.

If the selection syntax were something like -o field=value, then:

1. vgs -o uuid=12-34-56-78
2. vgs -o name=12-34-56-78

The first command would read all vg's and attempt to match the uuid arg with
a vg's uuid.  The second command would only read the named vg.  Importantly,
this is not only an output filter, but selects the objects to be processed.
Comment 3 Peter Rajnoha 2014-04-25 06:12:37 EDT
I've pushed a development branch 'dev-prajnoha-condout' with the code to support filtering report output (vgs, lvs, pvs, dmsetup as of now) based on conditions supplied: https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-prajnoha-condout. This is actually rebased original version of the patchset.
The original code is dated back in 2007 and was written by Jun'ichi Nomura: http://www.redhat.com/archives/dm-devel/2007-April/msg00025.html.

I've added some improvements like support for comparing sizes with units...

I'd still like to explore a possibility to reuse part of this logic for conditional execution as well (e.g. change all LVs that comply with the condition supplied).

Still needs more testing, but the reporting part seems to work basically. 

Example:

[1] raw/~ # pvs
  PV         VG         Fmt  Attr PSize   PFree  
  /dev/sda   vg1        lvm2 a--  480.00m 440.00m
  /dev/sdb   vg1        lvm2 a--   96.00m  96.00m
  /dev/sdc   vg1        lvm2 a--  124.00m 124.00m
  /dev/sde   vg2        lvm2 a--  124.00m  72.00m
  /dev/sdf   vg2        lvm2 a--  124.00m 124.00m
  /dev/sdg   vg2        lvm2 a--  124.00m 124.00m
  /dev/sdh   other      lvm2 a--  124.00m  28.00m
  /dev/sdi   other      lvm2 a--  124.00m 124.00m
  /dev/vda2  fedora_raw lvm2 a--    9.50g      0 
[1] raw/~ # vgs
  VG         #PV #LV #SN Attr   VSize   VFree  
  fedora_raw   1   2   0 wz--n-   9.50g      0 
  other        2   2   0 wz--n- 248.00m 152.00m
  vg1          3   4   0 wz--n- 700.00m 660.00m
  vg2          3   4   0 wz--n- 372.00m 320.00m
[1] raw/~ # lvs
  LV    VG         Attr       LSize  
  root  fedora_raw -wi-ao----   9.00g
  swap  fedora_raw -wi-ao---- 512.00m
  one   other      -wi-a-----  32.00m
  two   other      -wi-a-----  64.00m
  lvol0 vg1        -wi-a-----   4.00m
  lvol1 vg1        -wi-a-----   8.00m
  lvol2 vg1        -wi-a-----  12.00m
  lvol3 vg1        -wi-a-----  16.00m
  lvol0 vg2        -wi-a-----   4.00m
  lvol1 vg2        -wi-a-----  12.00m
  lvol2 vg2        -wi-a-----  16.00m
  lvol3 vg2        -wi-a-----  20.00m
[1] raw/~ # pvs --condition 'pv_size > 200m'
  PV         VG         Fmt  Attr PSize   PFree  
  /dev/sda   vg1        lvm2 a--  480.00m 440.00m
  /dev/vda2  fedora_raw lvm2 a--    9.50g      0 
[1] raw/~ # pvs --condition '(pv_size > 200m) && (pv_size < 9g)'
  PV         VG   Fmt  Attr PSize   PFree  
  /dev/sda   vg1  lvm2 a--  480.00m 440.00m
[1] raw/~ # lvs --condition 'lv_name=~"lvol[023]"'
  LV    VG   Attr       LSize 
  lvol0 vg1  -wi-a-----  4.00m
  lvol2 vg1  -wi-a----- 12.00m
  lvol3 vg1  -wi-a----- 16.00m
  lvol0 vg2  -wi-a-----  4.00m
  lvol2 vg2  -wi-a----- 16.00m
  lvol3 vg2  -wi-a----- 20.00m
[1] raw/~ # lvs --condition '(lv_name=~"lvol[023]") || (lv_size > 500m)'
  LV    VG         Attr       LSize  
  root  fedora_raw -wi-ao----   9.00g
  swap  fedora_raw -wi-ao---- 512.00m
  lvol0 vg1        -wi-a-----   4.00m
  lvol2 vg1        -wi-a-----  12.00m
  lvol3 vg1        -wi-a-----  16.00m
  lvol0 vg2        -wi-a-----   4.00m
  lvol2 vg2        -wi-a-----  16.00m
  lvol3 vg2        -wi-a-----  20.00m
[1] raw/~ # lvs --condition 'lv_attr=~"ao"'
  LV   VG         Attr       LSize  
  root fedora_raw -wi-ao----   9.00g
  swap fedora_raw -wi-ao---- 512.00m
Comment 4 Peter Rajnoha 2014-04-30 05:21:58 EDT
Second iteration of the patchset:
https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-prajnoha-selout

- renamed --condition to -S/--select
- added -S/--select to pv/vg/lvdisplay (with -C)
- AND is "+" and OR is "," now
- "==" is "=" now
- no need to use quotes, so this works now:
   lvs --select name=abc,vg_name=def
- added a check whether a field referenced in the selection is recognized (so "lvs --select unknown" produces:

[0] raw # lvs -S bla=abc 
  Unrecognized selection field: bla
  Selection syntax error at 'bla=abc'

...now I'll try to add the coverage for list matching (e.g. tag list). So that will be in next iteration I hope...
Comment 5 Peter Rajnoha 2014-05-29 06:20:11 EDT
Hopefully final candidate:
https://git.fedorahosted.org/cgit/lvm2.git/log/?h=dev-prajnoha-report-select
Comment 6 Peter Rajnoha 2014-06-24 03:15:14 EDT
The final released version looks like this (lvm2 v2.02.107/libdevmapper v1.02.86):

pvs/vgs/lvs/pvdisplay/vgdisplay/lvdisplay/lvm devtypes -S SELECTION

# pvs -S help

  Selection operands
  ------------------
    field               - Reporting field.
    number              - Non-negative integer value.
    size                - Floating point value with units, 'm' unit used by default if not specified.
    percent             - Non-negative integer with or without % suffix.
    string              - Characters quoted by ' or " or unquoted.
    string list         - Strings enclosed by [ ] and elements delimited by either
                          "all items must match" or "at least one item must match" operator.
    regular expression  - Characters quoted by ' or " or unquoted.
   
  Reserved values
  ---------------
    -1, undefined, undef, unknown   - Reserved value for undefined numeric value. [number]
    unmanaged                       - Reserved value for unmanaged number of metadata copies in VG. [number]
    auto                            - Reserved value for size that is automatically calculated. [size]
   
  Selection operators
  -------------------
    Comparison operators:
        =~  - Matching regular expression. [regex]
        !~  - Not matching regular expression. [regex]
         =  - Equal to. [number, size, percent, string, string list]
        !=  - Not equal to. [number, size, percent, string, string_list]
        >=  - Greater than or equal to. [number, size, percent]
         >  - Greater than. [number, size, percent]
        <=  - Less than or equal to. [number, size, percent]
         <  - Less than. [number, size, percent]
   
    Logical and grouping operators:
        &&  - All fields must match
         ,  - All fields must match
        ||  - At least one field must match
         #  - At least one field must match
         !  - Logical negation
         (  - Left parenthesis
         )  - Right parenthesis
         [  - List start
         ]  - List end


(the reserved values are specific to the tool, e.g. dmsetup doesn't define any of these reserved values)

From man lvm (8):

Informal grammar specification:

              STATEMENT = column cmp_op VALUE | STATEMENT log_op STATEMENT | (STATEMENT) | !(STATEMENT)

              VALUE = [VALUE log_op VALUE]
              For list-based types: string list. The log_op must always be of one type within the whole list value.

              VALUE = value
              For scalar types: number (integer), size (floating point number with size unit suffix), percent (floating point number with or without % suffix), string.
Comment 7 Peter Rajnoha 2014-06-24 03:39:10 EDT
I'm separating this original BZ in two:
  - criteria-based reporting (done, this bz)
  - criteria-based processing (to be done, bz #1112544)
Comment 8 Peter Rajnoha 2014-06-24 03:51:35 EDT
There's probably one more thing to cover - date/time fields (e.g. lv creation time). Currently, these fields are recognized as strings which means the comparison is done simply on strings.

It would be nice, if we could add a separate field type "DM_REPORT_FIELD_TYPE_DATE_TIME" and then we should be able to compare the date and time formats. I envision this to be very similar to what "git" command has, e.g. 'git log --since "two weeks ago"' - this would look like in LVM: 'lvs --select time < "two weeks"'. Or maybe a new time-comparing operator for this (like since, before...). But this is not high prio at all. Let's have a look at this if there's some spare time. Maybe lots of this code can be just reused: https://github.com/git/git/blob/master/date.c.

===

Another extensions may include shortcuts like:

'lvs --select name=[a || b || c]' which would be equal to 'lvs --select name=a || name=b || name=c' (this should be easy to add).

If found usable for some situations, maybe also transitive comparisons could be recognized:

'lvs --select lv_name=vg_name=abc' which would be equal to 'lvs --select lv_name=abc && vg_name=abc'
Comment 9 Peter Rajnoha 2014-06-24 08:01:20 EDT
(In reply to Peter Rajnoha from comment #8)
> There's probably one more thing to cover - date/time fields (e.g. lv
> creation time). Currently, these fields are recognized as strings which
> means the comparison is done simply on strings.
> 
> It would be nice, if we could add a separate field type
> "DM_REPORT_FIELD_TYPE_DATE_TIME" and then we should be able to compare the
> date and time formats. I envision this to be very similar to what "git"
> command has, e.g. 'git log --since "two weeks ago"' - this would look like
> in LVM: 'lvs --select time < "two weeks"'. Or maybe a new time-comparing
> operator for this (like since, before...). But this is not high prio at all.
> Let's have a look at this if there's some spare time. Maybe lots of this
> code can be just reused: https://github.com/git/git/blob/master/date.c.
> 

...tracked by bug #1112645

> 
> Another extensions may include shortcuts like:
> 
> 'lvs --select name=[a || b || c]' which would be equal to 'lvs --select
> name=a || name=b || name=c' (this should be easy to add).
> 
> If found usable for some situations, maybe also transitive comparisons could
> be recognized:
> 
> 'lvs --select lv_name=vg_name=abc' which would be equal to 'lvs --select
> lv_name=abc && vg_name=abc'

...tracked by bug #1112646

Note You need to log in before you can comment on or make changes to this bug.