1365772 – New reporting features related to report groups and formatting report output

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1365772 - New reporting features related to report groups and formatting report output

Summary: New reporting features related to report groups and formatting report output

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.3
Hardware:	All
OS:	Linux
Priority:	medium
Severity:	low
Target Milestone:	rc
Target Release:	---
Assignee:	Peter Rajnoha
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1366364
TreeView+	depends on / blocked

Reported:	2016-08-10 08:11 UTC by Peter Rajnoha
Modified:	2021-09-03 12:51 UTC (History)
CC List:	10 users (show)
Fixed In Version:	lvm2-2.02.163-1.el7
Doc Type:	Enhancement
Doc Text:	This release contains several enhancements in reporting infrastructure introduced in libdevmapper which LVM2 uses and various supporting and related enhancements. The libdevmapper within its reporting infrastructure now allows grouping several reports together and it makes it possible to switch report output into a different format for the whole group. Output format for LVM2 reports is configured by new report/output_format lvm.conf setting or directly on command line with new --reportformat option. By default, the original output is used, which is now called "basic" report output. The newly supported report output format is "json". This encompasses all reporting commands (e.g. pvs, vgs, lvs) as well as command log report. The command log report is a new feature that comes together with enhancements in reporting infrastructure. With this feature, it's possible to report comand's log which contains per-item processing status as well as any messages which are issued during LVM command processing. This feature is enabled with new log/report_command_log lvm.conf setting and it can be customized globally with new log/command_log_{cols,sort,selection} lvm.conf settings. The command log is presented as report on output, thus it's also possible to switch its output format as it is for any other report. To support more reports per command (e.g. main command's report and log report in addition to that), there's a new --configreport <report_name> option to make it possible to select for which report any subsequent -o\|--option, -O\|--sort and -S\|--select applies to. The log report also contains information about overall command execution status (mainly its return code) and the log report's coverage is wider than just processing status if LVM commands are executed from within LVM shell. To support querying any last command's log collected while running an LVM command in LVM shell, there's a new "lastlog" command that also accepts -S\|--select with which it's possible to select only lines from the log which are of interest to users and applications using LVM shell. With this group of new features within LVM reporting also comes new "lvm fullreport" command that makes it possible to gather all information about PVs, PV segments, VGs, LVs and LV segments all at once for each VG and guarded by per-VG lock for consistency. New LVM_OUT_FD, LVM_ERR_FD and LVM_REPORT_FD environment variables that LVM accepts makes it possible to redirect common output, error output and report output into different and custom file descriptors which are prepared before running an LVM command.
Clone Of:	1334682
Clones:	1366364 (view as bug list)
Environment:
Last Closed:	2016-11-04 04:16:45 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:1445	0	normal	SHIPPED_LIVE	lvm2 bug fix and enhancement update	2016-11-03 13:46:41 UTC

Description Peter Rajnoha 2016-08-10 08:11:03 UTC

This is a clone for RHEL 7.3 of the original upstream feature-tracking bugzilla report (bug #1334682). These features are now available in 7.3 and they're summarized below:

--> Enhance reporting code in libdevmapper to support report groups with different output formats
 - new "struct dm_report_group" handler
 - new "dm_report_group_type_t" type
 - new functions:
    struct dm_report_group *dm_report_group_create(dm_report_group_type_t type, void *data);
  int dm_report_group_push(struct dm_report_group *group, struct dm_report *report, void *data);
  int dm_report_group_pop(struct dm_report_group *group);
  int dm_report_group_output_and_pop_all(struct dm_report_group *group);
  int dm_report_group_destroy(struct dm_report_group *group);


--> Make it possible to report output from lvm2 reporting commands in JSON format
  - new "report/output_format=json" lvm.conf option to switch output into JSON format
  - new "--reportformat json" lvm2 cmd line option to switch output into JSON directly for each command (also overrides the lvm.conf setting)


--> Add new lvm2 command to do complete report per VG
  - new "lvm fullreport" command to do PV, PV segment, VG, LV, LV segment report per each VG (note: due to its extent, it's more readable/parseable with JSON output)
  - new lvm.conf settings to set up fullreport: report/{pvs,pvsegs,vgs,lvs,segs}_{cols,sort}_full (the same principle as existing report/{pvs,pvsegs,vgs,lvs,segs_{cols,sort} settings, but designated for the fullreport, not single reports


--> Issue warnings, errors and status per object from lvm2 command line tools via report
   - new "log/report_command_log" lvm.conf setting to enable/disable log report
   - new "log/command_log_{cols,sort} lvm.conf setting to set up columns/sorting for log report
   - new "log/command_log_selection" lvm.conf setting to define automatic selection used for log report
   - note: if log report is enabled, the report is appended to any existing reports and added to report group that is used for the command

Related to both fullreport and log report: since we enabled several reports per command now, we have a new "--configreport <report_name>" lvm2 cmd line option to switch among reports for which the -o|--options, -O|--sort, -S|--select option should be applied. The report names are: "pv", "pvseg", "vg", "lv", "seg" (all of them are contained in fullreport) and "log" report name for the log report.


--> Make it possible to query lvm2 command status report
  - new "lastlog" command designated for use in lvm shell to display the most recent log report that is recorded (so it contains log report for any previous command except lastlog command itself)
  - the "lastlog" command accepts -S|--select to select only lines we're interested in from the log report
  - the "lastlog" command can be called more than once and we can use different selection criteria each time

Comment 2 Peter Rajnoha 2016-08-16 12:10:07 UTC

+ there are new LVM_OUT_FD (common output - this is stdout by default), LVM_ERR_FD (error output - this is stderr by default), LVM_REPORT_FD (report output - this is stdout by default) environment variables which LVM commands honours and with which it's possible to define custom file descriptor to which the concrete output type is redirected.

Comment 3 Peter Rajnoha 2016-08-16 12:36:59 UTC

I'm adding some examples from which it should be clear what this is all about...

(In reply to Peter Rajnoha from comment #0)
> --> Enhance reporting code in libdevmapper to support report groups with
> different output formats
>  - new "struct dm_report_group" handler
>  - new "dm_report_group_type_t" type
>  - new functions:
>     struct dm_report_group *dm_report_group_create(dm_report_group_type_t
> type, void *data);
>   int dm_report_group_push(struct dm_report_group *group, struct dm_report
> *report, void *data);
>   int dm_report_group_pop(struct dm_report_group *group);
>   int dm_report_group_output_and_pop_all(struct dm_report_group *group);
>   int dm_report_group_destroy(struct dm_report_group *group);
> 

Available as part of device-mapper-libs/device-mapper-devel package (libdevmapper.so + libdevmapper.h).

> 
> --> Make it possible to report output from lvm2 reporting commands in JSON
> format
>   - new "report/output_format=json" lvm.conf option to switch output into
> JSON format
>   - new "--reportformat json" lvm2 cmd line option to switch output into
> JSON directly for each command (also overrides the lvm.conf setting)
> 

(I'm using report/compact_output=1 in all examples so the output is shorter to fit on this page better...)

[0] rhel7-a/~ # lvmconfig --type diff
report {
	compact_output=1
}

[0] rhel7-a/~ # lvs
  LV   VG   Attr       LSize   
  root rhel -wi-ao----   9.02g
  swap rhel -wi-ao---- 500.00m

[0] rhel7-a/~ # vim /etc/lvm/lvm.conf 

[0] rhel7-a/~ # lvmconfig --type diff
report {
	output_format="json"
	compact_output=1
}

[0] rhel7-a/~ # lvs
  {
      "report": [
          {
              "lv": [
                  {"lv_name":"root", "vg_name":"rhel", "lv_attr":"-wi-ao----", "lv_size":"9.02g"},
                  {"lv_name":"swap", "vg_name":"rhel", "lv_attr":"-wi-ao----", "lv_size":"500.00m"}
              ]
          }
      ]
  }

[0] rhel7-a/~ # lvs --reportformat basic
  LV   VG   Attr       LSize   
  root rhel -wi-ao----   9.02g
  swap rhel -wi-ao---- 500.00m

[0] rhel7-a/~ # lvs --reportformat json 
  {
      "report": [
          {
              "lv": [
                  {"lv_name":"root", "vg_name":"rhel", "lv_attr":"-wi-ao----", "lv_size":"9.02g"},
                  {"lv_name":"swap", "vg_name":"rhel", "lv_attr":"-wi-ao----", "lv_size":"500.00m"}
              ]
          }
      ]
  }

> 
> --> Add new lvm2 command to do complete report per VG
>   - new "lvm fullreport" command to do PV, PV segment, VG, LV, LV segment
> report per each VG (note: due to its extent, it's more readable/parseable
> with JSON output)
>   - new lvm.conf settings to set up fullreport:
> report/{pvs,pvsegs,vgs,lvs,segs}_{cols,sort}_full (the same principle as
> existing report/{pvs,pvsegs,vgs,lvs,segs_{cols,sort} settings, but
> designated for the fullreport, not single reports
> 

[0] rhel7-a/~ # lvmconfig --type diff
report {
	output_format="json"
	compact_output=1
	vgs_cols_full="vg_name"
	pvs_cols_full="pv_name,vg_name"
	lvs_cols_full="lv_name,vg_name"
	pvsegs_cols_full="pv_name,pvseg_start,pvseg_size"
	segs_cols_full="lv_name,seg_start,seg_size"
}

[0] rhel7-a/~ # lvm fullreport
  {
      "report": [
          {
              "vg": [
                  {"vg_name":"rhel"}
              ]
              ,
              "pv": [
                  {"pv_name":"/dev/vda2", "vg_name":"rhel"}
              ]
              ,
              "lv": [
                  {"lv_name":"root", "vg_name":"rhel"},
                  {"lv_name":"swap", "vg_name":"rhel"}
              ]
              ,
              "pvseg": [
                  {"pv_name":"/dev/vda2", "pvseg_start":"0", "pvseg_size":"2309"},
                  {"pv_name":"/dev/vda2", "pvseg_start":"2309", "pvseg_size":"125"}
              ]
              ,
              "seg": [
                  {"lv_name":"swap", "seg_start":"0 ", "seg_size":"500.00m"},
                  {"lv_name":"root", "seg_start":"0 ", "seg_size":"9.02g"}
              ]
          }
      ]
  }

> 
> --> Issue warnings, errors and status per object from lvm2 command line
> tools via report
>    - new "log/report_command_log" lvm.conf setting to enable/disable log
> report
>    - new "log/command_log_{cols,sort} lvm.conf setting to set up
> columns/sorting for log report
>    - new "log/command_log_selection" lvm.conf setting to define automatic
> selection used for log report
>    - note: if log report is enabled, the report is appended to any existing
> reports and added to report group that is used for the command

[0] rhel7-a/~ # lvmconfig --type diff
report {
	compact_output=1
}
log {
	report_command_log=1
	command_log_selection="all"
}


(reporting command - the command log is appended to lvs main report)

[0] rhel7-a/~ # lvs
  Logical Volume
  ==============
  LV   VG   Attr       LSize   
  root rhel -wi-ao----   9.02g
  swap rhel -wi-ao---- 500.00m
  
  Command Log
  ===========
  Seq LogType Context    ObjType ObjName ObjID                                  ObjGrp  ObjGrpID                               Msg     Errno RetCode
    1 status  processing lv      root    yWDUvw-9zpE-n2EI-TeoN-POzW-3wHt-B4uDpX rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT success     0       1
    2 status  processing lv      swap    JcSyXd-KByX-2sKE-QDuQ-T10R-Q5kI-mn8OcA rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT success     0       1
    3 status  processing vg      rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT                                                success     0       1


(non-reporting command - there's only log report)

[0] rhel7-a/~ # vgcreate vg /dev/sda
  Command Log
  ===========
  Seq LogType Context    ObjType ObjName   ObjID                                  ObjGrp  ObjGrpID                               Msg                                    Errno RetCode
    1 status  processing pv      /dev/vda2 2cBtQX-IKw0-vsJi-ZLB3-pUfS-w48b-2kA4H9 rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT success                                    0       1
    2 status  processing vg      rhel      LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT                                                success                                    0       1
    3 status  processing orphan  lvm1                                                                                            success                                    0       1
    4 status  processing orphan  pool                                                                                            success                                    0       1
    5 status  processing pv      /dev/sda  XEP0SI-mFGo-C1UT-WvaF-UYIw-5ZfL-DLGWB1                                                success                                    0       1
    6 status  processing orphan  lvm2                                                                                            success                                    0       1
    7 print   processing                                                                                                         Volume group "vg" successfully created     0       0


(changing log/command_log_selection back to default command_log_selection = "!(log_type=status && message=success)" - so we don't display "success", just error states)

[0] rhel7-a/~ # lvmconfig --type diff
report {
	compact_output=1
}
log {
	report_command_log=1
}

[0] rhel7-a/~ # lvs
  Logical Volume
  ==============
  LV   VG   Attr       LSize   
  root rhel -wi-ao----   9.02g
  swap rhel -wi-ao---- 500.00m
  
  Command Log
  ===========
  Seq LogType Context ObjType ObjName ObjID   ObjGrp  ObjGrpID Msg     Errno RetCode

[0] rhel7-a/~ # lvs vg/i_dont_exist
  
  Command Log
  ===========
  Seq LogType Context    ObjType ObjName      ObjID                                  ObjGrp  ObjGrpID                               Msg                                             Errno RetCode
    1 error   processing lv      i_dont_exist                                        vg      Hafilo-sFEy-vLz0-bLk7-vvPs-UmsF-hcF13b Failed to find logical volume "vg/i_dont_exist"    -1       0
    2 status  processing lv      i_dont_exist                                        vg      Hafilo-sFEy-vLz0-bLk7-vvPs-UmsF-hcF13b failure                                            -1       5
    3 status  processing vg      vg           Hafilo-sFEy-vLz0-bLk7-vvPs-UmsF-hcF13b                                                failure                                            -1       5


[0] rhel7-a/~ # lvmconfig --type diff
report {
	compact_output=1
}
log {
	report_command_log=1
	command_log_cols="log_type,log_context,log_object_type,log_object_name,log_message"
}

[0] rhel7-a/~ # lvs vg/i_dont_exist
  
  Command Log
  ===========
  LogType Context    ObjType ObjName      Msg                                            
  error   processing lv      i_dont_exist Failed to find logical volume "vg/i_dont_exist"
  status  processing lv      i_dont_exist failure                                        
  status  processing vg      vg           failure         


> 
> Related to both fullreport and log report: since we enabled several reports
> per command now, we have a new "--configreport <report_name>" lvm2 cmd line
> option to switch among reports for which the -o|--options, -O|--sort,
> -S|--select option should be applied. The report names are: "pv", "pvseg",
> "vg", "lv", "seg" (all of them are contained in fullreport) and "log" report
> name for the log report.
> 

[0] rhel7-a/~ # lvmconfig --type diff
report {
	compact_output=1
}
log {
	report_command_log=1
	command_log_cols="log_type,log_context,log_object_type,log_object_name,log_message"
}

[0] rhel7-a/~ # lvs --configreport lv -o lv_name --configreport log -o+log_seq_num -S all rhel/root vg/i_dont_exist
  Logical Volume
  ==============
  LV  
  root
  
  Command Log
  ===========
  LogType Context    ObjType ObjName      Msg                                             Seq
  status  processing lv      root         success                                           1
  status  processing vg      rhel         success                                           2
  error   processing lv      i_dont_exist Failed to find logical volume "vg/i_dont_exist"   3
  status  processing lv      i_dont_exist failure                                           4
  status  processing vg      vg           failure                                           5


> 
> --> Make it possible to query lvm2 command status report
>   - new "lastlog" command designated for use in lvm shell to display the
> most recent log report that is recorded (so it contains log report for any
> previous command except lastlog command itself)
>   - the "lastlog" command accepts -S|--select to select only lines we're
> interested in from the log report
>   - the "lastlog" command can be called more than once and we can use
> different selection criteria each time

[0] rhel7-a/~ # lvm
lvm> lvmconfig --type diff
report {
	compact_output=1
}
log {
	report_command_log=1
	command_log_selection="all"
}
  Command Log
  ===========
  Seq LogType Context ObjType ObjName   ObjID   ObjGrp  ObjGrpID Msg     Errno RetCode
    1 status  shell   cmd     lvmconfig                          success     0       1
lvm> lvs rhel/root vg/i_dont_exist
  Logical Volume
  ==============
  LV   VG   Attr       LSize 
  root rhel -wi-ao---- 9.02g
  
  Command Log
  ===========
  Seq LogType Context    ObjType ObjName ObjID                                  ObjGrp  ObjGrpID                               Msg                            Errno RetCode
    1 status  processing lv      root    yWDUvw-9zpE-n2EI-TeoN-POzW-3wHt-B4uDpX rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT success                            0       1
    2 status  processing vg      rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT                                                success                            0       1
    3 error   processing vg      vg                                                                                            Volume group "vg" not found       -1       0
    4 error   processing vg      vg                                                                                            Cannot process volume group vg    -1       0
    5 status  processing vg      vg                                                                                            failure                           -1       5
    6 status  shell      cmd     lvs                                                                                           failure                            0       5
lvm> lastlog
  Command Log
  ===========
  Seq LogType Context    ObjType ObjName ObjID                                  ObjGrp  ObjGrpID                               Msg                            Errno RetCode
    1 status  processing lv      root    yWDUvw-9zpE-n2EI-TeoN-POzW-3wHt-B4uDpX rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT success                            0       1
    2 status  processing vg      rhel    LDTYQz-81Zb-qwBC-Arek-EZ3t-Qnjd-Yu4yWT                                                success                            0       1
    3 error   processing vg      vg                                                                                            Volume group "vg" not found       -1       0
    4 error   processing vg      vg                                                                                            Cannot process volume group vg    -1       0
    5 status  processing vg      vg                                                                                            failure                           -1       5
    6 status  shell      cmd     lvs                                                                                           failure                            0       5
lvm> lastlog -S log_type=error
  Command Log
  ===========
  Seq LogType Context    ObjType ObjName ObjID   ObjGrp  ObjGrpID Msg                            Errno RetCode
    3 error   processing vg      vg                               Volume group "vg" not found       -1       0
    4 error   processing vg      vg                               Cannot process volume group vg    -1       0
lvm> lastlog -S log_context=shell
  Command Log
  ===========
  Seq LogType Context ObjType ObjName ObjID   ObjGrp  ObjGrpID Msg     Errno RetCode
    6 status  shell   cmd     lvs                              failure     0       5

Comment 4 Peter Rajnoha 2016-08-16 12:46:41 UTC

(In reply to Peter Rajnoha from comment #2)
> + there are new LVM_OUT_FD (common output - this is stdout by default),
> LVM_ERR_FD (error output - this is stderr by default), LVM_REPORT_FD (report
> output - this is stdout by default) environment variables which LVM commands
> honours and with which it's possible to define custom file descriptor to
> which the concrete output type is redirected.

[0] rhel7-a/~ # lvmconfig --type diff
report {
	compact_output=1
}

===

[0] rhel7-a/~ # LVM_OUT_FD=3 LVM_ERR_FD=4 LVM_REPORT_FD=5 lvs rhel/root vg/i_dont_exist 3>out 4>err 5>report

[0] rhel7-a/~ # cat out 

[0] rhel7-a/~ # cat err
  Volume group "vg" not found
  Cannot process volume group vg

[0] rhel7-a/~ # cat report 
  LV   VG   Attr       LSize 
  root rhel -wi-ao---- 9.02g


===


[0] rhel7-a/~ # LVM_OUT_FD=3 LVM_ERR_FD=4 LVM_REPORT_FD=5 lvchange --addtag x rhel/root vg/i_dont_exist 3>out 4>err 5>report

[0] rhel7-a/~ # cat out 
  Logical volume rhel/root changed.

[0] rhel7-a/~ # cat err
  Volume group "vg" not found
  Cannot process volume group vg

[0] rhel7-a/~ # cat report

Comment 5 Peter Rajnoha 2016-09-14 14:36:25 UTC

See also "man lvmreport" for more info which contains all about reporting infrastructure, including the new features mentioned in this report.

Comment 6 Roman Bednář 2016-09-19 12:29:43 UTC

Marking as verified with latest rpms. 

3.10.0-505.el7.x86_64

lvm2-2.02.165-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016
lvm2-libs-2.02.165-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016
lvm2-cluster-2.02.165-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016
device-mapper-1.02.134-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016
device-mapper-libs-1.02.134-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016
device-mapper-event-1.02.134-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016
device-mapper-event-libs-1.02.134-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 12:29:13 CEST 2016
cmirror-2.02.165-2.el7    BUILT: Wed Sep 14 16:01:43 CEST 2016

Comment 8 errata-xmlrpc 2016-11-04 04:16:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1445.html

Note You need to log in before you can comment on or make changes to this bug.