Bug 981545

Summary: iostat2pcp cannot parse iostat output
Product: [Fedora] Fedora EPEL Reporter: Marko Myllynen <myllynen>
Component: pcpAssignee: Nathan Scott <nathans>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: el6CC: fche, kenj, mgoodwin, nathans
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pcp-3.8.2-1.el5 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-02 03:47:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
iostat output
none
patch for iostat none

Description Marko Myllynen 2013-07-05 06:26:41 UTC 
Created attachment 769108 [details]
iostat output

Description of problem:
localhost:~> iostat -t -x 5 60 > foo
localhost:~> iostat2pcp foo bar
[17] 2013-07-05 09:17:33
Device: number of values? expected 12, found 2
localhost:~> 

Version-Release number of selected component (if applicable):
pcp-3.8.1-1.el6.x86_64
Comment 1 Ken McDonell 2013-07-05 20:46:56 UTC 
Marko,

This appears to be a date/time formatting issue.

Can you please tell me what env vars you may have set that would influence the formatting of the date from the iostat -t command?

Also, could you try to re-run the failing case with S_TIME_FORMAT=ISO set in the environment?
Comment 2 Ken McDonell 2013-07-05 23:05:04 UTC 
Created attachment 769420 [details]
patch for iostat

Marko, also could you try the attached patch for iostat2pcp ... this works for me with the first few samples from your failing iostat output, and passes QA 733 (so no regression, although I'm not sure how good the coverage is in that unit test).
Comment 3 Ken McDonell 2013-07-06 11:23:19 UTC 
OK, we need to demand that the user takes more control over the environment before iostat is run ... the time stamp format has more variations than I've had hot breakfasts, including _both_ the ambiguous MM/DD/YY and DD/MM/YY.

For all the locales I have locally, the following output is triples of lines:
1. LC_TIME
2. LC_TIME=... iostat -t timestamp
3. LC_TIME=... S_FORMAT_TIME=ISO iostat -t timestamp

As you'll see S_FORMAT_TIME=ISO is the only thing that over-rides $LC_TIME and produces a timestamp we can parse unambiguously.  I'll make the man page description very forceful, as we cannot untangle this mess in iostat2pcp.

$ for l in `locale -a`; do echo $l; LC_TIME=$l iostat -t | sed -n -e 3p; LC_TIME=$l S_TIME_FORMAT=ISO iostat -t | sed -n -e 3p; done
C
07/06/13 21:15:16
2013-07-06T21:15:16+1000
C.UTF-8
07/06/2013 21:15:16            <- US-style MM/DD
2013-07-06T21:15:16+1000
de_AT.utf8
2013-07-06 21:15:16
2013-07-06T21:15:16+1000
de_BE.utf8
2013-07-06 21:15:16
2013-07-06T21:15:16+1000
de_CH.utf8
06.07.2013 21:15:16
2013-07-06T21:15:16+1000
de_DE.utf8
06.07.2013 21:15:16
2013-07-06T21:15:16+1000
de_LI.utf8
06.07.2013 21:15:16
2013-07-06T21:15:16+1000
de_LU.utf8
2013-07-06 21:15:16
2013-07-06T21:15:16+1000
en_AG
06/07/13 21:15:16
2013-07-06T21:15:16+1000
en_AG.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
en_AU.utf8
06/07/13 21:15:16                   <- Australian-style DD/MM
2013-07-06T21:15:16+1000
en_BW.utf8
06/07/2013 21:15:16
2013-07-06T21:15:16+1000
en_CA.utf8
13-07-06 09:15:16 PM
2013-07-06T21:15:16+1000
en_DK.utf8
2013-07-06 21:15:16
2013-07-06T21:15:16+1000
en_GB.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
en_HK.utf8
Saturday, July 06, 2013 09:15:16 EST
2013-07-06T21:15:16+1000
en_IE.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
en_IN
Saturday 06 July 2013 09:15:16  EST
2013-07-06T21:15:16+1000
en_IN.utf8
Saturday 06 July 2013 09:15:16  EST
2013-07-06T21:15:16+1000
en_NG
06/07/2013 21:15:16
2013-07-06T21:15:16+1000
en_NG.utf8
06/07/2013 21:15:16
2013-07-06T21:15:16+1000
en_NZ.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
en_PH.utf8
Saturday, 06 July, 2013 09:15:16  EST
2013-07-06T21:15:16+1000
en_SG.utf8
06/07/2013 21:15:16
2013-07-06T21:15:16+1000
en_US.utf8
07/06/2013 09:15:16 PM
2013-07-06T21:15:16+1000
en_ZA.utf8
06/07/2013 21:15:16
2013-07-06T21:15:16+1000
en_ZM
06/07/13 21:15:16
2013-07-06T21:15:16+1000
en_ZM.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
en_ZW.utf8
06/07/2013 21:15:16
2013-07-06T21:15:16+1000
es_AR.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_BO.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_CL.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_CO.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_CR.utf8
06/07/2013 21:15:16
2013-07-06T21:15:16+1000
es_DO.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_EC.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_ES.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_GT.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_HN.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_MX.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_NI.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_PA.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_PE.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_PR.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_PY.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_SV.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_US.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_UY.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
es_VE.utf8
06/07/13 21:15:16
2013-07-06T21:15:16+1000
POSIX
07/06/13 21:15:16
2013-07-06T21:15:16+1000
pt_BR.utf8
06-07-2013 21:15:16
2013-07-06T21:15:16+1000
pt_PT.utf8
06-07-2013 21:15:16
2013-07-06T21:15:16+1000
zh_CN.utf8
2013年07月06日 21时15分16秒
2013-07-06T21:15:16+1000
zh_SG.utf8
2013年07月06日 21时15分16秒 EST
2013-07-06T21:15:16+1000
Comment 4 Marko Myllynen 2013-07-08 08:38:54 UTC 
Hi Ken,

thanks for looking into this.

I've got the following time related environment variable set but unsetting them does not seem to make a difference with 3.8.1:

localhost:~> echo $LANG $LC_TIME $TIME_STYLE
en_US.UTF-8 en_DK.UTF-8 long-iso
localhost:~> echo $LC_ALL $S_TIME_FORMAT    

localhost:~> iostat -t -x 5 60 > foo        
localhost:~> iostat2pcp foo bar     
[16] 2013-07-08 11:22:44
Device: number of values? expected 12, found 2
localhost:~> unset LC_TIME TIME_STYLE 
localhost:~> echo $LANG $LC_TIME $TIME_STYLE
en_US.UTF-8
localhost:~> iostat -t -x 5 60 > foo        
localhost:~> iostat2pcp foo bar             
[16] 07/08/2013 11:24:13 AM
Device: number of values? expected 12, found 3
localhost:~> 

Trying with your patch and the variables set again then everything works as expected:

localhost:~> echo $LANG $LC_TIME $TIME_STYLE
en_US.UTF-8 en_DK.UTF-8 long-iso
localhost:~> echo $LC_ALL $S_TIME_FORMAT    

localhost:~> iostat -t -x 5 60 > foo
localhost:~> iostat2pcp.new foo bar        
localhost:~> 

Thanks!
Comment 5 Nathan Scott 2013-07-25 22:53:09 UTC 
Ken's updates are merged upstream, will release with pcp-3.8.2 shortly.
Comment 6 Fedora Update System 2013-07-31 04:41:31 UTC 
pcp-3.8.2-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/pcp-3.8.2-1.fc19
Comment 7 Fedora Update System 2013-07-31 05:16:59 UTC 
pcp-3.8.2-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/pcp-3.8.2-1.fc18
Comment 8 Fedora Update System 2013-07-31 05:17:55 UTC 
pcp-3.8.2-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/pcp-3.8.2-1.fc17
Comment 9 Fedora Update System 2013-07-31 05:18:53 UTC 
pcp-3.8.2-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/pcp-3.8.2-1.el6
Comment 10 Fedora Update System 2013-07-31 05:19:44 UTC 
pcp-3.8.2-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/pcp-3.8.2-1.el5
Comment 11 Fedora Update System 2013-08-01 20:33:04 UTC 
Package pcp-3.8.2-1.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing pcp-3.8.2-1.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2013-11023/pcp-3.8.2-1.el6
then log in and leave karma (feedback).
Comment 12 Fedora Update System 2013-08-02 03:47:41 UTC 
pcp-3.8.2-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 13 Fedora Update System 2013-08-05 23:28:43 UTC 
pcp-3.8.2-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 14 Fedora Update System 2013-08-10 04:01:09 UTC 
pcp-3.8.2-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 15 Fedora Update System 2013-08-10 12:32:59 UTC 
pcp-3.8.2-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 16 Fedora Update System 2013-08-16 19:52:33 UTC 
pcp-3.8.2-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.