Bug 989650

Summary: user_action.log format
Product: OpenShift Online Reporter: Kenny Woodson <kwoodson>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: urgent Docs Contact:
Priority: high    
Version: 1.xCC: blentz, jhou, kwoodson
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-07 22:57:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenny Woodson 2013-07-29 17:05:04 UTC
Description of problem:
We parse user_action.log for good/bad statistics.  This file format currently has changed and also includes unexpected output.

This is really difficult to parse.

Here is some maven output that was catpured in the user_action.log:

SUCCESS DATE=2013-07-29 TIME=03:29:25 ACTION=ADD_APPLICATION REQ_ID=5ca0cf78444ca83d7982135495c87cdb USER_ID=51f2334ee0b8cd738300080c LOGIN=xxxxxxxxxxxxx DOMAIN=ormanli The initial build for the application failed: Shell command '/sbin/runuser -s /bin/sh 51f618c0e0b8cdda2c006a16 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c3,c613' /bin/sh -c \"gear postreceive >> /tmp/initial-build.log 2>&1\""' exceeded timeout of 232

.Last 10 kB of build output:
ublic/org/codehaus/plexus/plexus-utils/1.5.6/plexus-utils-1.5.6.pom (6 KB at 161.8 KB/sec)
Downloading: http://maven.repository.redhat.com/techpreview/all/org/codehaus/plexus/plexus/1.0.12/plexus-1.0.12.pom
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus/1.0.12/plexus-1.0.12.pom
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus/1.0.12/plexus-1.0.12.pom (10 KB at 416.1 KB/sec)
Downloading: http://maven.repository.redhat.com/techpreview/all/org/codehaus/plexus/plexus-interpolation/1.6/plexus-interpolation-1.6.pom
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus-interpolation/1.6/plexus-interpolation-1.6.pom
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus-interpolation/1.6/plexus-interpolation-1.6.pom (3 KB at 118.3 KB/sec)
Downloading: http://maven.repository.redhat.com/techpreview/all/org/apache/maven/maven-archiver/2.4.1/maven-archiver-2.4.1.jar
Downloading: http://maven.repository.redhat.com/techpreview/all/org/codehaus/plexus/plexus-archiver/1.2/plexus-archiver-1.2.jar
Downloading: http://maven.repository.redhat.com/techpreview/all/org/codehaus/plexus/plexus-io/1.0.1/plexus-io-1.0.1.jar
Downloading: http://maven.repository.redhat.com/techpreview/all/com/thoughtworks/xstream/xstream/1.3.1/xstream-1.3.1.jar
Downloading: http://maven.repository.redhat.com/techpreview/all/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.jar
Downloading: http://maven.repository.redhat.com/techpreview/all/org/apache/maven/shared/maven-filtering/1.0-beta-2/maven-filtering-1.0-beta-2.jar
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/apache/maven/maven-archiver/2.4.1/maven-archiver-2.4.1.jar
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus-archiver/1.2/plexus-archiver-1.2.jar
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus-io/1.0.1/plexus-io-1.0.1.jar
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/com/thoughtworks/xstream/xstream/1.3.1/xstream-1.3.1.jar
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.jar
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/apache/maven/maven-archiver/2.4.1/maven-archiver-2.4.1.jar (20 KB at 400.5 KB/sec)
Downloading: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/apache/maven/shared/maven-filtering/1.0-beta-2/maven-filtering-1.0-beta-2.jar
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus-io/1.0.1/plexus-io-1.0.1.jar (50 KB at 699.6 KB/sec)
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/xpp3/xpp3_min/1.1.4c/xpp3_min-1.1.4c.jar (25 KB at 348.2 KB/sec)
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/codehaus/plexus/plexus-archiver/1.2/plexus-archiver-1.2.jar (178 KB at 2219.8 KB/sec)
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/org/apache/maven/shared/maven-filtering/1.0-beta-2/maven-filtering-1.0-beta-2.jar (33 KB at 1012.5 KB/sec)
Downloaded: http://mirror1.ops.rhcloud.com:80/nexus/content/groups/public/com/thoughtworks/xstream/xstream/1.3.1/xstream-1.3.1.jar (422 KB at 3830.0 KB/sec)
[INFO] Packaging webapp
[INFO] Assembling webapp [prime-phonebook] in [/var/lib/openshift/51f618c0e0b8cdda2c006a16/app-root/runtime/repo/target/prime-phonebook-1.0-SNAPSHOT]
[INFO] Processing war project
[INFO] Copying webapp resources [/var/lib/openshift/51f618c0e0b8cdda2c006a16/app-root/runtime/repo/src/main/webapp]
[INFO] Webapp assembled in [309 msecs]
[INFO] Building war: /var/lib/openshift/51f618c0e0b8cdda2c006a16/app-root/runtime/repo/target/prime-phonebook-1.0-SNAPSHOT.war
[INFO] WEB-INF/web.xml already added, skipping
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:06.668s
[INFO] Finished at: Mon Jul 29 03:28:33 EDT 2013
[INFO] Final Memory: 14M/58M
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "openshift" could not be activated because it does not exist.
+ tmp=/var/lib/openshift/51f618c0e0b8cdda2c006a16/jbossews//tmp
+ '[' -d /var/lib/openshift/51f618c0e0b8cdda2c006a16/jbossews//tmp ']'
+ for d in '$tmp/*'
+ '[' -d '/var/lib/openshift/51f618c0e0b8cdda2c006a16/jbossews//tmp/*' ']'
+ '[' -d /var/lib/openshift/51f618c0e0b8cdda2c006a16/app-root/runtime/repo//webapps ']'
+ rm -rf /var/lib/openshift/51f618c0e0b8cdda2c006a16/jbossews//webapps/ROOT.war


Version-Release number of selected component (if applicable):
Current.

How reproducible:
I'd assume this is easily reproducible as I see multiple occurrences of it in the log file.

Steps to Reproduce:
1. 
2.
3.

I don't know the exact steps but it appears that application creation fails and it is logging that info to the user action log.  I would guess that a java gear  application creation occurs which then calls maven to build.  This build process fails and user_action.log collects the failure and logs it here.

Actual results:

The user_action.log should not log maven build logs to itself.

Our parsers are slow, inefficient, and often times do not give us accurate information.

Expected results:

Log Success/Failures based on the user's actions.  No more, no less.


Additional info:

We would really appreciate a proper log format for this file.  

The date in-line should be a unix timestamp instead of splitting the date and time up into different information.

Here is another example output:
----
FAILURE DATE=2013-07-29 TIME=09:44:48 ACTION=DELETE_APPLICATION REQ_ID=c45390b39011eb35ae84829dd5e00873 USER_ID=51e367bfe0b8cd86b900000e LOGIN=liveC99b0d26d05c APP=project DOMAIN=livec99b0d26d05c APP_UUID=51e367c04382ec15480001cc Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support..
Reference ID: c45390b39011eb35ae84829dd5e00873
---
We could clean this up greatly by logging this in a more parsable format:
-----
STATUS=FAILURE TIME=1375117122 ACTION=DELETE_APPLICATION REQ_ID=c45390b39011eb35ae84829dd5e00873 USER_ID=51e367bfe0b8cd86b900000e LOGIN=liveC99b0d26d05c APP=project DOMAIN=livec99b0d26d05c APP_UUID=51e367c04382ec15480001cc REFID=c45390b39011eb35ae84829dd5e00873
------

KEY=value pairs always.  This makes parsing much more friendly and also keeps consistency.

REFID=<id> instead of "Reference ID: <id>"

REFID=<id> on same line as the other data.  

I also noticed the REQ_ID is the same as the Reference ID.  If this is the case then we shouldn't need both.  I see we only get Ref ID when there is a failure.

Comment 1 Abhishek Gupta 2013-07-29 22:39:46 UTC
Does Ops not care about the description text in the user action log at all regardless of success or failure?

There are 3 classes of log entries here:
1. success
2. validation failures or other user errors
3. failures on account of internal system errors or node failures

I am planning to get rid of the messages in case of internal errors, since they can be long/generic/stack-traces and are anyways found in the broker logs. 

Question: Do you want me to get rid of text messages completely and just have key-value pairs? I would imagine that the messages at the end were helpful to debug any user reported issues since everything other than internal errors are reported as SUCCESS in the user action logs.


Note: Also, I will be adding TIMESTAMP=<epoch_time> and STATUS=[SUCCESS|FAILURE] in the log format, as requested.

Comment 2 Abhishek Gupta 2013-07-29 23:08:19 UTC
Initial fix while Ops responds to the above query.

https://github.com/openshift/origin-server/pull/3213

Comment 3 openshift-github-bot 2013-07-30 18:26:03 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/b5d1aaae17ad9eb7c9bfbffca882fe48b25baade
Fix for bug 989650, bug 988115, and added additional check in oo-admin-chk

Comment 4 Abhishek Gupta 2013-07-30 20:53:30 UTC
Fix to incorporate changes suggested by Ops --> https://github.com/openshift/origin-server/pull/3237

Comment 5 Jianwei Hou 2013-07-31 02:30:19 UTC
Verified on devenv_3588

Now the user_actions.log formant is udpated to key-value pairs, here is what the actual log looks like:
STATUS=SUCCESS TIMESTAMP=1375237289 DATE=2013-07-30 TIME=22:21:29 ACTION=LIST_USER REQ_ID=309ccb46dcc90b8b9dee58669dea58b1 USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou 
STATUS=SUCCESS TIMESTAMP=1375237289 DATE=2013-07-30 TIME=22:21:29 ACTION=AUTHENTICATE REQ_ID=747c01832bfff385239e20d7021e87d4 USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou IP=203.114.244.88 SCOPES=session Authenticated
STATUS=SUCCESS TIMESTAMP=1375237289 DATE=2013-07-30 TIME=22:21:29 ACTION=LIST_KEY REQ_ID=747c01832bfff385239e20d7021e87d4 USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou Found 1 ssh keys
.........
STATUS=SUCCESS TIMESTAMP=1375237754 DATE=2013-07-30 TIME=22:29:14 ACTION=DELETE_APPLICATION REQ_ID=aa22c31fafa9eec1853ea6209a4fd03e USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou APP=as1 DOMAIN=jhou APP_UUID=bdfde35af98711e2a04e123139401dc0 Application as1 is deleted.


Tried to create a jbossas app and do maven build, found that no maven build logs are appended to user_actions.log. I think this is what the bug requires.
Mark as verified.

Comment 7 Kenny Woodson 2013-07-31 13:11:55 UTC
(In reply to Hou Jianwei from comment #5)
> Verified on devenv_3588
> 
> Now the user_actions.log formant is udpated to key-value pairs, here is what
> the actual log looks like:
> STATUS=SUCCESS TIMESTAMP=1375237289 DATE=2013-07-30 TIME=22:21:29
> ACTION=LIST_USER REQ_ID=309ccb46dcc90b8b9dee58669dea58b1
> USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou 
> STATUS=SUCCESS TIMESTAMP=1375237289 DATE=2013-07-30 TIME=22:21:29
> ACTION=AUTHENTICATE REQ_ID=747c01832bfff385239e20d7021e87d4
> USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou IP=203.114.244.88
> SCOPES=session Authenticated
> STATUS=SUCCESS TIMESTAMP=1375237289 DATE=2013-07-30 TIME=22:21:29
> ACTION=LIST_KEY REQ_ID=747c01832bfff385239e20d7021e87d4
> USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou Found 1 ssh keys
> .........
> STATUS=SUCCESS TIMESTAMP=1375237754 DATE=2013-07-30 TIME=22:29:14
> ACTION=DELETE_APPLICATION REQ_ID=aa22c31fafa9eec1853ea6209a4fd03e
> USER_ID=51f86f3553d8ecfcec000001 LOGIN=jhou APP=as1 DOMAIN=jhou
> APP_UUID=bdfde35af98711e2a04e123139401dc0 Application as1 is deleted.
> 
> 
> Tried to create a jbossas app and do maven build, found that no maven build
> logs are appended to user_actions.log. I think this is what the bug requires.
> Mark as verified.

What happens on a "STATUS=FAILURE"?

Comment 8 Jianwei Hou 2013-08-01 05:27:49 UTC
Here is a failure scenario when I have mcollective stopped, there are also some 'unknown' status


RESULT=SUCCESS STATUS=ok TIMESTAMP=1375334369 DATE=2013-08-01 TIME=01:19:29 ACTION=LIST_DOMAIN REQ_ID=43c6ad841397565e8feba2e8780908bf USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou 
RESULT=SUCCESS STATUS=unknown TIMESTAMP=1375334369 DATE=2013-08-01 TIME=01:19:29 ACTION=AUTHENTICATE REQ_ID=6287aebb803b4260e41f351e930f38a9 USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou IP=203.114.244.88 SCOPES=session Authenticated
RESULT=SUCCESS STATUS=ok TIMESTAMP=1375334369 DATE=2013-08-01 TIME=01:19:29 ACTION=LIST_APPLICATION REQ_ID=6287aebb803b4260e41f351e930f38a9 USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou DOMAIN=jhou Found 0 applications.
RESULT=SUCCESS STATUS=ok TIMESTAMP=1375334707 DATE=2013-08-01 TIME=01:25:07 ACTION=LIST_CARTRIDGE REQ_ID=9cdaf13871acccb816a89693ed3d3ae7 USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou List all cartridges
RESULT=SUCCESS STATUS=unknown TIMESTAMP=1375334708 DATE=2013-08-01 TIME=01:25:08 ACTION=AUTHENTICATE REQ_ID=c960af93b7f05d92acb3ec91aed3baed USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou IP=203.114.244.88 SCOPES=session Authenticated
RESULT=SUCCESS STATUS=ok TIMESTAMP=1375334708 DATE=2013-08-01 TIME=01:25:08 ACTION=LIST_DOMAIN REQ_ID=c960af93b7f05d92acb3ec91aed3baed USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou 
RESULT=SUCCESS STATUS=unknown TIMESTAMP=1375334708 DATE=2013-08-01 TIME=01:25:08 ACTION=AUTHENTICATE REQ_ID=1b64f62b797c6b24bd63f00b7e2fc06b USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou IP=203.114.244.88 SCOPES=session Authenticated
RESULT=FAILURE STATUS=service_unavailable TIMESTAMP=1375334713 DATE=2013-08-01 TIME=01:25:13 ACTION=ADD_APPLICATION REQ_ID=1b64f62b797c6b24bd63f00b7e2fc06b USER_ID=51f9d65559e81d14b5000001 LOGIN=jhou DOMAIN=jhou Unable to complete the requested operation because the system is unavailable. If the problem persists please contact Red Hat support.