This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 789454

Summary: Drift detection fails for whole directory if special-file is present
Product: [Other] RHQ Project Reporter: Heiko W. Rupp <hrupp>
Component: driftAssignee: Jay Shaughnessy <jshaughn>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.2CC: loleary, spinder
Target Milestone: ---   
Target Release: JON 3.0.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 790598 (view as bug list) Environment:
Last Closed: 2013-09-03 11:05:25 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 728579, 790598    
Attachments:
Description Flags
Agent log documenting failed verification none

Description Heiko W. Rupp 2012-02-10 14:47:39 EST
DriftDetection fails if eht directory to be monitored e.g. contains a unix domain socket (see stack trace below) 

org.rhq.core.pc.drift.DriftDetector#doDirectoryScan

basically does
       for (File dir : getScanDirectories(basedir, includes)) {
            forEachFile(dir, new FilterFileVisitor(basedir, includes, excludes, new FileVisitor() {
                @Override
                public void visit(File file) {
                    try {
                     ... work_on_file... <<--- fails for socket, see below
                    } catch (IOException e) {
                        throw new DriftDetectionException(
                            "An error occurred while generating a coverage change set for " + schedule, e);
                    }
}

So if the _work_on_file part is throwing an IOException as it does for a socket, the catch-block is re-throwing that as DriftDetectionException and throwing out of  the whole set of loops, basically ignoring all files that are still to be processed. 
As the postgres socket name starts with a dot, it is sorted early in the list, so that it basically breaks drift detection for the directory completely.

While /tmp may be a corner case for drift detection, unix domain sockets (and other kinds of special files) can show up in all directories.

2012-02-10 20:36:59,963 ERROR [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Drift detection failed: An error occurred while generating a coverage change set for DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 10001, driftDefinitionName: fosdem]
org.rhq.core.pc.drift.DriftDetectionException: An error occurred while generating a coverage change set for DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 10001, driftDefinitionName: fosdem]
	at org.rhq.core.pc.drift.DriftDetector$2.visit(DriftDetector.java:667)
	at org.rhq.core.pc.drift.FilterFileVisitor.visit(FilterFileVisitor.java:136)
	at org.rhq.core.util.file.FileUtil.forEachFile(FileUtil.java:434)
	at org.rhq.core.pc.drift.DriftDetector.doDirectoryScan(DriftDetector.java:649)
	at org.rhq.core.pc.drift.DriftDetector.generateSnapshot(DriftDetector.java:627)
	at org.rhq.core.pc.drift.DriftDetector.run(DriftDetector.java:139)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.FileNotFoundException: /tmp/.s.PGSQL.5432 (Operation not supported on socket)
	at java.io.FileInputStream.open(Native Method)
Comment 1 Mike Foley 2012-02-10 14:57:29 EST
when verifying ... please also try edge cases such as:

links
named pipes
sockets
block devices

see section 3-1 in this document:
http://tldp.org/LDP/intro-linux/html/sect_03_01.html
Comment 2 Mike Foley 2012-02-13 11:51:33 EST
when verifying, please also (re) verify file permission testcases ...particularly files which the agent does not have permission for
Comment 3 Heiko W. Rupp 2012-02-14 11:17:12 EST
missing lines of the stack trace

Caused by: java.io.FileNotFoundException: /tmp/.s.PGSQL.5432 (Operation not supported on socket)
	at java.io.FileInputStream.open(Native Method)
	at java.io.FileInputStream.<init>(FileInputStream.java:120)
	at org.rhq.core.util.MessageDigestGenerator.calcDigest(MessageDigestGenerator.java:255)
	at org.rhq.core.util.MessageDigestGenerator.calcDigestString(MessageDigestGenerator.java:291)
	at org.rhq.core.pc.drift.DriftDetector.sha256(DriftDetector.java:689)
	at org.rhq.core.pc.drift.DriftDetector.access$200(DriftDetector.java:57)
	at org.rhq.core.pc.drift.DriftDetector$2.visit(DriftDetector.java:662)
Comment 4 Jay Shaughnessy 2012-02-14 17:08:31 EST
release/3.0.x commit: f55c68ee2db419cbbe2daef2f7721a7c68279039

 File.canRead() returning true means that the process has read access to
 a file, it is a security check. It does not mean that the file can be
 parsed as a Stream.  In particular, FileInputStream(File) can throw
 FileNotFoundException on a file it can't actually handle, like a
 socket file.

 I added more protection for this situation, for files which drift monitoring
 can not and should not be performed.  They will be skipped (and logged in
 debug).  Moreover, taking Heiko's suggestion, don't fail drift detection
 due to a problematic file, even if the problem is unexpected.
Comment 5 Simeon Pinder 2012-02-17 00:33:53 EST
Moving to ON_QA for testing with JON 3.0.1.GA RC5 or better:
https://brewweb.devel.redhat.com//buildinfo?buildID=199114
Comment 6 Mike Foley 2012-02-23 11:09:57 EST
I am marking this failed to verify in JON 3.01 RC5.

I created a Drift configuration named "mytmp" on my /tmp directory, which contained both special files, as well as files without permisssions.  Snapshot #0 is never created.

Attached is the agent log file documenting this test.  DEBUG was turned on.

Notice:
1)  at 10:48 ... Drift Detection is requested.

2012-02-23 10:47:19,117 DEBUG [WorkerThread#0[192.168.0.111:56452]] (rhq.core.pc.drift.DriftManager)- DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 10012, driftDefinitionName: mytmp] has been added to ScheduleQueue[DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 10012, driftDefinitionName: mytmp], DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 10001, driftDefinitionName: mydrift]]

2) at 10:48 ...Drift Detection has begun

2012-02-23 10:48:14,847 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Adding /tmp/imageio8311409316851006636.tmp to coverage change set for DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 0, driftDefinitionName: mytmp]

3) at 10:48 ...Drift Detection correctly skips over some files 

2012-02-23 10:48:18,293 DEBUG [pool-3-thread-1] (rhq.core.pc.drift.DriftDetector)- Skipping /tmp/orbit-mfoley/linc-9ffa-0-662d6c6de3691 since it is missing or is not a physically readable file.

4) ...by 10:49 ...Drift Detection has stopped 

[Nothing in logs relating to Drift]

[The UI does not show Snapshot #0 ]


5)  at 10:58 (which is 10 minutes later) ... I request Drift Detection again 

012-02-23 10:58:25,542 DEBUG [WorkerThread#0[192.168.0.111:59770]] (rhq.core.pc.drift.DriftManager)- DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 10012, driftDefinitionName: mytmp] has been added to ScheduleQueue[DriftDetectionSchedule[resourceId: 10001, driftDefinitionId: 10012,

6) at 10:58+ ... there is no more Drift Detection activity in the agent log
7) at 10:58+ ... the UI is not showing me snapshot #0 
8) at 10:59 .... I create a file in /tmp


echo hello > mynewfile3.txt


9) at 11:07 .... nothing more in the agent logs, still no snapshot 0
Comment 7 Mike Foley 2012-02-23 11:10:42 EST
Created attachment 565326 [details]
Agent log documenting failed verification
Comment 8 Mike Foley 2012-02-23 11:25:25 EST
just adding onto comment #6 ...

30 minutes later ... still no snapshot #0 in the UI.  no more drift activity in agent logs.  drift is blocked or hung.
Comment 9 Mike Foley 2012-02-27 13:42:57 EST
observed by jay

analysis...  root cause is special file permissions ... 'prw' ...fifo ...or named pipe 

esoteric edge case ... but it does hang drift if it occurs.

[root@foleymonsterbox1 icedteaplugin-mfoley]# ls -al
<mfoley> prw-------.  1 mfoley mfoley     0 Jul 12  2011 24966-icedteanp-appletviewer-to-plugin
Comment 10 Jay Shaughnessy 2012-02-27 14:36:20 EST
I found that the problem here is not related to the files being handled
by this BZ.  A new BZ will be written up about named pipe files.

As for this, I created a drift def using mfoley's system, which included
unreadable (non pipe) files as well as regular text files and my test
passed fine.  I used filters to avoid the pipe files.

I recommend this be moved to VERIFIED.
Comment 11 Mike Foley 2012-02-27 14:39:45 EST
added https://bugzilla.redhat.com/show_bug.cgi?id=798006 per comment #10.
Comment 12 Mike Foley 2012-02-27 14:57:41 EST
agree with comment #10.

additionally, i expressly verified that drift definitions do not hang on socket files (the original issue).
Comment 13 Heiko W. Rupp 2013-09-03 11:05:25 EDT
Bulk closing of old issues in VERIFIED state.