1326881 – PV Recycler fails to clean a HostPath volume

Bug 1326881 - PV Recycler fails to clean a HostPath volume

Summary: PV Recycler fails to clean a HostPath volume

Keywords:
Status:	CLOSED DUPLICATE of bug 1332688
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Storage
Sub Component:
Version:	3.1.0
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Sami Wagiaalla
QA Contact:	Jianwei Hou
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-04-13 16:00 UTC by Boris Kurktchiev
Modified:	2016-06-07 22:46 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-05-17 15:01:06 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
recycler binary (771.94 KB, application/x-gzip) 2016-04-29 18:13 UTC, Sami Wagiaalla	no flags	Details
recycler binary (774.03 KB, application/x-gzip) 2016-05-03 17:04 UTC, Sami Wagiaalla	no flags	Details
recycler binary (774.99 KB, application/x-gzip) 2016-05-03 18:21 UTC, Sami Wagiaalla	no flags	Details
Show Obsolete (2) View All

Description Boris Kurktchiev 2016-04-13 16:00:41 UTC

Description of problem:
I have a HostPath volume attached to the metrics infrastructure. When I release it and redeploy the PV fails to be cleaned up with the following message: Scrub failed: /scrub/commitlog (drwxr-xr-x), calling walkFn: remove /scrub/commitlog: directory not empty

Version-Release number of selected component (if applicable):
oc v3.1.1.6-33-g81eabcc
kubernetes v1.1.0-origin-1107-g4c8e6f4

How reproducible:
Add HostPath PV, claim it, then release it. Go to node the pv pod was created on and run docker ps -a | grep pv, docker logs $dockerIDofPV and you should see Scrub failed: /scrub/commitlog (drwxr-xr-x), calling walkFn: remove /scrub/commitlog: directory not empty

Note that I have a bug opened about the error message the system gives when this happens here https://bugzilla.redhat.com/show_bug.cgi?id=1326879

Comment 1 Jan Safranek 2016-04-14 08:31:24 UTC

It seems that the scrubber left some files there. Can you please manually check what files are there and what owner/permission do they have? E.g. `ls -laR <the directory>`

Comment 2 Boris Kurktchiev 2016-04-14 12:20:56 UTC

For what its worth this seems to only be a problem with the Metrics infrastructure/Cassandra. In any other project I can grab/release a PV (thats setup the exact same way as the metrics volume)

root@osmaster0s:/ose_pv_infra_storage/metrics:
----> ls -laR
.:
total 16
drwxrwxr-x. 4 root       root 4096 Apr 12 13:59 .
drwxrwxrwx. 4 root       root 4096 Mar 24 15:43 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 13 08:50 commitlog
drwxr-xr-x. 7 1000020000 root 4096 Apr 12 14:01 data

./commitlog:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 13 08:50 .
drwxrwxr-x. 4 root       root 4096 Apr 12 13:59 ..

./data:
total 28
drwxr-xr-x.  7 1000020000 root 4096 Apr 12 14:01 .
drwxrwxr-x.  4 root       root 4096 Apr 12 13:59 ..
drwxr-xr-x. 11 1000020000 root 4096 Apr 12 14:01 hawkular_metrics
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 system
drwxr-xr-x.  6 1000020000 root 4096 Apr 12 14:00 system_auth
drwxr-xr-x.  4 1000020000 root 4096 Apr 12 14:00 system_distributed
drwxr-xr-x.  4 1000020000 root 4096 Apr 12 14:00 system_traces

./data/hawkular_metrics:
total 52
drwxr-xr-x. 11 1000020000 root  4096 Apr 12 14:01 .
drwxr-xr-x.  7 1000020000 root  4096 Apr 12 14:01 ..
drwxr-xr-x.  3 1000020000 root 12288 Apr 13 08:06 data-a1b4908000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 03:11 leases-a2e03f4000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 06:34 metrics_idx-a21bb6c000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 03:11 metrics_tags_idx-a1e2303000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 14:01 retentions_idx-a240caf000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 14:01 task_queue-a2be5f6000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 14:01 tasks-a28e9cd000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 14:01 tenants-a14701a000d811e685a0b16bcccac1a5
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 03:11 tenants_by_time-a17600e000d811e685a0b16bcccac1a5

./data/hawkular_metrics/data-a1b4908000d811e685a0b16bcccac1a5:
total 46764
drwxr-xr-x.  3 1000020000 root    12288 Apr 13 08:06 .
drwxr-xr-x. 11 1000020000 root     4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root     4096 Apr 12 14:01 backups
-rw-r--r--.  1 1000020000 root    14891 Apr 13 03:11 la-17-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 34373990 Apr 13 03:11 la-17-big-Data.db
-rw-r--r--.  1 1000020000 root        8 Apr 13 03:11 la-17-big-Digest.adler32
-rw-r--r--.  1 1000020000 root      312 Apr 13 03:11 la-17-big-Filter.db
-rw-r--r--.  1 1000020000 root   179527 Apr 13 03:11 la-17-big-Index.db
-rw-r--r--.  1 1000020000 root     6831 Apr 13 03:11 la-17-big-Statistics.db
-rw-r--r--.  1 1000020000 root      417 Apr 13 03:11 la-17-big-Summary.db
-rw-r--r--.  1 1000020000 root       94 Apr 13 03:11 la-17-big-TOC.txt
-rw-r--r--.  1 1000020000 root     1251 Apr 13 04:17 la-18-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  2827562 Apr 13 04:17 la-18-big-Data.db
-rw-r--r--.  1 1000020000 root        9 Apr 13 04:17 la-18-big-Digest.adler32
-rw-r--r--.  1 1000020000 root      288 Apr 13 04:17 la-18-big-Filter.db
-rw-r--r--.  1 1000020000 root    21263 Apr 13 04:17 la-18-big-Index.db
-rw-r--r--.  1 1000020000 root     6707 Apr 13 04:17 la-18-big-Statistics.db
-rw-r--r--.  1 1000020000 root      407 Apr 13 04:17 la-18-big-Summary.db
-rw-r--r--.  1 1000020000 root       94 Apr 13 04:17 la-18-big-TOC.txt
-rw-r--r--.  1 1000020000 root     1059 Apr 13 05:11 la-19-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  2335962 Apr 13 05:11 la-19-big-Data.db
-rw-r--r--.  1 1000020000 root        9 Apr 13 05:11 la-19-big-Digest.adler32
-rw-r--r--.  1 1000020000 root      288 Apr 13 05:11 la-19-big-Filter.db
-rw-r--r--.  1 1000020000 root    21263 Apr 13 05:11 la-19-big-Index.db
-rw-r--r--.  1 1000020000 root     6707 Apr 13 05:11 la-19-big-Statistics.db
-rw-r--r--.  1 1000020000 root      407 Apr 13 05:11 la-19-big-Summary.db
-rw-r--r--.  1 1000020000 root       94 Apr 13 05:11 la-19-big-TOC.txt
-rw-r--r--.  1 1000020000 root      899 Apr 13 05:56 la-20-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1935649 Apr 13 05:56 la-20-big-Data.db
-rw-r--r--.  1 1000020000 root        9 Apr 13 05:56 la-20-big-Digest.adler32
-rw-r--r--.  1 1000020000 root      288 Apr 13 05:56 la-20-big-Filter.db
-rw-r--r--.  1 1000020000 root    21263 Apr 13 05:56 la-20-big-Index.db
-rw-r--r--.  1 1000020000 root     6707 Apr 13 05:56 la-20-big-Statistics.db
-rw-r--r--.  1 1000020000 root      407 Apr 13 05:56 la-20-big-Summary.db
-rw-r--r--.  1 1000020000 root       94 Apr 13 05:56 la-20-big-TOC.txt
-rw-r--r--.  1 1000020000 root     1371 Apr 13 07:07 la-21-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  3110621 Apr 13 07:07 la-21-big-Data.db
-rw-r--r--.  1 1000020000 root        9 Apr 13 07:07 la-21-big-Digest.adler32
-rw-r--r--.  1 1000020000 root      288 Apr 13 07:07 la-21-big-Filter.db
-rw-r--r--.  1 1000020000 root    21263 Apr 13 07:07 la-21-big-Index.db
-rw-r--r--.  1 1000020000 root     6707 Apr 13 07:07 la-21-big-Statistics.db
-rw-r--r--.  1 1000020000 root      407 Apr 13 07:07 la-21-big-Summary.db
-rw-r--r--.  1 1000020000 root       94 Apr 13 07:07 la-21-big-TOC.txt
-rw-r--r--.  1 1000020000 root     1163 Apr 13 08:06 la-22-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  2584542 Apr 13 08:06 la-22-big-Data.db
-rw-r--r--.  1 1000020000 root       10 Apr 13 08:06 la-22-big-Digest.adler32
-rw-r--r--.  1 1000020000 root      288 Apr 13 08:06 la-22-big-Filter.db
-rw-r--r--.  1 1000020000 root    21263 Apr 13 08:06 la-22-big-Index.db
-rw-r--r--.  1 1000020000 root     6707 Apr 13 08:06 la-22-big-Statistics.db
-rw-r--r--.  1 1000020000 root      407 Apr 13 08:06 la-22-big-Summary.db
-rw-r--r--.  1 1000020000 root       94 Apr 13 08:06 la-22-big-TOC.txt

./data/hawkular_metrics/data-a1b4908000d811e685a0b16bcccac1a5/backups:
total 16
drwxr-xr-x. 2 1000020000 root  4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 12288 Apr 13 08:06 ..

./data/hawkular_metrics/leases-a2e03f4000d811e685a0b16bcccac1a5:
total 116
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 03:11 .
drwxr-xr-x. 11 1000020000 root  4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root  4096 Apr 12 14:01 backups
-rw-r--r--.  1 1000020000 root    43 Apr 12 14:11 la-1-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   188 Apr 12 14:11 la-1-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 12 14:11 la-1-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    24 Apr 12 14:11 la-1-big-Filter.db
-rw-r--r--.  1 1000020000 root   220 Apr 12 14:11 la-1-big-Index.db
-rw-r--r--.  1 1000020000 root  4629 Apr 12 14:11 la-1-big-Statistics.db
-rw-r--r--.  1 1000020000 root    92 Apr 12 14:11 la-1-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 12 14:11 la-1-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 12 14:41 la-2-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   498 Apr 12 14:41 la-2-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 12 14:41 la-2-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    48 Apr 12 14:41 la-2-big-Filter.db
-rw-r--r--.  1 1000020000 root   638 Apr 12 14:41 la-2-big-Index.db
-rw-r--r--.  1 1000020000 root  4990 Apr 12 14:41 la-2-big-Statistics.db
-rw-r--r--.  1 1000020000 root    92 Apr 12 14:41 la-2-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 12 14:41 la-2-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 03:11 la-3-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 12391 Apr 13 03:11 la-3-big-Data.db
-rw-r--r--.  1 1000020000 root     9 Apr 13 03:11 la-3-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   952 Apr 13 03:11 la-3-big-Filter.db
-rw-r--r--.  1 1000020000 root 16456 Apr 13 03:11 la-3-big-Index.db
-rw-r--r--.  1 1000020000 root  8155 Apr 13 03:11 la-3-big-Statistics.db
-rw-r--r--.  1 1000020000 root   192 Apr 13 03:11 la-3-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 03:11 la-3-big-TOC.txt

./data/hawkular_metrics/leases-a2e03f4000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 03:11 ..

./data/hawkular_metrics/metrics_idx-a21bb6c000d811e685a0b16bcccac1a5:
total 208
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 06:34 .
drwxr-xr-x. 11 1000020000 root  4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root  4096 Apr 12 14:01 backups
-rw-r--r--.  1 1000020000 root   123 Apr 13 03:11 la-8-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 99035 Apr 13 03:11 la-8-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 03:11 la-8-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    24 Apr 13 03:11 la-8-big-Filter.db
-rw-r--r--.  1 1000020000 root  2231 Apr 13 03:11 la-8-big-Index.db
-rw-r--r--.  1 1000020000 root  4609 Apr 13 03:11 la-8-big-Statistics.db
-rw-r--r--.  1 1000020000 root   118 Apr 13 03:11 la-8-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 03:11 la-8-big-TOC.txt
-rw-r--r--.  1 1000020000 root    75 Apr 13 06:34 la-9-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 40188 Apr 13 06:34 la-9-big-Data.db
-rw-r--r--.  1 1000020000 root     9 Apr 13 06:34 la-9-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    24 Apr 13 06:34 la-9-big-Filter.db
-rw-r--r--.  1 1000020000 root   356 Apr 13 06:34 la-9-big-Index.db
-rw-r--r--.  1 1000020000 root  4582 Apr 13 06:34 la-9-big-Statistics.db
-rw-r--r--.  1 1000020000 root   118 Apr 13 06:34 la-9-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 06:34 la-9-big-TOC.txt

./data/hawkular_metrics/metrics_idx-a21bb6c000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 06:34 ..

./data/hawkular_metrics/metrics_tags_idx-a1e2303000d811e685a0b16bcccac1a5:
total 156
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 03:11 .
drwxr-xr-x. 11 1000020000 root  4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root  4096 Apr 12 14:01 backups
-rw-r--r--.  1 1000020000 root   115 Apr 13 03:11 la-5-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 95617 Apr 13 03:11 la-5-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 03:11 la-5-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    96 Apr 13 03:11 la-5-big-Filter.db
-rw-r--r--.  1 1000020000 root 19279 Apr 13 03:11 la-5-big-Index.db
-rw-r--r--.  1 1000020000 root  4957 Apr 13 03:11 la-5-big-Statistics.db
-rw-r--r--.  1 1000020000 root   280 Apr 13 03:11 la-5-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 03:11 la-5-big-TOC.txt

./data/hawkular_metrics/metrics_tags_idx-a1e2303000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 03:11 ..

./data/hawkular_metrics/retentions_idx-a240caf000d811e685a0b16bcccac1a5:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 11 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 14:01 backups

./data/hawkular_metrics/retentions_idx-a240caf000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:01 ..

./data/hawkular_metrics/task_queue-a2be5f6000d811e685a0b16bcccac1a5:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 11 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 14:01 backups

./data/hawkular_metrics/task_queue-a2be5f6000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:01 ..

./data/hawkular_metrics/tasks-a28e9cd000d811e685a0b16bcccac1a5:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 11 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 14:01 backups

./data/hawkular_metrics/tasks-a28e9cd000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:01 ..

./data/hawkular_metrics/tenants-a14701a000d811e685a0b16bcccac1a5:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 11 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 14:01 backups

./data/hawkular_metrics/tenants-a14701a000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:01 ..

./data/hawkular_metrics/tenants_by_time-a17600e000d811e685a0b16bcccac1a5:
total 36
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 03:11 .
drwxr-xr-x. 11 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 14:01 backups
-rw-r--r--.  1 1000020000 root   43 Apr 13 03:11 la-5-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 2018 Apr 13 03:11 la-5-big-Data.db
-rw-r--r--.  1 1000020000 root    9 Apr 13 03:11 la-5-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   32 Apr 13 03:11 la-5-big-Filter.db
-rw-r--r--.  1 1000020000 root  594 Apr 13 03:11 la-5-big-Index.db
-rw-r--r--.  1 1000020000 root 4549 Apr 13 03:11 la-5-big-Statistics.db
-rw-r--r--.  1 1000020000 root   92 Apr 13 03:11 la-5-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 03:11 la-5-big-TOC.txt

./data/hawkular_metrics/tenants_by_time-a17600e000d811e685a0b16bcccac1a5/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 03:11 ..

./data/system:
total 112
drwxr-xr-x. 22 1000020000 root  4096 Apr 12 13:59 .
drwxr-xr-x.  7 1000020000 root  4096 Apr 12 14:01 ..
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 13:59 available_ranges-c539fcabd65a31d18133d25605643ee3
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 13:59 batchlog-0290003c977e397cac3efdfdc01d626b
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 05:11 compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca
drwxr-xr-x.  4 1000020000 root  8192 Apr 13 03:11 compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 13:59 hints-2666e20573ef38b390fefecf96e8f0c7
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 13:59 IndexInfo-9f5c6374d48532299a0a5094af9ad1e3
drwxr-xr-x.  3 1000020000 root  8192 Apr 13 08:50 local-7ad54392bcdd35a684174e047860b377
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 13:59 paxos-b7b7f0c2fd0a34108c053ef614bb7c2d
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 13:59 peer_events-59dfeaea8db2334191ef109974d81484
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 peers-37f71aca7dc2383ba70672528af04d4f
drwxr-xr-x.  3 1000020000 root  4096 Apr 12 13:59 range_xfers-55d764384e553f8b9f6e676d4af3976d
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 schema_aggregates-a5fc57fc9d6c3bfda3fc01ad54686fea
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 schema_columns-296e9c049bec3085827dc17d3df2122a
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 schema_functions-d1b675fe2b503ca48e49c0f81989dcad
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 schema_keyspaces-b0f2235744583cdb9631c43e59ce3676
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 schema_triggers-0359bc7171233ee19a4ab9dfb11fc125
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 schema_usertypes-3aa752254f82350b8d5c430fa221fa0a
drwxr-xr-x.  3 1000020000 root 12288 Apr 13 08:11 size_estimates-618f817b005f3678b8a453f3930b8e86
drwxr-xr-x.  3 1000020000 root 12288 Apr 13 08:11 sstable_activity-5a1ff267ace03f128563cfae6103c65e

./data/system/available_ranges-c539fcabd65a31d18133d25605643ee3:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups

./data/system/available_ranges-c539fcabd65a31d18133d25605643ee3/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 13:59 ..

./data/system/batchlog-0290003c977e397cac3efdfdc01d626b:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups

./data/system/batchlog-0290003c977e397cac3efdfdc01d626b/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 13:59 ..

./data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca:
total 108
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 05:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 12 15:41 la-5-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 2258 Apr 12 15:41 la-5-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 12 15:41 la-5-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   40 Apr 12 15:41 la-5-big-Filter.db
-rw-r--r--.  1 1000020000 root  690 Apr 12 15:41 la-5-big-Index.db
-rw-r--r--.  1 1000020000 root 4704 Apr 12 15:41 la-5-big-Statistics.db
-rw-r--r--.  1 1000020000 root  116 Apr 12 15:41 la-5-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 12 15:41 la-5-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 12 16:41 la-6-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  748 Apr 12 16:41 la-6-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 12 16:41 la-6-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   24 Apr 12 16:41 la-6-big-Filter.db
-rw-r--r--.  1 1000020000 root  180 Apr 12 16:41 la-6-big-Index.db
-rw-r--r--.  1 1000020000 root 4516 Apr 12 16:41 la-6-big-Statistics.db
-rw-r--r--.  1 1000020000 root  116 Apr 12 16:41 la-6-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 12 16:41 la-6-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 04:11 la-7-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 1168 Apr 13 04:11 la-7-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 04:11 la-7-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   24 Apr 13 04:11 la-7-big-Filter.db
-rw-r--r--.  1 1000020000 root  240 Apr 13 04:11 la-7-big-Index.db
-rw-r--r--.  1 1000020000 root 4653 Apr 13 04:11 la-7-big-Statistics.db
-rw-r--r--.  1 1000020000 root  116 Apr 13 04:11 la-7-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 04:11 la-7-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 05:11 la-8-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  458 Apr 13 05:11 la-8-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 05:11 la-8-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 05:11 la-8-big-Filter.db
-rw-r--r--.  1 1000020000 root   90 Apr 13 05:11 la-8-big-Index.db
-rw-r--r--.  1 1000020000 root 4538 Apr 13 05:11 la-8-big-Statistics.db
-rw-r--r--.  1 1000020000 root  116 Apr 13 05:11 la-8-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 05:11 la-8-big-TOC.txt

./data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 05:11 ..

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b:
total 56
drwxr-xr-x.  4 1000020000 root 8192 Apr 13 03:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 13 03:11 la-14-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  217 Apr 13 03:11 la-14-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 03:11 la-14-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 03:11 la-14-big-Filter.db
-rw-r--r--.  1 1000020000 root   30 Apr 13 03:11 la-14-big-Index.db
-rw-r--r--.  1 1000020000 root 4462 Apr 13 03:11 la-14-big-Statistics.db
-rw-r--r--.  1 1000020000 root  116 Apr 13 03:11 la-14-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 03:11 la-14-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 03:11 la-15-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   42 Apr 13 03:11 la-15-big-Data.db
-rw-r--r--.  1 1000020000 root    9 Apr 13 03:11 la-15-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 03:11 la-15-big-Filter.db
-rw-r--r--.  1 1000020000 root   30 Apr 13 03:11 la-15-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 13 03:11 la-15-big-Statistics.db
-rw-r--r--.  1 1000020000 root  116 Apr 13 03:11 la-15-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 03:11 la-15-big-TOC.txt
drwxr-xr-x.  7 1000020000 root 4096 Apr 13 03:11 snapshots

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/backups:
total 12
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 4 1000020000 root 8192 Apr 13 03:11 ..

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots:
total 32
drwxr-xr-x. 7 1000020000 root 4096 Apr 13 03:11 .
drwxr-xr-x. 4 1000020000 root 8192 Apr 13 03:11 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 1460483987167-compactions_in_progress
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 1460484070014-compactions_in_progress
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:11 1460484713985-compactions_in_progress
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:41 1460486497102-compactions_in_progress
drwxr-xr-x. 2 1000020000 root 4096 Apr 13 03:11 1460531466482-compactions_in_progress

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/1460483987167-compactions_in_progress:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 13 03:11 ..
-rw-r--r--. 1 1000020000 root   13 Apr 12 13:59 manifest.json

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/1460484070014-compactions_in_progress:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 13 03:11 ..
-rw-r--r--. 1 1000020000 root   13 Apr 12 14:01 manifest.json

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/1460484713985-compactions_in_progress:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:11 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 13 03:11 ..
-rw-r--r--. 1 1000020000 root   13 Apr 12 14:11 manifest.json

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/1460486497102-compactions_in_progress:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:41 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 13 03:11 ..
-rw-r--r--. 1 1000020000 root   13 Apr 12 14:41 manifest.json

./data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/snapshots/1460531466482-compactions_in_progress:
total 48
drwxr-xr-x. 2 1000020000 root 4096 Apr 13 03:11 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 13 03:11 ..
-rw-r--r--. 1 1000020000 root   43 Apr 12 14:41 la-5-big-CompressionInfo.db
-rw-r--r--. 1 1000020000 root   42 Apr 12 14:41 la-5-big-Data.db
-rw-r--r--. 1 1000020000 root   10 Apr 12 14:41 la-5-big-Digest.adler32
-rw-r--r--. 1 1000020000 root   16 Apr 12 14:41 la-5-big-Filter.db
-rw-r--r--. 1 1000020000 root   30 Apr 12 14:41 la-5-big-Index.db
-rw-r--r--. 1 1000020000 root 4450 Apr 12 14:41 la-5-big-Statistics.db
-rw-r--r--. 1 1000020000 root  116 Apr 12 14:41 la-5-big-Summary.db
-rw-r--r--. 1 1000020000 root   94 Apr 12 14:41 la-5-big-TOC.txt
-rw-r--r--. 1 1000020000 root   43 Apr 12 14:41 la-6-big-CompressionInfo.db
-rw-r--r--. 1 1000020000 root  202 Apr 12 14:41 la-6-big-Data.db
-rw-r--r--. 1 1000020000 root   10 Apr 12 14:41 la-6-big-Digest.adler32
-rw-r--r--. 1 1000020000 root   16 Apr 12 14:41 la-6-big-Filter.db
-rw-r--r--. 1 1000020000 root   90 Apr 12 14:41 la-6-big-Index.db
-rw-r--r--. 1 1000020000 root 4472 Apr 12 14:41 la-6-big-Statistics.db
-rw-r--r--. 1 1000020000 root  116 Apr 12 14:41 la-6-big-Summary.db
-rw-r--r--. 1 1000020000 root   94 Apr 12 14:41 la-6-big-TOC.txt
-rw-r--r--. 1 1000020000 root   50 Apr 13 03:11 manifest.json

./data/system/hints-2666e20573ef38b390fefecf96e8f0c7:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups

./data/system/hints-2666e20573ef38b390fefecf96e8f0c7/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 13:59 ..

./data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups

./data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 13:59 ..

./data/system/local-7ad54392bcdd35a684174e047860b377:
total 60
drwxr-xr-x.  3 1000020000 root 8192 Apr 13 08:50 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 13 03:11 la-27-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 5764 Apr 13 03:11 la-27-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 03:11 la-27-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 03:11 la-27-big-Filter.db
-rw-r--r--.  1 1000020000 root   19 Apr 13 03:11 la-27-big-Index.db
-rw-r--r--.  1 1000020000 root 4458 Apr 13 03:11 la-27-big-Statistics.db
-rw-r--r--.  1 1000020000 root   83 Apr 13 03:11 la-27-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 03:11 la-27-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 08:50 la-28-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   77 Apr 13 08:50 la-28-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 08:50 la-28-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 08:50 la-28-big-Filter.db
-rw-r--r--.  1 1000020000 root   19 Apr 13 08:50 la-28-big-Index.db
-rw-r--r--.  1 1000020000 root 4434 Apr 13 08:50 la-28-big-Statistics.db
-rw-r--r--.  1 1000020000 root   83 Apr 13 08:50 la-28-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 08:50 la-28-big-TOC.txt

./data/system/local-7ad54392bcdd35a684174e047860b377/backups:
total 12
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 8192 Apr 13 08:50 ..

./data/system/paxos-b7b7f0c2fd0a34108c053ef614bb7c2d:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups

./data/system/paxos-b7b7f0c2fd0a34108c053ef614bb7c2d/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 13:59 ..

./data/system/peer_events-59dfeaea8db2334191ef109974d81484:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups

./data/system/peer_events-59dfeaea8db2334191ef109974d81484/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 13:59 ..

./data/system/peers-37f71aca7dc2383ba70672528af04d4f:
total 44
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 13 03:11 la-1-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   30 Apr 13 03:11 la-1-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 03:11 la-1-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 03:11 la-1-big-Filter.db
-rw-r--r--.  1 1000020000 root   18 Apr 13 03:11 la-1-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 13 03:11 la-1-big-Statistics.db
-rw-r--r--.  1 1000020000 root   80 Apr 13 03:11 la-1-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 03:11 la-1-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 04:11 la-2-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   30 Apr 13 04:11 la-2-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 04:11 la-2-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 04:11 la-2-big-Filter.db
-rw-r--r--.  1 1000020000 root   18 Apr 13 04:11 la-2-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 13 04:11 la-2-big-Statistics.db
-rw-r--r--.  1 1000020000 root   80 Apr 13 04:11 la-2-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 04:11 la-2-big-TOC.txt

./data/system/peers-37f71aca7dc2383ba70672528af04d4f/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d:
total 12
drwxr-xr-x.  3 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups

./data/system/range_xfers-55d764384e553f8b9f6e676d4af3976d/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 13:59 ..

./data/system/schema_aggregates-a5fc57fc9d6c3bfda3fc01ad54686fea:
total 44
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 12 15:41 la-5-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   32 Apr 12 15:41 la-5-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 12 15:41 la-5-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 12 15:41 la-5-big-Filter.db
-rw-r--r--.  1 1000020000 root   20 Apr 12 15:41 la-5-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 12 15:41 la-5-big-Statistics.db
-rw-r--r--.  1 1000020000 root   86 Apr 12 15:41 la-5-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 12 15:41 la-5-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 04:11 la-6-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   32 Apr 13 04:11 la-6-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 04:11 la-6-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 04:11 la-6-big-Filter.db
-rw-r--r--.  1 1000020000 root   20 Apr 13 04:11 la-6-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 13 04:11 la-6-big-Statistics.db
-rw-r--r--.  1 1000020000 root   86 Apr 13 04:11 la-6-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 04:11 la-6-big-TOC.txt

./data/system/schema_aggregates-a5fc57fc9d6c3bfda3fc01ad54686fea/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697:
total 48
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root  4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root  4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root    51 Apr 13 04:11 la-21-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 12335 Apr 13 04:11 la-21-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 04:11 la-21-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    24 Apr 13 04:11 la-21-big-Filter.db
-rw-r--r--.  1 1000020000 root   134 Apr 13 04:11 la-21-big-Index.db
-rw-r--r--.  1 1000020000 root  4510 Apr 13 04:11 la-21-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 04:11 la-21-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 04:11 la-21-big-TOC.txt

./data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/schema_columns-296e9c049bec3085827dc17d3df2122a:
total 52
drwxr-xr-x.  3 1000020000 root  4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root  4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root  4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root    51 Apr 13 04:11 la-21-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root 20351 Apr 13 04:11 la-21-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 04:11 la-21-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    24 Apr 13 04:11 la-21-big-Filter.db
-rw-r--r--.  1 1000020000 root   343 Apr 13 04:11 la-21-big-Index.db
-rw-r--r--.  1 1000020000 root  4604 Apr 13 04:11 la-21-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 04:11 la-21-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 04:11 la-21-big-TOC.txt

./data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad:
total 44
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 12 15:41 la-5-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   32 Apr 12 15:41 la-5-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 12 15:41 la-5-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 12 15:41 la-5-big-Filter.db
-rw-r--r--.  1 1000020000 root   20 Apr 12 15:41 la-5-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 12 15:41 la-5-big-Statistics.db
-rw-r--r--.  1 1000020000 root   86 Apr 12 15:41 la-5-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 12 15:41 la-5-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 04:11 la-6-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   32 Apr 13 04:11 la-6-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 04:11 la-6-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 04:11 la-6-big-Filter.db
-rw-r--r--.  1 1000020000 root   20 Apr 13 04:11 la-6-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 13 04:11 la-6-big-Statistics.db
-rw-r--r--.  1 1000020000 root   86 Apr 13 04:11 la-6-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 04:11 la-6-big-TOC.txt

./data/system/schema_functions-d1b675fe2b503ca48e49c0f81989dcad/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676:
total 36
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 13 04:11 la-25-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  309 Apr 13 04:11 la-25-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 04:11 la-25-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   24 Apr 13 04:11 la-25-big-Filter.db
-rw-r--r--.  1 1000020000 root  134 Apr 13 04:11 la-25-big-Index.db
-rw-r--r--.  1 1000020000 root 4482 Apr 13 04:11 la-25-big-Statistics.db
-rw-r--r--.  1 1000020000 root  113 Apr 13 04:11 la-25-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 04:11 la-25-big-TOC.txt

./data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125:
total 44
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 12 15:41 la-5-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   32 Apr 12 15:41 la-5-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 12 15:41 la-5-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 12 15:41 la-5-big-Filter.db
-rw-r--r--.  1 1000020000 root   20 Apr 12 15:41 la-5-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 12 15:41 la-5-big-Statistics.db
-rw-r--r--.  1 1000020000 root   86 Apr 12 15:41 la-5-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 12 15:41 la-5-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 13 04:11 la-6-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   32 Apr 13 04:11 la-6-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 04:11 la-6-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 04:11 la-6-big-Filter.db
-rw-r--r--.  1 1000020000 root   20 Apr 13 04:11 la-6-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 13 04:11 la-6-big-Statistics.db
-rw-r--r--.  1 1000020000 root   86 Apr 13 04:11 la-6-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 04:11 la-6-big-TOC.txt

./data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a:
total 48
drwxr-xr-x.  3 1000020000 root 4096 Apr 13 04:11 .
drwxr-xr-x. 22 1000020000 root 4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root 4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root   43 Apr 13 04:11 la-10-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   32 Apr 13 04:11 la-10-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 13 04:11 la-10-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 13 04:11 la-10-big-Filter.db
-rw-r--r--.  1 1000020000 root   20 Apr 13 04:11 la-10-big-Index.db
-rw-r--r--.  1 1000020000 root 4450 Apr 13 04:11 la-10-big-Statistics.db
-rw-r--r--.  1 1000020000 root   86 Apr 13 04:11 la-10-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 13 04:11 la-10-big-TOC.txt
-rw-r--r--.  1 1000020000 root   43 Apr 12 15:41 la-9-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  809 Apr 12 15:41 la-9-big-Data.db
-rw-r--r--.  1 1000020000 root   10 Apr 12 15:41 la-9-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   16 Apr 12 15:41 la-9-big-Filter.db
-rw-r--r--.  1 1000020000 root   50 Apr 12 15:41 la-9-big-Index.db
-rw-r--r--.  1 1000020000 root 4531 Apr 12 15:41 la-9-big-Statistics.db
-rw-r--r--.  1 1000020000 root  106 Apr 12 15:41 la-9-big-Summary.db
-rw-r--r--.  1 1000020000 root   94 Apr 12 15:41 la-9-big-TOC.txt

./data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 13 04:11 ..

./data/system/size_estimates-618f817b005f3678b8a453f3930b8e86:
total 164
drwxr-xr-x.  3 1000020000 root 12288 Apr 13 08:11 .
drwxr-xr-x. 22 1000020000 root  4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root  4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root    43 Apr 13 03:11 la-16-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   930 Apr 13 03:11 la-16-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 03:11 la-16-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    16 Apr 13 03:11 la-16-big-Filter.db
-rw-r--r--.  1 1000020000 root   114 Apr 13 03:11 la-16-big-Index.db
-rw-r--r--.  1 1000020000 root  4617 Apr 13 03:11 la-16-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 03:11 la-16-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 03:11 la-16-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 04:11 la-17-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1294 Apr 13 04:11 la-17-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 04:11 la-17-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    16 Apr 13 04:11 la-17-big-Filter.db
-rw-r--r--.  1 1000020000 root   114 Apr 13 04:11 la-17-big-Index.db
-rw-r--r--.  1 1000020000 root  4573 Apr 13 04:11 la-17-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 04:11 la-17-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 04:11 la-17-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 05:11 la-18-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1229 Apr 13 05:11 la-18-big-Data.db
-rw-r--r--.  1 1000020000 root     9 Apr 13 05:11 la-18-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    16 Apr 13 05:11 la-18-big-Filter.db
-rw-r--r--.  1 1000020000 root   114 Apr 13 05:11 la-18-big-Index.db
-rw-r--r--.  1 1000020000 root  4573 Apr 13 05:11 la-18-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 05:11 la-18-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 05:11 la-18-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 06:11 la-19-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1235 Apr 13 06:11 la-19-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 06:11 la-19-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    16 Apr 13 06:11 la-19-big-Filter.db
-rw-r--r--.  1 1000020000 root   114 Apr 13 06:11 la-19-big-Index.db
-rw-r--r--.  1 1000020000 root  4573 Apr 13 06:11 la-19-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 06:11 la-19-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 06:11 la-19-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 07:11 la-20-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1295 Apr 13 07:11 la-20-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 07:11 la-20-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    16 Apr 13 07:11 la-20-big-Filter.db
-rw-r--r--.  1 1000020000 root   114 Apr 13 07:11 la-20-big-Index.db
-rw-r--r--.  1 1000020000 root  4573 Apr 13 07:11 la-20-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 07:11 la-20-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 07:11 la-20-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 08:11 la-21-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1225 Apr 13 08:11 la-21-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 08:11 la-21-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    16 Apr 13 08:11 la-21-big-Filter.db
-rw-r--r--.  1 1000020000 root   114 Apr 13 08:11 la-21-big-Index.db
-rw-r--r--.  1 1000020000 root  4573 Apr 13 08:11 la-21-big-Statistics.db
-rw-r--r--.  1 1000020000 root   113 Apr 13 08:11 la-21-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 08:11 la-21-big-TOC.txt

./data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/backups:
total 16
drwxr-xr-x. 2 1000020000 root  4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 12288 Apr 13 08:11 ..

./data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e:
total 176
drwxr-xr-x.  3 1000020000 root 12288 Apr 13 08:11 .
drwxr-xr-x. 22 1000020000 root  4096 Apr 12 13:59 ..
drwxr-xr-x.  2 1000020000 root  4096 Apr 12 13:59 backups
-rw-r--r--.  1 1000020000 root    43 Apr 13 03:11 la-18-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1053 Apr 13 03:11 la-18-big-Data.db
-rw-r--r--.  1 1000020000 root     9 Apr 13 03:11 la-18-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    80 Apr 13 03:11 la-18-big-Filter.db
-rw-r--r--.  1 1000020000 root  1179 Apr 13 03:11 la-18-big-Index.db
-rw-r--r--.  1 1000020000 root  4904 Apr 13 03:11 la-18-big-Statistics.db
-rw-r--r--.  1 1000020000 root   174 Apr 13 03:11 la-18-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 03:11 la-18-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 04:11 la-19-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root  1790 Apr 13 04:11 la-19-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 04:11 la-19-big-Digest.adler32
-rw-r--r--.  1 1000020000 root   120 Apr 13 04:11 la-19-big-Filter.db
-rw-r--r--.  1 1000020000 root  4226 Apr 13 04:11 la-19-big-Index.db
-rw-r--r--.  1 1000020000 root  4930 Apr 13 04:11 la-19-big-Statistics.db
-rw-r--r--.  1 1000020000 root   180 Apr 13 04:11 la-19-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 04:11 la-19-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 05:11 la-20-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   538 Apr 13 05:11 la-20-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 05:11 la-20-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    24 Apr 13 05:11 la-20-big-Filter.db
-rw-r--r--.  1 1000020000 root   455 Apr 13 05:11 la-20-big-Index.db
-rw-r--r--.  1 1000020000 root  4573 Apr 13 05:11 la-20-big-Statistics.db
-rw-r--r--.  1 1000020000 root   182 Apr 13 05:11 la-20-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 05:11 la-20-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 06:11 la-21-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   602 Apr 13 06:11 la-21-big-Data.db
-rw-r--r--.  1 1000020000 root     9 Apr 13 06:11 la-21-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    32 Apr 13 06:11 la-21-big-Filter.db
-rw-r--r--.  1 1000020000 root   549 Apr 13 06:11 la-21-big-Index.db
-rw-r--r--.  1 1000020000 root  4611 Apr 13 06:11 la-21-big-Statistics.db
-rw-r--r--.  1 1000020000 root   178 Apr 13 06:11 la-21-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 06:11 la-21-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 07:11 la-22-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   673 Apr 13 07:11 la-22-big-Data.db
-rw-r--r--.  1 1000020000 root    10 Apr 13 07:11 la-22-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    32 Apr 13 07:11 la-22-big-Filter.db
-rw-r--r--.  1 1000020000 root   650 Apr 13 07:11 la-22-big-Index.db
-rw-r--r--.  1 1000020000 root  4651 Apr 13 07:11 la-22-big-Statistics.db
-rw-r--r--.  1 1000020000 root   178 Apr 13 07:11 la-22-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 07:11 la-22-big-TOC.txt
-rw-r--r--.  1 1000020000 root    43 Apr 13 08:11 la-23-big-CompressionInfo.db
-rw-r--r--.  1 1000020000 root   711 Apr 13 08:11 la-23-big-Data.db
-rw-r--r--.  1 1000020000 root     9 Apr 13 08:11 la-23-big-Digest.adler32
-rw-r--r--.  1 1000020000 root    32 Apr 13 08:11 la-23-big-Filter.db
-rw-r--r--.  1 1000020000 root   697 Apr 13 08:11 la-23-big-Index.db
-rw-r--r--.  1 1000020000 root  4670 Apr 13 08:11 la-23-big-Statistics.db
-rw-r--r--.  1 1000020000 root   178 Apr 13 08:11 la-23-big-Summary.db
-rw-r--r--.  1 1000020000 root    94 Apr 13 08:11 la-23-big-TOC.txt

./data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/backups:
total 16
drwxr-xr-x. 2 1000020000 root  4096 Apr 12 13:59 .
drwxr-xr-x. 3 1000020000 root 12288 Apr 13 08:11 ..

./data/system_auth:
total 24
drwxr-xr-x. 6 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 resource_role_permissons_index-5f2fbdad91f13946bd25d5da3a5c35ec
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 role_members-0ecdaa87f8fb3e6088d174fb36fe5c0d
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 role_permissions-3afbe79f219431a7add7f5ab90d8ec9c
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:01 roles-5bc52802de2535edaeab188eecebb090

./data/system_auth/resource_role_permissons_index-5f2fbdad91f13946bd25d5da3a5c35ec:
total 12
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 6 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups

./data/system_auth/resource_role_permissons_index-5f2fbdad91f13946bd25d5da3a5c35ec/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 ..

./data/system_auth/role_members-0ecdaa87f8fb3e6088d174fb36fe5c0d:
total 12
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 6 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups

./data/system_auth/role_members-0ecdaa87f8fb3e6088d174fb36fe5c0d/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 ..

./data/system_auth/role_permissions-3afbe79f219431a7add7f5ab90d8ec9c:
total 12
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 6 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups

./data/system_auth/role_permissions-3afbe79f219431a7add7f5ab90d8ec9c/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 ..

./data/system_auth/roles-5bc52802de2535edaeab188eecebb090:
total 32
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:01 .
drwxr-xr-x. 6 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups
-rw-r--r--. 1 1000020000 root   43 Apr 12 14:01 la-1-big-CompressionInfo.db
-rw-r--r--. 1 1000020000 root  161 Apr 12 14:01 la-1-big-Data.db
-rw-r--r--. 1 1000020000 root   10 Apr 12 14:01 la-1-big-Digest.adler32
-rw-r--r--. 1 1000020000 root   16 Apr 12 14:01 la-1-big-Filter.db
-rw-r--r--. 1 1000020000 root   23 Apr 12 14:01 la-1-big-Index.db
-rw-r--r--. 1 1000020000 root 4434 Apr 12 14:01 la-1-big-Statistics.db
-rw-r--r--. 1 1000020000 root   95 Apr 12 14:01 la-1-big-Summary.db
-rw-r--r--. 1 1000020000 root   94 Apr 12 14:01 la-1-big-TOC.txt

./data/system_auth/roles-5bc52802de2535edaeab188eecebb090/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:01 ..

./data/system_distributed:
total 16
drwxr-xr-x. 4 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 repair_history-759fffad624b318180eefa9a52d1f627

./data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14:
total 12
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 4 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups

./data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 ..

./data/system_distributed/repair_history-759fffad624b318180eefa9a52d1f627:
total 12
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 4 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups

./data/system_distributed/repair_history-759fffad624b318180eefa9a52d1f627/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 ..

./data/system_traces:
total 16
drwxr-xr-x. 4 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 7 1000020000 root 4096 Apr 12 14:01 ..
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 events-8826e8e9e16a372887533bc1fc713c25
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 sessions-c5e99f1686773914b17e960613512345

./data/system_traces/events-8826e8e9e16a372887533bc1fc713c25:
total 12
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 4 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups

./data/system_traces/events-8826e8e9e16a372887533bc1fc713c25/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 ..

./data/system_traces/sessions-c5e99f1686773914b17e960613512345:
total 12
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 4 1000020000 root 4096 Apr 12 14:00 ..
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 backups

./data/system_traces/sessions-c5e99f1686773914b17e960613512345/backups:
total 8
drwxr-xr-x. 2 1000020000 root 4096 Apr 12 14:00 .
drwxr-xr-x. 3 1000020000 root 4096 Apr 12 14:00 ..




root@osmaster0s:/ose_pv_infra_storage/metrics:
----> oc describe pv metrics
Name:		metrics
Labels:		size=200Gi,type=infra
Status:		Failed
Claim:		openshift-infra/metrics-cassandra-1
Reclaim Policy:	Recycle
Access Modes:	RWO,RWX
Capacity:	200Gi
Message:	Recycling error: Pod failed, pod.Status.Message unknown.
Source:
    Type:	HostPath (bare host directory volume)
    Path:	/ose_pv_infra_storage/metrics/



root@osmaster0s:/ose_pv_infra_storage/metrics:
----> oc get pv metrics -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  creationTimestamp: 2016-03-24T19:43:16Z
  labels:
    size: 200Gi
    type: infra
  name: metrics
  resourceVersion: "1012279"
  selfLink: /api/v1/persistentvolumes/metrics
  uid: a776b85f-f1f8-11e5-9b14-005056a6874f
spec:
  accessModes:
  - ReadWriteOnce
  - ReadWriteMany
  capacity:
    storage: 200Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: metrics-cassandra-1
    namespace: openshift-infra
    resourceVersion: "967460"
    uid: 189f14a8-00d6-11e6-baef-005056a68fb4
  hostPath:
    path: /ose_pv_infra_storage/metrics/
  persistentVolumeReclaimPolicy: Recycle
status:
  message: 'Recycling error: Pod failed, pod.Status.Message unknown.'
  phase: Failed

Comment 3 Bradley Childs 2016-04-14 20:05:26 UTC

I see a couple of issues here.  First the data is owned by user 1000020000 and group root.  However the group permission bits aren't set as writeable.  The recycler pod would have to run as UID 1000020000 or chmod -R the directories to give the root user access prior to running the scrub script.

I believe the fix for this is modification to the SCC that the recycler POD runs under.  Assigning to Sami who should be back on Monday.

Comment 4 Boris Kurktchiev 2016-04-14 20:23:19 UTC

I want to point out that the recycler works just fine for all other PVs that are added to projects. This problem seems to be specific to the metrics infrastructure and the way it creates data in its PV

Comment 5 Sami Wagiaalla 2016-04-19 13:43:04 UTC

hmm.. the recycler script should become root then try to delete the file (https://github.com/openshift/origin/blob/master/pkg/cmd/recycle/walker.go#L79)

I will try to reproduce this using the permissions provided above but just to rule SELinux out could you try to reproduce with SELinux disabled ?

Comment 6 Boris Kurktchiev 2016-04-19 13:46:57 UTC

Is there a specific SELinux ruleset that you think is the culprit here? I am hesitant the just turn off SElinux flat out as I dont see how we will determine what was triggered. Also, just to re-iterate, this seems to only be happening in regards to the metrics infrastructure as any other PV claim from normal projects gets recycled just fine. All of my PVs are setup the exact same way.

Comment 7 Sami Wagiaalla 2016-04-19 18:01:16 UTC

> Is there a specific SELinux ruleset that you think is the culprit here? I am
> hesitant the just turn off SElinux flat out as I dont see how we will
> determine what was triggered.

disabling SELinux is a blunt instrument just to determine which path to take when diagnosing.

To do it more precisely:

tail -f /var/log/audit/audit.log
then trigger recycling and look for denials while recycling.

Also if you can post the labels for the files in the volume. Lets focus on things in the commitlog subdirectory:

ls -laZ /ose_pv_infra_storage/metrics/commitlog

> Also, just to re-iterate, this seems to only
> be happening in regards to the metrics infrastructure as any other PV claim
> from normal projects gets recycled just fine. All of my PVs are setup the
> exact same way.

Got it.. we need to figure out what the application is doing when creating its files that is preventing the recycling script from working.

Comment 8 Boris Kurktchiev 2016-04-19 18:23:30 UTC

root@osmaster0s:~:
----> ls -laZ /ose_pv_infra_storage/metrics/commitlog
drwxr-xr-x. 1000020000 root system_u:object_r:nfs_t:s0       .
drwxrwxr-x. root       root system_u:object_r:nfs_t:s0       ..

I did a splunk search of the audit log for that system around the time I had the failure occur and have not seen anything. Let me look and see if I can make a new volume for metrics and re-deploy it again while checking the audit log

Comment 9 Sami Wagiaalla 2016-04-19 20:22:52 UTC

So I tried to reproduce this with the above permissions and selinux labels but was not able to.

Boris, let me know what you find in the audit logs. It might be worth while disabling dontaudit:  

$semodule -DB

while you look at the logs.

Comment 10 Boris Kurktchiev 2016-04-19 20:37:50 UTC

Just failed again and this is all there wa in the audit.log, I am not seeing anything that jumps at me immediately:

type=NETFILTER_CFG msg=audit(1461098183.429:27272): table=nat family=2 entries=175
type=SYSCALL msg=audit(1461098183.429:27272): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=18852c0 items=0 ppid=28972 pid=66944 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098183.439:27273): table=nat family=2 entries=175
type=SYSCALL msg=audit(1461098183.439:27273): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1f8a090 items=0 ppid=28972 pid=66946 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098183.449:27274): table=nat family=2 entries=175
type=SYSCALL msg=audit(1461098183.449:27274): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=11102c0 items=0 ppid=28972 pid=66951 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098183.457:27275): table=nat family=2 entries=175
type=SYSCALL msg=audit(1461098183.457:27275): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=15b8a00 items=0 ppid=28972 pid=66953 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098185.477:27276): table=nat family=2 entries=155
type=SYSCALL msg=audit(1461098185.477:27276): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=933cc0 items=0 ppid=28972 pid=66960 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098185.486:27277): table=nat family=2 entries=155
type=SYSCALL msg=audit(1461098185.486:27277): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=22b97c0 items=0 ppid=28972 pid=66962 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098187.552:27278): table=nat family=2 entries=150
type=SYSCALL msg=audit(1461098187.552:27278): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=222af40 items=0 ppid=28972 pid=66975 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098187.563:27279): table=nat family=2 entries=150
type=SYSCALL msg=audit(1461098187.563:27279): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=25ff8d0 items=0 ppid=28972 pid=66977 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.594:27280): table=nat family=2 entries=145
type=SYSCALL msg=audit(1461098188.594:27280): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=25d11c0 items=0 ppid=28972 pid=66983 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.602:27281): table=nat family=2 entries=145
type=SYSCALL msg=audit(1461098188.602:27281): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=b28a50 items=0 ppid=28972 pid=66985 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.616:27282): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.616:27282): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1097a00 items=0 ppid=28972 pid=66990 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.623:27283): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.623:27283): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=25f59b0 items=0 ppid=28972 pid=66992 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.668:27284): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.668:27284): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=cc8a00 items=0 ppid=28972 pid=66997 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.675:27285): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.675:27285): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=18db900 items=0 ppid=28972 pid=66999 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.689:27286): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.689:27286): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=e64a00 items=0 ppid=28972 pid=67004 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.696:27287): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.696:27287): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1743940 items=0 ppid=28972 pid=67006 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.720:27288): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.720:27288): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=21fda00 items=0 ppid=28972 pid=67011 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.728:27289): table=nat family=2 entries=133
type=SYSCALL msg=audit(1461098188.728:27289): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=f597b0 items=0 ppid=28972 pid=67013 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.741:27290): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098188.741:27290): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=2648410 items=0 ppid=28972 pid=67018 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.750:27291): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098188.750:27291): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=a95360 items=0 ppid=28972 pid=67020 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.785:27292): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098188.785:27292): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=125c410 items=0 ppid=28972 pid=67025 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.794:27293): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098188.794:27293): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=182d040 items=0 ppid=28972 pid=67027 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.815:27294): table=nat family=2 entries=127
type=SYSCALL msg=audit(1461098188.815:27294): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=18d3e20 items=0 ppid=28972 pid=67035 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098188.830:27295): table=nat family=2 entries=127
type=SYSCALL msg=audit(1461098188.830:27295): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1f21aa0 items=0 ppid=28972 pid=67037 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.451:27296): table=nat family=2 entries=127
type=SYSCALL msg=audit(1461098190.451:27296): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1c65e20 items=0 ppid=28972 pid=67043 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.460:27297): table=nat family=2 entries=127
type=SYSCALL msg=audit(1461098190.460:27297): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=256fe00 items=0 ppid=28972 pid=67045 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.484:27298): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098190.484:27298): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=fcd410 items=0 ppid=28972 pid=67050 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.492:27299): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098190.492:27299): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=fce170 items=0 ppid=28972 pid=67052 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.707:27300): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098190.707:27300): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=120b410 items=0 ppid=28972 pid=67057 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.715:27301): table=nat family=2 entries=130
type=SYSCALL msg=audit(1461098190.715:27301): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1ffca90 items=0 ppid=28972 pid=67059 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.732:27302): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.732:27302): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=17e4bd0 items=0 ppid=28972 pid=67064 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.740:27303): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.740:27303): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=172c9c0 items=0 ppid=28972 pid=67066 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.750:27304): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.750:27304): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=17e2bd0 items=0 ppid=28972 pid=67071 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.758:27305): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.758:27305): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=2552cd0 items=0 ppid=28972 pid=67073 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.789:27306): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.789:27306): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1fbfbd0 items=0 ppid=28972 pid=67078 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.797:27307): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.797:27307): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1a84b90 items=0 ppid=28972 pid=67080 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.901:27308): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.901:27308): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=8e5bd0 items=0 ppid=28972 pid=67086 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.912:27309): table=nat family=2 entries=142
type=SYSCALL msg=audit(1461098190.912:27309): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1eaad40 items=0 ppid=28972 pid=67088 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.934:27310): table=nat family=2 entries=145
type=SYSCALL msg=audit(1461098190.934:27310): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=1eca1c0 items=0 ppid=28972 pid=67093 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)
type=NETFILTER_CFG msg=audit(1461098190.942:27311): table=nat family=2 entries=145
type=SYSCALL msg=audit(1461098190.942:27311): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=11d4ff0 items=0 ppid=28972 pid=67095 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables-restor" exe="/usr/sbin/xtables-multi" subj=system_u:system_r:unconfined_service_t:s0 key=(null)

Comment 11 Sami Wagiaalla 2016-04-19 21:02:34 UTC

Hmm... yeah nothing there.. lets try with setenforce 0 just so we can figure out whether we should dig further down the SELinux path look elsewhere.

Comment 12 Boris Kurktchiev 2016-04-20 13:02:14 UTC

 setenforce 0 did not remedy the situation:
NAME         LABELS                    CAPACITY   ACCESSMODES   STATUS      CLAIM                                 REASON    AGE
metrics      size=200Gi,type=infra     200Gi      RWO,RWX       Failed      openshift-infra/metrics-cassandra-1             2m

Comment 13 Sami Wagiaalla 2016-04-20 17:43:38 UTC

hmm... so we have ruled out SELinux and file permissions. I'll try to reproduce this using the metrics component

Comment 14 Boris Kurktchiev 2016-04-26 18:40:23 UTC

so anything new we can do here?

Comment 15 Sami Wagiaalla 2016-04-26 20:39:18 UTC

Hi Boris,

I still cannot reproduce this. Can you give me move information about how you are setting things up ? The smaller the reproducer the more helpful it will be

Comment 16 Boris Kurktchiev 2016-04-26 20:41:30 UTC

Create a hostpath PV, add PV to metrics project's initial deploy, then tell metric to redeploy with REDPLOY=true, watch the recycler not be able to recycle.

Comment 17 Sami Wagiaalla 2016-04-27 17:58:14 UTC

I followed your instructions and I am not able to reproduce. The metrics project redeploy's properly and the volume is recycled and ready for use and is bound by the newly deployed project.

Can you try to reproduce with origin master ?

Comment 18 Boris Kurktchiev 2016-04-27 18:02:46 UTC

I cannot as I am running OSE 3.1.1 as indicated in the Description.

Comment 20 Boris Kurktchiev 2016-04-28 19:39:28 UTC

@Eric Jones... what info is needed that is not in the bug report?

Comment 21 Eric Jones 2016-04-28 19:44:20 UTC

Hi Boris, That needinfo flag was aimed more for engineering than for you, I apologize.

Comment 22 Sami Wagiaalla 2016-04-29 18:13:11 UTC

Created attachment 1152370 [details]
recycler binary

Comment 23 Sami Wagiaalla 2016-04-29 18:15:44 UTC

Hi Boris,

I was not able to recreate this locally so lets debug on your system. I have attached the recycler binary. Can you try to scrub the volume using that.

So
./recycler /ose_pv_infra_storage/metrics/

Do that after the failure of course

Comment 24 Boris Kurktchiev 2016-05-02 15:14:14 UTC

I will run this as soon as I can today. For what is worth if we can move the debug process faster I am on IRC #openshift-dev as @ebalsumgo

Comment 25 Sami Wagiaalla 2016-05-02 15:15:35 UTC

> For what is worth if we can move the debug process faster I am on IRC #openshift-dev as @ebalsumgo

Sounds good

Comment 26 Boris Kurktchiev 2016-05-02 15:39:47 UTC

root@osmaster0s:~/recycler:
----> ./recycler /ose_pv_infra_storage/metrics/
Scrub OK

And now the directory is empty. Can we make sure that the binary is the same binary as whatever was shipped with: oc v3.1.1.6-33-g81eabcc

Comment 27 Sami Wagiaalla 2016-05-03 17:04:43 UTC

Created attachment 1153515 [details]
recycler binary

Comment 28 Sami Wagiaalla 2016-05-03 18:21:48 UTC

Created attachment 1153537 [details]
recycler binary

Comment 29 Boris Kurktchiev 2016-05-03 19:18:56 UTC

So after about 2 days of debugging this in private Sami and I came up with the problem and have filed a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1332688 with a work around.

Comment 30 Boris Kurktchiev 2016-05-03 19:28:42 UTC

I have also shot out a PR on github https://github.com/openshift/origin-metrics/pull/117

Comment 31 Boris Kurktchiev 2016-05-04 11:43:06 UTC

I think it is safe to close this bug and point to https://bugzilla.redhat.com/show_bug.cgi?id=1332688's solution as a workaround, @Matt merged the PR i sent his way which should keep Metrics from mis-behaving with NFSv3 (note that v4 should be fine as it does away with the weird locking mechanism) mounts, it wont solve the problem for any other projects that may delete pods and pvcs at the same time though.

Comment 32 Bradley Childs 2016-05-17 15:01:06 UTC


*** This bug has been marked as a duplicate of bug 1332688 ***

Note You need to log in before you can comment on or make changes to this bug.