1728350 – seeing duplicated row entry when doing a report

Bug 1728350 - seeing duplicated row entry when doing a report

Summary: seeing duplicated row entry when doing a report

Keywords:
Status:	CLOSED UPSTREAM
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Metering Operator
Sub Component:
Version:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	medium
Target Milestone:	---
Target Release:	4.4.0
Assignee:	tflannag
QA Contact:	Peter Ruan
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-07-09 18:34 UTC by Peter Ruan
Modified:	2020-04-16 14:07 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	Known Issue
Doc Text:	Cause: There's a problem where Hive Metastore cannot drop a table in an s3 bucket path once a user has deleted a Report. Consequence: In the case where a user then re-creates that same deleted Report, the reporting data will still exist which results in duplicate row entries. Workaround (if any): You would need to manually delete this data yourself. Result:
Clone Of:
Environment:
Last Closed:	2020-02-05 21:29:19 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Peter Ruan 2019-07-09 18:34:15 UTC

Description of problem:
  I've seen once that querying for the report the output comes back with duplicate row entries.  When running in aws with s3 as storage

  

Version-Release number of selected component (if applicable):


How reproducible:
rarely

Steps to Reproduce:
1. create metreingconfig calling ./hack/openshift-install.sh with METERING_CR_FILE env set to point to the input yaml below.
apiVersion: metering.openshift.io/v1alpha1
kind: MeteringConfig
metadata:
  name: "operator-metering"
spec:
  storage:
    type: "hive"
    hive:
      type: "s3"
      s3:
        bucket: "qe-metering"
        region: "us-east-2"
        secretName: "my-aws-secret"
        createBucket: true

2. create a report using the following input yaml
apiVersion: metering.openshift.io/v1alpha1
kind: Report
metadata:
  name: namespace-memory-usage-now
spec:
  query: "namespace-memory-usage"
  reportingStart: '2019-01-01T00:00:00Z'
  reportingEnd: '2019-12-30T23:59:59Z'
  runImmediately: true

Actual results:
pruan@MacBook-Pro ~/workspace $ display_report_using_exposed_route namespace-memory-usage-now                                                                         
period_start			period_end			namespace						pod_usage_memory_byte_seconds
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	default							2312939028480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	default							2668162007040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	install-test						938529177600.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	install-test						1084615311360.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	kube-service-catalog					799721226240.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	kube-service-catalog					922536591360.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	kube-system						9666583756800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	kube-system						11024166420480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ansible-service-broker			99876372480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ansible-service-broker			115310100480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-apiserver					1508030054400.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-apiserver					47269559992320.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-apiserver					58099110297600.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-apiserver-operator				123364638720.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-apiserver-operator				2731592663040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-apiserver-operator				3542916464640.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-authentication				269240156160.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-authentication				5802167255040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-authentication				7614269521920.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-authentication-operator			119551918080.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-authentication-operator			6345431531520.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-authentication-operator			7298463989760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cloud-credential-operator			113928683520.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cloud-credential-operator			2232316723200.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cloud-credential-operator			2950679101440.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-machine-approver			32527073280.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-machine-approver			761420021760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-machine-approver			976890101760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-node-tuning-operator			490467409920.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-node-tuning-operator			10403614064640.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-node-tuning-operator			13572570071040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-samples-operator			88716165120.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-samples-operator			2278741524480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-samples-operator			2929561190400.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-storage-operator			28674293760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-storage-operator			544376340480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-storage-operator			718870855680.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-version				115138068480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-version				2405867274240.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-cluster-version				3149834403840.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-console					1140702658560.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-console					22266413383680.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-console					29454580531200.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-console-operator				71019479040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-console-operator				1526581002240.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-console-operator				1990649856000.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-controller-manager				691809730560.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-controller-manager				12602371031040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-controller-manager				16785822597120.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-controller-manager-operator			103004897280.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-controller-manager-operator			2176246824960.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-controller-manager-operator			2841802752000.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-dns						376257331200.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-dns						9053308108800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-dns						11687601684480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-dns-operator					67477340160.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-dns-operator					1355759861760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-dns-operator					1785667338240.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-etcd						2857158082560.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-etcd						148115934658560.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-etcd						173050256916480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-image-registry				196858429440.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-image-registry				4296096153600.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-image-registry				5602465873920.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ingress					245066219520.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ingress					5313516257280.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ingress					6900835860480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ingress-operator				57156894720.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ingress-operator				1077710684160.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ingress-operator				1424653516800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-apiserver				6564240506880.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-apiserver				267481522913280.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-apiserver				318906842972160.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-apiserver-operator			184155340800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-apiserver-operator			3995225948160.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-apiserver-operator			5241864929280.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-controller-manager			929479802880.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-controller-manager			19271975976960.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-controller-manager			25354390978560.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-controller-manager-operator		160453509120.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-controller-manager-operator		3597994475520.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-controller-manager-operator		4671742279680.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-scheduler				397834321920.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-scheduler				8334371635200.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-scheduler				10896791715840.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-scheduler-operator			140081971200.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-scheduler-operator			2861664829440.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-kube-scheduler-operator			3757653196800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-machine-api					292320215040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-machine-api					6191637872640.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-machine-api					8081748541440.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-machine-config-operator			538617692160.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-machine-config-operator			11398444154880.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-machine-config-operator			14887580712960.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-marketplace					257288110080.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-marketplace					6369365114880.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-marketplace					8174083276800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-metering					240077783040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-metering					255833049661440.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-metering					311776609812480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-monitoring					5019342028800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-monitoring					237745466081280.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-monitoring					286711471964160.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-multus					760356864000.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-multus					14720508887040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-multus					19495549992960.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-network-operator				60216852480.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-network-operator				1241221939200.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-network-operator				1627715911680.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-node						179405045760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-node						205917880320.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-operator-lifecycle-manager			563577815040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-operator-lifecycle-manager			21044405207040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-operator-lifecycle-manager			25320943779840.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-sdn						2268324741120.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-sdn						48844918702080.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-sdn						63767396597760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-ca					276382924800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-ca					4998769950720.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-ca					6613650309120.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-ca-operator				94495703040.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-ca-operator				2085602672640.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-ca-operator				2717191864320.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-catalog-apiserver-operator		80678092800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-catalog-apiserver-operator		1730164654080.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-catalog-apiserver-operator		2256934993920.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-catalog-controller-manager-operator	89373327360.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-catalog-controller-manager-operator	1939539640320.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-service-catalog-controller-manager-operator	2543743057920.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ssh-bastion					13031669760.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ssh-bastion					307475005440.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-ssh-bastion					397929676800.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-template-service-broker			479105433600.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-template-service-broker			552392294400.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-web-console					148593131520.000000
2019-01-01 00:00:00 +0000 UTC	2019-12-30 23:59:59 +0000 UTC	openshift-web-console					173445611520.000000

Expected results:
no duplicate row en

Additional info:

Comment 1 Chance Zibolski 2019-07-10 20:18:17 UTC

I believe this is related to using s3 for storage and deleting the namespace to uninstall. Because our operator and pods run in the namespace where we instruct users to create reports, sometimes, metering may be unable to clean up resources outside the cluster before the pods are deleted.

We have a few approaches we will need to investigate:

- Running metering components/operators/pods in a namespace separate from where reporting resources are created.
- Enabling finalizers by default.
- Handling bucket deletion when we created the bucket.

Comment 2 Chance Zibolski 2019-08-19 19:02:07 UTC

Enabling finalizers is a lot trickier than I'd hoped, and will likely require us to change how we deploy things (running them in a separate namespace), so I think that's off the table in the short term.

Handling bucket deletion seems relatively easy but when I tested it seems like namespace deletion often prevents the operator from having time to do clean up. I'm wondering if documenting that the bucket needs to manually cleaned up is sufficient in the short term. Alternatively, we can also disable bucket creation so that users are always 100% in control of the bucket, making it more clear we will not do clean up. Longer term, I believe having bucket creation handled by something like the AWS service operator may make sense, and then creation/cleanup can be done by that operator instead of ours. Especially since the lifecycle of the bucket and metering may be different.

Comment 3 Chance Zibolski 2019-08-28 15:26:50 UTC

I'm moving this to target 4.3 and we will cover this in our documentation for 4.2.

Comment 5 Kevin Lamenzo 2020-02-04 15:00:12 UTC

Added more info to the docs here https://github.com/openshift/openshift-docs/pull/19498

Note You need to log in before you can comment on or make changes to this bug.