Bug 1728350
| Summary: | seeing duplicated row entry when doing a report | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Peter Ruan <pruan> |
| Component: | Metering Operator | Assignee: | tflannag |
| Status: | CLOSED UPSTREAM | QA Contact: | Peter Ruan <pruan> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.2.0 | CC: | emoss, klamenzo, sd-operator-metering, wsun |
| Target Milestone: | --- | ||
| Target Release: | 4.4.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Known Issue | |
| Doc Text: |
Cause:
There's a problem where Hive Metastore cannot drop a table in an s3 bucket path once a user has deleted a Report.
Consequence:
In the case where a user then re-creates that same deleted Report, the reporting data will still exist which results in duplicate row entries.
Workaround (if any):
You would need to manually delete this data yourself.
Result:
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-02-05 21:29:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I believe this is related to using s3 for storage and deleting the namespace to uninstall. Because our operator and pods run in the namespace where we instruct users to create reports, sometimes, metering may be unable to clean up resources outside the cluster before the pods are deleted. We have a few approaches we will need to investigate: - Running metering components/operators/pods in a namespace separate from where reporting resources are created. - Enabling finalizers by default. - Handling bucket deletion when we created the bucket. Enabling finalizers is a lot trickier than I'd hoped, and will likely require us to change how we deploy things (running them in a separate namespace), so I think that's off the table in the short term. Handling bucket deletion seems relatively easy but when I tested it seems like namespace deletion often prevents the operator from having time to do clean up. I'm wondering if documenting that the bucket needs to manually cleaned up is sufficient in the short term. Alternatively, we can also disable bucket creation so that users are always 100% in control of the bucket, making it more clear we will not do clean up. Longer term, I believe having bucket creation handled by something like the AWS service operator may make sense, and then creation/cleanup can be done by that operator instead of ours. Especially since the lifecycle of the bucket and metering may be different. I'm moving this to target 4.3 and we will cover this in our documentation for 4.2. Added more info to the docs here https://github.com/openshift/openshift-docs/pull/19498 |
Description of problem: I've seen once that querying for the report the output comes back with duplicate row entries. When running in aws with s3 as storage Version-Release number of selected component (if applicable): How reproducible: rarely Steps to Reproduce: 1. create metreingconfig calling ./hack/openshift-install.sh with METERING_CR_FILE env set to point to the input yaml below. apiVersion: metering.openshift.io/v1alpha1 kind: MeteringConfig metadata: name: "operator-metering" spec: storage: type: "hive" hive: type: "s3" s3: bucket: "qe-metering" region: "us-east-2" secretName: "my-aws-secret" createBucket: true 2. create a report using the following input yaml apiVersion: metering.openshift.io/v1alpha1 kind: Report metadata: name: namespace-memory-usage-now spec: query: "namespace-memory-usage" reportingStart: '2019-01-01T00:00:00Z' reportingEnd: '2019-12-30T23:59:59Z' runImmediately: true Actual results: pruan@MacBook-Pro ~/workspace $ display_report_using_exposed_route namespace-memory-usage-now period_start period_end namespace pod_usage_memory_byte_seconds 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC default 2312939028480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC default 2668162007040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC install-test 938529177600.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC install-test 1084615311360.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC kube-service-catalog 799721226240.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC kube-service-catalog 922536591360.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC kube-system 9666583756800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC kube-system 11024166420480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ansible-service-broker 99876372480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ansible-service-broker 115310100480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-apiserver 1508030054400.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-apiserver 47269559992320.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-apiserver 58099110297600.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-apiserver-operator 123364638720.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-apiserver-operator 2731592663040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-apiserver-operator 3542916464640.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-authentication 269240156160.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-authentication 5802167255040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-authentication 7614269521920.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-authentication-operator 119551918080.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-authentication-operator 6345431531520.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-authentication-operator 7298463989760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cloud-credential-operator 113928683520.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cloud-credential-operator 2232316723200.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cloud-credential-operator 2950679101440.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-machine-approver 32527073280.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-machine-approver 761420021760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-machine-approver 976890101760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-node-tuning-operator 490467409920.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-node-tuning-operator 10403614064640.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-node-tuning-operator 13572570071040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-samples-operator 88716165120.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-samples-operator 2278741524480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-samples-operator 2929561190400.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-storage-operator 28674293760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-storage-operator 544376340480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-storage-operator 718870855680.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-version 115138068480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-version 2405867274240.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-cluster-version 3149834403840.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-console 1140702658560.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-console 22266413383680.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-console 29454580531200.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-console-operator 71019479040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-console-operator 1526581002240.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-console-operator 1990649856000.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-controller-manager 691809730560.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-controller-manager 12602371031040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-controller-manager 16785822597120.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-controller-manager-operator 103004897280.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-controller-manager-operator 2176246824960.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-controller-manager-operator 2841802752000.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-dns 376257331200.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-dns 9053308108800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-dns 11687601684480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-dns-operator 67477340160.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-dns-operator 1355759861760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-dns-operator 1785667338240.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-etcd 2857158082560.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-etcd 148115934658560.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-etcd 173050256916480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-image-registry 196858429440.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-image-registry 4296096153600.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-image-registry 5602465873920.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ingress 245066219520.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ingress 5313516257280.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ingress 6900835860480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ingress-operator 57156894720.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ingress-operator 1077710684160.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ingress-operator 1424653516800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-apiserver 6564240506880.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-apiserver 267481522913280.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-apiserver 318906842972160.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-apiserver-operator 184155340800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-apiserver-operator 3995225948160.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-apiserver-operator 5241864929280.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-controller-manager 929479802880.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-controller-manager 19271975976960.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-controller-manager 25354390978560.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-controller-manager-operator 160453509120.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-controller-manager-operator 3597994475520.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-controller-manager-operator 4671742279680.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-scheduler 397834321920.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-scheduler 8334371635200.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-scheduler 10896791715840.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-scheduler-operator 140081971200.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-scheduler-operator 2861664829440.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-kube-scheduler-operator 3757653196800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-machine-api 292320215040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-machine-api 6191637872640.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-machine-api 8081748541440.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-machine-config-operator 538617692160.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-machine-config-operator 11398444154880.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-machine-config-operator 14887580712960.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-marketplace 257288110080.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-marketplace 6369365114880.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-marketplace 8174083276800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-metering 240077783040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-metering 255833049661440.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-metering 311776609812480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-monitoring 5019342028800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-monitoring 237745466081280.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-monitoring 286711471964160.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-multus 760356864000.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-multus 14720508887040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-multus 19495549992960.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-network-operator 60216852480.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-network-operator 1241221939200.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-network-operator 1627715911680.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-node 179405045760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-node 205917880320.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-operator-lifecycle-manager 563577815040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-operator-lifecycle-manager 21044405207040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-operator-lifecycle-manager 25320943779840.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-sdn 2268324741120.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-sdn 48844918702080.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-sdn 63767396597760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-ca 276382924800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-ca 4998769950720.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-ca 6613650309120.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-ca-operator 94495703040.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-ca-operator 2085602672640.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-ca-operator 2717191864320.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-catalog-apiserver-operator 80678092800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-catalog-apiserver-operator 1730164654080.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-catalog-apiserver-operator 2256934993920.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-catalog-controller-manager-operator 89373327360.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-catalog-controller-manager-operator 1939539640320.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-service-catalog-controller-manager-operator 2543743057920.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ssh-bastion 13031669760.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ssh-bastion 307475005440.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-ssh-bastion 397929676800.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-template-service-broker 479105433600.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-template-service-broker 552392294400.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-web-console 148593131520.000000 2019-01-01 00:00:00 +0000 UTC 2019-12-30 23:59:59 +0000 UTC openshift-web-console 173445611520.000000 Expected results: no duplicate row en Additional info: