Bug 1763305 - Metering aws-billing integration not working
Summary: Metering aws-billing integration not working
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Metering Operator
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.2.z
Assignee: tflannag
QA Contact: Peter Ruan
URL:
Whiteboard:
Depends On: 1763303
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-18 17:48 UTC by Chance Zibolski
Modified: 2020-02-24 16:53 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
We broke AWS-billing integration during the release-4.2 cycle, and user's weren't able to use this functionality. This was caused by the following: - Quoting of partition values/locations is incorrect - Incorrectly checking database name for AWS billing datasources can cause a panic - aws-billing datasource is not an input in the aws-ec2-billing-raw query, causing "table not found" errors when the datasource is still initializing due to lack of input dependency. - Managing partitions of HiveTables is not using hive database/schema name when altering partitions. This prevented partitions from being managed. - HiveTables spec.fileFormat was incorrectly being ignored, preventing the table format from being correctly specified. Now, the correct adjustments have been made to the aforementioned struct fields, and values.yaml fields to fix aws-billing integration.
Clone Of: 1763303
Environment:
Last Closed: 2020-02-24 16:52:45 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github operator-framework operator-metering pull 1009 None closed [release-4.2] bug 1763305: Fix AWS billing ReportDataSource functionality 2020-04-09 19:59:05 UTC
Github operator-framework operator-metering pull 1028 None closed [release-4.2] bug 1763305: Fix handling of AdditionalArtifactKeys in AWS billing report manifests 2020-04-09 19:59:05 UTC
Red Hat Product Errata RHBA-2020:0460 None None None 2020-02-24 16:52:59 UTC

Description Chance Zibolski 2019-10-18 17:48:19 UTC
+++ This bug was initially created as a clone of Bug #1763303 +++

Description of problem: A user is seeing the aws billing reportdatasource for AWS billing correlation use the wrong reportQuery.
https://github.com/operator-framework/operator-metering/issues/993

Version-Release number of selected component (if applicable): 4.2.0


How reproducible: Always


Steps to Reproduce:
1. Install metering via install scripts or OperatorHub
2. Enable AWS billing datasource
3.

Actual results:
time="2019-10-17T19:02:58Z" level=error msg="error queuing Report dependents of ReportDataSource pod-memory-request-raw" app=metering component=reportDataSourceWorker error="unable to get dependencies for ReportQueryView ReportDataSource aws-ec2-billing-data-raw-raw: reportquery.metering.openshift.io \"aws-ec2-billing-data-raw-raw\" not found" logID=mL1IelFccR namespace=metering reportDataSource=pod-memory-request-raw


Expected results:

No errors

Additional info:

Comment 1 Chance Zibolski 2019-10-28 19:21:23 UTC
Text copied from the 4.3 bug:

The scope of this has grown. There are a few additional issues preventing this feature from working once you rename the reportQuery and reportDataSources to not have the second `-raw` suffix. 

There's a few other issues:

- Quoting of partition values/locations is incorrect
- Incorrectly checking database name for AWS billing datasources can cause a panic
- aws-billing datsaource is not an input in the aws-ec2-billing-raw query, causing table not found errors when the datasource is still initializing due to lack of input dependency
- Managing partitions of HiveTables is not using hive database/schema name when altering partitions, preventing partitions from being managed.
- HiveTables spec.fileFormat was incorrectly being ignored, preventing the table format from being correctly specified.

Comment 4 Peter Ruan 2020-01-15 06:14:42 UTC
Report can't be reported due to name mismatch against the reportdatasource, which has the name `aws-ec2-billing-data-raw-raw`


pruan@MacBook-Pro ~/workspace/gocode/src/github.com/operator-framework/operator-metering (release-4.2●)$ oc get reportdatasource                                   [ruby-2.6.3]
NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
aws-billing                                                                                                                                                     3h58m
aws-ec2-billing-data-raw-raw                                                                                                                                    3h58m

pruan@MacBook-Pro ~/workspace/gocode/src/github.com/operator-framework/operator-metering (release-4.2●)$ oc get report pod-memory-request-billing-run-once -o yaml [ruby-2.6.3]
apiVersion: metering.openshift.io/v1
kind: Report
metadata:
  creationTimestamp: "2020-01-15T04:03:22Z"
  generation: 2
  name: pod-memory-request-billing-run-once
  namespace: openshift-metering
  resourceVersion: "203941"
  selfLink: /apis/metering.openshift.io/v1/namespaces/openshift-metering/reports/pod-memory-request-billing-run-once
  uid: f8bf152e-374b-11ea-ab76-02946fa4ea42
spec:
  query: pod-memory-request-aws
  reportingEnd: "2020-11-14T00:00:00Z"
  reportingStart: "2019-11-12T00:00:00Z"
  runImmediately: true
status:
  conditions:
  - lastTransitionTime: "2020-01-15T04:03:22Z"
    lastUpdateTime: "2020-01-15T04:03:22Z"
    message: 'failed to resolve ReportQuery dependencies pod-memory-request-aws: reportdatasource.metering.openshift.io
      "aws-ec2-billing-data-raw" not found'
    reason: InvalidReport
    status: "False"
    type: Running
  tableRef: {}

Comment 6 Peter Ruan 2020-02-04 23:21:10 UTC
verified with 4.2.0-0.nightly-2020-02-04-145549 

pruan@desktop ~METERING_HOME (master●)$ oc get reportdatasource
NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
aws-billing                                                                                                                                                     167m
aws-ec2-billing-data-raw                                                                                                                                        167m
cluster-cpu-capacity-raw                                                                                                                                        167m

Comment 8 errata-xmlrpc 2020-02-24 16:52:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0460


Note You need to log in before you can comment on or make changes to this bug.