Bug 1763305

Summary: Metering aws-billing integration not working
Product: OpenShift Container Platform Reporter: Chance Zibolski <chancez>
Component: Metering OperatorAssignee: tflannag
Status: CLOSED ERRATA QA Contact: Peter Ruan <pruan>
Severity: medium Docs Contact:
Priority: high    
Version: 4.2.0CC: emoss, pruan, sd-operator-metering
Target Milestone: ---   
Target Release: 4.2.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
We broke AWS-billing integration during the release-4.2 cycle, and user's weren't able to use this functionality. This was caused by the following: - Quoting of partition values/locations is incorrect - Incorrectly checking database name for AWS billing datasources can cause a panic - aws-billing datasource is not an input in the aws-ec2-billing-raw query, causing "table not found" errors when the datasource is still initializing due to lack of input dependency. - Managing partitions of HiveTables is not using hive database/schema name when altering partitions. This prevented partitions from being managed. - HiveTables spec.fileFormat was incorrectly being ignored, preventing the table format from being correctly specified. Now, the correct adjustments have been made to the aforementioned struct fields, and values.yaml fields to fix aws-billing integration.
Story Points: ---
Clone Of: 1763303 Environment:
Last Closed: 2020-02-24 16:52:45 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1763303    
Bug Blocks:    

Description Chance Zibolski 2019-10-18 17:48:19 UTC
+++ This bug was initially created as a clone of Bug #1763303 +++

Description of problem: A user is seeing the aws billing reportdatasource for AWS billing correlation use the wrong reportQuery.
https://github.com/operator-framework/operator-metering/issues/993

Version-Release number of selected component (if applicable): 4.2.0


How reproducible: Always


Steps to Reproduce:
1. Install metering via install scripts or OperatorHub
2. Enable AWS billing datasource
3.

Actual results:
time="2019-10-17T19:02:58Z" level=error msg="error queuing Report dependents of ReportDataSource pod-memory-request-raw" app=metering component=reportDataSourceWorker error="unable to get dependencies for ReportQueryView ReportDataSource aws-ec2-billing-data-raw-raw: reportquery.metering.openshift.io \"aws-ec2-billing-data-raw-raw\" not found" logID=mL1IelFccR namespace=metering reportDataSource=pod-memory-request-raw


Expected results:

No errors

Additional info:

Comment 1 Chance Zibolski 2019-10-28 19:21:23 UTC
Text copied from the 4.3 bug:

The scope of this has grown. There are a few additional issues preventing this feature from working once you rename the reportQuery and reportDataSources to not have the second `-raw` suffix. 

There's a few other issues:

- Quoting of partition values/locations is incorrect
- Incorrectly checking database name for AWS billing datasources can cause a panic
- aws-billing datsaource is not an input in the aws-ec2-billing-raw query, causing table not found errors when the datasource is still initializing due to lack of input dependency
- Managing partitions of HiveTables is not using hive database/schema name when altering partitions, preventing partitions from being managed.
- HiveTables spec.fileFormat was incorrectly being ignored, preventing the table format from being correctly specified.

Comment 4 Peter Ruan 2020-01-15 06:14:42 UTC
Report can't be reported due to name mismatch against the reportdatasource, which has the name `aws-ec2-billing-data-raw-raw`


pruan@MacBook-Pro ~/workspace/gocode/src/github.com/operator-framework/operator-metering (release-4.2●)$ oc get reportdatasource                                   [ruby-2.6.3]
NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
aws-billing                                                                                                                                                     3h58m
aws-ec2-billing-data-raw-raw                                                                                                                                    3h58m

pruan@MacBook-Pro ~/workspace/gocode/src/github.com/operator-framework/operator-metering (release-4.2●)$ oc get report pod-memory-request-billing-run-once -o yaml [ruby-2.6.3]
apiVersion: metering.openshift.io/v1
kind: Report
metadata:
  creationTimestamp: "2020-01-15T04:03:22Z"
  generation: 2
  name: pod-memory-request-billing-run-once
  namespace: openshift-metering
  resourceVersion: "203941"
  selfLink: /apis/metering.openshift.io/v1/namespaces/openshift-metering/reports/pod-memory-request-billing-run-once
  uid: f8bf152e-374b-11ea-ab76-02946fa4ea42
spec:
  query: pod-memory-request-aws
  reportingEnd: "2020-11-14T00:00:00Z"
  reportingStart: "2019-11-12T00:00:00Z"
  runImmediately: true
status:
  conditions:
  - lastTransitionTime: "2020-01-15T04:03:22Z"
    lastUpdateTime: "2020-01-15T04:03:22Z"
    message: 'failed to resolve ReportQuery dependencies pod-memory-request-aws: reportdatasource.metering.openshift.io
      "aws-ec2-billing-data-raw" not found'
    reason: InvalidReport
    status: "False"
    type: Running
  tableRef: {}

Comment 6 Peter Ruan 2020-02-04 23:21:10 UTC
verified with 4.2.0-0.nightly-2020-02-04-145549 

pruan@desktop ~METERING_HOME (master●)$ oc get reportdatasource
NAME                                         EARLIEST METRIC        NEWEST METRIC          IMPORT START           IMPORT END             LAST IMPORT TIME       AGE
aws-billing                                                                                                                                                     167m
aws-ec2-billing-data-raw                                                                                                                                        167m
cluster-cpu-capacity-raw                                                                                                                                        167m

Comment 8 errata-xmlrpc 2020-02-24 16:52:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0460