Bug 1479645 - failed to parse [collectd.processes.ps_vm] Numeric value (4298063872) out of range of int
failed to parse [collectd.processes.ps_vm] Numeric value (4298063872) out of ...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.6.z
Assigned To: Rich Megginson
Anping Li
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-08 23:32 EDT by Rich Megginson
Modified: 2017-10-25 09:04 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The collectd schema was using "int" instead of "long" for the field values. Consequence: Long integer values could not be stored. Fix: Use "long" instead of "int" in the schema for those field. Result: All long integer values can be stored.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-25 09:04:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rich Megginson 2017-08-08 23:32:14 EDT
Description of problem:
Collectd records cannot be stored - get the following error in the bulk index response:

"status"=>400,
"error"=>{
  "type"=>"mapper_parsing_exception",
  "reason"=>"failed to parse [collectd.statsd.host_nic_tx]",
  "caused_by"=>{"type"=>"json_parse_exception",
  "reason"=>"Numeric value (68554252414) out of range of int\n at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@32b1a580; line: 1, column: 195]"}}}

I also see the same error for the field collectd.swap.swap, collectd.processes.ps_vm, collectd.processes.ps_data, and more

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Probably many/most of these fields need to be changed from integer to long: https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/collectd.yml

as per the Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/mapping-types.html
Comment 4 Rich Megginson 2017-08-23 16:53:32 EDT
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=587926
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:rhaos-3.6-rhel-7-docker-candidate-27813-20170823203718
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.173.0.5
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-elasticsearch:v3.6.173.0.5-6
Comment 5 Xia Zhao 2017-08-24 01:59:05 EDT
@Rich,

Could you please help advise the scenario where I can verify the bug fix? 

Thanks,
Xia
Comment 6 Rich Megginson 2017-08-24 12:16:42 EDT
Shirly, is this something that RHV QE can help with?

- create a namespace

    oadm new-project ovirt-metrics-collectd-test --node-selector=''

- get the uuid of the project

    uuid=$( oc get project ovirt-metrics-collectd-test -o jsonpath='{ .metadata.uid }' )

- create an index name

    indexname="project.ovirt-metrics-collectd-test.$uuid.$(date -u +'%Y.%m.%d')"

- create a test record

cat > test.json <<EOF
{
  "collectd": {
    "processes": {
      "ps_vm": 4298063872
    }
  }
}
EOF

- add record

    cat test.json | oc exec -i $espod -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/$indexname/multi-tenancy-test/" -XPOST --data-binary @- | python -mjson.tool
Comment 7 Shirly Radco 2017-08-27 03:46:58 EDT
(In reply to Rich Megginson from comment #6)
> Shirly, is this something that RHV QE can help with?
> 
> - create a namespace
> 
>     oadm new-project ovirt-metrics-collectd-test --node-selector=''
> 
> - get the uuid of the project
> 
>     uuid=$( oc get project ovirt-metrics-collectd-test -o jsonpath='{
> .metadata.uid }' )
> 
> - create an index name
> 
>     indexname="project.ovirt-metrics-collectd-test.$uuid.$(date -u
> +'%Y.%m.%d')"
> 
> - create a test record
> 
> cat > test.json <<EOF
> {
>   "collectd": {
>     "processes": {
>       "ps_vm": 4298063872
>     }
>   }
> }
> EOF
> 
> - add record
> 
>     cat test.json | oc exec -i $espod -- curl -s -k --cert
> /etc/elasticsearch/secret/admin-cert --key
> /etc/elasticsearch/secret/admin-key
> https://localhost:9200/$indexname/multi-tenancy-test/" -XPOST --data-binary
> @- | python -mjson.tool

How should they install viaq for testing?
Comment 8 Xia Zhao 2017-09-05 06:29:37 EDT
(In reply to Shirly Radco from comment #7)
> (In reply to Rich Megginson from comment #6)
> > Shirly, is this something that RHV QE can help with?

> 
> How should they install viaq for testing?

Does this refer to the logging system with have viaq template embedded?

I'm having issue inserting the record into es, the cursor hangs at curl command:


# oc exec logging-es-data-master-4w1n5zto-1-3xqql -- curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/_cat/indices
green open project.default.507ff221-91e0-11e7-8996-fa163e14042b.2017.09.05      1 0   2773 0   1.7mb   1.7mb 
green open project.install-test.6403e17c-91e1-11e7-8996-fa163e14042b.2017.09.05 1 0   2812 0   1.7mb   1.7mb 
green open project.logging.085f90a5-91e1-11e7-8996-fa163e14042b.2017.09.05      1 0   2814 0   2.2mb   2.2mb 
green open .kibana                                                              1 0      1 0   3.1kb   3.1kb 
green open .searchguard.logging-es-data-master-4w1n5zto                         1 0      5 0    28kb    28kb 
green open .operations.2017.09.05                                               1 0 642576 0 264.9mb 264.9mb 


# oc rsh logging-es-data-master-4w1n5zto-1-3xqql
sh-4.2$ cat > test.json <<EOF
> {
>   "collectd": {
>     "processes": {
>       "ps_vm": 4298063872
>     }
>   }
> }
> EOF

sh-4.2$ ls -al | grep test.json
-rw-r--r--. 1 1000060000 root   75 Sep  5 10:17 test.json

sh-4.2$ curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/project.logging.085f90a5-91e1-11e7-8996-fa163e14042b.2017.09.05/multi-tenancy-test/" -XPOST --data-binary @test.json
> 




@Shirly Radco Is this something you can help with? Let me know if you need the openshift env with logging stacks installed, I can provide it in case you have the libra.pem key file.
Comment 9 Rich Megginson 2017-09-05 09:33:44 EDT
Try running the curl command with `-v` - curl -s -v -k .....

Also, try using `--data` instead of `--data-binary`
Comment 10 Anping Li 2017-09-08 03:26:58 EDT
The post [1,2] succeed with openshift3/logging-elasticsearch:v3.6.173.0.5. But the post failed when the number is in (2^63-192) to (2^63-1). As per https://www.elastic.co/guide/en/elasticsearch/reference/2.4/number.html, (2^63-1) should be an acceptable value. What happened to the back-end?  Is (2^63-193) enough for our requirement?

1. "ps_vm": 4298063872
2. "ps_vm": 9223372036854775807 (2^63-193)

3. "ps_vm": (2^63-192) to (2^63-1)
cat test.json| curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/project.ovirt-metrics-collectd-test.7d94b436-945f-11e7-bc00-fa163e8efe63.2017.09.08/multi-tenancy-test/ -XPOST --data-binary  @-
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse [_source]"}],"type":"mapper_parsing_exception","reason":"failed to parse [_source]","caused_by":{"type":"illegal_state_exception","reason":"No matching token for number_type [BIG_INTEGER]"}},"status":400}
Comment 11 Rich Megginson 2017-09-08 13:56:58 EDT
(In reply to Anping Li from comment #10)
> The post [1,2] succeed with openshift3/logging-elasticsearch:v3.6.173.0.5.
> But the post failed when the number is in (2^63-192) to (2^63-1). As per
> https://www.elastic.co/guide/en/elasticsearch/reference/2.4/number.html,
> (2^63-1) should be an acceptable value. What happened to the back-end?  Is
> (2^63-193) enough for our requirement?
> 
> 1. "ps_vm": 4298063872
> 2. "ps_vm": 9223372036854775807 (2^63-193)
> 
> 3. "ps_vm": (2^63-192) to (2^63-1)
> cat test.json| curl -s -k --cert /etc/elasticsearch/secret/admin-cert --key
> /etc/elasticsearch/secret/admin-key
> https://localhost:9200/project.ovirt-metrics-collectd-test.7d94b436-945f-
> 11e7-bc00-fa163e8efe63.2017.09.08/multi-tenancy-test/ -XPOST --data-binary 
> @-
> {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed
> to parse [_source]"}],"type":"mapper_parsing_exception","reason":"failed to
> parse [_source]","caused_by":{"type":"illegal_state_exception","reason":"No
> matching token for number_type [BIG_INTEGER]"}},"status":400}

Hmm - this sounds like a bug in Elasticsearch - they document that a "long" field can store 2^63-1

@lvlcek @pportant - are you guys aware of any Elasticsearch bugs or issues related to this?
Comment 12 Peter Portante 2017-09-08 14:08:44 EDT
I my ES instance, I see:

$ curl -X PUT elasticsearch.example.com/test-big-num/bignum/1 -d '{ "ps_vm": 9223372036854775808 }'
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse [ps_vm]"}],"type":"mapper_parsing_exception","reason":"failed to parse [ps_vm]","caused_by":{"type":"json_parse_exception","reason":"Numeric value (9223372036854775808) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: org.elasticsearch.common.io.stream.InputStreamStreamInput@c59bdad; line: 1, column: 31]"}},"status":400}
$ curl -X GET elasticsearch.example.com/
{
  "name" : "staging-perf34",
  "cluster_name" : "elasticsearch.perf-dept",
  "cluster_uuid" : "3UFFaKFYSOqcR59S-9Zm1A",
  "version" : {
    "number" : "2.4.6",
    "build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd",
    "build_timestamp" : "2017-07-18T12:17:44Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.4"
  },
  "tagline" : "You Know, for Search"
}
Comment 13 Peter Portante 2017-09-08 14:11:30 EDT
From https://github.com/elastic/elasticsearch/issues/21728#issuecomment-262235966, ES does not support unsigned long.

You can store the field as text or as a double with a mapping.
Comment 14 Rich Megginson 2017-09-08 14:31:04 EDT
(In reply to Peter Portante from comment #13)
> From
> https://github.com/elastic/elasticsearch/issues/21728#issuecomment-262235966,
> ES does not support unsigned long.

The value 2^63-1 is 9223372036854775807

This value is within the range of Elasticsearch signed long: -9223372036854775808 - 9223372036854775807 - assuming the range is inclusive, then 2^63-1 is a legal value.

This appears to be a bug in Elasticsearch - it doesn't accept a value that it says it should accept.

> 
> You can store the field as text or as a double with a mapping.
Comment 15 Peter Portante 2017-09-08 14:35:35 EDT
curl -X PUT elasticsearch.example.com/test-big-num/bignum/1 -d '{ "ps_vm": 9223372036854775807 }'
{"_index":"test-big-num","_type":"bignum","_id":"1","_version":1,"_shards":{"total":1,"successful":1,"failed":0},"created":true}

Works fine, so what is the bug?
Comment 16 Rich Megginson 2017-09-08 15:06:03 EDT
(In reply to Peter Portante from comment #15)
> curl -X PUT elasticsearch.example.com/test-big-num/bignum/1 -d '{ "ps_vm":
> 9223372036854775807 }'
> {"_index":"test-big-num","_type":"bignum","_id":"1","_version":1,"_shards":
> {"total":1,"successful":1,"failed":0},"created":true}
> 
> Works fine, so what is the bug?

Anping - can you verify the above result?
Comment 17 Anping Li 2017-09-10 22:16:31 EDT
It works. move to verified.

2^63-1 is 9223372036854775807. There is a numerical error. The online scientific calculator give me a wrong value (2^63=2^63=9223372036854776000).
Comment 19 errata-xmlrpc 2017-10-25 09:04:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049

Note You need to log in before you can comment on or make changes to this bug.