Bug 1294911
Summary: | oslo_messaging._drivers.amqpdriver has no "backoff" for throttling logging of errors? | |||
---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Robin Cernin <rcernin> | |
Component: | python-oslo-log | Assignee: | Victor Stinner <vstinner> | |
Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Udi Shkalim <ushkalim> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 7.0 (Kilo) | CC: | aguetta, apevec, ebarrera, fdinitto, fpercoco, hjensas, itamar.landsman, lars, lhh, michele, pablo.iranzo, rcernin, royoung, srevivo, vstinner | |
Target Milestone: | --- | Keywords: | Reopened, ZStream | |
Target Release: | 7.0 (Kilo) | |||
Hardware: | Unspecified | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Bug Fix | ||
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1301751 (view as bug list) | Environment: | ||
Last Closed: | 2016-12-20 11:39:34 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: |
Description
Robin Cernin
2015-12-30 22:47:31 UTC
Hello, It might be good to sync this with others, like another error message that may occur in the nova-conductor.log: [-] Unexpected exception occurred N time(s)... retrying. So it would be like, [-] Failed to process incoming message occurred N time(s), retrying... Thank you, Regards, Robin Černín oslo.messaging is a library consumed by other OpenStack projects and it has no control on the logging configuration. Each project provides a config file for logging (i.e: https://github.com/openstack/glance/blob/master/etc/glance-api.conf#L384 ) that can be tuned for this. Projects use oslo.log[0] to handle most of these configurations and instantiate the various loggers. Therefore, I'm moving this BZ under oslo.log and check if there's a way to throttle logs using this library. I'm not aware of a way to do that right now. Perhaps it could be done using syslog. [0] https://github.com/openstack/oslo.log What if we were to (a) stop logging to syslog, (b) stop logging to files, and (c) just log to stdout/stderr (so all log messages would be handled by journald, and could eventually end up in syslog anyway). Then we could take advantage of the rate limiting support in journald: RateLimitInterval=, RateLimitBurst= Configures the rate limiting that is applied to all messages generated on the system. If, in the time interval defined by RateLimitInterval=, more messages than specified in RateLimitBurst= are logged by a service, all further messages within the interval are dropped until the interval is over. A message about the number of dropped messages is generated. This rate limiting is applied per-service, so that two services which log do not interfere with each other's limits. Defaults to 1000 messages in 30s. The time specification for RateLimitInterval= may be specified in the following units: "s", "min", "h", "ms", "us". To turn off any kind of rate limiting, set either value to 0. FYI my change https://review.openstack.org/#/c/322263/ was approved but failed to be merged. I'm still investigating why tests failed. Because of the lack of feedback, I consider that the issue was worked again and so that this side effect is not more an issue anymore. Anyway, a generic solution to limit logs is to use journald which supports log rate limit. Moreover, in parallel, a rate limiting feature was added to the development version of OpenStack (in Oslo Log). |