Description of problem: CFME (mangeiq) generate lots of api objects, mostly per call. by tracing the app and adding additional log before the api object creation can bee we can count the number of api objects. [root@4af2d41755bf vmdb]# cat log/evm.log |grep trace |wc -l 169 patched code @app/models/manageiq/providers/redhat/infra_manager/api_integration.rb: _log.info("perf trace - connect again") connection = self.class.public_send(connect_method, server, port, path, username, password, service) Version-Release number of selected component (if applicable): upstream. master.20161222164848_ using 1 global connection per RHV provider, might reduce dramatically the performance. currently due to that situation RHV setup suffering from too many DB connections and and 10% overhead for the RHV DB machine. seems like we can implements the same logic for any provider. How reproducible: 100% Steps to Reproduce: 1. 2. 3. Actual results: lots of api objects probably per api call. Expected results: single api object to server any api call. Additional info:
I'm not aware of any evidence that this will make the performance any better. Oved?
Not that I know of, but let's hear Juan.
(In reply to Oved Ourfali from comment #3) > Not that I know of, but let's hear Juan. I already talk to him see https://github.com/ManageIQ/manageiq/pull/13315 but it was just for PoC, now we need to take it further.
I agree that reducing the number of connection objects is good, both for CFME and RHV. But I don't have numbers to support that. If we do that, we have to be very careful, as connections need to be closed. This is specially important for the RHV side, as each connections hold resources that need to be released. My suggestion is to create a singleton that contains a hash. That hash should contain the connections, indexed by the URL and user name. Those connections should be closed from time to time, at least when the CFME process goes down. Greg, how do you manage this kind of object caching/reusing in other parts of CFME?
As I understood, CFME doesn't actually hold open connections to RHV for API calls since it's going over the REST API, right? Or, did that change? CFME opens multiple HTTP connections in separate threads during inventory collection to collect inventory objects in parallel. In VMWare, CFME does inventory object caching and connection pooling by using the VIM Broker, and we've talked to Piotr a bit about the possibility of RHV implementing a similar RHV Inventory Broker. But, the connections to VMWare are over SOAP and not REST. CFME *does* open direct DB connections to gather metrics, so that could result in extra DB Connections. Not sure if any of that helps...
The use of the 'rest-client' gem implies that there are no open HTTP/TCP connections. But regardless of that a new RHV authentication session is created each time that a new instance of the 'Ovirt::Service' class is created. A new class of that instance is created when the provider calls the 'connect' method of the provider. That is for the existing 'ovirt' gem, to work with version 3 of the RHV API. With the new RHV Ruby SDK this is a bit different: the SDK does keep HTTP/TCP connections open, in addition to the authentication session. But the rest is the same: there is still one instance of the 'OvirtSDK4::Connection' instance created each time the provider calls the 'connect' method. I both cases those objects need to be cleaned up, as they hold resources that are expensive in the RHV side. As the reference to that instance is held as an attribute of the EMS, my understanding is that one will be created each time the EMS is loaded from the database. So the question is there is any recommended, already existing, mechanism in CFME to cache these instances, so that they are created only once, if possible, and cleaned when the Ruby process finishes. If this were plain Ruby I'd suggest to use a singleton to store a hash, with the instance of these objects indexed by the URL and credentials of the provider. I'd then suggest to clean them using the Ruby 'at_exit' method. Does that sound reasonable to you?
VMware solves caching connections with the VimBroker. It doesn't keep the connection count at 1, but it does pool the connections and reuse them as necessary. You could talk to Adam Grare about the VimBroker connection pool to learn more about it.
Oved, could you set some target release to this bug
(In reply to Eldad Marciano from comment #9) > Oved, could you set some target release to this bug This is on future, as we haven't prioritized it yet. So I'm not setting any target for now.
Please assess the importance of this issue and update the priority accordingly. Somewhere it was missed in the bug triage process. Please refer to https://bugzilla.redhat.com/page.cgi?id=fields.html#priority for a reminder on each priority's definition. If it's something like a tracker bug where it doesn't matter, please set it to Low/Low.
Clearing need info on me, as I am not suppose to set the priority, but the PM.
This issue is partially addressed by the following pull request, which is already merged: Add connection manager https://github.com/ManageIQ/manageiq-providers-ovirt/pull/92 I say partially because it solves the problem only when using version 4 of the API via the oVirt Ruby SDK, it doesn't change the behaviour for version 3. As we want are not doing improvements to version 3, I think we can re-target this for CFME 5.9, and move it to POST.
Juan, How this bug can be verified please?
The provider has been modified so that messages like the following will appear in the ManageIQ logs each time a new connection to oVirt is created: ---8<--- Creating new connection for EMS with identifier '1' and URL 'https://engine42.local:443/ovirt-engine/api' --->8--- Those messages should appear at most once per Ruby process and EMS identifier. If you see more than that, then the bug isn't fixed. A simple way to verify is to add a provider for an oVirt environment with a reasonable size, perform some activity in the oVirt side to trigger events and refreshes (add virtual machines, start virtual machines, remove virtual machines, etc) and grep the logs to check that the message isn't generated repeatedly: should appear only once per Ruby process and EMS identifier.
Verified on CFME-5.9.0.14/RHV-4.2.1 The provider works correctly: it creates only one connection (per process): vmdb/log# grep "Creating new connection for EMS" rhevm.log, showed several messages like the following, and each one had a unique PID [----] I, [2017-12-21T03:30:07.736478 #13573:12f9134] INFO -- : Creating new connection for EMS with identifier '' and URL 'https://<engine fqdn>/ovirt-engine/api'.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0380