Bug 1891200 - rabbitmq-limits validation is broken because max_fds is no longer defined from rabbit's point of view
Summary: rabbitmq-limits validation is broken because max_fds is no longer defined fro...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-validations
Version: 16.1 (Train)
Hardware: All
OS: All
low
low
Target Milestone: z8
: 16.1 (Train on RHEL 8.2)
Assignee: chrisbro@redhat.com
QA Contact: nlevinki
URL:
Whiteboard:
: 1896409 (view as bug list)
Depends On: 1912879
Blocks: 1987090
TreeView+ depends on / blocked
 
Reported: 2020-10-24 10:05 UTC by Alex Stupnikov
Modified: 2023-12-15 19:54 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-validations-11.3.2-1.20210913123540.4db92ba.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1987090 (view as bug list)
Environment:
Last Closed: 2022-01-25 16:49:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 761399 0 None MERGED replace max_fds with sysctl call 2021-02-16 15:38:22 UTC
OpenStack gerrit 799451 0 None MERGED Corrected the task 'Get file_descriptors total_limit' to get the 'fs.file-max' value. 2021-07-26 09:35:57 UTC
OpenStack gerrit 802247 0 None MERGED Corrected the task 'Get file_descriptors total_limit' to get the 'fs.file-max' value. 2021-07-28 20:40:32 UTC
OpenStack gerrit 802248 0 None MERGED Corrected the task 'Get file_descriptors total_limit' to get the 'fs.file-max' value. 2021-07-28 20:40:29 UTC
OpenStack gerrit 802249 0 None MERGED Corrected the task 'Get file_descriptors total_limit' to get the 'fs.file-max' value. 2022-01-03 09:28:21 UTC
OpenStack gerrit 802250 0 None MERGED Corrected the task 'Get file_descriptors total_limit' to get the 'fs.file-max' value. 2021-07-28 20:40:34 UTC
Red Hat Issue Tracker OSP-3089 0 None None None 2022-01-25 16:41:09 UTC
Red Hat Product Errata RHBA-2021:0817 0 None None None 2021-03-17 15:39:05 UTC

Description Alex Stupnikov 2020-10-24 10:05:45 UTC
Description of problem:

rabbitmq-limits validation uses "rabbitmqctl eval" command to obtain max_fds value from RabbitMQ instance running inside podman container [1].

In customer's environment and in my RHOSP 16.1 lab this check is broken: max_fds value is undefined from rabbit perspective:

  # podman exec 1e1970fab36f rabbitmqctl eval 'proplists:get_value(max_fds, erlang:system_info(check_io)).'
  undefined

And check fails with the following error:
    "task": {
        "hosts": {
            "overcloud-controller-0": {
                "_ansible_no_log": false,
                "action": "fail",
                "changed": false,
                "failed": true,
                "failed_when_result": true,
                "msg": "undefined must be greater than or equal to 16384"
            }
        },
        "name": "Verify the actual limit exceeds the minimal value",
        "status": "FAILED"
    }

I am not sure if this check is still relevant in its current state: AFAIU we enforce various limitations using podman settings. In any case would like to escalate to developers and ask for a fix.

[1]
https://github.com/openstack/tripleo-validations/blob/stable/train/roles/rabbitmq_limits/tasks/main.yml#L13

Comment 2 Michele Baldessari 2020-11-24 18:00:13 UTC
*** Bug 1896409 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2021-03-17 15:33:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.4 director bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0817

Comment 16 errata-xmlrpc 2021-03-17 15:38:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.4 director bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0817

Comment 17 Chris Fields 2021-04-16 21:06:52 UTC
This validator is still broken so I'm opening this bug back up.  

run: openstack tripleo validator run --validation  rabbitmq-limits

run: openstack tripleo validator show run  03d3af96-8612-4f15-b5a2-c6b999b75f78




"hosts": {
        "overcloud-controller-0": {
            "_ansible_no_log": false,
            "action": "command",
            "changed": false,
            "cmd": "\"podman\" exec $(\"podman\" ps -q --filter \"name=rabbitmq\" | head -1) rabbitmqctl eval 'sysctl -n fs.file-max'",
            "delta": "0:00:01.557734",
            "end": "2021-04-16 20:58:32.249211",
            "failed": true,
            "invocation": {
                "module_args": {
                    "_raw_params": "\"podman\" exec $(\"podman\" ps -q --filter \"name=rabbitmq\" | head -1) rabbitmqctl eval 'sysctl -n fs.file-max'",
                    "_uses_shell": true,
                    "argv": null,
                    "chdir": null,
                    "creates": null,
                    "executable": null,
                    "removes": null,
                    "stdin": null,
                    "stdin_add_newline": true,
                    "strip_empty_ends": true,
                    "warn": true
                }
            },
            "msg": "non-zero return code",
            "rc": 64,
            "start": "2021-04-16 20:58:30.691477",
            "stderr": "Error (argument validation): \"syntax error before: fs\"\nArguments given:\n\teval sysctl -n fs.file-max\n\n\u001b[1mUsage\u001b[0m\n\nrabbitmqctl [--node <node>] [--longnames] [--quiet] eval <expression>\nError: non zero exit code: 64: OCI runtime error",
            "stderr_lines": [
                "Error (argument validation): \"syntax error before: fs\"",
                "Arguments given:",
                "\teval sysctl -n fs.file-max",
                "",
                "\u001b[1mUsage\u001b[0m",
                "",
                "rabbitmqctl [--node <node>] [--longnames] [--quiet] eval <expression>",
                "Error: non zero exit code: 64: OCI runtime error"
            ],
            "stdout": "",
            "stdout_lines": []
        }
    },
    "name": "Get file_descriptors total_limit",
    "status": "FAILED"
}

Comment 18 Chris Fields 2021-04-16 21:07:19 UTC
 The 'eval' statement expect ERLANG code and no longer belongs here.  This works:

podman exec $(podman ps -q --filter \"name=rabbitmq\" | head -1) sysctl -n fs.file-max

Comment 20 Jiri Podivin 2021-07-26 10:37:07 UTC
Backports of the patch with fix are waiting for review.

Comment 22 Uemit Seren 2021-09-20 11:49:53 UTC
This is still occuring on OSP 16.2

Comment 25 Red Hat Bugzilla 2023-09-15 01:31:05 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days


Note You need to log in before you can comment on or make changes to this bug.