RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1691511 - Repeated systemd-run --scope -- mount -t tmpfs tmpfs /<path> under directories bind-mounted to themselves result in E2BIG failures
Summary: Repeated systemd-run --scope -- mount -t tmpfs tmpfs /<path> under directorie...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.6
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: ---
Assignee: systemd-maint
QA Contact: Frantisek Sumsal
URL:
Whiteboard:
Depends On:
Blocks: 1722576 1786340
TreeView+ depends on / blocked
 
Reported: 2019-03-21 18:58 UTC by Kyle Walker
Modified: 2023-12-15 16:24 UTC (History)
11 users (show)

Fixed In Version: systemd-219-65.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1722576 1786340 (view as bug list)
Environment:
Last Closed: 2019-08-06 12:43:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:2091 0 None None None 2019-08-06 12:44:19 UTC

Description Kyle Walker 2019-03-21 18:58:52 UTC
Description of problem:
 With openshift workloads, commands such as the following are issued repeatedly:

    systemd-run --description='Kubernetes transient mount for /var/lib/origin/openshift.local.volumes/pods/<val>/volumes/kubernetes.io~secret/<val>' --scope -- mount -t tmpfs tmpfs /var/lib/origin/openshift.local.volumes/pods/<val>/volumes/kubernetes.io~secret/<val>

When the mount point is within a bind-mounted directory, the behaviour results in a large number of "loaded inactive dead" mount points. Eventually the commands will fail with the following error message visible in the system logs.

    systemd: Failed to set up mount unit: Argument list too long


This issue was reported upstream in the following:

    https://github.com/kubernetes/kubernetes/issues/57345


Which pushed the investigation to an upstream systemd issue documented in:

    https://github.com/systemd/systemd/issues/7798


Version-Release number of selected component (if applicable):

    systemd-219-62.el7_6.2.x86_64


How reproducible:

    Difficult, though the underlying suspected problem is fairly simple

Steps to Reproduce:
1. Create a directory and bind mount it to itself

    # mkdir /mnt/test && mount --bind /mnt/test /mnt/test

2. Mount additional directories under the above

    # mkdir /mnt/test/subdir1 && mount -t tmpfs tmpfs /mnt/test/subdir1

3. Verify the state of the underlying subdir

    # systemctl list-units --type mount --all | grep test


Actual results:

    # systemctl list-units --type mount --all | grep test
    mnt-test-subdir1.mount        loaded inactive dead    /mnt/test/subdir1
    mnt-test.mount                loaded active   mounted /mnt/test


Expected results:

    # systemctl list-units --type mount --all | grep test
    mnt-test-subdir1.mount        loaded active   mounted /mnt/test/subdir1
    mnt-test.mount                loaded active   mounted /mnt/test    


Additional info:

    The cause of the "systemd: Failed to set up mount unit: Argument list too long" error message is suspected to be continued growth of "inactive dead" mount units that cannot be removed and eventual failure in the following codepath:

    src/core/unit.c
    int unit_add_name(Unit *u, const char *text) {
            _cleanup_free_ char *s = NULL, *i = NULL;
            UnitType t;
            int r;
    <snip>
            if (hashmap_size(u->manager->units) >= MANAGER_MAX_NAMES)
                    return -E2BIG;
    <snip>

Comment 2 Kyle Walker 2019-03-21 19:25:04 UTC
Opened a downstream PR with a backport of the following commit:

    core: Fix edge case when processing /proc/self/mountinfo

    Currently, if there are two /proc/self/mountinfo entries with the same
    mount point path, the mount setup flags computed for the second of
    these two entries will overwrite the mount setup flags computed for
    the first of these two entries. This is the root cause of issue systemd#7798.
    This patch changes mount_setup_existing_unit to prevent the
    just_mounted mount setup flag from being overwritten if it is set to
    true. This will allow all mount units created from /proc/self/mountinfo
    entries to be initialized properly.

PR:

    https://github.com/lnykryn/systemd-rhel/pull/318

Comment 5 Lukáš Nykrýn 2019-04-02 09:10:30 UTC
fix merged to staging branch -> https://github.com/lnykryn/systemd-rhel/pull/318 -> post

Comment 12 errata-xmlrpc 2019-08-06 12:43:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2091


Note You need to log in before you can comment on or make changes to this bug.