Bug 1938239 - [RFE] Extend DNA plugin to support intervals sizes for subuids
Summary: [RFE] Extend DNA plugin to support intervals sizes for subuids
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: 389-ds-base
Version: 8.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.5
Assignee: thierry bordaz
QA Contact: RHDS QE
Marc Muehlfeld
URL:
Whiteboard: sync-to-jira
Depends On:
Blocks: 1803943
TreeView+ depends on / blocked
 
Reported: 2021-03-12 15:09 UTC by Christian Heimes
Modified: 2021-11-09 22:27 UTC (History)
5 users (show)

Fixed In Version: 389-ds-1.4-8050020210531183345.1a75f91c
Doc Type: Enhancement
Doc Text:
.The `dnaInterval` configuration attribute is now supported With this update, Red Hat Directory Server supports setting the `dnaInterval` attribute of the Distributed Numeric Assignment (DNA) plug-in in the `cn=__<DNA_config_entry>__,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config` entry. The DNA plug-in generates unique values for specified attributes. In a replication environment, servers can share the same range. To avoid overlaps on different servers, you can set the `dnaInterval` attribute to skip some values. For example, if the interval is `3` and the first number in the range is `1`, the next number used in the range is `4`, then `7`, then `10`. For further details, see the link:https://access.redhat.com/documentation/en-us/red_hat_directory_server/11/html/configuration_command_and_file_reference/plug_in_implemented_server_functionality_reference#dnaInterval[dnaInterval] parameter description.
Clone Of:
Environment:
Last Closed: 2021-11-09 18:11:20 UTC
Type: Enhancement
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Fedora Pagure freeipa issue 8361 0 None None None 2021-03-12 15:11:44 UTC
Github 389ds 389-ds-base issues 4773 0 None open [RFE] Enable interval feature of DNA plugin to support intervals sizes for subuids 2021-05-20 14:18:45 UTC
Github SSSD sssd issues 5197 0 None open Support subid resources in ipa provider 2021-03-12 15:11:44 UTC
Github shadow-maint shadow issues 154 0 None open subid ranges sourced from the network store 2021-03-12 15:11:44 UTC
Red Hat Product Errata RHBA-2021:4203 0 None None None 2021-11-09 18:12:06 UTC

Description Christian Heimes 2021-03-12 15:09:39 UTC
IdM is working on central management of subordinate ids for RHEL 8.5. The feature is also known as subid, subuid, or subgid, man page subuid(5). The feature is important for user namespace mapping in containers. Several major customers have requested central management of subids in LDAP.

Subids are blocks of uids or gids that are owned by a user. Typical block size is 65,536 ids per user. IdM would like to use 389-DS's DNA plugin for automatic allocation and assignment of subids. The DNA plugin only supports increments by one. I like to request an enhancement for DNA plugin.

Design proposal:

- The plugin configuration for "Distributed Numeric Assignment Plugin" instances gets a new "dnaIntervalAttr". The atribute is a multi-value text type.
- When dnaType attribute is equal to dnaMagicRegen, the DNA plugin increments the internal counter by the value of the attributes in "dnaIntervalAttr". If no "dnaIntervalAttr" is configured, 389-DS assumes "increment by one" (current behavior).
- DNA plugin operation should fail when
    - one or more dnaIntervalAttr attributes are not a single-value integer attribute
    - the entry does not have at least one of the specified count attribute
    - if two or more attributes exists and contain different values
    - value is smaller than 1
- DNA plugin has to work for at least full range of uint32_t datatype. It looks like DNA supports even larger ran. The config entry uses PRUint64 for nextval and maxval.
- To keep things simple, IdM will assume that ipaSubUidNumber == ipaSubGidNumber and ipaSubUidCount == ipaSubGidCount == 65536 for now. The object class for subids require all four attributes.
- You could also hard-code the increment for the first implementation, but then you might have to touch the code again. We might need to implement flexible intervals in the future. uid_t and gid_t are limited to uint32_t.



IdM would configure the DNA plugin instance for subordinate IDs like this:

dn: cn=Subordinate IDs,cn=Distributed Numeric Assignment plugin,cn=plugins,cn=config
objectClass: extensibleObject
objectClass: top
cn: Subordinate IDs
dnaExcludeScope: cn=provisioning,dc=ipa,dc=example
dnaFilter: (|(objectClass=ipaSubordinateUserID)(objectClass=ipaSubordinateGroupID))
dnaMagicRegen: -1
# UINT32_MAX - 1
dnaMaxValue: 4294967295
# 2** 31
dnaNextValue: 2147483648
dnaScope: dc=ipa,dc=example
dnaSharedCfgDN: cn=subordinate-ids,cn=dna,cn=ipa,cn=etc,dc=ipa,dc=example
# dnaThreshold: ???
dnaType: ipaSubUidNumber
dnaType: ipaSubGidNumber
dnaIntervalAttr: ipaSubUidCount
dnaIntervalAttr: ipaSubGidCount


A new entry before DNA op may look like this (I have omitted irrelevant fields):

dn: uid=bob,cn=users,cn=accounts,dc=ipa,dc=example
uid: bob
ipaSubUidNumber: -1
ipaSubUidCount: 65536
ipaSubGidNumber: -1
ipaSubGidCount: 65536

Schema (tentative)

attributeTypes: ( 2.16.840.1.113730.3.8.11.78 NAME 'ipaSubUidNumber' DESC 'numerical subordinate user ID' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE  X-ORIGIN 'IPA v4.9')
attributeTypes: ( 2.16.840.1.113730.3.8.11.79 NAME 'ipaSubUidCount' DESC 'numerical subordinate user ID count' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE  X-ORIGIN 'IPA v4.9')
attributeTypes: ( 2.16.840.1.113730.3.8.11.80 NAME 'ipaSubGidNumber' DESC 'numerical subordinate user ID' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE  X-ORIGIN 'IPA v4.9')
attributeTypes: ( 2.16.840.1.113730.3.8.11.81 NAME 'ipaSubGidCount' DESC 'numerical subordinate user ID count' EQUALITY integerMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE  X-ORIGIN 'IPA v4.9')

objectClasses: (2.16.840.1.113730.3.8.12.40 NAME 'ipaSubordinateID' DESC 'Subordinate user and group id' SUP posixAccount AUXILIARY MUST ( ipaSubUidNumber $ ipaSubUidCount $ ipaSubGidNumber $ ipaSubGidCount) X-ORIGIN 'IPA v4.9')

Comment 3 Alexey Tikhonov 2021-03-23 15:25:23 UTC
(sorry for a potentially lame question, but I'm really not fluent in this area)


(In reply to Christian Heimes from comment #0)
> 
> Design proposal:
> 

Does this take different replicas in account?

I.e. does https://directory.fedoraproject.org/docs/389ds/design/dna-plugin.html :: "Usage with Multi-Master Replication" apply here?

How "The entire range of the high bit" is going to be split between replicas?

Comment 4 Christian Heimes 2021-03-24 08:49:26 UTC
The DNA plugin (distributed numeric assignment) handles cluster-wide assignments and allocation automatically. IPA will set up the new DNA plugin configuration similar to the existing "cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config" configuration. dnaNextValue is set to the beginning of the range, dnaMaxValue to the end of the range. 389-DS automatically takes care of the splitting internally.

Comment 6 thierry bordaz 2021-05-03 10:13:53 UTC
DNA allocates a number pickup from a given range and is able to request/grant new range.
The known interval of allocation (default 1, else dnaIntervalAttr) is used to compute remaining available numbers and threshold to request/(grant ?) ranges. The interval will change on each allocation, so the remaining available numbers is always wrong and threshold will not be correctly enforced.

DNA allocates a number in anticipating the next allocated value ('nextval'), with a known interval this is easy. But with dereferenced attribute it is not possible to precompute the 'nextval'. For example, current being '10', entry E1.dnaIntervalAttr_val: 65536 and E2.dnaIntervalAttr: 5. It is expected that E1 gets 65546 and E2 gets 65551 but the current mechanism may apply E1=10, E2=65546. It could be easy to change but mean that 'nextval' may become useless.

The prototype https://github.com/tiran/389-ds-base/tree/dna_interval_attr/ldap/servers/plugins/dna, implement a fixed interval assuming that all dereferenced attributes have the same value. This solution is easy to implement and actually does not need a new attribute dnaMaxInterval as I think it could be done with dnaInterval configuration (so without change in the plugin).

In conclusion:
Regarding the implementation, derefencing the attribute would require to add target entry parameter to several functions (dna_get_next_value, dna_notice_allocation, dna_parse_config_entry...
Requesting of ranges (remaining and threshold) may be based on the last dnaIntervallAttr_val but changes the behavior and require tests.
Needs an evaluation if 'nextval' is still usable or becomes useless with referenced interval

Comment 8 Christian Heimes 2021-05-05 11:02:30 UTC
Thierry,

I see two problems with dnaInterval:

* It is disabled by default and not compiled in. Mark mentioned some problems with the feature. I don't recall the details.
* It may not do what I need.

If I understand the examples at https://directory.fedoraproject.org/docs/389ds/design/dna-plugin.html#interval-method correctly, then dnaInterval does not give correct results.


    Master 1
        dnaNextVal = 1
        dnaMaxVal = 300
        dnaInterval = 3
    Master 2
        dnaNextVal = 2
        dnaMaxVal = 300
        dnaInterval = 3
    Master 3
        dnaNextVal = 3
        dnaMaxVal = 300
        dnaInterval = 3

Master 1 gets 1,4,7,10,etc.
Master 2 gets 2,5,8,11,etc.
Master 3 gets 3,6,9,12,etc.

For subordinate ids the DNA plugin must not use any values in the gaps. When a master assigned value N, then it must consider value "N" to "N + INTERVAL - 1" as used. In the example above Master 2 and 3 must not use values 2,5,8 and 3,6,9.

Comment 11 mreynolds 2021-05-21 17:14:23 UTC
Change merged upstream, but still need to add CI tests.  Leaving in ASSIGNED for now...

https://github.com/389ds/389-ds-base/issues/4773

Comment 12 sgouvern 2021-06-02 08:38:29 UTC
]# PYTHONPATH=src/lib389/ py.test -sv dirsrvtests/tests/suites/plugins/dna_interval_test.py 
re-exec with libfaketime dependencies



===================================== test session starts =====================================
platform linux -- Python 3.6.8, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3.6
cachedir: .pytest_cache
metadata: {'Python': '3.6.8', 'Platform': 'Linux-4.18.0-310.el8.x86_64-x86_64-with-redhat-8.5-Ootpa', 'Packages': {'pytest': '6.2.4', 'py': '1.10.0', 'pluggy': '0.13.1'}, 'Plugins': {'metadata': '1.11.0', 'html': '3.1.1', 'libfaketime': '0.1.2', 'flaky': '3.7.0'}}
389-ds-base: 1.4.3.23-2.module+el8.5.0+11209+cb479c8d
nss: 3.53.1-17.el8_3
nspr: 4.25.0-2.el8_2
openldap: 2.4.46-16.el8
cyrus-sasl: 2.1.27-5.el8
FIPS: disabled
rootdir: /mnt/tests/rhds/tests/upstream/ds/dirsrvtests, configfile: pytest.ini
plugins: metadata-1.11.0, html-3.1.1, libfaketime-0.1.2, flaky-3.7.0
collected 1 item                                                                              

dirsrvtests/tests/suites/plugins/dna_interval_test.py::test_dna_interval INFO:lib389.topologies:Instance with parameters {'ldap-port': 38901, 'ldap-secureport': 63601, 'server-id': 'standalone1', 'suffix': 'dc=example,dc=com'} was created.
INFO:dirsrvtests.tests.suites.plugins.dna_interval_test:Add dna plugin config entry...
INFO:dirsrvtests.tests.suites.plugins.dna_interval_test:Enable the DNA plugin and restart...
INFO:dirsrvtests.tests.suites.plugins.dna_interval_test:Test DNA is working...
INFO:dirsrvtests.tests.suites.plugins.dna_interval_test:Adding user1
INFO:dirsrvtests.tests.suites.plugins.dna_interval_test:Test DNA interval assignment is working...
PASSEDInstance slapd-standalone1 removed.


========================================================== 1 passed in 12.39s ===========================================================


marking as verified/tested

Comment 16 sgouvern 2021-06-03 12:28:38 UTC
As per comment 12, marking as VERIFIED

Comment 17 Marc Muehlfeld 2021-08-30 13:37:53 UTC
Thierry, can you please review the release note (see Doc Text field)?
Thanks.

Comment 21 errata-xmlrpc 2021-11-09 18:11:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (389-ds-base bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4203


Note You need to log in before you can comment on or make changes to this bug.