Bug 2544950 (CVE-2026-17507) - CVE-2026-17507 bouncycastle: Bouncy Castle: Denial of Service via improper leaf index validation in MLS
Summary: CVE-2026-17507 bouncycastle: Bouncy Castle: Denial of Service via improper le...
Keywords:
Status: NEW
Alias: CVE-2026-17507
Product: Security Response
Classification: Other
Component: vulnerability
Version: unspecified
Hardware: All
OS: Linux
high
high
Target Milestone: ---
Assignee: Product Security
QA Contact:
URL:
Whiteboard:
Depends On: 2545140 2545141
Blocks:
TreeView+ depends on / blocked
 
Reported: 2026-10-02 07:42 UTC by OSIDB Bzimport
Modified: 2026-10-02 12:25 UTC (History)
0 users

Fixed In Version:
Clone Of:
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)

Description OSIDB Bzimport 2026-10-02 07:42:21 UTC
In Bouncy Castle for Java before 1.86, the MLS implementation (org.bouncycastle.mls) holds RFC 9420's uint32 leaf_index in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input, and it must still decode, since the MLS interop test vectors round-trip the full range. GroupKeySet.SecretTree.hasLeaf and Group.validateRemove compared the decoded value directly against the tree's leaf count, and a signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check. In the hasLeaf case the SenderData of an unprotected PrivateMessage could then drive LeafIndex.directPath through NodeIndex.parent() arithmetic that never reaches the tree root, growing the resulting node list without bound until the JVM exhausted its heap. A single small message from any current group member could therefore deny service to every other member of the group. Both comparisons now interpret the value as unsigned via Integer.toUnsignedLong, rejecting an out-of-range sender however it was encoded; well-formed leaf indices are unaffected.


Note You need to log in before you can comment on or make changes to this bug.