Bug 2544950 (CVE-2026-17507)

Summary: CVE-2026-17507 bouncycastle: Bouncy Castle: Denial of Service via improper leaf index validation in MLS
Product: [Other] Security Response Reporter: OSIDB Bzimport <bzimport>
Component: vulnerabilityAssignee: Product Security <prodsec-ir-bot>
Status: NEW --- QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedKeywords: Security
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
A flaw was found in Bouncy Castle's Messaging Layer Security (MLS) implementation. Due to an improper signed integer comparison, an out-of-range leaf identifier incorrectly bypasses group membership checks. A group member can exploit this flaw by sending a specially crafted message, causing unbounded memory allocation during tree path calculations. This flaw exhausts available memory, resulting in a Denial of Service (DoS) for all participants in the group.
 Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2545140, 2545141    
Bug Blocks:    

Description OSIDB Bzimport 2026-10-02 07:42:21 UTC 
In Bouncy Castle for Java before 1.86, the MLS implementation (org.bouncycastle.mls) holds RFC 9420's uint32 leaf_index in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input, and it must still decode, since the MLS interop test vectors round-trip the full range. GroupKeySet.SecretTree.hasLeaf and Group.validateRemove compared the decoded value directly against the tree's leaf count, and a signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check. In the hasLeaf case the SenderData of an unprotected PrivateMessage could then drive LeafIndex.directPath through NodeIndex.parent() arithmetic that never reaches the tree root, growing the resulting node list without bound until the JVM exhausted its heap. A single small message from any current group member could therefore deny service to every other member of the group. Both comparisons now interpret the value as unsigned via Integer.toUnsignedLong, rejecting an out-of-range sender however it was encoded; well-formed leaf indices are unaffected.