Bug 2255135 (CVE-2023-50981)

Summary: CVE-2023-50981 cryptopp: malformed DER public key file can trigger infinite loop condition
Product: [Other] Security Response Reporter: Robb Gatica <rgatica>
Component: vulnerabilityAssignee: Product Security <prodsec-ir-bot>
Status: NEW --- QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: unspecifiedKeywords: Security
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
A flaw was discovered in Crypto++ (cryptopp). A remote attacker may craft a malformed public key file which can trigger an infinite loop condition, leading to a denial of service.
 Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2255136, 2255137    
Bug Blocks: 2255124    

Description Robb Gatica 2023-12-18 19:55:57 UTC 
Reference: https://github.com/weidai11/cryptopp/issues/1249

-----
Hi, recently I found a security issue in the ModularSquareRoot function of Crypto++ library that would cause an infinite loop, since this function is being used in ECP::DecodePoint, an attacker could potentially craft a malformed DER public key file, and any user or server attempting to read this public key file in processes such as ECDSA may be susceptible to a DOS attack.
Issue

The issue lies in the second while loop in this function, the loop starts with n = 2 and increments n by one each time until we find an n such that Jacobi(n, p) = -1. However, this overlooks the case when p is in the form of m^2 and m is an odd number.

In this case, jacobi(n, p) = jacobi(n, m^2) can be converted into the product of a series of squares of Jacobi symbols using the properties of Jacobi symbols, this means that its value can only be 1 or 0 but not -1, therefore, no matter how n continues to increase, it will never satisfy Jacobi(n, p) = -1 to break out of the loop, thus leading to an infinite loop.
Comment 1 Robb Gatica 2023-12-18 19:56:14 UTC 
Created cryptopp tracking bugs for this issue:

Affects: epel-all [bug 2255136]
Affects: fedora-all [bug 2255137]