Bug 2326335 (CVE-2024-11249, GHSA-j3px-q95c-9683)

Summary: CVE-2024-11249 zlib-rs: zlib-rs stack overflow during decompression with malicious input
Product: [Other] Security Response Reporter: OSIDB Bzimport <bzimport>
Component: vulnerabilityAssignee: Product Security DevOps Team <prodsec-dev>
Status: NEW --- QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: unspecifiedKeywords: Security
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
A flaw was found in zlib-rs, a Rust implementation of the zlib compression library. This vulnerability allows for a denial of service (DoS) through a stack overflow triggered by specially crafted input.
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2326411, 2326412, 2326413, 2326414    
Bug Blocks:    

Description OSIDB Bzimport 2024-11-14 19:01:52 UTC
A denial of service vulnerability was found in zlib-rs, triggered by specially constructed input. This input causes a stack overflow, resulting in the process using zlib-rs to crash.

### Impact

Due to the way LLVM handles the zlib-rs codebase, tail calls were not guaranteed. This caused certain input patterns to result in a large number of stack frames being required, quickly resulting in a stack overflow. These are unlikely to occur in practice, but a dedicated attacker can construct malicious input files.

After stack overflows were found by @inahga with a fuzzer, we dove into the assembly, and found some cases where the stack grew

```asm
.LBB109_326:
    mov rdi, rbx
    call zlib_rs::inflate::State::type_do
    jmp .LBB109_311

.LBB109_311:
    lea rsp, [rbp - 40]
    pop rbx
    pop r12
    pop r13
    pop r14
    pop r15
    pop rbp
    .cfi_def_cfa rsp, 8
    ret
```

LLVM wants to centralize the cleanup before the return (many other blocks jump to `LBB109_311`), thereby invalidating a tail call to `type_do`. We were not able to get rid of this call without introducing one elsewhere: we just don't currently have the power to tell LLVM what we want it to do.

So, we switch back to loop+match waiting for changes to rust to make a more efficient implementation possible. Performance-wise, the damage is relatively minimal: we're just slower in cases where we already were slower than C. We are faster in cases where the relevant code is barely touched (in these cases the logic quickly moves into a hot inner loop and just spends most of its time there).

### Patches
Version 0.4.0 patches the problem and is no longer vulnerable.

### Workarounds
Users of zlib-rs should upgrade to the latest version. Users could alternatively run zlib-rs in a separate process to prevent a stack overflow crashing the entire program. In some situations a signal handler can be used to catch a stack overflow happening.