Bug 1875596 - fwrite on stream opened with open_memstream() truncates the buffer size
Summary: fwrite on stream opened with open_memstream() truncates the buffer size
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: glibc
Version: 8.2
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 8.0
Assignee: DJ Delorie
QA Contact: qe-baseos-tools-bugs
URL:
Whiteboard:
Depends On:
Blocks: 1875597
TreeView+ depends on / blocked
 
Reported: 2020-09-03 20:31 UTC by Paulo Andrade
Modified: 2020-10-31 21:50 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1875597 (view as bug list)
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
memstream-test.c (1.08 KB, text/plain)
2020-09-03 20:31 UTC, Paulo Andrade
no flags Details
test-open_memstream.c (5.40 KB, text/x-csrc)
2020-09-03 20:58 UTC, Paulo Andrade
no flags Details


Links
System ID Priority Status Summary Last Updated
Sourceware 26557 P2 UNCONFIRMED Incorrect behavior with open_memstream in recent version(s) of glibc 2021-01-23 10:19:18 UTC

Description Paulo Andrade 2020-09-03 20:31:52 UTC
Created attachment 1713678 [details]
memstream-test.c

Current output:

"""
Running test: fopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: fmemopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: open_memstream()
position after fwrite: 24
position after SEEK_END: 4

open_memstream() final buffer size: 4
"""

Expected ouput:
"""
Running test: fopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: fmemopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: open_memstream()
position after fwrite: 24
position after SEEK_END: 24

open_memstream() final buffer size: 24
"""

  The seek to the start of the buffer, and write data smaller than
the stream length truncates the memory stream.

Comment 1 Mason Loring Bliss 2020-09-03 20:41:45 UTC
For comparison, on FreeBSD 12.1:

$ make glibcbug
cc -O2 -pipe  glibcbug.c  -o glibcbug
$ ./glibcbug 
Running test: fopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: fmemopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: open_memstream()
position after fwrite: 24
position after SEEK_END: 24

open_memstream() final buffer size: 24

Comment 2 Paulo Andrade 2020-09-03 20:58:52 UTC
Created attachment 1713683 [details]
test-open_memstream.c

Minor edit to http://web.mit.edu/freebsd/head/tools/regression/lib/libc/stdio/test-open_memstream.c to compile cleanly with glibc.

It appears it would fail on open_group_test() for this specific
report, but fails before for yet another test...

Comment 5 DJ Delorie 2020-09-19 04:28:39 UTC
This appears to be the way that open_memstream is designed.  The internal structure only keeps track of a read pointer and a write pointer, so the call to seek to the beginning and write four bytes resets the "end of file" to be at the four byte mark.

Note that fmemopen uses a completely different internal implementation, and has separate variables for current and maximum position.

Comment 6 Carlos Santos 2020-09-21 16:37:32 UTC
(In reply to DJ Delorie from comment #5)

My understanding is that the behavior in RHEL-8 matches the documentation
but the results are not consistent between rhel-7, rhel-8 and FreeBSD:

* RHEL-7

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 (Maipo)
$ gcc -o memstream-test-rhel-7 memstream-test.c
$ ./memstream-test-rhel-7
Running test: fopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: fmemopen()
position after fwrite: 24
position after SEEK_END: 64

Running test: open_memstream()
position after fwrite: 24
position after SEEK_END: 48

open_memstream() final buffer size: 48

* RHEL-8

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 (Ootpa)
$ gcc -o memstream-test-rhel-8 memstream-test.c
$ ./memstream-test-rhel-8
Running test: fopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: fmemopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: open_memstream()
position after fwrite: 24
position after SEEK_END: 4

open_memstream() final buffer size: 4

* FreeBSD 12.1

$ uname -a
FreeBSD freebsd-12-1.example.com 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
$ cc -o memstream-test-fbsd-12 memstream-test.c
$ ./memstream-test-fbsd-12
Running test: fopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: fmemopen()
position after fwrite: 24
position after SEEK_END: 24

Running test: open_memstream()
position after fwrite: 24
position after SEEK_END: 24

open_memstream() final buffer size: 24

Comment 7 Carlos O'Donell 2020-09-24 13:37:23 UTC
DJ and I reviewed this bug.

It's a problem with the standard. The specification of SEEK_END is underspecified.

We aren't going to make any changes in Red Hat Enterprise Linux 7.

In Red Hat Enterprise Linux 8 we can make some adjustments based on the outcome of an upstream conversation to harmonize the behaviour.

I reached out to Rich Felker (musl) and discussed the specification. He agrees it is underspecified.

Comment 9 bob.huemmer 2020-09-27 14:51:43 UTC
I passed the recent updates along to our developers and they aren't too pleased about RHEL 7 not being targeted for a fix as we still use RHEL7 internally a lot in addition to RHEL8.  They also don't understand why folks believe that the specification of SEEK_END is underspecified for fmemopen(). The problem from an ISV and developer is that the interpretation of SEEK_END is already defined for file-based streams, and the behavior for memory-based streams is not consistent with it. Consistency is our issue. If there is something in the spec of fmemopen() or open_memstream() that implies the resulting stream should behave differently from fopen(), then that could be an issue. But as far as I'm aware, the resulting memory-based stream should have the same behavior as a file-based stream, only it's in memory.

Comment 10 Carlos O'Donell 2020-09-27 15:01:00 UTC
(In reply to bob.huemmer from comment #9)
> I passed the recent updates along to our developers and they aren't too
> pleased about RHEL 7 not being targeted for a fix as we still use RHEL7
> internally a lot in addition to RHEL8.  They also don't understand why folks
> believe that the specification of SEEK_END is underspecified for fmemopen().
> The problem from an ISV and developer is that the interpretation of SEEK_END
> is already defined for file-based streams, and the behavior for memory-based
> streams is not consistent with it. Consistency is our issue. If there is
> something in the spec of fmemopen() or open_memstream() that implies the
> resulting stream should behave differently from fopen(), then that could be
> an issue. But as far as I'm aware, the resulting memory-based stream should
> have the same behavior as a file-based stream, only it's in memory.

Red Hat Enterprise Linux 7 is in Maintenance Support 2 and as such only urgent priority bug fixes will be considered. Existing customers on RHEL 7 are likely to have applications that may be depenent on the *existing* open_memstream() behaviour in terms of reported buffer sizes. If we change the behaviour of the API to make it consistent with RHEL 8 we risk breaking those existing applications. Conservatively we aim to keep RHEL 7 applications working as this is one of the important requirements from an enterprise operating system point of view.

Customers looking to release and test a single binary application across RHEL 7 and RHEL 8 deployments will see these differences and will need to account for them. This is always going to be the case as older products enter later life-cycle phases.

Please feel free to point your developers here: https://www.openwall.com/lists/libc-coord/2020/09/24/1, if they wish to engage in the libc coordination list discussion with various authors of C libraries to weigh in on the specification of SEEK_END. We would be more than happy to have input from application developers regarding the direction the libraries should take.

Thank you for your input.

Comment 11 bob.huemmer 2020-09-28 11:32:17 UTC
Feed back from our developers regarding the standard:

"If a writable FILE* stream returned by open_memstream() behaves any differently from one returned by fopen(), then open_memstream() is useless to me, because my basic expectation of being able to pass either one to code that has been written/tested against the latter is broken. The application that uncovered this issue in the first place was a WAV audio file writer. In cases where the audio data size is not initially known, you first have to write out the data, then seek to the beginning, and write the data size into the file header. I see no reason why the standard should be elaborated in a way that this simple use case works with one type of stream, but not the other."

Comment 12 DJ Delorie 2020-09-28 20:41:30 UTC
> Feed back from our developers

Thanks for the feedback, and normally I'd agree with you, but the spec specifically states that a seek/write changes the value of "buffer size" returned after fflush/fclose (value pointed to by sizep), and that an fseek wrt SEEK_SET is shown resetting the buffer size to the "end".  That confuses the issue enough to warrant asking them to clarify.  Once they clarify, all the implementations can decide what (if any) fixes are required to meet the spec.  And since three projects came to three different conclusions about "what makes sense", I'd say it's not obvious, and perhaps there's some unsaid requirement or expectation that we don't know about.

Also, the problem case is open_memstream, not fmemopen.  They are specified differently upstream, which also adds to the confusion.

Comment 13 DJ Delorie 2020-09-28 21:27:59 UTC
Reported to the Austin Group...

https://www.austingroupbugs.net/view.php?id=1406

Comment 14 bob.huemmer 2020-09-29 12:11:21 UTC
Additional feedback from SAS Development regarding this issue and the latest response from Red Hat:

"I think this issue needs to be approached as an oversight/error in the specification, then. I have tested open_memstream() on multiple systems aside from GNU/Linux, and in every instance where it is available, the expected file-like behavior is observed: FreeBSD 11.1, MacOS X 10.13 (i.e. mainstream FreeBSD), NetBSD 8.1, even AIX 7.1. It is clear that in-the-wild implementations have taken a pragmatic approach, rather than follow what appears to be the (ill-advised) letter of the spec. If the GNU/Linux interpretation is upheld, I foresee that new standards work would be needed in this area to provide for the behavior that everyone assumed was the correct one."


Note You need to log in before you can comment on or make changes to this bug.