Description of problem: It seems the inplace edit is broken in perl 5.28.0: in some circumstances the file is not updated with the output from STDOUT. In the example below the -i script reads the complete input and then writes the output. This works for all perl versions I have tested so far, but not with 5.28.0 Version-Release number of selected component (if applicable): 5.28.0 How reproducible: The problem can be easily reproduced, find below a short test script. Steps to Reproduce: $ perl -e 'print "Perl $^V: FAILED\n"' > testfile $ perl -i -e 'local $/; my $s=<>; $s=~s/FAILED/OK/; print $s;' testfile $ cat testfile Actual results: Perl v5.28.0: FAILED Expected results: Perl v5.28.0: OK Additional info: I have tested with a few older versions, which deliver the expected output: Perl v5.26.1: OK Perl v5.24.1: OK Perl v5.16.3: OK Per: OK # v5.8.8 without $^V ### perl -V: Summary of my perl5 (revision 5 version 28 subversion 0) configuration: Platform: osname=linux osvers=4.18.10-200.fc28.x86_64 archname=x86_64-linux-thread-multi uname='linux buildvm-15.phx2.fedoraproject.org 4.18.10-200.fc28.x86_64 #1 smp wed sep 26 09:48:36 utc 2018 x86_64 x86_64 x86_64 gnulinux ' config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Dldflags=-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dlddlflags=-shared -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.28.0 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize -Duse64bitint' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='gcc' ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' optimize=' -g' cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include' ccversion='' gccversion='8.2.1 20180801 (Red Hat 8.2.1-2)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='gcc' ldflags ='-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib libs=-lpthread -lresolv -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lpthread -lresolv -ldl -lm -lcrypt -lutil -lc libc=libc-2.28.so so=so useshrplib=true libperl=libperl.so gnulibc_version='2.28' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' cccdlflags='-fPIC' lddlflags='-lpthread -shared -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -fstack-protector-strong' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_OP_PARENT PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API USE_SITECUSTOMIZE Locally applied patches: Fedora Patch1: Removes date check, Fedora/RHEL specific Fedora Patch2: support for libdir64 Fedora Patch3: use libresolv instead of libbind Fedora Patch4: USE_MM_LD_RUN_PATH Fedora Patch5: Provide MM::maybe_command independently (bug #1129443) Fedora Patch6: Dont run one io test due to random builder failures Fedora Patch8: Define SONAME for libperl.so Fedora Patch9: Install libperl.so to -Dshrpdir value Fedora Patch10: Document Math::BigInt::CalcEmu requires Math::BigInt (CPAN RT#85015) Fedora Patch11: Make *DBM_File desctructors thread-safe (RT#61912) Fedora Patch12: Replace EU::MakeMaker dependency with EU::MM::Utils in IPC::Cmd (bug #1129443) Fedora Patch13: Fix executing arybase::_tie_it() in Safe compartement (RT#131588) Fedora Patch14: Link XS modules to pthread library to fix linking with -z defs Fedora Patch15: Adjust tests to gdbm-1.15 (RT#133295) Fedora Patch16: Fix an integer wrap when allocating memory for an environment variable (RT#133204) Fedora Patch17: Fix printing a warning about a wide character when matching a regular expression while ISO-8859-1 locale is in effect Fedora Patch18: Fix invoking a check for wide characters while ISO-8859-1 locale is in effect Fedora Patch19: Fix index() and rindex() optimization in given-when boolean context (RT#133368) Fedora Patch20: Fix build conditions in locale.c Fedora Patch21: Fix a file descriptor leak in in-place edits (RT#133314) Fedora Patch22: Fix a file descriptor leak in in-place edits (RT#133314) Fedora Patch23: Fix a buffer overrun in deprecated S_is_utf8_common() Fedora Patch24: Fix a time race in Time-HiRes/t/itimer.t test Fedora Patch25: Fix matching an ASCII digit followed by a non-ASCII digit using a script run Fedora Patch26: Fix Time::Piece to handle objects in overloaded methods correctly Fedora Patch27: Fix an assignment to a lexical variable in multiconcatenation expressions (RT#133441) Fedora Patch28: Fix a spurious warning about uninitialized value in warn (RT#132683) Fedora Patch30: Pass the correct CFLAGS to dtrace Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux Built under linux Compiled at Oct 25 2018 13:15:23 @INC: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 --- Flags: category=core severity=high --- Site configuration information for perl 5.28.0: Configured by Red Hat, Inc. at Thu Oct 25 13:14:11 UTC 2018. Summary of my perl5 (revision 5 version 28 subversion 0) configuration: Platform: osname=linux osvers=4.18.10-200.fc28.x86_64 archname=x86_64-linux-thread-multi uname='linux buildvm-15.phx2.fedoraproject.org 4.18.10-200.fc28.x86_64 #1 smp wed sep 26 09:48:36 utc 2018 x86_64 x86_64 x86_64 gnulinux ' config_args='-des -Doptimize=none -Dccflags=-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Dldflags=-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dccdlflags=-Wl,--enable-new-dtags -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dlddlflags=-shared -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.28.0 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize -Duse64bitint' hint=recommended useposix=true d_sigaction=define useithreads=define usemultiplicity=define use64bitint=define use64bitall=define uselongdouble=undef usemymalloc=n default_inc_excludes_dot=define bincompat5005=undef Compiler: cc='gcc' ccflags ='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' optimize=' -g' cppflags='-D_REENTRANT -D_GNU_SOURCE -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fwrapv -fno-strict-aliasing -I/usr/local/include' ccversion='' gccversion='8.2.1 20180801 (Red Hat 8.2.1-2)' gccosandvers='' intsize=4 longsize=8 ptrsize=8 doublesize=8 byteorder=12345678 doublekind=3 d_longlong=define longlongsize=8 d_longdbl=define longdblsize=16 longdblkind=3 ivtype='long' ivsize=8 nvtype='double' nvsize=8 Off_t='off_t' lseeksize=8 alignbytes=8 prototype=define Linker and Libraries: ld='gcc' ldflags ='-Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib64 /lib64 /usr/lib64 /usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib libs=-lpthread -lresolv -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat perllibs=-lpthread -lresolv -ldl -lm -lcrypt -lutil -lc libc=libc-2.28.so so=so useshrplib=true libperl=libperl.so gnulibc_version='2.28' Dynamic Linking: dlsrc=dl_dlopen.xs dlext=so d_dlsymun=undef ccdlflags='-Wl,--enable-new-dtags -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld' cccdlflags='-fPIC' lddlflags='-lpthread -shared -Wl,-z,relro -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -fstack-protector-strong' Locally applied patches: Fedora Patch1: Removes date check, Fedora/RHEL specific Fedora Patch2: support for libdir64 Fedora Patch3: use libresolv instead of libbind Fedora Patch4: USE_MM_LD_RUN_PATH Fedora Patch5: Provide MM::maybe_command independently (bug #1129443) Fedora Patch6: Dont run one io test due to random builder failures Fedora Patch8: Define SONAME for libperl.so Fedora Patch9: Install libperl.so to -Dshrpdir value Fedora Patch10: Document Math::BigInt::CalcEmu requires Math::BigInt (CPAN RT#85015) Fedora Patch11: Make *DBM_File desctructors thread-safe (RT#61912) Fedora Patch12: Replace EU::MakeMaker dependency with EU::MM::Utils in IPC::Cmd (bug #1129443) Fedora Patch13: Fix executing arybase::_tie_it() in Safe compartement (RT#131588) Fedora Patch14: Link XS modules to pthread library to fix linking with -z defs Fedora Patch15: Adjust tests to gdbm-1.15 (RT#133295) Fedora Patch16: Fix an integer wrap when allocating memory for an environment variable (RT#133204) Fedora Patch17: Fix printing a warning about a wide character when matching a regular expression while ISO-8859-1 locale is in effect Fedora Patch18: Fix invoking a check for wide characters while ISO-8859-1 locale is in effect Fedora Patch19: Fix index() and rindex() optimization in given-when boolean context (RT#133368) Fedora Patch20: Fix build conditions in locale.c Fedora Patch21: Fix a file descriptor leak in in-place edits (RT#133314) Fedora Patch22: Fix a file descriptor leak in in-place edits (RT#133314) Fedora Patch23: Fix a buffer overrun in deprecated S_is_utf8_common() Fedora Patch24: Fix a time race in Time-HiRes/t/itimer.t test Fedora Patch25: Fix matching an ASCII digit followed by a non-ASCII digit using a script run Fedora Patch26: Fix Time::Piece to handle objects in overloaded methods correctly Fedora Patch27: Fix an assignment to a lexical variable in multiconcatenation expressions (RT#133441) Fedora Patch28: Fix a spurious warning about uninitialized value in warn (RT#132683) Fedora Patch30: Pass the correct CFLAGS to dtrace Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux --- @INC for perl 5.28.0: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 --- Environment for perl 5.28.0: HOME=/home/m LANG=de_DE.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/home/m/.local/bin:/home/m/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin PERL_BADLANG (unset) SHELL=/bin/bash
This looks like a glitch when more secure file replacement was added into perl code. $ cat test 1 $ strace perl -i -e '$/=undef; $a=<>; $a=~s/1/2/; print $a' test openat(AT_FDCWD, "test", O_RDONLY|O_CLOEXEC) = 3 ioctl(3, TCGETS, 0x7ffeb193f510) = -1 ENOTTY (Inappropriate ioctl for device) lseek(3, 0, SEEK_CUR) = 0 fstat(3, {st_mode=S_IFREG|0664, st_size=2, ...}) = 0 umask(0177) = 002 getpid() = 1192 openat(AT_FDCWD, "XXH6qplJ", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 4 fcntl(4, F_GETFD) = 0x1 (flags FD_CLOEXEC) umask(002) = 0177 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 ioctl(4, TCGETS, 0x7ffeb193f5a0) = -1 ENOTTY (Inappropriate ioctl for device) lseek(4, 0, SEEK_CUR) = 0 fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 getpid() = 1192 openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5 fstat(5, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=180, ...}) = 0 brk(NULL) = 0x55f03b859000 brk(0x55f03b87b000) = 0x55f03b87b000 fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 fchmod(4, 0100664) = 0 fstat(3, {st_mode=S_IFREG|0664, st_size=2, ...}) = 0 read(3, "1\n", 8192) = 2 read(3, "", 8192) = 0 [...] write(4, "2\n", 2) = 2 close(3) = 0 getpid() = 1192 close(4) = 0 unlinkat(5, "XXH6qplJ", 0) = 0 [...] close(5) = 0 exit_group(0) = ? It unlinks the output file instead of renaming it to the original input file name as it happens when editing per-line: $ strace perl -i -p -e 's/1/2/' test [...] openat(AT_FDCWD, "test", O_RDONLY|O_CLOEXEC) = 3 ioctl(3, TCGETS, 0x7fff259cc550) = -1 ENOTTY (Inappropriate ioctl for device) lseek(3, 0, SEEK_CUR) = 0 fstat(3, {st_mode=S_IFREG|0664, st_size=2, ...}) = 0 umask(0177) = 002 getpid() = 1196 openat(AT_FDCWD, "XXJAnFeW", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 4 fcntl(4, F_GETFD) = 0x1 (flags FD_CLOEXEC) umask(002) = 0177 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 ioctl(4, TCGETS, 0x7fff259cc5e0) = -1 ENOTTY (Inappropriate ioctl for device) lseek(4, 0, SEEK_CUR) = 0 fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 getpid() = 1196 openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5 fstat(5, {st_mode=S_IFDIR|S_ISVTX|0777, st_size=180, ...}) = 0 brk(NULL) = 0x5559d5c60000 brk(0x5559d5c82000) = 0x5559d5c82000 fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 fchmod(4, 0100664) = 0 read(3, "1\n", 8192) = 2 read(3, "", 8192) = 0 write(4, "2\n", 2) = 2 close(4) = 0 getpid() = 1196 renameat(5, "XXJAnFeW", 5, "test") = 0 close(5) = 0 close(3) = 0 [...] exit_group(0) = ?
Playing with developmental Perl tree, this works: $ ./perl -Ilib -i -e '$/=undef; while ($_=<>) {s/1/2/; print}' /tmp/test This does not work: $ ./perl -Ilib -i -e '$/=undef; $_=<>; {s/1/2/; print}' /tmp/test It seems there is a special handling for "while(<>)" that triggers the final rename. I will forward this bug report to Perl 5 Porters.
Thanks, Petr!
Perl 5 porters recommend calling "close ARGVOUT" explicitly as a workaround because the file replacement is triggered by closing the file handle (that normally happens by consecutive "<>" call that's called only once in your code): $ ./perl -i -e 'local $/; my $s=<>; $s=~s/FAILED/OK/; print $s; close ARGVOUT'
Thanks again for the analysis and both workarounds!
perl-5.28.1-427.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-55eef04b2c
perl-5.28.1-427.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-55eef04b2c
An update associated with this bug has been pushed to stable.