Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1163650

Summary: [linearstore] segfault when EFP tries to generate new journal but ulimit prevents it
Product: Red Hat Enterprise MRG Reporter: Pavel Moravec <pmoravec>
Component: qpid-cppAssignee: Kim van der Riet <kim.vdriet>
Status: CLOSED ERRATA QA Contact: Michal Toth <mtoth>
Severity: high Docs Contact:
Priority: high    
Version: 3.0CC: freznice, iboverma, jross, kim.vdriet, mcressma, mtoth, pmoravec, zkraus
Target Milestone: 3.2Keywords: Patch, WorkAround
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: qpid-cpp-0.30-5 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-08 13:09:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2014-11-13 08:37:44 UTC
Description of problem:
When verifying https://bugzilla.redhat.com/show_bug.cgi?id=1150397#c12, another way of segfaulting broker was identified. It is assumed to be reproducible when broker hits ulimit on open files (/me failed to repro in a short time, though).

Despite broker should be run with vast ulimit settings, it should be robust enough to handle this kind of situation - anyhow better than segfaulting.


Version-Release number of selected component (if applicable):
qpid-cpp-server-linearstore-0.22-50.el6.x86_64


How reproducible:
Quoting QA: Bug was reproduced in variable time (5 to 16, to 63minutes).


Steps to Reproduce:
1. See https://bugzilla.redhat.com/show_bug.cgi?id=1150397#c13 in general.
2. In detail, have reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=1150397#c1, increase queues_per_topic=80 and have ulimit -n 1024.
3.


Actual results:
Broker segfaults, logging "Too many open files" prior the segfault. Backtrace see Additional info.


Expected results:
Broker/linearstore needs to raise some exception when good() returns false.


Additional info:
From backtrace:

#0  0x00000034d429c1b9 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::rfind(char, unsigned long) const () from /usr/lib64/libstdc++.so.6
#1  0x00007fe466719b50 in qpid::linearstore::journal::EmptyFilePool::takeEmptyFile (this=0xddb540, destDirectory="/var/lib/qpidd/qls/jrnl/Durable_4_8")
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/linearstore/journal/EmptyFilePool.cpp:109
#2  0x00007fe4667327a0 in qpid::linearstore::journal::LinearFileController::pullEmptyFileFromEfp (this=0x3478288)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/linearstore/journal/LinearFileController.cpp:239
#3  0x00007fe466748bfd in qpid::linearstore::journal::wmgr::flush_check (this=0x34784f8, res=@0x7fe462f55f90, cont=@0x7fe462f55f9f, done=@0x7fe462f55f9c)
    at /usr/src/debug/qpid-0.22/cpp/src/qpid/linearstore/journal/wmgr.cpp:651
#4  0x00007fe46674c134 in qpid::linearstore::journal::wmgr::enqueue (this=0x34784f8, data_buff=<value optimized out>, tot_data_len=177, this_data_len=<value optimized out>, dtokp=
    0x7fe3e583ed60, xid_ptr=0x0, xid_len=0, tpc_flag=false, transient=false, external=false) at /usr/src/debug/qpid-0.22/cpp/src/qpid/linearstore/journal/wmgr.cpp:223

and the fact that takeEmptyFile called std::basic_string::rfind for an empty string, the bug is here in code:

// WARNING: this method needs to be called under the scope of emptyFileListMutex_ lock
void EmptyFilePool::createEmptyFile() {
    ::file_hdr_t fh;
    ::file_hdr_create(&fh, QLS_FILE_MAGIC, QLS_JRNL_VERSION, QLS_JRNL_FHDR_RES_SIZE_SBLKS, partitionPtr_->getPartitionNumber(), efpDataSize_kib_);
    std::string efpfn = getEfpFileName();
    std::ofstream ofs(efpfn.c_str(), std::ofstream::out | std::ofstream::binary);
    if (ofs.good()) {
        ofs.write((char*)&fh, sizeof(::file_hdr_t));
        uint64_t rem = ((efpDataSize_kib_ + (QLS_JRNL_FHDR_RES_SIZE_SBLKS * QLS_SBLK_SIZE_KIB)) * 1024) - sizeof(::file_hdr_t);
        while (rem--)
            ofs.put('\0');
        ofs.close();
        emptyFileList_.push_back(efpfn);
//std::cout << "WARNING: EFP " << efpDirectory << " is empty - created new journal file " << efpfn.substr(efpfn.rfind('/') + 1) << " on the fly" << std::endl; // DEBUG
    } else {
//std::cerr << "ERROR: Unable to open file \"" << efpfn << "\"" << std::endl; // DEBUG
    }
}

if ofs.good() is false (i.e. due to ulimits, linearstore cant open a new file), "else" branch is "executed" doing nothing. That causes emptyFileList is not populated by a file - then method:

std::string EmptyFilePool::popEmptyFile() {
    std::string emptyFileName;
    {
        slock l(emptyFileListMutex_);
        if (emptyFileList_.empty()) {
            createEmptyFile();
        }
        emptyFileName = emptyFileList_.front();
        emptyFileList_.pop_front();
    }
    return emptyFileName;
}

returns empty string as emptyFileName.

To fix: the else branch in createEmptyFile needs to raise some journal exception.

Same logic (and same flaw in it) is seen in upstream qpid, though particular method differs:

std::string EmptyFilePool::createEmptyFile() {
    std::string efpfn = getEfpFileName();
    if (!overwriteFileContents(efpfn)) {
        // TODO: handle failure to prepare new file here  <---- HERE to raise the exception
    }
    return efpfn;
}

Comment 1 Pavel Moravec 2014-11-13 09:06:27 UTC
Potential upstream patch:

Index: /data_xfs/qpid/cpp/src/qpid/linearstore/journal/EmptyFilePool.cpp
===================================================================
--- /data_xfs/qpid/cpp/src/qpid/linearstore/journal/EmptyFilePool.cpp	(revision 1639253)
+++ /data_xfs/qpid/cpp/src/qpid/linearstore/journal/EmptyFilePool.cpp	(working copy)
@@ -239,6 +239,9 @@
         return true;
 //std::cout << "*** WARNING: EFP " << efpDirectory_ << " is empty - created new journal file " << fqFileName.substr(fqFileName.rfind('/') + 1) << " on the fly" << std::endl; // DEBUG
     } else {
+        std::ostringstream oss;
+        oss << "std::ofstream ofs: file=\"" << fqFileName.c_str() << "\"" << " failed to be open";
+        throw jexception(jerrno::JERR_JNLF_OPEN, oss.str(), "EmptyFilePool", "validateEmptyFile");
 //std::cerr << "*** ERROR: Unable to open file \"" << fqFileName << "\"" << std::endl; // DEBUG
     }
     return false;

(equivalent 3 lines in EmptyFilePool::createEmptyFile would be a patch in downstream)

Kim, could you please review?

Comment 2 Kim van der Riet 2014-11-13 15:07:38 UTC
Thanks Pavel for not only catching this bug, but also proposing a patch. I have reviewed the patch, and it looks good. However, the error code you use is for the JournalFile class, I would add a JERR_EFP_FOPEN error to error.h/error.cpp and use that code.

As you point out, the upstream code has changed somewhat in this part, but the handling of the error is still in a TODO status.

Comment 4 Pavel Moravec 2014-11-18 15:15:19 UTC
Committed revision 1640357.

Comment 13 Pavel Moravec 2015-03-24 08:17:52 UTC
gdb shows the backtrace and problem matches:

(gdb) frame 1
#1  0x00007f9da0d759b2 in qpid::linearstore::journal::EmptyFilePool::takeEmptyFile (this=0x133c030, destDirectory="/var/lib/qpidd/qls/jrnl/Durable_10_66")
    at /usr/src/debug/qpid-cpp-0.30/src/qpid/linearstore/journal/EmptyFilePool.cpp:109
109	    std::string newFileName = destDirectory + emptyFileName.substr(emptyFileName.rfind('/')); // NOTE: substr() includes leading '/'
(gdb) p emptyFileName 
$1 = ""
(gdb)


I *think* here is the problem:

bool EmptyFilePool::overwriteFileContents(const std::string& fqFileName) {
    ::file_hdr_t fh;
    ::file_hdr_create(&fh, QLS_FILE_MAGIC, QLS_JRNL_VERSION, QLS_JRNL_FHDR_RES_SIZE_SBLKS, partitionPtr_->getPartitionNumber(), efpDataSize_kib_);
    std::ofstream ofs(fqFileName.c_str(), std::ofstream::out | std::ofstream::binary);
    if (ofs.good()) {
        ofs.write((char*)&fh, sizeof(::file_hdr_t));
        uint64_t rem = ((efpDataSize_kib_ + (QLS_JRNL_FHDR_RES_SIZE_SBLKS * QLS_SBLK_SIZE_KIB)) * 1024) - sizeof(::file_hdr_t);
        while (rem--)
            ofs.put('\0');
        ofs.close();
        return true;
//std::cout << "WARNING: EFP " << efpDirectory << " is empty - created new journal file " << efpfn.substr(efpfn.rfind('/') + 1) << " on the fly" << std::endl; // DEBUG
    } else {
//std::cerr << "ERROR: Unable to open file \"" << efpfn << "\"" << std::endl; // DEBUG
    }
    return false;
}

If ofs.good() returns false (i.e. unable to open the file), the file is not created and method returns false. Then createEmptyFile does not call emptyFileList_.push_back(efpfn); what means popEmptyFile returns empty string as filename. That is taken as emptyFileName in takeEmptyFile.

Yes, this sequence of steps *should* cause segfault already on line "emptyFileName = emptyFileList_.front();" and not in string::rfind. But per C++ reference, "Calling this function on an empty container causes undefined behavior." (same holds for pop_front), so it is imho possible explanation.

Regarless the above steps could cause the segfault, I would encourage to raising the newly added exception instead of "return false" in the above code. As in any case it leads to _some_ segfault.

Just my 2c.

Comment 18 Kim van der Riet 2015-06-04 15:54:38 UTC
(In reply to Pavel Moravec from comment #13)
> gdb shows the backtrace and problem matches:
> 
> (gdb) frame 1
> #1  0x00007f9da0d759b2 in
> qpid::linearstore::journal::EmptyFilePool::takeEmptyFile (this=0x133c030,
> destDirectory="/var/lib/qpidd/qls/jrnl/Durable_10_66")
>     at
> /usr/src/debug/qpid-cpp-0.30/src/qpid/linearstore/journal/EmptyFilePool.cpp:
> 109
> 109	    std::string newFileName = destDirectory +
> emptyFileName.substr(emptyFileName.rfind('/')); // NOTE: substr() includes
> leading '/'
> (gdb) p emptyFileName 
> $1 = ""
> (gdb)
> 
> 
> I *think* here is the problem:
> 
> bool EmptyFilePool::overwriteFileContents(const std::string& fqFileName) {
>     ::file_hdr_t fh;
>     ::file_hdr_create(&fh, QLS_FILE_MAGIC, QLS_JRNL_VERSION,
> QLS_JRNL_FHDR_RES_SIZE_SBLKS, partitionPtr_->getPartitionNumber(),
> efpDataSize_kib_);
>     std::ofstream ofs(fqFileName.c_str(), std::ofstream::out |
> std::ofstream::binary);
>     if (ofs.good()) {
>         ofs.write((char*)&fh, sizeof(::file_hdr_t));
>         uint64_t rem = ((efpDataSize_kib_ + (QLS_JRNL_FHDR_RES_SIZE_SBLKS *
> QLS_SBLK_SIZE_KIB)) * 1024) - sizeof(::file_hdr_t);
>         while (rem--)
>             ofs.put('\0');
>         ofs.close();
>         return true;
> //std::cout << "WARNING: EFP " << efpDirectory << " is empty - created new
> journal file " << efpfn.substr(efpfn.rfind('/') + 1) << " on the fly" <<
> std::endl; // DEBUG
>     } else {
> //std::cerr << "ERROR: Unable to open file \"" << efpfn << "\"" <<
> std::endl; // DEBUG
>     }
>     return false;
> }
> 
> If ofs.good() returns false (i.e. unable to open the file), the file is not
> created and method returns false. Then createEmptyFile does not call
> emptyFileList_.push_back(efpfn); what means popEmptyFile returns empty
> string as filename. That is taken as emptyFileName in takeEmptyFile.
> 
> Yes, this sequence of steps *should* cause segfault already on line
> "emptyFileName = emptyFileList_.front();" and not in string::rfind. But per
> C++ reference, "Calling this function on an empty container causes undefined
> behavior." (same holds for pop_front), so it is imho possible explanation.
> 
> Regarless the above steps could cause the segfault, I would encourage to
> raising the newly added exception instead of "return false" in the above
> code. As in any case it leads to _some_ segfault.
> 
> Just my 2c.

It looks to me that this part of the code was updated and improved as part of the linearstore partitions implementation, and which is now included in 3.2. The check for IO success is now conducted by a function and which throws an exception when it fails. I recommend that this bug be re-evaluated using 3.2 packages.

Comment 20 Michal Toth 2015-08-17 09:13:34 UTC
I have seen few times $err_msg while testing bug 1158903 with default ulimit on opne files and while deliberately setting ulimit -n to 512. 
Qpid broker never segfaulted on failure of not being able to open new file handler.
Because the reproducer is same as for 1158903 bug, there has been segfault, but only because of system ran out of memory (oom-killer).

err_msg="[System] error Could not accept socket: Too many open files (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/sys/posix/BSDSocket.cpp:214)"

Marking as VERIFIED on rhel 7.1, and both archs on 6.7.
qpid-cpp-server-linearstore-0.34-1.el{6,7}
qpid-cpp-server-0.34-1.el{6,7}

Comment 22 errata-xmlrpc 2015-10-08 13:09:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-1879.html