Bug 1163650
| Summary: | [linearstore] segfault when EFP tries to generate new journal but ulimit prevents it | ||
|---|---|---|---|
| Product: | Red Hat Enterprise MRG | Reporter: | Pavel Moravec <pmoravec> |
| Component: | qpid-cpp | Assignee: | Kim van der Riet <kim.vdriet> |
| Status: | CLOSED ERRATA | QA Contact: | Michal Toth <mtoth> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.0 | CC: | freznice, iboverma, jross, kim.vdriet, mcressma, mtoth, pmoravec, zkraus |
| Target Milestone: | 3.2 | Keywords: | Patch, WorkAround |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | qpid-cpp-0.30-5 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-10-08 13:09:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Pavel Moravec
2014-11-13 08:37:44 UTC
Potential upstream patch:
Index: /data_xfs/qpid/cpp/src/qpid/linearstore/journal/EmptyFilePool.cpp
===================================================================
--- /data_xfs/qpid/cpp/src/qpid/linearstore/journal/EmptyFilePool.cpp (revision 1639253)
+++ /data_xfs/qpid/cpp/src/qpid/linearstore/journal/EmptyFilePool.cpp (working copy)
@@ -239,6 +239,9 @@
return true;
//std::cout << "*** WARNING: EFP " << efpDirectory_ << " is empty - created new journal file " << fqFileName.substr(fqFileName.rfind('/') + 1) << " on the fly" << std::endl; // DEBUG
} else {
+ std::ostringstream oss;
+ oss << "std::ofstream ofs: file=\"" << fqFileName.c_str() << "\"" << " failed to be open";
+ throw jexception(jerrno::JERR_JNLF_OPEN, oss.str(), "EmptyFilePool", "validateEmptyFile");
//std::cerr << "*** ERROR: Unable to open file \"" << fqFileName << "\"" << std::endl; // DEBUG
}
return false;
(equivalent 3 lines in EmptyFilePool::createEmptyFile would be a patch in downstream)
Kim, could you please review?
Thanks Pavel for not only catching this bug, but also proposing a patch. I have reviewed the patch, and it looks good. However, the error code you use is for the JournalFile class, I would add a JERR_EFP_FOPEN error to error.h/error.cpp and use that code. As you point out, the upstream code has changed somewhat in this part, but the handling of the error is still in a TODO status. Committed revision 1640357. gdb shows the backtrace and problem matches:
(gdb) frame 1
#1 0x00007f9da0d759b2 in qpid::linearstore::journal::EmptyFilePool::takeEmptyFile (this=0x133c030, destDirectory="/var/lib/qpidd/qls/jrnl/Durable_10_66")
at /usr/src/debug/qpid-cpp-0.30/src/qpid/linearstore/journal/EmptyFilePool.cpp:109
109 std::string newFileName = destDirectory + emptyFileName.substr(emptyFileName.rfind('/')); // NOTE: substr() includes leading '/'
(gdb) p emptyFileName
$1 = ""
(gdb)
I *think* here is the problem:
bool EmptyFilePool::overwriteFileContents(const std::string& fqFileName) {
::file_hdr_t fh;
::file_hdr_create(&fh, QLS_FILE_MAGIC, QLS_JRNL_VERSION, QLS_JRNL_FHDR_RES_SIZE_SBLKS, partitionPtr_->getPartitionNumber(), efpDataSize_kib_);
std::ofstream ofs(fqFileName.c_str(), std::ofstream::out | std::ofstream::binary);
if (ofs.good()) {
ofs.write((char*)&fh, sizeof(::file_hdr_t));
uint64_t rem = ((efpDataSize_kib_ + (QLS_JRNL_FHDR_RES_SIZE_SBLKS * QLS_SBLK_SIZE_KIB)) * 1024) - sizeof(::file_hdr_t);
while (rem--)
ofs.put('\0');
ofs.close();
return true;
//std::cout << "WARNING: EFP " << efpDirectory << " is empty - created new journal file " << efpfn.substr(efpfn.rfind('/') + 1) << " on the fly" << std::endl; // DEBUG
} else {
//std::cerr << "ERROR: Unable to open file \"" << efpfn << "\"" << std::endl; // DEBUG
}
return false;
}
If ofs.good() returns false (i.e. unable to open the file), the file is not created and method returns false. Then createEmptyFile does not call emptyFileList_.push_back(efpfn); what means popEmptyFile returns empty string as filename. That is taken as emptyFileName in takeEmptyFile.
Yes, this sequence of steps *should* cause segfault already on line "emptyFileName = emptyFileList_.front();" and not in string::rfind. But per C++ reference, "Calling this function on an empty container causes undefined behavior." (same holds for pop_front), so it is imho possible explanation.
Regarless the above steps could cause the segfault, I would encourage to raising the newly added exception instead of "return false" in the above code. As in any case it leads to _some_ segfault.
Just my 2c.
(In reply to Pavel Moravec from comment #13) > gdb shows the backtrace and problem matches: > > (gdb) frame 1 > #1 0x00007f9da0d759b2 in > qpid::linearstore::journal::EmptyFilePool::takeEmptyFile (this=0x133c030, > destDirectory="/var/lib/qpidd/qls/jrnl/Durable_10_66") > at > /usr/src/debug/qpid-cpp-0.30/src/qpid/linearstore/journal/EmptyFilePool.cpp: > 109 > 109 std::string newFileName = destDirectory + > emptyFileName.substr(emptyFileName.rfind('/')); // NOTE: substr() includes > leading '/' > (gdb) p emptyFileName > $1 = "" > (gdb) > > > I *think* here is the problem: > > bool EmptyFilePool::overwriteFileContents(const std::string& fqFileName) { > ::file_hdr_t fh; > ::file_hdr_create(&fh, QLS_FILE_MAGIC, QLS_JRNL_VERSION, > QLS_JRNL_FHDR_RES_SIZE_SBLKS, partitionPtr_->getPartitionNumber(), > efpDataSize_kib_); > std::ofstream ofs(fqFileName.c_str(), std::ofstream::out | > std::ofstream::binary); > if (ofs.good()) { > ofs.write((char*)&fh, sizeof(::file_hdr_t)); > uint64_t rem = ((efpDataSize_kib_ + (QLS_JRNL_FHDR_RES_SIZE_SBLKS * > QLS_SBLK_SIZE_KIB)) * 1024) - sizeof(::file_hdr_t); > while (rem--) > ofs.put('\0'); > ofs.close(); > return true; > //std::cout << "WARNING: EFP " << efpDirectory << " is empty - created new > journal file " << efpfn.substr(efpfn.rfind('/') + 1) << " on the fly" << > std::endl; // DEBUG > } else { > //std::cerr << "ERROR: Unable to open file \"" << efpfn << "\"" << > std::endl; // DEBUG > } > return false; > } > > If ofs.good() returns false (i.e. unable to open the file), the file is not > created and method returns false. Then createEmptyFile does not call > emptyFileList_.push_back(efpfn); what means popEmptyFile returns empty > string as filename. That is taken as emptyFileName in takeEmptyFile. > > Yes, this sequence of steps *should* cause segfault already on line > "emptyFileName = emptyFileList_.front();" and not in string::rfind. But per > C++ reference, "Calling this function on an empty container causes undefined > behavior." (same holds for pop_front), so it is imho possible explanation. > > Regarless the above steps could cause the segfault, I would encourage to > raising the newly added exception instead of "return false" in the above > code. As in any case it leads to _some_ segfault. > > Just my 2c. It looks to me that this part of the code was updated and improved as part of the linearstore partitions implementation, and which is now included in 3.2. The check for IO success is now conducted by a function and which throws an exception when it fails. I recommend that this bug be re-evaluated using 3.2 packages. I have seen few times $err_msg while testing bug 1158903 with default ulimit on opne files and while deliberately setting ulimit -n to 512. Qpid broker never segfaulted on failure of not being able to open new file handler. Because the reproducer is same as for 1158903 bug, there has been segfault, but only because of system ran out of memory (oom-killer). err_msg="[System] error Could not accept socket: Too many open files (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/sys/posix/BSDSocket.cpp:214)" Marking as VERIFIED on rhel 7.1, and both archs on 6.7. qpid-cpp-server-linearstore-0.34-1.el{6,7} qpid-cpp-server-0.34-1.el{6,7} Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2015-1879.html |