Description of problem: Unable to collect logs to an FTP server. Version-Release number of selected component (if applicable): 5.3.2.6 - 5.3.2.6.20150108100920_387a856 How reproducible: Always Steps to Reproduce: 1. Add an FTP log collection point to cfme 2. Run log collection Actual results: Error in Web UI and evm.log; no logs collected. Expected results: No errors; logs are successfully uploaded to the FTP server. Additional info: This is happening because of the following line: vmdb/app/models/file_depot_ftp.rb:86 ``` ftp.mkdir(destination_path) ``` The directory path cannot be created all at once (unless its just a single folder). So when called with e.g. "/default_1/EVM_1" it fails with error 550. The path has to be iterated over and created one directory at a time.
*** Bug 1188359 has been marked as a duplicate of this bug. ***
Thanks Jan, yes, it's a crazy mess I've been looking at today. Apparently, IIS and other ftp servers can raise a 550 on both a missing file and already existing file. See: http://support.microsoft.com/kb/969061 In other words, it's not easy to tell for sure what the issue is... creating a directory/file when it already exists or checking for a file/directory when it's missing... it's a 550 error in either case. I have a WIP branch that appears to fix some of this for me but it's so horrible, I can't open a PR on it: https://github.com/jrafanie/manageiq/compare/ftp_550_error If you have any ideas, I'd love to hear them. I hated net/ftp when we wrote this code originally and I still hate it.
Jan, it looks like this commit associated with a refactoring is what broke this functionality: https://github.com/ManageIQ/manageiq/commit/32df530deeb39a54d713dfea71149b71ce215bc4 I'll look to resurrect it.
Upstream pull request: https://github.com/ManageIQ/manageiq/pull/1603
New commit detected on manageiq/master: https://github.com/ManageIQ/manageiq/commit/13ca633e6c4485f6b4b784b3ba324ec1796e79b6 commit 13ca633e6c4485f6b4b784b3ba324ec1796e79b6 Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:02:42 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Feb 6 15:46:42 2015 -0500 Refactor log_header into a method. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
New commit detected on manageiq/master: https://github.com/ManageIQ/manageiq/commit/ef2663d76ced898be756c4fbe41cc0ab53106500 commit ef2663d76ced898be756c4fbe41cc0ab53106500 Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:05:35 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Feb 6 15:46:43 2015 -0500 Expose @ftp as an attr_reader so we don't need to pass it around. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
New commit detected on manageiq/master: https://github.com/ManageIQ/manageiq/commit/39bf19c9b20aa1d1e238db96cf0c7feeab28be12 commit 39bf19c9b20aa1d1e238db96cf0c7feeab28be12 Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:12:26 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Feb 6 15:46:43 2015 -0500 Change base_path and destination_path to return Pathnames. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
New commit detected on manageiq/master: https://github.com/ManageIQ/manageiq/commit/3f7aaec0d5333a88d07a0e394e78d768d32954b0 commit 3f7aaec0d5333a88d07a0e394e78d768d32954b0 Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:24:16 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Feb 6 18:51:22 2015 -0500 Rewrite file/directory existence and upload mechanisms. The creation of intermediate directories was removed in commit 32df530deeb39a as some ftp systems will allow you to create a directory without the intermediate directories. This commit is more conservative and assumes you must create each intermediate directory one at a time. file_exists? now encapsulates issuing ftp 'NLST' with exception handling on missing file/directories. create_directory_structure will now traverse the desired directory path and create missing intermediate directories. Move the actual uploading into a private method. net/ftp methods don't support Pathnames so convert them to Strings. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 44 +++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 20 deletions(-)
New commit detected on manageiq/master: https://github.com/ManageIQ/manageiq/commit/34b127fa4e1e15a1c6f05740089b495ea8eb614b commit 34b127fa4e1e15a1c6f05740089b495ea8eb614b Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Feb 6 15:26:41 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Feb 6 18:51:35 2015 -0500 Update FileDepotFtpAnonymous to match changes to FileDepotFtp. destination_file_exists? was renamed to file_exists? since it's used for the destination directories in addition to the destination file. destination_path needs to return a Pathname object that responds to join. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp_anonymous.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
New commit detected on manageiq/master: https://github.com/ManageIQ/manageiq/commit/a49ae387ea9b5cad0857fb6ec4b58c0ef6a021d0 commit a49ae387ea9b5cad0857fb6ec4b58c0ef6a021d0 Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Feb 6 15:29:46 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Fri Feb 6 18:51:35 2015 -0500 Update tests to reflect the intermediate directory creation. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/spec/models/file_depot_ftp_spec.rb | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
I'm copying the description of the upstream pull request here for convenience (https://github.com/ManageIQ/manageiq/pull/1603): The creation of intermediate directories was removed in commit 32df530 as some ftp systems will allow you to create a directory without the intermediate directories. Some ftp servers, including iis, don't support doing this and raise a 550 error on the first missing directory. This pull request is more conservative and assumes you must create each intermediate directory one at a time. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 In a follow up pull request, I'll be adding camcorder tests to test the net/ftp interactions as the specs don't cover end to end testing. The logging within FileDepotFtp was slightly changed. Below is an example snippet when the ftp server doesn't have some of the intermediate directories: [----] I, [2015-02-06T16:03:43.710741 #51742:3fd4dd465be4] INFO -- : MIQ(FileDepotFtp#with_connection) Connecting through FileDepotFtp: [test ftp] [----] I, [2015-02-06T16:03:43.710859 #51742:3fd4dd465be4] INFO -- : MIQ(FileDepotFtp#connect) Connecting to FileDepotFtp: test ftp host: test-ftp.foo.bar.com... [----] I, [2015-02-06T16:03:43.833582 #51742:3fd4dd465be4] INFO -- : MIQ(FileDepotFtp#connect) Connected to FileDepotFtp: test ftp host: test-ftp.foo.bar.com [----] I, [2015-02-06T16:03:44.407360 #51742:3fd4dd465be4] INFO -- : MIQ(FileDepotFtp#create_directory_structure) creating tmp/user1/default_1000000000001 [----] I, [2015-02-06T16:03:44.541827 #51742:3fd4dd465be4] INFO -- : MIQ(FileDepotFtp#create_directory_structure) creating tmp/user1/default_1000000000001/EVM_1000000000001 [----] I, [2015-02-06T16:03:44.564096 #51742:3fd4dd465be4] INFO -- : MIQ(FileDepotFtp#upload) Uploading file: tmp/user1/default_1000000000001/EVM_1000000000001/Current_region_1_default_1000000000001_EVM_1000000000001_20150113_213629_20150206_160343.zip to File Depot: test ftp... [----] I, [2015-02-06T16:03:50.652019 #51742:3fd4dd465be4] INFO -- : MIQ(FileDepotFtp#upload) Uploading file: tmp/user1/default_1000000000001/EVM_1000000000001/Current_region_1_default_1000000000001_EVM_1000000000001_20150113_213629_20150206_160343.zip... Complete
New commit detected on cfme/5.3.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=0513e87e5d47522d6315d14ca6bb293986e80c93 commit 0513e87e5d47522d6315d14ca6bb293986e80c93 Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:02:42 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Mon Feb 9 11:32:22 2015 -0500 Refactor log_header into a method. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
New commit detected on cfme/5.3.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=91e662c7385c288b8f37e912ee725a11528a43d7 commit 91e662c7385c288b8f37e912ee725a11528a43d7 Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:05:35 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Mon Feb 9 11:32:28 2015 -0500 Expose @ftp as an attr_reader so we don't need to pass it around. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
New commit detected on cfme/5.3.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a222a416c0f617854d3c0eef3872636afb363db5 commit a222a416c0f617854d3c0eef3872636afb363db5 Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:12:26 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Mon Feb 9 11:32:32 2015 -0500 Change base_path and destination_path to return Pathnames. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
New commit detected on cfme/5.3.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a4c1f8711ef6a851c710c0ac723036e6a28119c4 commit a4c1f8711ef6a851c710c0ac723036e6a28119c4 Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Feb 6 15:26:41 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Mon Feb 9 11:32:39 2015 -0500 Update FileDepotFtpAnonymous to match changes to FileDepotFtp. destination_file_exists? was renamed to file_exists? since it's used for the destination directories in addition to the destination file. destination_path needs to return a Pathname object that responds to join. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp_anonymous.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
New commit detected on cfme/5.3.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=05d6825ffed8380bd0c704692a1b86002a4b3c1b commit 05d6825ffed8380bd0c704692a1b86002a4b3c1b Author: Joe Rafaniello <jrafanie> AuthorDate: Tue Feb 3 16:24:16 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Mon Feb 9 11:32:35 2015 -0500 Rewrite file/directory existence and upload mechanisms. The creation of intermediate directories was removed in commit 32df530deeb39a as some ftp systems will allow you to create a directory without the intermediate directories. This commit is more conservative and assumes you must create each intermediate directory one at a time. file_exists? now encapsulates issuing ftp 'NLST' with exception handling on missing file/directories. create_directory_structure will now traverse the desired directory path and create missing intermediate directories. Move the actual uploading into a private method. net/ftp methods don't support Pathnames so convert them to Strings. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/app/models/file_depot_ftp.rb | 44 +++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 20 deletions(-)
New commit detected on cfme/5.3.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=2c8fb4b07d27aab91a711587a0c315c35d438b86 commit 2c8fb4b07d27aab91a711587a0c315c35d438b86 Author: Joe Rafaniello <jrafanie> AuthorDate: Fri Feb 6 15:29:46 2015 -0500 Commit: Joe Rafaniello <jrafanie> CommitDate: Mon Feb 9 11:32:42 2015 -0500 Update tests to reflect the intermediate directory creation. https://bugzilla.redhat.com/show_bug.cgi?id=1184465 vmdb/spec/models/file_depot_ftp_spec.rb | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
An additional issue was found and fixed in the 5.3 version of this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1190812#c3
Verified in 5.4.0.0.11. Folders are created, logs uploaded and no error appears
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1100.html
https://github.com/ManageIQ/manageiq/pull/9127