Bug 435155

Hey Neil... the above spec/src.rpm links seem to give a 404 error here. 

Any chance for updated links? 

Sorry, silly finger check on my part.  Corrected  links
http://people.redhat.com/nhorman/s3fs.spe
http://people.redhat.com/nhorman/s3fs-0.4-1.fc8.src.rpm

s3fs is confusing :
http://code.google.com/p/s3fs/

There are other fuse-fs driver in Fedora :
fuse-smb
fuse-sshfs
fuse-encfs

fuse-s3 ?

Man page mount.s3fs is packaged. But mount.s3fs is not.

Not sure that I see s3fs as confusing in any particular way, but to preserve
naming, I can rename to fuse-s3fs (keeping eith the fuse-<meaningful name>fs
convention.  As for the mount.s3fs, I've recently removed it from the project as
its superfulous.  I'll repackage shortly

(In reply to comment #5)
> Not sure that I see s3fs as confusing in any particular way

I was locking for s3fs rpm (google code) and reached
https://fedorahosted.org/s3fs/ :-)
http://www.google.fr/search?hl=fr&q=s3fs+rpm&btnG=Rechercher&meta=

New spec/rpm

http://people.redhat.com/nhorman/fuse-s3fs.spec
http://people.redhat.com/nhorman/fuse-s3fs-0.4-2.fc8.src.rpm

I've renamed the package to fuse-s3fs and removed the mount.s3fs binary.  as for
the google code reference, yes, I found that after I created the project on
fedora hosted.  I'd really rather not give up the name, given that the google
code project isn't packaged and its particularly descriptive.  It looks like
there are actually multiple google code projects for s3fs.  One (s3fs-fuse) is a
python implementation that hasn't been touched in over a year.  The other, s3fs,
is a c++ implementation that has only seen recent activity (much like my
project).  As there are actively multiple project competing for the name, I'd
prefer not to give up my claim on it.  Neither of the other projects have been
packaged, or are looking to be packaged for Fedora or any other distro (at least
not publically that I can find).

It's "Amazon Simple Storage Service (Amazon S3)".
http://aws.amazon.com/s3
fuse-as3fs ?

http://aws.amazon.com/ (Amazon Web Services)
fuse-awss3fs ?

> the project on fedora hosted.

It's the latest project.
To be honest I don't really care. Your (a|aws|any way)s3fs seems very promising
and packaging it is really helpful. Many thanks.
But I think https://fedorahosted.org/s3fs/ should at least mention that "s3fs"
is used by other similar projets.

Some feedback :
With the "-C" flag ?
"s3fs -c|-r|-d|-f" require one argument (the bucket).

"s3fs bucket mountpoint" requires two arguments (btw, does not work, only "s3fs
-o bucket=xxx mountpoint" works).

Proposition :
s3fs -h # help
s3fs -c|-d|-r|-f bucket
s3fs -m bucket mountpoint # '-m' optional ?
s3fs -u mountpoint # umount (or the documentation should point to "fusermount -u")

Options :
[-o AWS_ACCESS_KEY_ID=xxx,AWS_SECRET_ACCESS_KEY=xxx]
Or -a xxx (access key) -s xxx (secret key)

Environment variables are fine also and convenient.

"But I think https://fedorahosted.org/s3fs/ should at least mention that "s3fs"
is used by other similar projets"
Thats an excellent suggestion.  I've updated the fedora hosted site to clarify
the naming contention.  In that effort I went looking, and as it turns out there
are at least 2 more projects (hosted on sourceforge) that make use of some
variant of this name :).  At any rate, I've added a paragraph to the hosted site
to make a point of relating this package to the hosted project.


Regarding your suggestions, you're right the man page is well out of date.  I'll
update and repost the package soon.  thanks!

New package available:

http://people.redhat.com/nhorman/fuse-s3fs.spec
http://people.redhat.com/nhorman/fuse-s3fs-0.4-3.fc8.src.rpm

Thanks & Regards!
Neil

(In reply to comment #10)
> http://people.redhat.com/nhorman/fuse-s3fs.spec

The line "mkdir -p $RPM_BUILD_ROOT/usr/share/man/man8" is useless.

I build the package in mock (--root=fedora-8-x86_64) and result/build.log show :
/usr/lib/rpm/pythondeps.sh: line 8: python: command not found
/usr/lib/rpm/pythondeps.sh: line 8: python: command not found
/usr/lib/rpm/pythondeps.sh: line 8: python: command not found
/usr/lib/rpm/pythondeps.sh: line 8: python: command not found

"BuildRequires:  python" fixe this.
In this case "Requires: python" is useless.
I get for "rpm -q --requires -p fuse-s3fs-0.4-3.fc8.mat.2.noarch.rpm" (my package) :
/usr/bin/python  
fuse-python
python-boto
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1

fuse-s3fs-0.4.3 provides the option "-p <pubkey>". AFAIK it's not a public key
(like pgp/x.509 public key). Same apply to the man page.

Amazon use "Access Key" and "Secret Access Key". Amazon keep the "Secret Access
Key". This is like login/password.

In contrast, for X.509 certificate (used by ec2), Amazon use "Public Key" and
"Private Key". Amazon do not keep the "Private Key".

Perhaps I am wrong.

Spec file updated with corrections from comment 11

As for the public/private key thing.  Its meant to look like a public/private
key setup, I think , but you are correct, in reality they're  not that.  I've
corrected the package to be more consistent in the naming

New packages, built successfully in koji available for reivew:
http://people.redhat.com/nhorman/fuse-s3fs.spec
http://people.redhat.com/nhorman/fuse-s3fs-0.4-4.fc8.src.rpm

Thanks!

Some random comments (details).

fuse-s3fs (could) do with real user data. I think the description of fuse-s3fs
(rpm and/or man page) should have a BIG warning until fuse-s3fs get stabilised.


From the man page :
- "s3fs will mount an amazon s3 bucket (that has been properly formatted)"
- "-f <bucket> Format  an  s3  bucket to make it suitable for mounting"
- "Directory  information  is stored  in  the  filesystem  metadata that is
retrieved during the mount operation from a file called fsdata.  This file is a
python class that has  been  pickled,  and  maintains  all  the directory 
heirarchy."

What is this format ?
The same as s3fs from google code ? Is it a standard from Amazon ? Is it a
fuse-s3fs (Fedora) specific format ?
Is it compatible with other tools (like s3fox) or should I only use fuse-s3fs ?
It's not clear and it's an important topic.

From de man page :
- "-f <bucket> Format  an  s3  bucket to make it suitable for mounting"

Format and so destroy all previous data ?


The man page does not give the default values.

The man page use "drive" instead of "bucket" in some places.


Many thanks for this great tool.

"rpm and/or man page) should have a BIG warning until fuse-s3fs get stabilised"
I actually had a warning on the project page and removed it as I am reasonably
confident in its stability.  I do however take your point since its not seen
widespread use.  I'll add a warning to the rpm description.

"...What is this format ?"
It is precisely what I indicated it was.  All the metadata for the fs is stored
in a python class that is pickled.  If you want to see what the data structure
looks like, all you need to do is look at the s3fs code.  Trying to describe it
in words I don't think would serve any additional purpose.  If you need
specifics, you can look at the S3DriveMetaData class.  thats the one that gets
pickled.   This is of course not compatible with most other tools.  However,
metadata aside, the files themselves that you store in s3fs are stored directly
as individual files (stored as their fully qualified path names).  As such, even
if the metadata gets corrupted (or reformatted), the files are still present on
S3.  While they won't be seen from an s3fs mount, they can still be retrieved
via a web interface (like www.s3browse.com).


"From de man page :
- "-f <bucket> Format  an  s3  bucket to make it suitable for mounting"

Format and so destroy all previous data ?"

no acutally (as per my previous paragraph)

"The man page does not give the default values"
for the mount options?  I can add that.

I'll also reconcile the drive/bucket discrepancy, as they are, as you have
likely guessed, synonomous.

I have a class to teach tomorrow, but I'll try to get to these and repost in the
afternoon.
Thanks!

Thanks for the reply (and sorry for my poor english).

> It is precisely what I indicated it was.  All the metadata for the fs is
stored in a python class that is pickled.  If you want to see what the data
structure looks like, all you need to do is look at the s3fs code.

For me it's also obvious.
My point of view was the "average joe".
For exemple the underlying FS of fuse-smb is smb. fuse-smb is compatible with
any smb implementation, can use any smb implementation. Fuse-smb is a client of
an smb server (which is a file system). For fuse-sshfs, it's ssh/scp. These fuse
file systems "export" _already existing_ file systems.

Fuse-s3fs does not have any underlying FS. S3 is not a file system. Unlike smb
or ssh, (fuse-)s3fs is not a "standard". fuse-s3fs define an entirely new file
system (like s3fs form google code do, etc). It's clear for "advanced" users, I
think it's less obvious for the average joe.
I think we should point this somewhere in the documentation. That is, fuse-s3fs
form Fedora is not compatible with s3fs from goole and so on.
It's an important point for the "average joe". He should not think "great I can
use my S3 storage with linux (fuse-s3fs) and with Windows (win-s3fs or like)".

> for the mount options?

Yes.


The whole propose of this bug report is the approval of fuse-s3fs to Fedora.
For me fuse-s3fs is fine for Fedora.
Sure, it should have bugs and miss features (it does not support S3 Europe :-)).
But the goal of Fedora is "the rapid progress of Free, Open Source software and
content" and share innovations (even if it's risky) with the broad Fedora community.

Thanks.

Ok, fair points.

I've updated the spec file to include a big warning in the description about
data integrity

I've updated the man page:
* to include defaults for all mount options
* to reconcile the drive/bucket naming consistency
* to include a section on compatibility with other s3fs implementations and
access methods.

New packages:
http://people.redhat.com/nhorman/fuse-s3fs.spec
http://people.redhat.com/nhorman/fuse-s3fs-0.4-5.fc8.src.rpm

Thanks!

Some thoughts.

Version convention.
http://people.redhat.com/nhorman/fuse-s3fs-0.4-2.fc8.src.rpm
...
http://people.redhat.com/nhorman/fuse-s3fs-0.4-5.fc8.src.rpm

These packages should have the same upstream release. It's an upstream issue.
http://fedoraproject.org/wiki/Packaging/NamingGuidelines

You can request advices here :
https://www.redhat.com/mailman/listinfo/fedora-packaging


By default fuse-s3fs use /tmp/{bucket} for its cache. It's "annoying" i think.
And from the man page :
- "preserve_cache=[no|yes]
If  set  to yes, the cache for the given mount will not be deleted on unmount,
saving the need for downloads on a subsequent remount (default: no)"

But Fedora (like many Unix system) use tmpwatch :
- "The tmpwatch utility recursively searches through specified
directories and removes files which have not been accessed in a
specified period of time.  Tmpwatch is normally used to clean up
directories which are used for temporarily holding files (for example,
/tmp)."


From the FHS (Filesystem Hierarchy Standard) it seems /var/cache is suitable for
your need for the root account :
http://www.pathname.com/fhs/pub/fhs-2.3.html#VARCACHEAPPLICATIONCACHEDATA

Root (system wide mount) : /var/cache/fuse-s3fs-cache/{bucket}
Users : $HOME/.fuse-s3fs-cache/{bucket}  (like firefox do).

"These packages should have the same upstream release. It's an upstream issue"
I'm not giving upstream a release number, only a version number and the package
version does currently match (I'm reuploading the same version to the release
area as you have me go through all this churn).  Its not the best thing to do,
but I really don't want to go bumping upstream version numbers for a man page
fix here and there.  Suffice it to say the version numbers match correctly at
the moment, and the release number is irrelevant to upstream.

As for the tmp file changes, fine.  New packages attached
http://people.redhat.com/nhorman/fuse-s3fs.spec
http://people.redhat.com/nhorman/fuse-s3fs-0.4-6.fc8.src.rpm

(In reply to comment #19)
> Its not the best thing to do,
> but I really don't want to go bumping upstream version numbers

You are the "boss", you are the developer and I really appreciate you work. You
have the last word.

But I am not requesting for "bumping upstream version numbers", but the respect
of a convention that, AFAIK everybody respect.

> I really don't want to go bumping upstream version numbers for a man page
fix here and there.

I understand.

But does it matter if fuse-s3fs-0.4-1.fc8.src.rpm through
fuse-s3fs-0.4-6.fc8.src.rpm was fuse-s3fs-0.4.1-1.fc8.src.rpm and
fuse-s3fs-0.4.6-1.fc8.src.rpm ?

Is it matter if I have :
$ rpm -q -l -p fuse-s3fs-0.4-4.fc8.src.rpm fuse-s3fs-0.4-6.fc8.src.rpm
fuse-s3fs-0.4.tbz2
fuse-s3fs.spec
fuse-s3fs-0.4.tbz2 # "wrong"
fuse-s3fs.spec

Or :
$ rpm -q -l -p fuse-s3fs-0.4-4.fc8.src.rpm fuse-s3fs-0.4-6.fc8.src.rpm
fuse-s3fs-0.4.tbz2
patch-cvs-20080310.diff
fuse-s3fs.spec
fuse-s3fs-0.4.tbz2
patch-cvs-20080313.diff
fuse-s3fs.spec

Or : 
$ rpm -q -l -p fuse-s3fs-0.5.cvs20080310-0.fc8.src.rpm
fuse-s3fs-0.5.cvs20080313-0.fc8-0.src.rpm
fuse-s3fs-0.5.cvs20080310.tbz2
fuse-s3fs.spec
fuse-s3fs-0.5.cvs20080313.tbz2
fuse-s3fs.spec


I don't know exactly how to deal with this. But I am quite sure some packagers
guru have the right answer.

Also, I am not sure koji can handle "soft" tagged version. I suppose koji have
some cache system.

> version numbers for a man page

If you release fuse-s3fs 0.4.1, 0.4.2, etc, use "this man page cover version
0.4.x (or just 0.4)".


Thanks!

Perhaps we're misunderstanding each other here.

Its my understanding that the version number in the rpm should match the
upstream source of the same version number (whats pointed to by the Source0: tag
in my spec file).  On that I think we both agree.  Please correct me if I'm wrong

What you've noticed is that I've posted several rpms to this bug, all of which
have the same version number, different revision numbers but contain the same
file set (i.e. no patches against the new revisions).  The implication that
you're concerned with here, if I'm understanding your concern, is that clearly
I've then modified the pristine source tarball that I'm pointing at.  And thats
badness. 

Nominally thats just not something that should be done.  I've been making the
changes in my upstream git tree in parallel, updating the tarball on
fedorahosted, and rebuilding the pakage with the same version.  The only reason
that I'm doing it now is that you're having me make minor cleanups to this
package in preparation for release to Fedora, and I'd really like to have the
first release of the package be identical to the first release of the tarball on
 the hosted site (silly I know, its just an OCD idosyncracy of mine).  Its
really just for convienience right now while you're reviewing this package. 
Once you give it the green light, I'll certainly not make changes to the 0.4
revision upstream.  That tag will stay fixed, and I'll bump the version number.

Does that adequately describe and satisfy your concerns?

Thanks for all your dilligent work on this review!

(In reply to comment #21)
> The only reason
> that I'm doing it now is that you're having me make minor cleanups to this
> package in preparation for release to Fedora

That's fine.

> Does that adequately describe and satisfy your concerns?

Fully.

> Once you give it the green light

For me you (fuse-s3fs) already have the green light.
I am not a mentor or sponsor or something like that. I am not confident with the
Fedora project in making approval package.
If you need someone to approuve your package and if it can please you that I
endorse this role, tell me and point me to some documentations to get the
process go ahead.

But be aware I am not a GNU/Linux developer.
Just an enthusiastic and happy Fedora's user. Currently I am mostly programming
for Windows :-(


Let me strongly suggest you to have a x.y.z pattern for the version number.
For the current fuse-s3fs this means fuse-s3fs-0.4.0-1.f8.noarch.rpm (and
fuse-s3fs-0.4.0.tar.gz for the upstream package). Keep the "version 0.4" in the
documentation. So you can update the upstream version (for minor bugs, etc which
will take place in 0.4.x) without touching the documentation. This just costs 2
chars ".0".

With x.y.z version number you can have :
- x : minor improvement, bug fix, ...
- y : major evolution, command line option change, ...
- z : format change, migration path needed.


I just check fuse-s3fs-0.4-6.
- default=$HOME/.fuse-s3fs-cache/

Perhaps not a gread idea :-)

New proposal :
- default system config : /etc/fuse-s3fs
- default system cache : /etc/fuse-s3fs/cache   (will be a symlink to
/var/cache/fuse-s3fs)
- default user config : $HOME/.fuse-s3fs
- default user cache : $HOME/.fuse-s3fs/cache

NB: default system cache is only for the root user (typically for mount at boot
time).
This permit in futur release to have a defaut config file :
$HOME/.fuse-s3fs/config .
Or a file for secret access key : $HOME/.fuse-s3fs/secret (mode 0700 only) .

"For me you (fuse-s3fs) already have the green light.
I am not a mentor or sponsor or something like that. I am not confident with the"

Please stop commenting on this bug then.  While you're comments are appreciated
, this bug is open specifically to request a package review for inclusion in
Fedora.  By actively holding a conversation on it, you may well be keeping
actual reviewers from taking the time to look at it, and that will prevent the
package from ever getting reviewed, and subsequently accepted.  Once this
package is in, I'll happily talk to you about enhancements.  Until then, please
let the reviewers do their job.

Setting fedora-review flag to (none). This flag must be set
by the reviewer.

> (In reply to comment #23)
> Please stop commenting on this bug then.

OK.

But you add https://bugzilla.redhat.com/show_bug.cgi?id=435155 in
https://fedorahosted.org/s3fs/ .

Note :
> Opened by Neil Horman (nhorman)  	 on 2008-02-27 12:40 EST
> Comment #1 From Kevin Fenzi (kevin)  	 on 2008-03-09 22:54 EST
> Comment #2 From Neil Horman (nhorman)  	 on 2008-03-10 09:09 EST  
> Comment #3 From Féliciano Matias (feliciano.matias)  	 on 2008-03-10
21:03 EST  	

There was no real review during 2 weeks.

> Assigned to : Nobody's working on this, feel free to take it
(nobody)

No "official" reviewer.

http://fedoraproject.org/wiki/PackageReviewProcess
> A Reviewer is defined as the person who chooses to review a package. For the
sake of clarity, one person takes ownership of the review. __Other people are
encouraged to comment on the review as well, either in the bug or on the mailing
list__.


> Once this package is in, I'll happily talk to you about enhancements.

I do not request any enhancements or bug fix. Just clarification. A good
documentation help reviewing a package. The /tmp issue was just playing well
with other packages.


> Until then, please let the reviewers do their job.

If you don't want me as (unofficial or official) reviewer, no problem.
I don't want to be the annoying man.
But please do not tell I am abusing the reviewing process or holding the
conversation or something like.


Best regards.

Hey Neil. I would love to formally review this package... 
look for a full review in a bit. 

OK - Package meets naming and packaging guidelines
OK - Spec file matches base package name.
OK - Spec has consistant macro usage.
OK - Meets Packaging Guidelines.
OK - License (GPLv2)
OK - License field in spec matches
OK - License file included in package
OK - Spec in American English
OK - Spec is legible.
See below - Sources match upstream md5sum:
See below - BuildRequires correct
OK - Package has %defattr and permissions on files is good.
OK - Package has a correct %clean section.
OK - Package has correct buildroot
OK - Package is code or permissible content.
OK - Packages %doc files don't affect runtime.
OK - Package has rm -rf RPM_BUILD_ROOT at top of %install

OK - Package compiles and builds on at least one arch.
OK - Package has no duplicate files in %files.
OK - Package doesn't own any directories other packages own.
OK - Package owns all the directories it creates.
See below - No rpmlint output.
OK - final provides and requires are sane.

SHOULD Items:

OK - Should build in mock.
OK - Should build on all supported archs
OK - Should function as described.
OK - Should have dist tag
OK - Should package latest version

Issues:

1. Your sources don't match with upstream:
d5904f2d8feae8c1e946b5cc3f4af82e  fuse-s3fs-0.4.tbz2
0821843e99f686a2854b2b12b3c5b06a  fuse-s3fs-0.4.tbz2.orig

I am looking at the src.rpm from comment #19.
Have you changed the upstream source since then without changing
the release?

2. rpmlint says:

fuse-s3fs.src: W: strange-permission fuse-s3fs.spec 0600

Can you make the spec mode 644 or the like?

3. Do you really need
BuildRequires: python ?
Nothing in the build seems to be needing python that I can see.

4. Might use '-p' with your install lines to preserve the timestamps
from the upstream package on install.

Ok, I've figured out the md5sum issue.  The makefile that I use to generate the
rpm and the release tarball had a bad dependency, and as such I was regenerating
the source tarball when building the rpm.  I've fixed that in my git tree.

As for the spec file permission, I had noticed that too.  I'm happy to change
them, but I'm curious as to why its occuring in the first place.  My makefile
uses rpmbuild -ts on the tarball that I generated (which incudes the spec file.
 You'll note that the spec file in the tarball is mode 0664), but when rpmbuild
-ts translates it into a src.rpm file, it adjusts the permissions to be 0600. 
I'm not sure why that would be, but I assumed that it was either a outdated
check in rpmlint, or a bug in rpmbuild.  Either way, I wasn't exactly sure what
to do about it.  I'd like to be able to use rpmsuild -ts to handle this, but am
not sure what the proper fix is.  Thoughts?

Thanks!

1. Cool. Thanks for tracking that down... it's important for sources to match up. 

2. Yeah, I think I have seen this before, but not sure where and what causes it.
;( Whats your umask when you run the make? If you run the rpmbuild with '-vv'
does it say why it's doing that?

Any thoughts on items 3 and 4?

1. Yeah, that will be correct on my next post of the spec/rpm file here
2. the only relevant output from rpmbuild -vv -ts <tarball> is:
D: fini      100600  1 (   0,   0)      2077
/home/nhorman/rpmbuild/SPECS/fuse-s3fs.spec 

but if I untar the file, the permissions are 664.  So it seems rpmbuild is doing
this, but I don't know why.

3.  I didn't think I'd need it either, but Mr. Matias was correct in comment 11,
building in mock does produce that error, which the BuildRequires line fixes. 
If its not needed in koji itself, I'll happily remove it

4. I'll update that and it will be in my next spec/rpm post (once we decide what
to do about the spec file mode issue).

Kevin, any thoughts as to how we should handle the permission issue on the spec
file?  Shal I just not use rpmbuild -ts at the moment, or should we not worry
about the permissions + file a bug against rpm-build?

Thanks!

Well, I could swear I have seen this before, but now I can't find a bug or a
workaround for it. ;( 

Perhaps post to fedora-devel? or fedora-packaging?

In the mean time if you are willing to just manually check in your spec to cvs
or generate a src.rpm thats got a spec with the correct perms, we shouldn't
consider this a blocker. 

Can you spin up a version with the final fixes? 

Ok, New packages available for review:
http://people.redhat.com/nhorman/fuse-s3fs.spec
http://people.redhat.com/nhorman/fuse-s3fs-0.4-8.fc8.src.rpm

Thanks!

and just to be clear, if/when you green light this, I'll just fix up the spec
permissions manually in  cvs when I import it.

Everything looks ok from what I can see now... so this package is APPROVED. 

> and just to be clear, if/when you green light this, I'll just fix up the spec
> permissions manually in  cvs when I import it.

Yeah, but note that cvs is bad about permissions. Once you check something in it
doesn't like changing it, so you should make sure and check in the 644 one in
the initial import here. 

New Package CVS Request
=======================
Package Name:  fuse-s3fs
Short Description: Fuse file system using amazon.com S3 storage service 
Owners:  nhorman
Branches:  devel, F-8
InitialCC:  nhorman
Cvsextras Commits: yes

cvs done.

Package Change Request
======================
Package Name:  fuse-s3fs
New Branches:  EL-5

The RHEL kernel has no support for FUSE as far as I can tell... 
so there wouldn't be any point in a EL-5 branch. 

Unless I am missing something? 

my bad, you're right.  I was messing with s3fs on a RHEL5 install, but I'd
forgotten that I'd built an upstream kernel on it, along with the fuse support
pacakges.

Closing this Review Request...