My thinking is, since extension code is currently not packaged in Fedora (do you
have plans to attempt this?) it seems only logical to put it in /etc/drupal, so
it can be manually managed by the admin.  Are you suggesting that I put the
current contents of /etc/drupal in var/lib/drupal/sites, then symlink default
and all back to /etc/drupal?

If this is a prelude to packaging extensions, that's fine, but otherwise I'm not
sure of the value.  I could make all of /etc/drupal config(noreplace) and that
would protect user-installed extensions.

Open to thoughts.

I wasn't thinking about packaging extensions, though that is a thought worth
considering. Let me try to describe what I *was* thinking.

In practice, *every* drupal installation involves many extensions. The person
doing the setup hunts them down and sticks them under the sites tree, either in
sites/all or in the subtree for the particular config. The issue isn't whether
you, as packager, are packaging add-ons. The issue is that the drupal extension
mechanism requires [*] that extensions go in this particular directory
hierarchy. By putting the directory hierarchy in /etc, you inadvertently force
the user to put all of their add-on extensions down in /etc.

I think that the right thing to do is put the content of /etc/default in
/var/lib/drupal/sites, point the /usr/share/drupal/sites symlink at
/var/lib/drupal/sites, and drop /etc/default entirely.

Unfortunately, doing a post-install script that gets this right without breaking
the world is going to be tricky. I have some ideas about that, but lets converge
on objective before I offer thoughts on implementation that you may not need.

Well, not *every* installation.  I run vanilla. :)

See, I understand the concern, and in fact the current drupal RPM file layout
would mean that any hypothetical extension RPM would have all their files in
/etc/drupal/somewhere.  The problem I have is that config files go in /etc. 
Period.  Every other webapp in Fedora that I've looked into with the exception
of Moodle (which I maintain but did not initially package, so I intend to fix
this at some point) keeps config data in /etc.  See Squirrelmail.  Most people
using Linux expect this behaviour for most app, that the config will live either
in /etc or $HOME/.appname*.  I really don't want to break this behaviour without
a good reason.  

I would also point out that I consider, for drupal, extensions to be a part of
the configuration.  They are code, granted, but their availability and behaviour
are dependent on where they are placed within the sites folder.  To take this a
step further, if we were to package drupal extensions, how would we determine
where to place the extensions by default?  In sites/all?  in sites/default?  In
/usr/share/drupal-extension-name/, with a README saying "symlink from where you
want this to here"?  Seems sort of ugly.

This aside, what is the downside to forcing extensions to live in /etc/drupal? 
I'm curious.

To clarify, the problem is that site-specific modules and themes are put in
subdirectories under /etc/drupal/sites/wwww.example.com, right?

I don't think these subdirectories are that size variable after the modules and
extensions are added.  They grow quickly during the development process, as
Jonathan suggests, and through the sites existence.  These files are also larger
than typically found under /etc.  Also, they're not configuratoin files, they're
very much library code or PHP/HTML templates.  Therefore, people may not have
partitions large enough.  On my system, the drupal directory in /etc is second
only in size to the GNOME configuration tool (gconf).

This seems to be a shortcoming of Drupal, than of the packaging.  A workaround
-- "manually managed by the admin" -- would create symlinks for modules and
themese from /etc/drupal to somewhere under /var.  An optimal layout would be
(with arrows noting symlinks):

/var/lib/drupal/sites -> /etc/drupal
/etc/drupal/www.example.com/modules -> /var/lib/www.example.com/modules
/etc/drupal/www.example.com/themes -> /var/lib/www.example.com/themes

Perhaps the package should install a README file in the /etc/drupal suggesting
users create these symlinks back to /var/lib?

Why not:

/var/lib/drupal/sites -> /etc/drupal
/etc/drupal/www.example.com/modules -> /var/lib/drupal_sites/www.example.com/modules
/etc/drupal/www.example.com/themes -> /var/lib/drupal_sites/www.example.com/themes

to avoid cluttering /varlib, or possible name collision, lick if I had a drupal
site about mock, for example?

My word, I can't type.

s/\/varlib/\/varl\/lib/
s/lick/like/

:)

Grrr. . . . 

s/\/varl\/lib/\/var\/lib/

The other option is putting it in /home/ which is where a lot of sysadmins are
putting things that "grow".  The other option is putting it in the Apache directory.

$ rpm -qf /var/www
httpd-2.2.4-2.fc6

Does Drupal require httpd?  I forgot.

So here's some proprosals

/var/lib/drupal/sites -> /etc/drupal
/etc/drupal/www.example.com/modules /var/www/www.example.com/modules
/etc/drupal/www.example.com/themes /var/www/www.example.com/themes

Or the /home approach would substitute "home" for "var".

/var/lib/drupal/sites -> /etc/drupal
/etc/drupal/www.example.com/modules /home/www/www.example.com/modules
/etc/drupal/www.example.com/themes /home/www/www.example.com/themes

Could these be suggested in a README file in /etc/drupal?

I still think we need to insert drupal_sites before the URL in either proposal.

No, it doesn't require httpd directly, but I believe it's pulled in my php,
which it does require.

So am I correct in thinking the consensus we're slowly approaching amounts to an
updated README?

FYI, currently working on module packaging.  Will be going in
/usr/share/drupal/modules, currently.

I haven't been tracking the exchange since I filed the original bug, but if you
mean that the sites/ subdir will be going into the module/ subdir, this is not
the right thing.

Concerning using /home, the desired convention would probably be to use
/srv/www, since that is already set up properly for SELinux. Hmm. Given the
selinux issue, perhaps /srv/www/drupal-sites?

I can't remember. Is this directory supposed to be accessible to httpd? If so,
then somewhere under /srv/www seems right. Otherwise, that would be a really bad
choice.

No, that's where the modules will go.  /etc is unaffected.

Other web apps for fedora put modules in /usr/share, like squirrelmail.  This is
because the app calls the module, not the webserver.

/home is not the answer, that is frequently on a remove filesystem.  Mine is
most of the time.

s/remove/remote/

Somebody just released a Drupal update that broke working installs by
re-installing the /usr/share/drupal/sites and /usr/share/drupal/files symlinks
without checking whether there were actual sites installed first. On our site,
the results were sufficiently bad to crash httpd, and of course the entire site,
which was our primary corporate developer portal, stopped working.

People need to be able to re-point those symlinks without having them broken by
updates.

Given the discussion above, I suspect this update may be related to the current bug.

First rule of updates: NEVER BREAK AN EXISTING, WORKING SITE.

I'm now considering shifting away from the RPM-based drupal install because I
cannot afford to have my sites disrupted by updates. Would you like me to file a
separate, halt-the-world bug against the spec file, or is this bug thread the
right one? I'm not asking to be obnoxious; just not clear how you would prefer
to track this in the bug system.

So, if I understand it correctly, you:

1. Modified files owned by a package, files that are not config files.
2. Updated the package.
3. Noticed (rather dramatically) that the files you'd modified were restored to
the state specified by the package.

If I do what I think you want me to do to correct this, which is essentially
make the synlinks %config(noreplace), it will make it impossible for me to make
any changes to the location of the symlink target in the future, because
everyone's symlinks will freeze at the state at which I made them %config,
without manual intervention.  I suppose I could theoretically check the link
targets and leave them alone if they're modified, but then how do I let admins
know there's a new suggested default?  README?  Reinvent .rpmnew for symlinks?

I'm genuinely sorry your site broke, but there's really no bug here.  That's the
way RPM works.  How does the maintainer of a given package know you're going to
modify files owned by the package?  Were you betting that I'd make the changes
discussed here in the next update, which hadn't even been finalized, or did you
not understand RPM update behavior?

I don't want to break anyone's system, but I'm also bound by Fedora's filesystem
conventions.  Ideally, what would you like me to do?  What configuration
arrangement would satisfy both your needs, RPMs behavior, and
http://fedoraproject.org/wiki/Packaging/Guidelines#head-e1c5548cbbe551c7a43d375c524ab2ea0188557e
?

Well, that would be one way to look at matters, but I think that the content of
the current bug indicates that it's not the right way.

The drupal "sites" and "files" symlinks are well known to be things that drupal
administrators may need to modify. There are a variety of reasons that they
*must* be modified, including (for one) setting up site-specific files
directories and dealing with site trees that (e.g.) do not belong in /etc/
(because they are code) and do not belong in /var/ (because the SELinux config
isn't set up for them there, and the file space requirements may not be possible
to meet there on some systems).

Given that legitimate administration of Drupal sometimes *requires* modifying
these symlinks for security or configuration reasons, it really isn't
appropriate for them to be owned by the package. I understand that the
config(noreplace) option is unattractive, but for better or worse it seems to
reflect how Drupal is designed to work.

Respectfully, you are NOT bound by Fedora's filesystem conventions. Note the
word "conventions". The first priority is not to break the subsystem that is
being packaged (i.e. not to break its theory of operation -- this is orthogonal
to breaking working sites). Where the subsystem can be adapted to the file
system conventions in a way that is maintainable and durable, well and good.
Unfortunately, Drupal isn't an example where this seems to work very well (as
the current bug shows). It may turn out to be a case where the packaging
guidelines cannot be met. If that is the case, the packaging guidelines must
yield to the requirement for effective operation.

To answer your question, I think that the following assumptions have to be made:

  1. Once Drupal has been installed, you have to assume that sites have
     been set up at the locations established by the package.

  2. Unless you can determine that no sites have been set up, you have
     to leave the symlinks that exist unmolested.

  3. Given that normal drupal configuration sometimes requires changing
     those symlinks, I fear that they must indeed be configured as
     %config(noreplace).

I believe it is safe to change the symlink if BOTH of the following conditions hold:

  1. The symlink still has the originally installed values.
  2. No unpackaged files appear in the target directory (implying that no
     sites have been installed).

I sympathize with the issue that you can't fix things once this is done. This is
actually why I opened this bug in the first place -- the current desire to
satisfy the packaging guidelines doesn't conform to the Drupal theory of
operation, and the result is simply a mess.

But given where you are, I think that at some level the only answer that is
possible today is to declare that you can't screw around with the install tree
configuration during a minor release cycle, and resolving this mess is going to
have to wait for the next major Drupal release. In the meantime, the best you
can do is not defeat the administrators who are attempting to make Drupal work.
That will merely serve to force them off of RPM altogether, at which point you
will lose the ability to deliver security updates (which is VERY bad).

Note this means that you must not move the modules/ directory either until the
next major release cycle. Same issue.

So basically, you're saying %config(noreplace) the symlinks, and only change the
default targets at 6.x, 7.x, etc.

To play devil's advocate, if filesystem space allocation is an issue, I symlink
from there the RPM wants things to be to the spot where I have the room.  For
example, I have one at /var/lib/mysql to /var/lib/storage/mysql, because
/var/storage is the mount point for a RAID5 array and this buys me space and
performance.  Neither the RPM nor MySql itself cares if I do this.  Why not put
something in the README to advise this for Drupal?  The sites directory itself
is already %config(noreplace), so if you move that, replace it with a symlink,
the symlinks owned by the packages are still good, nothing breaks, and you can
put sites wherever you feel like and the RPM won't break.

I agree that the practice you describe will work. The problem with this as a
recommended practice is that it does not agree with the Drupal documentation,
which recommends altering those symlinks at the source.

One thing seems clear: if the package does not deliver clear and complete
documentation describing how the Fedora configuration diverges from the large
body of Drupal documentation, the package shouldn't diverge at all. Conversely,
if it *does* diverge, it *must* supply a suitable INSTALL and/or README file
that clearly explains what is going on.

Meta-comment: if the target directory is %config(noreplace), then the symlink
might as well be.

Hmm. Now that I think about it, I simply don't recall whether the original
upstream distribution has a symlink for the sites/ subdir at all, but the docs
do talk about moving that subdir someplace else and replacing it with a symlink
because it isn't in the right file system.

I think a fair summary of this issue is that the upstream drupal tree layout
wasn't particularly well thought out in the first place, which has driven
administrators to fix it for themselves, which is now creating a problem for you
as a packager. I wish I could suggest a good solution to this, but it's a sticky
wicket. The right answer, ultimately, is not to use a subsystem built by people
who don't understand deployment issues, but that doesn't help you as the poor
guy who is presently trying to package that subsystem. :-(

How about this README?:


Some system configurations do not provide large partitions for the
/etc directory.  To avoid filling this location with large Drupal
modules and themes, consider locating the modules and themes
directories for each site to another location, like /var or /srv.

Do not change the symlink from drupal/sites to /etc/drupal, by either
making it a different symlink or making it a directory.  This symlink
is maintained by the RPM-based package and will be overwritten on
updates.  Instead, consider doing the following.

First, make a subdirectory for the configuration of Drupal site:

  mkdir /etc/drupal/example.com

Then, choose a location to place your Drupal site-specific data (here
we're using /var/www/drupal, but /srv/www/drupal also works):

  mkdir /var/www/drupal

Make a subdirectory for your site:

  mkdir /var/www/drupal/example.com

And make the themes and modules subdirectories

  mkdir /var/www/drupal/example.com/modules
  mkdir /var/www/drupal/example.com/themes

Then make symlinks from /etc/drupal to /var/www/drupal for your site:

  cd /etc/drupal/example.com
  ln -s /var/www/drupal/example.com/modules
  ln -s /var/www/drupal/example.com/themes

The README is fine. The advice that it gives is truly terrible. You are
basically setting up a "symlink to a symlink" structure, apparently because you
are reluctant to let the user change the drupal/sites symlink. What is
protecting that symlink buying you that is worth imposing the pain of the
structure you are outlining onto the end user? The current structure is already
FAR too complicated. This is MORE complicated. What operational benefit is it
providing? Leaving the current suboptimal structure alone is a significant
improvement over this proposal.

The symlinks to modules and themes need to be explained better. I *think* that
drupal implements a search path for these such that the distributed modules and
themes are picked up from drupal/sites/default/modules and
drupal/sites/default/themes. Unless the user is installing custom
modules/themes, there should be no need to create those symlinks. If they *are*
installing custom modules/themes, putting them in /var/www/drupal seems
undesirable. Now, instead of having a single directory to manage for backup
purposes, there are umpteen directories to manage.

In past comments, you have expressed a strong motivation to comply with the
fedora filesystem layout guidelines -- enough so that I think this is your
primary motivation, and it is driving the creation of this rats nest of symlinks.

I suggest that you change your approach. Make compliance with guidelines be
objective number 2, and make simplicity of operational maintenance objective
number 1. The guidelines exist in order to make operations more straightforward.
To the extent that they are failing to achieve that, they should be set aside in
favor of a layout that is easy to use and maintain. The easiest layout decision
is achieved by letting the end user change the drupal/sites symlink, which is
what the Drupal people recommend.

If you succeed in complying with the layout guidelines at the cost of making a
structure that is not maintainable by administrators, people will "fix" the
problem by switching to a non-Fedora release. I'm a Fedora fan, but the "Fedora"
label on the OS isn't all that important to me in comparison to having
production systems that I can maintain.

I'm personally of the opinion that the current layout is fine and that admins
are generally capable of adjusting to their own needs.  To plagiarize my own
example, should I request that mysql-server move it's data from /var/lib/mysql
to some other place, because my system has /var on the smaller OS drive, and not
on my seperate RAID partition?  

Check out the review for a drupal module:
https://bugzilla.redhat.com/show_bug.cgi?id=359911

It's not going in /etc.  It's going in /usr/share/drupal/modules.  Installing a
module does not enable it for all sites.  Themes, once packaged, could go in
/usr/share/drupal/themes.  Installing a theme does not enable it for all sites.

How large are these site customizations that they are a space problem?  And as
far as backups go, are you not backing up /etc?  If you really want a module or
theme, and REALLY don't want it in /etc, why not ask me to package it?  I've got
5 module reviews up there for Infrastructure.  Just point me to a URL and I'll
even check the dependencies for you.

I'd really like to see hard numbers confirming that this is a real issue on a
production site before making any drastic changes.  I think custom-installed,
non-rpm modules and themes belong in /etc.  If they exist as RPMs, they will be
in /usr/share/drupal/. . . and it's a non-issue.

I think that compliance with guidelines and simplicity of operational
maintenance go hand in hand.  I suspect all we need is a README change.  Just
not the one in #19, at least not all of it.

I applaud your push to package modules and make them centrally available to a
Drupal installation.  Unfortunately for us Drupal users, site-specific modules
and themes is another layer of modularity that is necessary at this point in
time, given things like version conflicts between modules.  Anyway, revise the
README all you like and lets close this issue.

drupal-5.4-1.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.