Red Hat Bugzilla – Bug 101493
LTC3724-[BETA] Insufficient space on /usr volume to install all products
Last modified: 2007-11-30 17:06:57 EST
The following has be reported by IBM LTC:
[BETA] Insufficient space on /usr volume to install all products
Please fill in each of the sections below.
zSeries 9064-116, shark e20
taroon beta 1 under VM
Steps to Reproduce:
1. put / on one volume 3390-3 volume
2. put /usr on a second 3390-3 volume
3. install EVERYTHING!
Message: You don't appear to have enough disk space to install the packages
you've selected. You need more space on the following file systems:
Either some notion during the install process how big each package is to pick
and choose by size or some methodology to cut /usr into more reasonable chunks
The stand disk size, after formatting, of the standard 3390-3 image is about 2.3
GB. The message implies to me that /usr is then greater than 3.4 GB!
Many installations are bound by this "small" disk size alternative because they
are using older hardware or they have an installation standard of a 3390-3 image
on their RVA's or Sharks, or for performance reasons, they do not want too much
data behind a single disk - subchannel - for performance reasons.
The only alternative I can think of in installing a full system is to keep
installing pieces, then moving these pieces to some other dasd using soft links
/usr seems to be growing in leaps and bounds. If I had 30GB disk, no problem.
But in the zSeries world, 2.3 GB is still the standard.In addition, when you
specify, the install process loops. It only gives you the
option to say ok, which then goes back, reformats the file system, and gives the
same message! It should at least take you back to the point where you specify
the packages and let you deselect some.After install of a much smaller set of
choices, I did a df command
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/dasda1 2366164 114756 2131212 6% /
/dev/dasdb1 2366164 1613388 632580 72% /usr
So it looks like almost everything is dumped into the /usr directory! /opt and
/misc are empty.Jim - if you have a standard 2.3 GB disk, maybe you should not
However, your Comment#1 is a problem and needs to be fixed by Red Hat.In the
zSeries world, IBM set the standard geometry as 3390 when they went from
3390 to RAMACI in the late '80s. At that time, the device size was also 3390-3
(2.7 GB), so that became the defacto standard in the industry. Later devices
(RAMACII-III, RVA, RSA, Shark, and our competitors - EMC, hitachi) have pretty
much kept to this standard because that is what the MVS device driver is set
for. With some of the devices, such as RVA or Shark, you can increase the size
of the volume, most notably to 3390-9 (3 times larger than a 3390-3), but,
unless you have PAV's, you take a performance hit.
Under linux, the actual amount of data is 4096 byte blocks times 12 blocks per
track or 49152 bytes (48K) per track not the 56K you often hear because of the
ECKD overhead. That makes a 3390-3 of 3339 tracks (15 tracks/cyl) 2347.73 MB. A
3390-9, the next popular size would be 7043.2 GB.
This 48K track is not implemented directly on the track today as with real 3390,
but, since RAMAC I, this is really a virtual mapping to a SCSI device or raid
array. It is a logical, not physical construct. Why not change it? Because the
customers of IBM have stated that they do not want anymore changes to the
logical format. So, this standard, 3390-3, has in some sense solidified in the
industry on zSeries.
There is one device per subchannel. The way you get to a lot of data is that you
have a lot of subchannels/devices. To get to 1.2 TB of data is 512 3390-3 class
devices or 171 3390-9 class devices.
Each subchannel is either ESCON at 10 MB/sec or FICON at 40/MB sec, though ESCON
probably predominates still today.
Therein lies the problem. Without PAV's, you are confined to one file
transferring data at a time. With PAV's, you can one file per PAV transferring
data at a time.
2.4 GB behind a 10 MB/sec pipe is one thing. 7 GB behind a 10 MB/sec pipe is
quite another. So many people are reluctant to put go with larger volumes.
You will note that the notion of larger volumes on Shark did not really get
interesting until PAV's came around because putting a lot of active data on a
larger volume turns out to be a bottleneck in the hardware architecture. They
are now implementing 32GB devices. But that is not feasible without 1) PAV's and
2) FICON. Otherwise, you spend most of your I/O time queued behind the
subchannel, waiting for it to be free - huge performance bottleneck.
So, people who are not running the latest hardware really cannot do justice to
large volumes and get adequate performance.
And if, as a Linux customer, you have an old device that does not have the
capability of larger volume, you are pretty much stuck.
That is why we do our system volume testing for DB2 on 3390-3 volumes - because
then we know that it will accomodate the vast majority of customers, old and
new. When we try and force a configuration with larger devices, there is a lot
of customer resistance for various reasons.
So RedHat and the other distributors need to take this into account when
implementing these systems in a zSeries environment.
How can we load the a full system on a series of 3390-3 images? Jim - thanks for
your good explanation!
Glen/Greg - Two issues here for Red Hat to consider/fix:
1. How can we install a full system (everything) on a series of standard
3390-3 volumes (which is about 2.3GB) ?
2. When we specify which packages to install, and if there is not enough
disk space, the install process loops around. It should go back and
allow the user to re-specify which packages to install again.
We provide the capability to use either software raid or LVM so you can get
larger logical volumes out of ridiculously small drives.
------ Additional Comments From email@example.com 2003-01-08 22:07 -------
Ridiculous size volumes is purely a matter of opinion. Is Red Hat willing to make
this statement to the customers who want to buy RHEL for zSeries?
Generally, from what I have read in the Marist forum indicates that no-one
really wants to use raid or LVM for their root volumes because of the
unreliability during recovery situations.
A more intelligent approach would be to allow installation of the software to
alternate directories. Only the stuff that really needs to be there
during bootup needs to be in /usr.
I think the key point here is that the user chose to install "EVERYTHING!".
Even in taroon this is quite a lot of packages, and most of the bits go into
/usr. If the user has chosen the default package set, perhaps adding in a few
key apps, there would have been no problems.
Additionally as Jeremy helpfully pointed out, you can use either software raid
or LVM to glue smallish partitions into larger ones.
------ Additional Comments From firstname.lastname@example.org 2003-04-08 21:03 -------
Judging from the replies from RedHat, it seems they choose to take approach that
it is a "user problem" rather the recognizing the real problems and constraints
working with zSeries DASD. They should be thinking about creative solutions
needed to resolve the issue than a "one size fits all" approach.
First of all, I did pare things down significantly, and needed 78% of a 3390-3
volume to fit, for our data center, a useful system.
We build a single instance of a distribution, customize it for this data center,
then distribute it to all our users rather than building a customized system for
each user. Than manpower would be prohibitive to customize each instance, so our
compromise is to use a larger system that what would fit any one users needs.
Since we have 40 active systems in LPAR mode and probably another 60 systems in
VM/EC mode, using RH 7.1, RH 7.2, SuSE SLES7, and SuSE SLES8, we try to think
beyond the PC mentality.
As to the popularity of LVM for volumes needed during boot, RedHat ought to
review the thread entitled Root "LVM - Two questions" in the Marist Forum
http://www2.marist.edu/htbin/wlvgl?L=LINUX-VM&LOG=LOG0307 . The posters on
Marist do not think that LVM for boot volumes is a very good idea.
As to the size of zSeries volumes, that will continue to a performance
constraint until PAV's are available because of the architecture of the hardware.
If some of the elements of /usr were not needed during boot, they ought to be
somewhere else. It makes no sense to put source, for example, in the same
directory that one needs to start programs provided by RedHat but not officially
part of the kernel. It is interesing to note that the stuff required for the
kernel is only about 168 mb. The other several GB of stuff is dumped into /usr
by the vendor.
------ Additional Comments From email@example.com 2003-04-08 21:11 -------
There also seems to an assumption that I have to have one copy of /usr per
instance in the Linux design. Clearly, with many instances, this is a waste of
A notion that we are working on, but does not seem to fit the LVM approach to
/usr is a shared /usr. These should be, according to the posix spec, read only
files. Certain kinds of things (such as the source) could be place on other
volumes and mounted by fstab as ro during boot, saving the amount of space we
need for dozens of copies of a single distribution. It appears to me that LVM
really doesn't like to share volumes.
What |Removed |Added
------- Additional Comments From firstname.lastname@example.org 2005-06-14 19:10 EDT -------
Non-duplicate rejected bug that's been rejected for over 6 months. Marking
closed, you can re-open if needed.