Bug 3228 - rdist will not distribute hard link files
rdist will not distribute hard link files
Status: CLOSED NEXTRELEASE
Product: Red Hat Linux
Classification: Retired
Component: rdist (Show other bugs)
6.0
i386 Linux
medium Severity high
: ---
: ---
Assigned To: Jay Turner
:
: 3519 3740 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 1999-06-03 03:25 EDT by arto.selonen
Modified: 2015-01-07 18:37 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 1999-06-03 14:04:48 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description arto.selonen 1999-06-03 03:25:05 EDT
When distributing a whole directory recursively, rdist will
stop on hard links. It will report something like "can not
malloc memory for linkinfo", which is misleading. For
example, install gzip (which should produce /bin/gunzip
and /bin/zcat). Then try to distribute /bin to a target
that doesn't have gzip-package with rdist
using something like:

/tmp/distfile:
	( /bin ) -> ( target.machine )
		install -R ;

rdist -M 1 -f /tmp/distfile

/bin/gunzip gets copied, but /bin/zcat will produce an
error.

This should be easy enough to confirm. I have a patch
(two line fix), if you are interested in this.

We are using rdist extensively to distribute operating
system from one master machine to several others, and
with RedHat 6.0, this can not be done with standard rdist
out of rdist-6.1.5-7.i386.rpm package.

------- Email Received From  Arto Selonen <aselonen@lce.hut.fi> 06/03/99 03:43 -------
Comment 1 Jeff Johnson 1999-06-03 14:04:59 EDT
Fixed (by applying your patch) in rdist-6.1.5-8. Thanks for the
report.
Comment 2 Jeff Johnson 1999-06-17 14:30:59 EDT
*** Bug 3519 has been marked as a duplicate of this bug. ***

X-url: http://www.isi.edu/~johnh/
To: rdist-dev@usc.edu
Subject: correction to rdist memory consumption problem
(with fix)
Fcc: week, outbox
--------

On Feb. 4th I posted a message about a problem:

	I've found a serious memory consumption problem for rdist
when running
	over directories that contain many hard-linked files.  The
attached
	patch fixes this problem at the cost of ~12 more lines of
code and a
	little more dynamic memory allocation.

	Rdist allocates a fixed-sized buffer (struct linkbuf in
defs.h) for
	each file that has more than one hard link.  This buffer
includes
	pre-allocated space for 3 strings, each of length BUFSIZ.
Undef
	Redhat 5.2/linux-2.0.36 BUFSIZ is 8KB, so a lot of this
buffer goes to
	waste.

	For my mh mailbox directory (mh---even worse than news
spools at
	stressing file systems!), I have ~54k directory entires,
~35k files
	with only one link and ~8300 with 2-7 links.  With typical
rdist this
	was resulting memory usage of >90MB and much swapping on my
poor
	laptop.  With the included fix memory for this case is
<3MB.

(This problem can also be found at
http://developer.redhat.com/bugzilla/show_bug.cgi?id=1046&BUGLIST=1046.)

Unfortuantely, there was a bug in the patch that I posted
that
breaks rdist'ing files with multiple links in certain
circumstances.
(If Tdest isn't set when linkinfo is called.)

Here's the fix, relative to my prior patch:
> --- client.c-   Mon Jun  7 15:54:59 1999
> +++ client.c    Mon Jun  7 15:58:12 1999
> @@ -347,8 +347,8 @@
>         if (Tdest)
>                 lp->target = strdup(Tdest);
>         else
> -               lp->target = NULL;
> -       if (!lp->pathname || !lp->src || !(Tdest &&
lp->target))
> +               lp->target = strdup("");
> +       if (!lp->pathname || !lp->src || !lp->target)
>                 fatalerr("Cannot malloc memory in
linkinfo.");
>
>         return((struct linkbuf *) NULL);

(Basically if Tdest was null, lp->target was always null and
the
fatalerr was always triggered.)


Alternatively, here's a replacement original patch (with the
bug
fixed):

----------------------------------------
--- src/client.c-	Wed Jun 16 17:34:57 1999
+++ src/client.c	Wed Jun 16 17:35:57 1999
@@ -309,6 +309,18 @@
 	return(0);
 }

+void freelinkinfo(lp)
+	struct linkbuf *lp;
+{
+	if (lp->pathname)
+		free(lp->pathname);
+	if (lp->src)
+		free(lp->src);
+	if (lp->target)
+		free(lp->target);
+	free(lp);
+}
+
 /*
  * Save and retrieve hard link info
  */
@@ -317,6 +329,7 @@
 {
 	struct linkbuf *lp;

+	/* xxx: linear search doesn't scale with many links */
 	for (lp = ihead; lp != NULL; lp = lp->nextp)
 		if (lp->inum == statp->st_ino && lp->devnum ==
statp->st_dev) {
 			lp->count--;
@@ -329,12 +342,14 @@
 	lp->inum = statp->st_ino;
 	lp->devnum = statp->st_dev;
 	lp->count = statp->st_nlink - 1;
-	(void) strcpy(lp->pathname, target);
-	(void) strcpy(lp->src, source);
+	lp->pathname = strdup(target);
+	lp->src = strdup(source);
 	if (Tdest)
-		(void) strcpy(lp->target, Tdest);
+		lp->target = strdup(Tdest);
 	else
-		*lp->target = CNULL;
+		lp->target = strdup("");
+	if (!lp->pathname || !lp->src || !lp->target)
+		fatalerr("Cannot malloc memory in linkinfo.");

 	return((struct linkbuf *) NULL);
 }
--- src/docmd.c-	Wed Jun 16 17:35:07 1999
+++ src/docmd.c	Wed Jun 16 17:35:18 1999
@@ -586,7 +586,7 @@
 	if (!nflag) {
 		register struct linkbuf *nextl, *l;

-		for (l = ihead; l != NULL; free((char *)l), l = nextl) {
+		for (l = ihead; l != NULL; freelinkinfo(l), l = nextl) {
 			nextl = l->nextp;
 			if (contimedout || IS_ON(opts, DO_IGNLNKS) ||
 			    l->count == 0)
--- include/defs.h-	Wed Jun 16 17:35:14 1999
+++ include/defs.h	Wed Jun 16 17:35:18 1999
@@ -276,9 +276,9 @@
 	ino_t	inum;
 	dev_t	devnum;
 	int	count;
-	char	pathname[BUFSIZ];
-	char	src[BUFSIZ];
-	char	target[BUFSIZ];
+	char	*pathname;
+	char	*src;
+	char	*target;
 	struct	linkbuf *nextp;
 };

----------------------------------------

This patch can replace RedHat's rdist-6.1.5-links.patch to
fix the
problem.

To reproduce the problem, create a directory like
/home/johnh/TMP/RDIST_TEST, populate it with the commands
	cd /home/johnh/TMP/RDIST_TEST
	touch a b
	ln b bb
	cat >distfile <<END

HOSTS = ( boreas.isi.edu )

master: /home/johnh/TMP/RDIST_TEST -> ${HOSTS}
        install -w -y;
END

then exercise the bug
with
	rdist -d HOSTS=otherhost.of.yours.com -f
/home/johnh/TMP/RDIST_TEST/distfile

with the incorrect fix rdist will exit with the error
"Cannot malloc memory in linkinfo.".
The correct version will run to completion.

My apologies for sending in a buggy first fix.

   -John Heidemann


------- Additional Comments From jbj@redhat.com  06/17/99 11:27 -------
Could you reply to this message with ascii patch? I hate HTML ...
Thanks.

------- Email Received From  John Heidemann <johnh@isi.edu> 06/17/99 12:17 -------


------- Additional Comments From jbj@redhat.com  06/17/99 14:29 -------
This has already been fixed in #3228 although in a slightly
different fashion. Please reopen this bug if there is still a problem.
Comment 3 Jeff Johnson 1999-06-26 10:15:59 EDT
*** Bug 3740 has been marked as a duplicate of this bug. ***

The rdist program dies with a memory allocation error or
something similar (sorry, but I do not have the output
handy) when I try to rdist a large amount of stuff from one
RH 6.0 Linux box to another.  With the same distfile, the
rdist that shipped with RH 5.1 works fine.  Repeat by
rdisting a large fraction of the complete RH 6.0 release
from one host to another.

------- Additional Comments From   06/25/99 19:52 -------

The message rdist prints before it dies is
LOCAL ERROR: Cannot malloc memory in linkinfo

Note You need to log in before you can comment on or make changes to this bug.