Bug 245815 - DS Admin Migration framework
DS Admin Migration framework
Status: CLOSED CURRENTRELEASE
Product: 389
Classification: Community
Component: Migration (Show other bugs)
1.0.4
All Linux
low Severity low
: ---
: ---
Assigned To: Rich Megginson
Viktor Ashirov
:
Depends On:
Blocks: 152373 240316 FDS1.1.0
  Show dependency treegraph
 
Reported: 2007-06-26 16:09 EDT by Noriko Hosoi
Modified: 2015-12-07 12:14 EST (History)
0 users

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-07 12:14:19 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
cvs diffs (8.61 KB, patch)
2007-06-28 13:59 EDT, Noriko Hosoi
no flags Details | Diff
new AdminMigration.pm.in (14.79 KB, text/plain)
2007-06-29 14:00 EDT, Rich Megginson
no flags Details
new asmigrate.map.in (3.91 KB, text/plain)
2007-06-29 14:01 EDT, Rich Megginson
no flags Details
new migrate-ds-admin.pl.in (3.68 KB, text/plain)
2007-06-29 14:01 EDT, Rich Megginson
no flags Details
new migrate-ds-admin.res.in (1.13 KB, text/plain)
2007-06-29 14:02 EDT, Rich Megginson
no flags Details
new asmigrate.ldif.tmpl (2.65 KB, text/plain)
2007-06-29 14:03 EDT, Rich Megginson
no flags Details
diffs for adding adminserver migration (18.29 KB, patch)
2007-06-29 14:22 EDT, Rich Megginson
no flags Details | Diff
new DSMigration.pm.in (16.89 KB, text/plain)
2007-06-29 14:24 EDT, Rich Megginson
no flags Details
new Migration.pm.in (11.47 KB, text/plain)
2007-06-29 14:24 EDT, Rich Megginson
no flags Details
new migrate-ds.pl.in (2.77 KB, text/plain)
2007-06-29 14:25 EDT, Rich Megginson
no flags Details
new migrate-ds.res (359 bytes, text/plain)
2007-06-29 14:25 EDT, Rich Megginson
no flags Details
diffs to add perl module migration support for DS (23.35 KB, patch)
2007-06-29 14:28 EDT, Rich Megginson
no flags Details | Diff
cvs commit log - ds migration (3.60 KB, text/plain)
2007-06-29 17:16 EDT, Rich Megginson
no flags Details
cvs commit log - adminserver migration (3.23 KB, text/plain)
2007-06-29 17:30 EDT, Rich Megginson
no flags Details
cross platform - ldapserver (51.69 KB, patch)
2007-07-10 15:27 EDT, Rich Megginson
no flags Details | Diff
cross platform - adminserver (1.13 KB, patch)
2007-07-10 15:30 EDT, Rich Megginson
no flags Details | Diff
cvs commit log (Comment #19) (1.41 KB, text/plain)
2007-07-12 09:57 EDT, Rich Megginson
no flags Details
cvs diff Makefile.am (1003 bytes, patch)
2007-07-16 14:35 EDT, Noriko Hosoi
no flags Details | Diff

  None (edit)
Description Noriko Hosoi 2007-06-26 16:09:53 EDT
Description of problem:
As for the code organization, we should probably split out what is currently in
migrateTo11 into a DS perl module, maybe scripts/Migrate.pm.in or something like
that.  Then we can use that code for both base DS migration and for AS+DS migration.

Perhaps we should have a script called migrate-ds.pl.in in DS and
migrate-ds-admin.pl.in in AS, parallel to setup-ds and setup-ds-admin.  The AS
code would handle the filtering of the o=NetscapeRoot data, as well as the
conversion of the old NES based admin server config files. 

See also
http://directory.fedoraproject.org/wiki/Migration_From_10
http://directory.fedoraproject.org/wiki/DS_Admin_Migration
Comment 1 Noriko Hosoi 2007-06-28 13:59:47 EDT
Created attachment 158147 [details]
cvs diffs

Files:
 ldapserver:
   ldap/admin/src/scripts/Util.pm.in
 adminserver:
   admserv/newinst/src/configdsroot.map.in
   admserv/newinst/src/dirserver.map.in
   admserv/newinst/src/register_param.map.in
   admserv/newinst/src/adminserver.map.in

Description:
This proposal is an improvement of Util.pm to support this replacement:
> Needs to improve:
> 2) The check_and_add_entry needs to handle Key:"default_value" in the map
file, that is, if Key is defined in the inf files, use the value; otherwise,
use "default_value".  It'd be used for ds_secure_port = ServerSecurePort:"636".

Ok.
Comment 2 Rich Megginson 2007-06-29 14:00:53 EDT
Created attachment 158227 [details]
new AdminMigration.pm.in
Comment 3 Rich Megginson 2007-06-29 14:01:16 EDT
Created attachment 158228 [details]
new asmigrate.map.in
Comment 4 Rich Megginson 2007-06-29 14:01:39 EDT
Created attachment 158229 [details]
new migrate-ds-admin.pl.in
Comment 5 Rich Megginson 2007-06-29 14:02:13 EDT
Created attachment 158230 [details]
new migrate-ds-admin.res.in
Comment 6 Rich Megginson 2007-06-29 14:03:06 EDT
Created attachment 158231 [details]
new asmigrate.ldif.tmpl
Comment 7 Rich Megginson 2007-06-29 14:22:01 EDT
Created attachment 158235 [details]
diffs for adding adminserver migration
Comment 8 Rich Megginson 2007-06-29 14:24:16 EDT
Created attachment 158236 [details]
new DSMigration.pm.in
Comment 9 Rich Megginson 2007-06-29 14:24:37 EDT
Created attachment 158237 [details]
new Migration.pm.in
Comment 10 Rich Megginson 2007-06-29 14:25:00 EDT
Created attachment 158238 [details]
new migrate-ds.pl.in
Comment 11 Rich Megginson 2007-06-29 14:25:28 EDT
Created attachment 158239 [details]
new migrate-ds.res
Comment 12 Rich Megginson 2007-06-29 14:28:07 EDT
Created attachment 158241 [details]
diffs to add perl module migration support for DS
Comment 13 Noriko Hosoi 2007-06-29 15:20:15 EDT
I quickly went through the diffs in Comment #2 ~ #12.
They look GREAT.
I'd like to confirm one thing:
The scripts support the migration on a same host, right?
For example, if I make a tar ball containing the config files and ldif files
exported from the DSes, copy it and expand it on a new host.  It'd be nice if
the migration scripts take care of the scenario by pointing the location as an
old server root.  I guess one of the issues is changelogdb.  I'm going to
investigate it some more...
Comment 14 Rich Megginson 2007-06-29 15:27:21 EDT
(In reply to comment #13)
> I quickly went through the diffs in Comment #2 ~ #12.
> They look GREAT.
> I'd like to confirm one thing:
> The scripts support the migration on a same host, right?

Right.

> For example, if I make a tar ball containing the config files and ldif files
> exported from the DSes, copy it and expand it on a new host.  It'd be nice if
> the migration scripts take care of the scenario by pointing the location as an
> old server root.  I guess one of the issues is changelogdb.  I'm going to
> investigate it some more...

Right.  Migration has an option for actualsroot but this is not used yet. 
That's the next thing we have to do for migration.  If you build a tarball then
copy it then untar it on the new system, you would use --oldsroot for the
physical location of the server root on the new disk, and --actualsroot for the
physical locatoin of the server root on the old disk.
Comment 15 Rich Megginson 2007-06-29 17:16:27 EDT
Created attachment 158266 [details]
cvs commit log - ds migration

Reviewed by: nhosoi (Thanks!)
Fix Description: Created a Migration class that is very similar to the Setup
class - to act as a sort of global context for the migration process.  Moved
most of the guts of migrateTo11 into the new DSMigration class and the new
migrate-ds.pl - we should deprecate migrateTo11 in favor of migrate-ds.pl.  I
had to enhance the check_and_add_entry function to handle pseudo-LDIF change
records - pseudo because mozilla perldap LDIF has no real LDIF change record
support.
Fixed a bug in create_instance.c - creating an instance without starting it was

not working if the port number of an existing directory server was supplied.
Added a new method createDSInstance to Util - this just wraps ds_newinst.pl for

now.
Platforms tested: RHEL4
Doc: Yes.  We will need to document the migration procedures.
Flag day: Yes.	Autotool file changes.
Comment 16 Rich Megginson 2007-06-29 17:30:40 EDT
Created attachment 158269 [details]
cvs commit log - adminserver migration

Reviewed by: nhosoi (Thanks!)
Fix Description: Created an AdminMigration class to handle all of the Admin
Server and Configuration DS specific parts of migration.  This should work for
both the NES based Admin Server and the Apache based (FDS 1.0) Admin Server.
A lot of the code in AdminServer.pm was reused, or made more generic for use in
migration as well as creation/setup.
Added a function to register all directory server instances with the
configuration DS.  This can be used for update/reconfig too.
Platforms tested: RHEL4
Doc: Yes.  We will need to document the migration procedures.
Flag day: Yes.	Autotool file changes.
Comment 17 Noriko Hosoi 2007-06-29 19:22:39 EDT
(In reply to comment #14)
> Right.  Migration has an option for actualsroot but this is not used yet. 
> That's the next thing we have to do for migration.  If you build a tarball then
> copy it then untar it on the new system, you would use --oldsroot for the
> physical location of the server root on the new disk, and --actualsroot for the
> physical locatoin of the server root on the old disk.

I did a quick test to check if changelog is usable across the hosts or not:
https://idmwiki.sfbay.redhat.com/export/idmwiki/Directory_Server_8.0_Migration_Memo#MMR

The result does not look promising.  CSN in the changelog is not recognized on
the new system.  (most likely 64-bit vs. 32-bit issue)  Could there be anything
I can try to convert/adjust the format?  (Should we write a tool to do that?)
Comment 18 Rich Megginson 2007-07-02 10:23:04 EDT
(In reply to comment #17)
> I did a quick test to check if changelog is usable across the hosts or not:
>
https://idmwiki.sfbay.redhat.com/export/idmwiki/Directory_Server_8.0_Migration_Memo#MMR
> 
> The result does not look promising.  CSN in the changelog is not recognized on
> the new system.  (most likely 64-bit vs. 32-bit issue)  Could there be anything
> I can try to convert/adjust the format?  (Should we write a tool to do that?)

If the CSN is the only binary data we need to worry about, then perhaps it would
make sense to do something like this.  perl does have the ability to handle
binary data (see pack and unpack).  But I think there is more binary data in the
changelog than just the CSN.  We also have to handle other binary data like the
CSN generator state, and possibly the uuid generator state.
Comment 19 Rich Megginson 2007-07-10 15:27:10 EDT
Created attachment 158883 [details]
cross platform - ldapserver
Comment 20 Rich Megginson 2007-07-10 15:30:11 EDT
Created attachment 158885 [details]
cross platform - adminserver
Comment 21 Noriko Hosoi 2007-07-11 18:30:08 EDT
Your new implementation looks good to me.

Could you post how to run the migration script?  I'd love to try the cross
platform migration.  Thanks!!
Comment 22 Rich Megginson 2007-07-12 09:57:17 EDT
Created attachment 159053 [details]
cvs commit log (Comment #19)

Reviewed by: nhosoi (Thanks!)
Files: see diff
Branch: HEAD
Fix Description: There are basically three parts to cross platform support
1) Allow a different physical server root than the logical server root.  This
allows you to copy the old server root directory to the target machine, either
by making a tarball or by a network mount.  Then you can migrate from e.g.
/mnt/opt/fedora-ds, and specify that the real old server root was
/opt/fedora-ds.  This is the distinction between the --oldsroot and
--actualsroot parameters.
2) Cross platform database migration requires the old data is converted to LDIF
first.	Migration makes the simplifying assumption that the database LDIF file
is in the old db directory and has the name of <old backend name>.ldif e.g.
userRoot.ldif
3) Cross platform replication migration doesn't preserve the state, so the
changelog nor other associated state information can be migrated.
I rewrote the old migration script to use the FileConn - this theoretically
will allow us to support migration using an LDAP::Conn as well.
I had to make some fixes to FileConn, primarily to support the root DSE.
Platforms tested: RHEL4
Flag Day: no
Doc impact: Yes.
Comment 23 Rich Megginson 2007-07-12 10:00:44 EDT
(In reply to comment #20)
> Created an attachment (id=158885) [edit]
> cross platform - adminserver

Checking in adminserver/admserv/newinst/src/migrate-ds-admin.pl.in;
/cvs/dirsec/adminserver/admserv/newinst/src/migrate-ds-admin.pl.in,v  <-- 
migrate-ds-admin.pl.in
new revision: 1.3; previous revision: 1.2
done
Comment 24 Rich Megginson 2007-07-12 10:15:32 EDT
(In reply to comment #21)
> Your new implementation looks good to me.
> 
> Could you post how to run the migration script?  I'd love to try the cross
> platform migration.  Thanks!!

http://directory.fedoraproject.org/wiki/DS_Admin_Migration#Remote_Source_to_Local_Target

In my testing I did an NFS mount of a RHEL4 i386 /opt/fedora-ds on a FC6 x86_64
/mnt/opt/fedora-ds.  Then I ran migrate-ds-admin.pl like this:
migrate-ds-admin.pl -o /mnt/opt/fedora-ds -a /opt/fedora-ds -x ... args ...
If you use migrate-ds-admin.pl --help it will tell you the other required args.
Then run migrate-ds-admin.pl with the -o
Comment 25 Noriko Hosoi 2007-07-16 14:35:37 EDT
Created attachment 159352 [details]
cvs diff Makefile.am

Let's get rid of old migration scripts.

After the above update, these are the migration related files:
$ du -a | egrep migrate
8	./share/fedora-ds/data/asmigrate.ldif.tmpl
8	./share/fedora-ds/properties/migrate-ds-admin.res
8	./share/fedora-ds/properties/migrate-ds.res
12	./share/fedora-ds/inf/asmigrate.map
12	./sbin/migrate-ds-admin.pl
8	./bin/migratecred
32	./bin/migrateTo11
8	./bin/migrate-ds.pl
16	./bin/migratecred-bin
Comment 27 Nathan Kinder 2007-12-21 18:02:40 EST
Verified that the new migration code is being used.  The functionality is
covered by the migration tests.

Marking as VERIFIED.

Note You need to log in before you can comment on or make changes to this bug.