Description of problem: As for the code organization, we should probably split out what is currently in migrateTo11 into a DS perl module, maybe scripts/Migrate.pm.in or something like that. Then we can use that code for both base DS migration and for AS+DS migration. Perhaps we should have a script called migrate-ds.pl.in in DS and migrate-ds-admin.pl.in in AS, parallel to setup-ds and setup-ds-admin. The AS code would handle the filtering of the o=NetscapeRoot data, as well as the conversion of the old NES based admin server config files. See also http://directory.fedoraproject.org/wiki/Migration_From_10 http://directory.fedoraproject.org/wiki/DS_Admin_Migration
Created attachment 158147 [details] cvs diffs Files: ldapserver: ldap/admin/src/scripts/Util.pm.in adminserver: admserv/newinst/src/configdsroot.map.in admserv/newinst/src/dirserver.map.in admserv/newinst/src/register_param.map.in admserv/newinst/src/adminserver.map.in Description: This proposal is an improvement of Util.pm to support this replacement: > Needs to improve: > 2) The check_and_add_entry needs to handle Key:"default_value" in the map file, that is, if Key is defined in the inf files, use the value; otherwise, use "default_value". It'd be used for ds_secure_port = ServerSecurePort:"636". Ok.
Created attachment 158227 [details] new AdminMigration.pm.in
Created attachment 158228 [details] new asmigrate.map.in
Created attachment 158229 [details] new migrate-ds-admin.pl.in
Created attachment 158230 [details] new migrate-ds-admin.res.in
Created attachment 158231 [details] new asmigrate.ldif.tmpl
Created attachment 158235 [details] diffs for adding adminserver migration
Created attachment 158236 [details] new DSMigration.pm.in
Created attachment 158237 [details] new Migration.pm.in
Created attachment 158238 [details] new migrate-ds.pl.in
Created attachment 158239 [details] new migrate-ds.res
Created attachment 158241 [details] diffs to add perl module migration support for DS
I quickly went through the diffs in Comment #2 ~ #12. They look GREAT. I'd like to confirm one thing: The scripts support the migration on a same host, right? For example, if I make a tar ball containing the config files and ldif files exported from the DSes, copy it and expand it on a new host. It'd be nice if the migration scripts take care of the scenario by pointing the location as an old server root. I guess one of the issues is changelogdb. I'm going to investigate it some more...
(In reply to comment #13) > I quickly went through the diffs in Comment #2 ~ #12. > They look GREAT. > I'd like to confirm one thing: > The scripts support the migration on a same host, right? Right. > For example, if I make a tar ball containing the config files and ldif files > exported from the DSes, copy it and expand it on a new host. It'd be nice if > the migration scripts take care of the scenario by pointing the location as an > old server root. I guess one of the issues is changelogdb. I'm going to > investigate it some more... Right. Migration has an option for actualsroot but this is not used yet. That's the next thing we have to do for migration. If you build a tarball then copy it then untar it on the new system, you would use --oldsroot for the physical location of the server root on the new disk, and --actualsroot for the physical locatoin of the server root on the old disk.
Created attachment 158266 [details] cvs commit log - ds migration Reviewed by: nhosoi (Thanks!) Fix Description: Created a Migration class that is very similar to the Setup class - to act as a sort of global context for the migration process. Moved most of the guts of migrateTo11 into the new DSMigration class and the new migrate-ds.pl - we should deprecate migrateTo11 in favor of migrate-ds.pl. I had to enhance the check_and_add_entry function to handle pseudo-LDIF change records - pseudo because mozilla perldap LDIF has no real LDIF change record support. Fixed a bug in create_instance.c - creating an instance without starting it was not working if the port number of an existing directory server was supplied. Added a new method createDSInstance to Util - this just wraps ds_newinst.pl for now. Platforms tested: RHEL4 Doc: Yes. We will need to document the migration procedures. Flag day: Yes. Autotool file changes.
Created attachment 158269 [details] cvs commit log - adminserver migration Reviewed by: nhosoi (Thanks!) Fix Description: Created an AdminMigration class to handle all of the Admin Server and Configuration DS specific parts of migration. This should work for both the NES based Admin Server and the Apache based (FDS 1.0) Admin Server. A lot of the code in AdminServer.pm was reused, or made more generic for use in migration as well as creation/setup. Added a function to register all directory server instances with the configuration DS. This can be used for update/reconfig too. Platforms tested: RHEL4 Doc: Yes. We will need to document the migration procedures. Flag day: Yes. Autotool file changes.
(In reply to comment #14) > Right. Migration has an option for actualsroot but this is not used yet. > That's the next thing we have to do for migration. If you build a tarball then > copy it then untar it on the new system, you would use --oldsroot for the > physical location of the server root on the new disk, and --actualsroot for the > physical locatoin of the server root on the old disk. I did a quick test to check if changelog is usable across the hosts or not: https://idmwiki.sfbay.redhat.com/export/idmwiki/Directory_Server_8.0_Migration_Memo#MMR The result does not look promising. CSN in the changelog is not recognized on the new system. (most likely 64-bit vs. 32-bit issue) Could there be anything I can try to convert/adjust the format? (Should we write a tool to do that?)
(In reply to comment #17) > I did a quick test to check if changelog is usable across the hosts or not: > https://idmwiki.sfbay.redhat.com/export/idmwiki/Directory_Server_8.0_Migration_Memo#MMR > > The result does not look promising. CSN in the changelog is not recognized on > the new system. (most likely 64-bit vs. 32-bit issue) Could there be anything > I can try to convert/adjust the format? (Should we write a tool to do that?) If the CSN is the only binary data we need to worry about, then perhaps it would make sense to do something like this. perl does have the ability to handle binary data (see pack and unpack). But I think there is more binary data in the changelog than just the CSN. We also have to handle other binary data like the CSN generator state, and possibly the uuid generator state.
Created attachment 158883 [details] cross platform - ldapserver
Created attachment 158885 [details] cross platform - adminserver
Your new implementation looks good to me. Could you post how to run the migration script? I'd love to try the cross platform migration. Thanks!!
Created attachment 159053 [details] cvs commit log (Comment #19) Reviewed by: nhosoi (Thanks!) Files: see diff Branch: HEAD Fix Description: There are basically three parts to cross platform support 1) Allow a different physical server root than the logical server root. This allows you to copy the old server root directory to the target machine, either by making a tarball or by a network mount. Then you can migrate from e.g. /mnt/opt/fedora-ds, and specify that the real old server root was /opt/fedora-ds. This is the distinction between the --oldsroot and --actualsroot parameters. 2) Cross platform database migration requires the old data is converted to LDIF first. Migration makes the simplifying assumption that the database LDIF file is in the old db directory and has the name of <old backend name>.ldif e.g. userRoot.ldif 3) Cross platform replication migration doesn't preserve the state, so the changelog nor other associated state information can be migrated. I rewrote the old migration script to use the FileConn - this theoretically will allow us to support migration using an LDAP::Conn as well. I had to make some fixes to FileConn, primarily to support the root DSE. Platforms tested: RHEL4 Flag Day: no Doc impact: Yes.
(In reply to comment #20) > Created an attachment (id=158885) [edit] > cross platform - adminserver Checking in adminserver/admserv/newinst/src/migrate-ds-admin.pl.in; /cvs/dirsec/adminserver/admserv/newinst/src/migrate-ds-admin.pl.in,v <-- migrate-ds-admin.pl.in new revision: 1.3; previous revision: 1.2 done
(In reply to comment #21) > Your new implementation looks good to me. > > Could you post how to run the migration script? I'd love to try the cross > platform migration. Thanks!! http://directory.fedoraproject.org/wiki/DS_Admin_Migration#Remote_Source_to_Local_Target In my testing I did an NFS mount of a RHEL4 i386 /opt/fedora-ds on a FC6 x86_64 /mnt/opt/fedora-ds. Then I ran migrate-ds-admin.pl like this: migrate-ds-admin.pl -o /mnt/opt/fedora-ds -a /opt/fedora-ds -x ... args ... If you use migrate-ds-admin.pl --help it will tell you the other required args. Then run migrate-ds-admin.pl with the -o
Created attachment 159352 [details] cvs diff Makefile.am Let's get rid of old migration scripts. After the above update, these are the migration related files: $ du -a | egrep migrate 8 ./share/fedora-ds/data/asmigrate.ldif.tmpl 8 ./share/fedora-ds/properties/migrate-ds-admin.res 8 ./share/fedora-ds/properties/migrate-ds.res 12 ./share/fedora-ds/inf/asmigrate.map 12 ./sbin/migrate-ds-admin.pl 8 ./bin/migratecred 32 ./bin/migrateTo11 8 ./bin/migrate-ds.pl 16 ./bin/migratecred-bin
Verified that the new migration code is being used. The functionality is covered by the migration tests. Marking as VERIFIED.