Database Migrations Using DBXExpress Metadata

by John 12/19/2007 5:56:00 AM
Overview

I recently spent some time looking at Ruby On Rails. One of the many things about Rails that caught my interest was the way that schema changes to databases are performed using Ruby. I think it was this post by Steve Shaughnessy about the new dbExpress 4.0 MetaData where it occurred to me perhaps I could use the DBExpress MetaData to do my something similar in Delphi.

I had several goals in mind:-
  • Produce a repeatable series of steps to upgrade database schemas.
  • Stay away from having to remember SQL syntax.
  • Automate the upgrade process
  • Target multiple db servers


Before presenting my solution, I would like to thank Leonel Togniolli, CodeGear R&D for helping me with how to go about calling the DBExpress MetaData API.

Project Layout

I have bundled everything together in a projectgroup which contains five projects.
Moshine.Migration.Framework.dll - Contains most of the core code
Moshine.Migration.Framework.Tests.dll - Contains some unit tests
Moshine.Migration.Test.dll - is an example of a database migration
Moshine.MigrationRunner - Is a VCL.Net application to execute migrations
Moshine.MSBuildMigration.dll - Contains a collection of MSBuild tasks that can be used from the IDE or standalone from the RAD Studio Command Line

Example

A database schema generation or migration is composed of multiple steps. A step is the equivalent to a Delphi class.  A step can perform such actions as creating tables, adding columns, adding indexes, adding foreign keys, and executing sql. Sql could be used to add stored procedures and insert domain data into tables.

So if you take a look at Moshine.MigrationTest.dll, this contains seven units which each contain a Delphi Class. So after running the migration the database would be at version 7. For example this is the declaration for the first step

[Migration(1)]
TBookMarksFirstMigration=class(TMigration)
public
    procedure Up();override;
    procedure Down(); override;
end;

Migrations can go backwards and forwards so I have Up and Down methods. The idea is that to create a step you create a new class derived from TMigration and decorate with an attribute which describes which step this is. Migrations must start at 1 and be sequential.

If you take a close look at the Up method in TBookMarksFirstMigration, its creating a table called Bookmarks using the DBXExpress metadata API. After table creation the column ID is set as the primary key.

The corresponding Down method for this class would drop the table.

TMigration


All migration steps must be derived from TMigration. This base class is in Moshine.Migration.Framework in Moshine.Migration.Framework.MigrationBase.pas. I have provided a number of protected methods that Migration developers can call

procedure AddColumn(tableName:string;columnName:string;dataType:integer);
procedure RemoveColumn(tableName,columnName:string); procedure AddIndex(tableName, columnName: string; indexName: String);
procedure RemoveIndex(tableName, columnName: string; indexName: String);
procedure AddPrimaryKey(columnName,TableName:string); procedure AddForeignKey(columnName,tableName,primaryTableName,primaryColumn:string);
procedure RemoveForeignKey(columnName,tableName,primaryTableName,primaryColumn:string);

There are also 2 properties

property Provider:TAdoMetaDataProvider read FProvider write FProvider;
property Connection:DbConnection read FConnection write FConnection;

Connection is a database connection but Provider is the most important. Its a TAdoMetaDataProvider object and is essentially the DBXExpress metadata. The protected methods I supply are basically calling methods in the metadata API. Most of the time I would expect that you would call the helper methods but you can use use the Provider or Connection to execute raw sql.

MSBuild Setup


I have been using MSBuild target files to define targets that call the various tasks. The Moshine.MigrationTest.dll project contains one such targets file called TesTargets.Target

The MSBuild tasks are driven by two properties, a connection and assembly that contains your migration.

Moshine.MigrationTest is a sample migration. The project contains an app.config

This contains your connection specifications. For example mine contains

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <connectionStrings>
        <add name="SomeConnectionString" connectionString="database=C:\develop\moshine\Migration\Moshine.MigrationRunner\App_Data\migration.jds;user=SYSDBA;password=masterkey;host=localhost;protocol=TCP;create=true" providerName="Borland.Data.BlackfishSQL.RemoteClient"/>
    </connectionStrings>
</configuration>

a single Blackfish connection called "SomeConnectionString"

The other item that must be added to a project are the MSBuild tasks. I have added TestTarget.targets to Moshine.MigrationTests.dll

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <UsingTask TaskName="Moshine.Migration.MSBuild.TMigrateDownTask"
            AssemblyFile="C:\develop\moshine\Migration\Moshine.MSBuildMigration\bin\Moshine.MSBuildMigration.dll" />
    <UsingTask TaskName="Moshine.Migration.MSBuild.TMigrateUpTask"
            AssemblyFile="C:\develop\moshine\Migration\Moshine.MSBuildMigration\bin\Moshine.MSBuildMigration.dll" />
    <UsingTask TaskName="Moshine.Migration.MSBuild.TMigrationTask"
            AssemblyFile="C:\develop\moshine\Migration\Moshine.MSBuildMigration\bin\Moshine.MSBuildMigration.dll" />
    <UsingTask TaskName="Moshine.Migration.MSBuild.TMigrationSchemaVersionTask"
            AssemblyFile="C:\develop\moshine\Migration\Moshine.MSBuildMigration\bin\Moshine.MSBuildMigration.dll" />

    <PropertyGroup>
        <MigrationAssembly>C:\develop\moshine\Migration\Moshine.MigrationTest\bin\Moshine.MigrationTest.dll</MigrationAssembly>
        <ConnectionName>SomeConnectionString</ConnectionName>
    </PropertyGroup>

    <Target Name="Migration">
        <TMigrationTask  MigrationAssembly="$(MigrationAssembly)" ConnectionName="$(ConnectionName)" />
    </Target>

    <Target Name="Up">
        <TMigrateUpTask  MigrationAssembly="$(MigrationAssembly)" ConnectionName="$(ConnectionName)" />
    </Target>

    <Target Name="Down">
        <TMigrateDownTask  MigrationAssembly="$(MigrationAssembly)" ConnectionName="$(ConnectionName)" />
    </Target>

    <Target Name="SchemaVersion">
        <TMigrationSchemaVersionTask  MigrationAssembly="$(MigrationAssembly)" ConnectionName="$(ConnectionName)" />
    </Target>

</Project>

The contents is above. I have setup two properties and four targets. The two properties define the Connection to use and the assembly that contains the migrations that will be run by the MSBuild targets.

You should now be able to open a RAD Studio Command Prompt. It you change to the Moshine.MSBuildMigration directory you add run the migrations. For example

msbuild TestTarget.targets -t:up
msbuild TestTarget.targets -t:down
msbuild TestTarget.targets -t:SchemaVersion
msbuild TestTarget.targets -t:Migration

You can also run the migrations from the IDE project menu as well. In the project manager select TestTarget.targets in the Moshine.MigrationTest project, right click and select the targets menu item. Listed as submenu items of the targets menu will be the same four targets as above, as can be seen below.



The SchemaVersion targets displays the database schema version.
Migration executes all steps defined in the assembly.
Up and down step up and down the migration.

So say I have my database at database schema version 4 and I execute the migration target. Steps 5, 6 and 7 will be performed.


Running a Migration



The Migration Framework looks for a database table called Migration_Schema to determine the database version and then looks at the steps provided in the metadata assembly to work out the steps to run.

If you issue the Migration target and the number of steps doesn't match the schema version the appropriate steps will be run.

You can also use the Up and down targets step up and down the migrations. MigrationSchemaVersion outputs the database schema version.

Moshine.MigrationRunner

I have also provide a VCL.Net application which duplicates the MSBuild functionality. This is configured with app.config. An example of which can be seen below.

Note(s)

To be able to build the projectgroup you will need the December 2007 Rad Studio Update installed.

The Moshine.MSBuildMigration.dll assembly relies on Moshine.Migration.Framework.dll, so make sure that both assemblies are located wherever you set the using task.

i.e

<UsingTask TaskName="Moshine.Migration.MSBuild.TMigrateDownTask"
            AssemblyFile="C:\develop\moshine\Migration\Moshine.MSBuildMigration\bin\Moshine.MSBuildMigration.dll" />


In theory you should be able to perform migrations on any database that CodeGear provides MetaData for. I have done some testing with AdoDbx against MSSQL, Interbase and MySql. Most of the time I have been using the Blackfish native providers.

If this interests you, I have placed the code in codecentral at this location. Absolutely any feedback is welcome.

Currently rated 3.3 by 6 people

  • Currently 3.333333/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

CodeGear | VCL.Net | Database | msbuild | Blackfish | dbx4 | Delphi | migration

Powered by BlogEngine.NET 1.3.0.0
Theme by Mads Kristensen

About the author

Name of author John Moshakis
I'm a software developer living in Toronto..

E-mail me Send mail

Calendar

<<  September 2017  >>
MoTuWeThFrSaSu
28293031123
45678910
11121314151617
18192021222324
2526272829301
2345678

View posts in large calendar

Pages

    Recent posts

    Recent comments

    Authors

    Tags

    Don't show

      Disclaimer

      The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway.

      © Copyright 2017

      Sign in