Tuesday, March 8, 2011

Simple Drizzle Replication Example Using the Slave Plugin

In this blog post, I thought that I'd cover, in a bit more detail, setting up a simple replication setup between two drizzle servers using the new replication slave plugin. If you've used MySQL replication before, you should find some of the concepts very similar. I'll only cover the simplest of examples (single master, single slave) and also explain how to provision a new slave into an existing setup.

So you've downloaded the latest and greatest version of Drizzle and want to setup replication. Where do you start? The very first thing to do is to make certain that both master and slave share the same version of Drizzle to avoid any potential incompatibility issues. Then you setup your master.

Master Setup

Setting up the master is the easiest step. The only thing you really have to do here is to make sure that the master Drizzle database server is started with the --innodb.replication-log option. Something along the lines of:

    master> sbin/drizzled --datadir=$PWD/var --innodb.replication-log &


For more complex setups, you'll also want to consider using the --server-id option, but the default for that is fine for this example. That option is documented here.

With the master running, you can optionally now create a backup of any databases to be imported on the new slave. You would use drizzledump to make such a backup. In this example, I'm keeping it simple and assuming that we are starting with a fresh database with no data.

Slave Setup

Now that you have the master running, you can setup your slave. Starting the slave is almost as simple as starting the master. You will need to use two options to the Drizzle database server on the slave: --plugin-add=slave and --slave.config-file.

    slave> sbin/drizzled --datadir=$PWD/var \
                                    --plugin-add=slave \
                                    --slave.config-file=/tmp/slave.cfg &


These options tell the server to load the slave plugin, and then tells the slave plugin where to find the slave host configuration file. This configuration file has options to specify the master host and a few options to control how the slave operates. You can read more about the available configuration options in the replication slave plugin documentation. Below is a simple example:

    master-host = kodiak
    master-port = 3306
    master-user = kodiak_slave
    master-pass = my_password
    io-thread-sleep = 10
    applier-thread-sleep = 10

Most of these options have sensible defaults, and it should be pretty obvious what most of them are for. The plugin documentation referenced above describes them in more complete detail if you need more information.

So once you start the slave as described above, it will immediately connect to the master host specified in the configuration file and begin pulling events from the InnoDB-based transaction log. By default, a freshly provisioned slave will begin pulling from the beginning of this transaction log. Once all replication messages have been pulled from the master and stored locally on the slave host, the IO thread will sleep and periodically awaken to check for more messages. That's all fine and dandy for your first slave machine in a brand new replication setup, but how do you insert another slave host into an already existing replication architecture? I'm glad you asked!

Provisioning a New Slave Host

We've recently made some changes that makes provisioning a new slave host very easy.

So, the basic formula for creating a new slave host for an existing replication setup is:

  1. Make a backup of the master databases.
  2. Record the state of the master transaction log at the point the backup was made.
  3. Restore the backup on the new slave machine.
  4. Start the new slave and tell it to begin reading the transaction log from the point recorded in #2.
Steps #1 and #2 are covered with the drizzledump client program. If you use the --single-transaction option to drizzledump, it will place a comment near the beginning of the dump output with the InnoDB transaction log metadata. For example:

    master> drizzledump --all-databases --single-transaction > master.backup
    master> head -1 master.backup
    -- SYS_REPLICATION_LOG: COMMIT_ID = 33426, ID = 35074

The SYS_REPLICATION_LOG line gives us the replication log metadata we need when we start a new slave. It has two pieces of information:
  • COMMIT_ID - This value is the commit sequence number recorded for the most recently executed transaction stored in the transaction log. We can use this value to determine proper commit order within the log. The unique transaction ID cannot be used since that value is assigned when the transaction is started, not when it is committed.
  • ID - This is the unique transaction identifier associated with the most recently executed transaction stored in the transaction log.
With this step done, we can now do steps #3 and #4 to start the new slave. First, you must start the slave WITHOUT the replication slave plugin enabled. We don't want it reading from the master until we've imported the backup. So start it without the plugin enabled, import your backup, then shutdown the server:

    slave> sbin/drizzled --datadir=$PWD/var &
    slave> drizzle < master.backup
    slave> drizzle --shutdown

Now that the backup is imported, we can restart the slave with the replication slave plugin enabled and use a new option, --slave.max-commit-id, to force the slave to begin reading the master's transaction log at the proper location:

    slave> sbin/drizzled --datadir=$PWD/var \
                                    --plugin-add=slave \
                                    --slave.config-file=/tmp/slave.cfg \
                                    --slave.max-commit-id=33426 &


We give the --slave.max-commit-id the value from the comment in the master dump file which defines the maximum COMMIT_ID value (the latest transaction) represented by the slave's contents.

Conclusion

So that's all there is to it. I hope you find this example useful and it encourages you to begin trying out Drizzle replication. Be sure to report any bugs and enhancements you'd like to see to our bug system. And don't forget that we have Drizzle Developer Day after the MySQL User Conference this year on April 15th. Be sure to sign up for that and come chat with us face-to-face about replication and anything else Drizzle related.  :)

Happy Replicating.

3 comments:

  1. Are there contention issues with many connections trying to write to the replication log? What testing has been done with the current approach?

    ReplyDelete
  2. Hi Rob,

    Our tests so far haven't identified any heavy impacting contention issues. Of course we can't say that there is NO impact. Using any transaction log is always going to affect performance, but the InnoDB replication log has so far turned out to perform better than the file based log. See: http://www.flamingspork.com/blog/2010/12/06/replication-log-inside-innodb/

    Our QA guy, Patrick, has run many multi-concurrency tests with the log enabled using the randgen tester. We also run the standard sysbench suite of tests. I'll see if I can get Patrick to post a comment with his observations of it and more about how he does the testing.

    ReplyDelete
  3. Rob,

    Hi. I'm going to crib myself, but will give some love to the original source: Andrew Hutchings puts out a weekly update on his blog: http://www.linuxjedi.co.uk/?p=102

    Hope this info helps, if you have any further questions, I'd be more than happy to discuss them. It should also be noted that we have just updated our test-runner to also create tests for our standard test suite. This will enable us to do finer-grained testing of things like the REPLICATE=False option for tables, etc.

    From Andrew's blog:
    “In our testing, we create a master-slave setup, then run the transaction log tests that we have been using since we first started beating on the trx log.

    There are a variety of different grammars that produce transactions (autocommit=off) and single queries (with autocommit=on). The various grammars produce different levels of valid queries – some make more valid queries and produce more deadlocks in multi-threaded testing, others create invalid queries to test how the log handles bad input. We test with simple scenarios like 100 cycles and 1 thread, then move on to more complex tests like 10000 cycles and 10 threads as well as setup 1 million cycle tests to stress the server long-term.

    There are still bugs to be found, but we can say with confidence that basic replication, even in highly concurrent / high stress (lots of deadlocks, rollbacks, and good commits) scenarios is working well. Data is replicated correctly and that is HUGE!

    We’ll now be moving onto other tests like different configurations (master + slave restarts and crashes, adding a slave to a populated master, KILLing queries, etc).”

    ReplyDelete