Thursday, December 22, 2011

2011 Year In Review

Wow, what an insane year. Looking back on it, a lot of stuff has happened this year for me. Let's look back:
The new job has kept me unbelievably busy, so I haven't had the time to contribute more to Drizzle, other than a bug fix here or there. That makes me a sad panda, but priorities have changed. Perhaps next year will offer more opportunity.

I don't know what 2012 has in store for me, but hopefully it is just as exciting. I hope everyone has a Happy New Year.

Wednesday, December 21, 2011

Going to Percona Live in DC

I'm looking forward to attending Percona Live in DC next month. If you're on the east coast of the US and you use MySQL in any capacity, you need to be there. Who's coming?

Friday, April 15, 2011

MySQL Users Conference Slides

Slides for my talk at the 2011 MySQL Users Conference are now up:

All in all, the talk went pretty well. There was quite a bit of interest in the subject. After attending my talk, Jay Janssen of Yahoo! put up a good blog post about some replication features that he would like to see in future versions. That is exactly the type of feedback that we want. Open source projects should be community driven. We want to implement the features that YOU want.

Saturday, March 19, 2011

Multi-Master Support in Drizzle Replication

So Brian asked me the other day what it would take to support multiple masters in our new Drizzle slave plugin. Not master to master replication, but multiple masters sending replication events to a single slave that simply ignores any conflicts and just chugs along. I told him I didn't know, but considering how simple the code is, it probably wouldn't take much.

To get a better understanding of what exactly would be involved in supporting multiple masters, I decided to just start hacking it up. I did this mainly to get a sense of what would need to be changed, since my original design didn't allow for this at all. (Shortsightedness on my part I suppose.)

So I have a beta version of my results available in this Launchpad branch:
From my simple tests, it seems to work. I'm not real happy with the code (like I said, this was a hack), but functionality is there. I'm not promising this will go into Drizzle trunk just yet. I would like to make some improvements on it, and I'd really like to get some feedback from people on it.

To use it, you'll first need to create a modified slave configuration file. Here is a sample one:

master-host =
master-port = 3306
master-user = user1
master-pass = password

master-host =
master-port = 3306
master-user = user2
master-pass = password
Currently, a total of 10 masters are supported. This was an arbitrary number. It was simplest to just predetermine a set number of masters due to some complications with config file parsing which I wasn't prepared to solve (this is one of the things I want to see fixed). One IO thread per master will be started, though we still use a single applier thread for the time being.

You'll notice in the sample config a new option, ignore-errors. If this option is present, the slave ignores any errors from replication events received from the masters that it executes locally. I highly recommend you have this option enabled. Also note the addition of the [master1] and [master2] sections that define options for each master. You can go all the way to a [master10] section.

Nothing changes with how you start your slave or masters (see my post on setting up a simple replication example).

Give it a try and let me know how it works for you. Again, this is bleeding edge stuff (does any other database support this?  :) ), so be prepared for bugs.

Thursday, March 17, 2011

Installing Drizzle from Source on OS X

Installing Drizzle on OS X 10.6 is pretty simple. We have a page on our wiki that has the basic steps, but I thought that I'd detail what I do on my Macs in the hope that it may make someone's life easier. Note that we don't build on any 10.5 machines, and I don't use that version anymore, so YMMV with these instructions. Also, these instructions assume that you have the Xcode package already installed. I have Xcode 4 installed, but these instructions should work with Xcode 3, too. If they don't, let me know.

I used to use MacPorts on my Macs to install the necessary libraries that are needed by Drizzle. I've recently dumped that because I didn't like all of the extra stuff that was installed (do you really need to install a separate Perl installation?). And a recent b0rk of their Perl installations was the final straw.

It turns out that all you really need are just a few extra packages to build Drizzle on your Mac. Here are the packages that I currently have installed on my machines:

The first three (autoconf, automake, and libtool) aren't strictly necessary. I install those because I want newer versions of those tools than what OS X provides by default. It makes building a little bit nicer (the output is much cleaner). The last three are what you really need.

Each package has its own instructions for how to compile and install. I use the default installation path (/usr/local) for each. Basically, building and installing for each is simply:
  • ./configure
  • make
  • sudo make install
The Boost package is the lone exception:
  • ./
  • ./bjam
  • sudo ./bjam install
If you install the libtool package, there is one additional step you should do. That package installs the binaries libtool and libtoolize into /usr/local/bin. I rename these to glibtool and glibtoolize, respectively. The Drizzle build system looks for these program names.

The last thing I do is to make sure that /usr/local/bin is in my path. So in $HOME/.bash_profile, I have this line:
export PATH=/usr/local/bin:$PATH
With all that in place, to build Drizzle is just:

  • ./config/
  • ./configure
No need to add any extra options to configure to find libraries.

Wednesday, March 9, 2011

Simplest Test Replication Setup. Ever.

> cd tests
> ./dbqp --mode=randgen --start-and-exit --suite=slave_plugin


Tuesday, March 8, 2011

Simple Drizzle Replication Example Using the Slave Plugin

In this blog post, I thought that I'd cover, in a bit more detail, setting up a simple replication setup between two drizzle servers using the new replication slave plugin. If you've used MySQL replication before, you should find some of the concepts very similar. I'll only cover the simplest of examples (single master, single slave) and also explain how to provision a new slave into an existing setup.

So you've downloaded the latest and greatest version of Drizzle and want to setup replication. Where do you start? The very first thing to do is to make certain that both master and slave share the same version of Drizzle to avoid any potential incompatibility issues. Then you setup your master.

Master Setup

Setting up the master is the easiest step. The only thing you really have to do here is to make sure that the master Drizzle database server is started with the --innodb.replication-log option. Something along the lines of:

    master> sbin/drizzled --datadir=$PWD/var --innodb.replication-log &

For more complex setups, you'll also want to consider using the --server-id option, but the default for that is fine for this example. That option is documented here.

With the master running, you can optionally now create a backup of any databases to be imported on the new slave. You would use drizzledump to make such a backup. In this example, I'm keeping it simple and assuming that we are starting with a fresh database with no data.

Slave Setup

Now that you have the master running, you can setup your slave. Starting the slave is almost as simple as starting the master. You will need to use two options to the Drizzle database server on the slave: --plugin-add=slave and --slave.config-file.

    slave> sbin/drizzled --datadir=$PWD/var \
                                    --plugin-add=slave \
                                    --slave.config-file=/tmp/slave.cfg &

These options tell the server to load the slave plugin, and then tells the slave plugin where to find the slave host configuration file. This configuration file has options to specify the master host and a few options to control how the slave operates. You can read more about the available configuration options in the replication slave plugin documentation. Below is a simple example:

    master-host = kodiak
    master-port = 3306
    master-user = kodiak_slave
    master-pass = my_password
    io-thread-sleep = 10
    applier-thread-sleep = 10

Most of these options have sensible defaults, and it should be pretty obvious what most of them are for. The plugin documentation referenced above describes them in more complete detail if you need more information.

So once you start the slave as described above, it will immediately connect to the master host specified in the configuration file and begin pulling events from the InnoDB-based transaction log. By default, a freshly provisioned slave will begin pulling from the beginning of this transaction log. Once all replication messages have been pulled from the master and stored locally on the slave host, the IO thread will sleep and periodically awaken to check for more messages. That's all fine and dandy for your first slave machine in a brand new replication setup, but how do you insert another slave host into an already existing replication architecture? I'm glad you asked!

Provisioning a New Slave Host

We've recently made some changes that makes provisioning a new slave host very easy.

So, the basic formula for creating a new slave host for an existing replication setup is:

  1. Make a backup of the master databases.
  2. Record the state of the master transaction log at the point the backup was made.
  3. Restore the backup on the new slave machine.
  4. Start the new slave and tell it to begin reading the transaction log from the point recorded in #2.
Steps #1 and #2 are covered with the drizzledump client program. If you use the --single-transaction option to drizzledump, it will place a comment near the beginning of the dump output with the InnoDB transaction log metadata. For example:

    master> drizzledump --all-databases --single-transaction > master.backup
    master> head -1 master.backup
    -- SYS_REPLICATION_LOG: COMMIT_ID = 33426, ID = 35074

The SYS_REPLICATION_LOG line gives us the replication log metadata we need when we start a new slave. It has two pieces of information:
  • COMMIT_ID - This value is the commit sequence number recorded for the most recently executed transaction stored in the transaction log. We can use this value to determine proper commit order within the log. The unique transaction ID cannot be used since that value is assigned when the transaction is started, not when it is committed.
  • ID - This is the unique transaction identifier associated with the most recently executed transaction stored in the transaction log.
With this step done, we can now do steps #3 and #4 to start the new slave. First, you must start the slave WITHOUT the replication slave plugin enabled. We don't want it reading from the master until we've imported the backup. So start it without the plugin enabled, import your backup, then shutdown the server:

    slave> sbin/drizzled --datadir=$PWD/var &
    slave> drizzle < master.backup
    slave> drizzle --shutdown

Now that the backup is imported, we can restart the slave with the replication slave plugin enabled and use a new option, --slave.max-commit-id, to force the slave to begin reading the master's transaction log at the proper location:

    slave> sbin/drizzled --datadir=$PWD/var \
                                    --plugin-add=slave \
                                    --slave.config-file=/tmp/slave.cfg \
                                    --slave.max-commit-id=33426 &

We give the --slave.max-commit-id the value from the comment in the master dump file which defines the maximum COMMIT_ID value (the latest transaction) represented by the slave's contents.


So that's all there is to it. I hope you find this example useful and it encourages you to begin trying out Drizzle replication. Be sure to report any bugs and enhancements you'd like to see to our bug system. And don't forget that we have Drizzle Developer Day after the MySQL User Conference this year on April 15th. Be sure to sign up for that and come chat with us face-to-face about replication and anything else Drizzle related.  :)

Happy Replicating.

Friday, February 25, 2011

Drizzle's Slave Plugin and the Execute class

Well, you may have already read Andrew's and Patrick's blog posts on it, but if you haven't, we now have a Drizzle plugin that implements native replication between Drizzle servers.

So, the initial plan was to support work by the community to have Tungsten Replicator ready as a replication solution for our first GA release. However, this didn't look like it would get completed in time for GA, so the decision was made to make a strong push to get a native solution ready. So after a big group effort, and many, many late nights and long weekends, we are pleased to say that we have delivered the initial version of a native replication solution.

Now, does this version have all of the features that we want? No. It will certainly be added to, though, as we progress. Keep in mind that replication was totally ripped out of Drizzle, so it will take some time to build on its functionality.

But what makes this release so exciting if it doesn't have all the features you want? Well, thanks to the solid backbone of a well tested transaction log and a well designed and flexible replication architecture (big shout out to Jay), it took only 2 weeks (admittedly, 2 very hard weeks) and 1900 lines of C++ code to implement this plugin. With only 1900 LOC, it is extremely simple to understand and expand on and improve. A kid just out of college could understand this.  :)  So if you want to add a cool feature to the plugin, it shouldn't take too much effort to add your contribution. How easily could you do this in any other database?

Patrick has already blogged about how easy it is to begin using the replication slave plugin. It should hit our main trunk later today, so feel free to give it a try. In later posts, I'll attempt to cover some of the implementation details, but there is one REALLY cool feature in Drizzle that is used by the slave plugin that I want to quickly cover...

The slave plugin has to execute SQL statements on the local server to apply the changes from the master. In the MySQL code, executing SQL locally isn't easy to do. Logically, you would think that it should be easy for a database kernel to execute SQL on itself. Alas, this was not the case.

Thanks to some changes by Brian Aker to the Drizzle kernel, the slave plugin can very easily do this by using the drizzled::Execute class. This class has a run() method that takes a std::string with the SQL to execute. Here is a simple example from the slave plugin code:

Execute execute(*(_session.get()), true);
string sql("SELECT `trx_id` FROM `sys_replication`.`queue`"
               " WHERE `commit_order` IS NOT NULL ORDER BY `commit_order` ASC");
sql::ResultSet result_set(1);, result_set);

It's as simple as that. Execute wraps whatever SQL you give it (it can be multiple statements) in a transaction and executes it. The ResultSet object will contain your results and any exceptions that may have been thrown (we are trying to match the JDBC API here).  Some of this API is still a work-in-progress, but it has proven to be extremely useful and solid so far. If you are interested, you can look at the Execute class source in the drizzled/execute.h and drizzled/ files.

Thursday, January 20, 2011

Transaction Log Quality and the Importance of QA

If you read my post a few months back that gave a progress report on Drizzle Replication, you know that our major focus lately has been the quality of the transaction log. It has to be solid before other things can be built upon it. And fixing bugs in it usually means fixing bugs in the core replication message stream, so other replication plugins will benefit as well.

Well, Patrick Crews, the Drizzle QA Superman, has sent this post to the drizzle-discuss mailing list:

It looks like we finally have a high-quality transaction log to build on! With that solid foundation in place, we can now work on adding more functionality.

This, of course, couldn't have happened with some super high quality QA work. Any developers out there know that finding your own flaws in code you maintain is difficult to do. You're too close to the code, can't see the forest for the trees. Sure, you'll find the obvious ones whenever you get a core dump, or when some unit or regression test fails (you do have those, right?). It's the super-hard-to-find-make-you-wish-you-chose-another-career bugs that are the ones that are usually missed. Sometimes because once we code it, we want to move on to something else as quickly as possible, so we'd like to assume that it works. The reality is usually different...

This is where QA comes in. Their job is to pull the reins in on you, show you where you messed up, and make sure it gets fixed. It's usually a thankless job, but it's also one that is absolutely necessary. Patrick continually amazes me at what he does and how thorough he is. How we got lucky enough to snag him for the Drizzle team is beyond me, but I'm glad we did. Without his QA expertise, we'd certainly be quite lagging in the quality of our product. (He would try to give us developers the credit for fixing the bugs, but we are the ones introducing the bugs to begin with!)

So never underestimate the value of your QA team. You can't make a quality product without them. And have you thanked your QA team lately?  (Thanks Patrick!)

Wednesday, January 12, 2011

Change in Transaction Protobuf Message Segmenting

If you read my post on Drizzle transaction message limits, you know that it is possible to have multiple Transaction protobuf messages (segments) for a single database transaction. This was necessary to keep the Google protobuf messages from growing too large (there is a maximum limit on message size).

Before my most recent change, you would have to parse each Statement sub-message contained within the enclosing Transaction message to see if the Transaction was split up into multiple messages (they would be linked together by sharing the same transaction ID). This was kind of a pain, but since the segment information was only contained in the Statement, this was the only way to do it.

As of Bazaar revision number 2076 of the Drizzle trunk, we now have segment information stored in the Transaction message in addition to the Statement message. We added the values segment_id and end_segment to the Transaction message definition. These are just like the identically named values in the Statement message definition. From the drizzled/message/transaction.proto definition file:
message Transaction
  required TransactionContext transaction_context = 1;
  repeated Statement statement = 2;
  optional Event event = 3;

   * A single transaction in the database can possibly be represented with
   * multiple protobuf Transaction messages if the message grows too large.
   * This can happen if you have a bulk transaction, or a single statement
   * affecting a very large number of rows, or just a large transaction with
   * many statements/changes.
   * For the first two examples, it is likely that the Statement sub-message
   * itself will get segmented, causing another Transaction message to be
   * created to hold the rest of the Statement's row changes. In these cases,
   * it is enough to look at the segment information stored in the Statement
   * message.
   * For the last example, the Statement sub-messages may or may not be
   * segmented, but we could still need to split the Statements up into
   * multiple Transaction messages to keep the Transaction message size from
   * growing too large. In this case, the segment information in the Statement
   * submessages is not helpful if the Statement isn't segmented. We need this
   * information in the Transaction message itself.
   * These values should be set appropriately whether or not the Statement
   * sub-messages are segmented.
  optional uint32 segment_id = 4; /* Segment number of the Transaction msg */
  optional bool end_segment = 5;  /* FALSE if Transaction msg is split into multiples */
So other than making it easier to check to see if a Transaction is segmented, why add these new values?

Well, it turns out having the segment information only in the Statement doesn't allow us to segment large Transaction messages if none of the Statement sub-messages are themselves segmented. This is documented in this bug report.