Shrews News

2011 Year In Review

2011-12-22T11:04:00.000-05:00

Wow, what an insane year. Looking back on it, a lot of stuff has happened this year for me. Let's look back:

With the help of Patrick Crews and Andrew Hutchings, and building on the work of Jay Pipes, I managed to implement a pretty solid replication solution in Drizzle and made the slave plugin available.
Implemented multi-source replication (more accurate name than multi-master) in Drizzle.
I ventured into new territory by giving my first-ever presentation at the MySQL User Conference in Santa Clara, CA.
Rackspace made life interesting and forced me find another position. I joined a local startup company as their first real hire. Two of the founders are former coworkers and current friends of mine, and the timing worked out well for all parties.
For my new job, I had to learn Python, GIT, and how to work with AWS systems. Python rocks. I feel like I can fly now.
Lost 20 lbs! Woohoo!

The new job has kept me unbelievably busy, so I haven't had the time to contribute more to Drizzle, other than a bug fix here or there. That makes me a sad panda, but priorities have changed. Perhaps next year will offer more opportunity.

I don't know what 2012 has in store for me, but hopefully it is just as exciting. I hope everyone has a Happy New Year.

Going to Percona Live in DC

2011-12-21T08:45:00.000-05:00

I'm looking forward to attending Percona Live in DC next month. If you're on the east coast of the US and you use MySQL in any capacity, you need to be there. Who's coming?

MySQL Users Conference Slides

2011-04-15T15:52:00.000-04:00

Slides for my talk at the 2011 MySQL Users Conference are now up:

http://en.oreilly.com/mysql2011/public/schedule/detail/17410

All in all, the talk went pretty well. There was quite a bit of interest in the subject. After attending my talk, Jay Janssen of Yahoo! put up a good blog post about some replication features that he would like to see in future versions. That is exactly the type of feedback that we want. Open source projects should be community driven. We want to implement the features that YOU want.

Multi-Master Support in Drizzle Replication

2011-03-19T16:58:00.000-04:00

So Brian asked me the other day what it would take to support multiple masters in our new Drizzle slave plugin. Not master to master replication, but multiple masters sending replication events to a single slave that simply ignores any conflicts and just chugs along. I told him I didn't know, but considering how simple the code is, it probably wouldn't take much.

To get a better understanding of what exactly would be involved in supporting multiple masters, I decided to just start hacking it up. I did this mainly to get a sense of what would need to be changed, since my original design didn't allow for this at all. (Shortsightedness on my part I suppose.)

So I have a beta version of my results available in this Launchpad branch:

lp:~dshrews/drizzle/beta-multi-master

From my simple tests, it seems to work. I'm not real happy with the code (like I said, this was a hack), but functionality is there. I'm not promising this will go into Drizzle trunk just yet. I would like to make some improvements on it, and I'd really like to get some feedback from people on it.

To use it, you'll first need to create a modified slave configuration file. Here is a sample one:

ignore-errors

[master1]
master-host = foo.my.domain
master-port = 3306
master-user = user1
master-pass = password

[master2]
master-host = bar.my.domain
master-port = 3306
master-user = user2
master-pass = password

Currently, a total of 10 masters are supported. This was an arbitrary number. It was simplest to just predetermine a set number of masters due to some complications with config file parsing which I wasn't prepared to solve (this is one of the things I want to see fixed). One IO thread per master will be started, though we still use a single applier thread for the time being.

You'll notice in the sample config a new option, ignore-errors. If this option is present, the slave ignores any errors from replication events received from the masters that it executes locally. I highly recommend you have this option enabled. Also note the addition of the [master1] and [master2] sections that define options for each master. You can go all the way to a [master10] section.

Nothing changes with how you start your slave or masters (see my post on setting up a simple replication example).

Give it a try and let me know how it works for you. Again, this is bleeding edge stuff (does any other database support this? :) ), so be prepared for bugs.

Installing Drizzle from Source on OS X

2011-03-17T09:07:00.000-04:00

Installing Drizzle on OS X 10.6 is pretty simple. We have a page on our wiki that has the basic steps, but I thought that I'd detail what I do on my Macs in the hope that it may make someone's life easier. Note that we don't build on any 10.5 machines, and I don't use that version anymore, so YMMV with these instructions. Also, these instructions assume that you have the Xcode package already installed. I have Xcode 4 installed, but these instructions should work with Xcode 3, too. If they don't, let me know.

I used to use MacPorts on my Macs to install the necessary libraries that are needed by Drizzle. I've recently dumped that because I didn't like all of the extra stuff that was installed (do you really need to install a separate Perl installation?). And a recent b0rk of their Perl installations was the final straw.

It turns out that all you really need are just a few extra packages to build Drizzle on your Mac. Here are the packages that I currently have installed on my machines:

The first three (autoconf, automake, and libtool) aren't strictly necessary. I install those because I want newer versions of those tools than what OS X provides by default. It makes building a little bit nicer (the output is much cleaner). The last three are what you really need.

Each package has its own instructions for how to compile and install. I use the default installation path (/usr/local) for each. Basically, building and installing for each is simply:

./configure
make
sudo make install

The Boost package is the lone exception:

./bootstrap.sh
./bjam
sudo ./bjam install

If you install the libtool package, there is one additional step you should do. That package installs the binaries libtool and libtoolize into /usr/local/bin. I rename these to glibtool and glibtoolize, respectively. The Drizzle build system looks for these program names.

The last thing I do is to make sure that /usr/local/bin is in my path. So in $HOME/.bash_profile, I have this line:

export PATH=/usr/local/bin:$PATH

With all that in place, to build Drizzle is just:

./config/autorun.sh
./configure

No need to add any extra options to configure to find libraries.

Simplest Test Replication Setup. Ever.

2011-03-09T11:21:00.000-05:00

> cd tests
> ./dbqp --mode=randgen --start-and-exit --suite=slave_plugin

BOOM. DONE. WINNING.

Simple Drizzle Replication Example Using the Slave Plugin

2011-03-08T08:06:00.001-05:00

In this blog post, I thought that I'd cover, in a bit more detail, setting up a simple replication setup between two drizzle servers using the new replication slave plugin. If you've used MySQL replication before, you should find some of the concepts very similar. I'll only cover the simplest of examples (single master, single slave) and also explain how to provision a new slave into an existing setup.

So you've downloaded the latest and greatest version of Drizzle and want to setup replication. Where do you start? The very first thing to do is to make certain that both master and slave share the same version of Drizzle to avoid any potential incompatibility issues. Then you setup your master.

Master Setup

Setting up the master is the easiest step. The only thing you really have to do here is to make sure that the master Drizzle database server is started with the --innodb.replication-log option. Something along the lines of:

   master> sbin/drizzled --datadir=$PWD/var --innodb.replication-log &

For more complex setups, you'll also want to consider using the --server-id option, but the default for that is fine for this example. That option is documented here.

With the master running, you can optionally now create a backup of any databases to be imported on the new slave. You would use drizzledump to make such a backup. In this example, I'm keeping it simple and assuming that we are starting with a fresh database with no data.

Slave Setup

Now that you have the master running, you can setup your slave. Starting the slave is almost as simple as starting the master. You will need to use two options to the Drizzle database server on the slave: --plugin-add=slave and --slave.config-file.

   slave> sbin/drizzled --datadir=$PWD/var \
   --plugin-add=slave \
   --slave.config-file=/tmp/slave.cfg &

These options tell the server to load the slave plugin, and then tells the slave plugin where to find the slave host configuration file. This configuration file has options to specify the master host and a few options to control how the slave operates. You can read more about the available configuration options in the replication slave plugin documentation. Below is a simple example:

   master-host = kodiak
   master-port = 3306
   master-user = kodiak_slave
   master-pass = my_password
   io-thread-sleep = 10
   applier-thread-sleep = 10

Most of these options have sensible defaults, and it should be pretty obvious what most of them are for. The plugin documentation referenced above describes them in more complete detail if you need more information.

So once you start the slave as described above, it will immediately connect to the master host specified in the configuration file and begin pulling events from the InnoDB-based transaction log. By default, a freshly provisioned slave will begin pulling from the beginning of this transaction log. Once all replication messages have been pulled from the master and stored locally on the slave host, the IO thread will sleep and periodically awaken to check for more messages. That's all fine and dandy for your first slave machine in a brand new replication setup, but how do you insert another slave host into an already existing replication architecture? I'm glad you asked!

Provisioning a New Slave Host

We've recently made some changes that makes provisioning a new slave host very easy.

So, the basic formula for creating a new slave host for an existing replication setup is:

Make a backup of the master databases.
Record the state of the master transaction log at the point the backup was made.
Restore the backup on the new slave machine.
Start the new slave and tell it to begin reading the transaction log from the point recorded in #2.

Steps #1 and #2 are covered with the drizzledump client program. If you use the --single-transaction option to drizzledump, it will place a comment near the beginning of the dump output with the InnoDB transaction log metadata. For example:

master> drizzledump --all-databases --single-transaction > master.backup

master> head -1 master.backup

-- SYS_REPLICATION_LOG: COMMIT_ID = 33426, ID = 35074

The SYS_REPLICATION_LOG line gives us the replication log metadata we need when we start a new slave. It has two pieces of information:

COMMIT_ID - This value is the commit sequence number recorded for the most recently executed transaction stored in the transaction log. We can use this value to determine proper commit order within the log. The unique transaction ID cannot be used since that value is assigned when the transaction is started, not when it is committed.
ID - This is the unique transaction identifier associated with the most recently executed transaction stored in the transaction log.

With this step done, we can now do steps #3 and #4 to start the new slave. First, you must start the slave WITHOUT the replication slave plugin enabled. We don't want it reading from the master until we've imported the backup. So start it without the plugin enabled, import your backup, then shutdown the server:

slave> sbin/drizzled --datadir=$PWD/var &

slave> drizzle < master.backup

slave> drizzle --shutdown

Now that the backup is imported, we can restart the slave with the replication slave plugin enabled and use a new option, --slave.max-commit-id, to force the slave to begin reading the master's transaction log at the proper location:

slave> sbin/drizzled --datadir=$PWD/var \
--plugin-add=slave \

--slave.config-file=/tmp/slave.cfg \

--slave.max-commit-id=33426 &

We give the --slave.max-commit-id the value from the comment in the master dump file which defines the maximum COMMIT_ID value (the latest transaction) represented by the slave's contents.

Conclusion

So that's all there is to it. I hope you find this example useful and it encourages you to begin trying out Drizzle replication. Be sure to report any bugs and enhancements you'd like to see to our bug system. And don't forget that we have Drizzle Developer Day after the MySQL User Conference this year on April 15th. Be sure to sign up for that and come chat with us face-to-face about replication and anything else Drizzle related. :)

Happy Replicating.

Drizzle's Slave Plugin and the Execute class

2011-02-25T11:45:00.000-05:00

Well, you may have already read Andrew's and Patrick's blog posts on it, but if you haven't, we now have a Drizzle plugin that implements native replication between Drizzle servers.

So, the initial plan was to support work by the community to have Tungsten Replicator ready as a replication solution for our first GA release. However, this didn't look like it would get completed in time for GA, so the decision was made to make a strong push to get a native solution ready. So after a big group effort, and many, many late nights and long weekends, we are pleased to say that we have delivered the initial version of a native replication solution.

Now, does this version have all of the features that we want? No. It will certainly be added to, though, as we progress. Keep in mind that replication was totally ripped out of Drizzle, so it will take some time to build on its functionality.

But what makes this release so exciting if it doesn't have all the features you want? Well, thanks to the solid backbone of a well tested transaction log and a well designed and flexible replication architecture (big shout out to Jay), it took only 2 weeks (admittedly, 2 very hard weeks) and 1900 lines of C++ code to implement this plugin. With only 1900 LOC, it is extremely simple to understand and expand on and improve. A kid just out of college could understand this. :) So if you want to add a cool feature to the plugin, it shouldn't take too much effort to add your contribution. How easily could you do this in any other database?

Patrick has already blogged about how easy it is to begin using the replication slave plugin. It should hit our main trunk later today, so feel free to give it a try. In later posts, I'll attempt to cover some of the implementation details, but there is one REALLY cool feature in Drizzle that is used by the slave plugin that I want to quickly cover...

The slave plugin has to execute SQL statements on the local server to apply the changes from the master. In the MySQL code, executing SQL locally isn't easy to do. Logically, you would think that it should be easy for a database kernel to execute SQL on itself. Alas, this was not the case.

Thanks to some changes by Brian Aker to the Drizzle kernel, the slave plugin can very easily do this by using the drizzled::Execute class. This class has a run() method that takes a std::string with the SQL to execute. Here is a simple example from the slave plugin code:

Execute execute(*(_session.get()), true);

string sql("SELECT `trx_id` FROM `sys_replication`.`queue`"

" WHERE `commit_order` IS NOT NULL ORDER BY `commit_order` ASC");

sql::ResultSet result_set(1);

execute.run(sql, result_set);

It's as simple as that. Execute wraps whatever SQL you give it (it can be multiple statements) in a transaction and executes it. The ResultSet object will contain your results and any exceptions that may have been thrown (we are trying to match the JDBC API here). Some of this API is still a work-in-progress, but it has proven to be extremely useful and solid so far. If you are interested, you can look at the Execute class source in the drizzled/execute.h and drizzled/execute.cc files.

Transaction Log Quality and the Importance of QA

2011-01-20T13:55:00.000-05:00

If you read my post a few months back that gave a progress report on Drizzle Replication, you know that our major focus lately has been the quality of the transaction log. It has to be solid before other things can be built upon it. And fixing bugs in it usually means fixing bugs in the core replication message stream, so other replication plugins will benefit as well.

Well, Patrick Crews, the Drizzle QA Superman, has sent this post to the drizzle-discuss mailing list:

https://lists.launchpad.net/drizzle-discuss/msg07354.html

It looks like we finally have a high-quality transaction log to build on! With that solid foundation in place, we can now work on adding more functionality.

This, of course, couldn't have happened with some super high quality QA work. Any developers out there know that finding your own flaws in code you maintain is difficult to do. You're too close to the code, can't see the forest for the trees. Sure, you'll find the obvious ones whenever you get a core dump, or when some unit or regression test fails (you do have those, right?). It's the super-hard-to-find-make-you-wish-you-chose-another-career bugs that are the ones that are usually missed. Sometimes because once we code it, we want to move on to something else as quickly as possible, so we'd like to assume that it works. The reality is usually different...

This is where QA comes in. Their job is to pull the reins in on you, show you where you messed up, and make sure it gets fixed. It's usually a thankless job, but it's also one that is absolutely necessary. Patrick continually amazes me at what he does and how thorough he is. How we got lucky enough to snag him for the Drizzle team is beyond me, but I'm glad we did. Without his QA expertise, we'd certainly be quite lagging in the quality of our product. (He would try to give us developers the credit for fixing the bugs, but we are the ones introducing the bugs to begin with!)

So never underestimate the value of your QA team. You can't make a quality product without them. And have you thanked your QA team lately? (Thanks Patrick!)

Change in Transaction Protobuf Message Segmenting

2011-01-12T09:47:00.000-05:00

If you read my post on Drizzle transaction message limits, you know that it is possible to have multiple Transaction protobuf messages (segments) for a single database transaction. This was necessary to keep the Google protobuf messages from growing too large (there is a maximum limit on message size).

Before my most recent change, you would have to parse each Statement sub-message contained within the enclosing Transaction message to see if the Transaction was split up into multiple messages (they would be linked together by sharing the same transaction ID). This was kind of a pain, but since the segment information was only contained in the Statement, this was the only way to do it.

As of Bazaar revision number 2076 of the Drizzle trunk, we now have segment information stored in the Transaction message in addition to the Statement message. We added the values segment_id and end_segment to the Transaction message definition. These are just like the identically named values in the Statement message definition. From the drizzled/message/transaction.proto definition file:

message Transaction
{
  required TransactionContext transaction_context = 1;
  repeated Statement statement = 2;
  optional Event event = 3;

  /*
   * A single transaction in the database can possibly be represented with
   * multiple protobuf Transaction messages if the message grows too large.
   * This can happen if you have a bulk transaction, or a single statement
   * affecting a very large number of rows, or just a large transaction with
   * many statements/changes.
   *
   * For the first two examples, it is likely that the Statement sub-message
   * itself will get segmented, causing another Transaction message to be
   * created to hold the rest of the Statement's row changes. In these cases,
   * it is enough to look at the segment information stored in the Statement
   * message.
   *
   * For the last example, the Statement sub-messages may or may not be
   * segmented, but we could still need to split the Statements up into
   * multiple Transaction messages to keep the Transaction message size from
   * growing too large. In this case, the segment information in the Statement
   * submessages is not helpful if the Statement isn't segmented. We need this
   * information in the Transaction message itself.
   *
   * These values should be set appropriately whether or not the Statement
   * sub-messages are segmented.
   */
  optional uint32 segment_id = 4; /* Segment number of the Transaction msg */
  optional bool end_segment = 5; /* FALSE if Transaction msg is split into multiples */
}

So other than making it easier to check to see if a Transaction is segmented, why add these new values?

Well, it turns out having the segment information only in the Statement doesn't allow us to segment large Transaction messages if none of the Statement sub-messages are themselves segmented. This is documented in this bug report.

Drizzle Transaction Log and Statement Rollbacks

2010-12-10T09:39:00.001-05:00

First off, apologies to those who left comments on my past posts and your comments didn't show up. Apparently, I didn't have my blog settings setup correctly, and I just noticed (and approved) them.

So for the past few weeks, Patrick Crews and myself have been trying to chase down a really nasty transaction log bug that has been showing up under heavy load. The transaction log would end up not accurately representing the changes that took place in the server. Obviously, that's bad... unless you don't care if your slaves are an accurate representation of your master.

This has been a crazy difficult bug to track down, partly due to the fact that the cause appears to be multiple bugs. We've identified at least two bugs that are leading to transaction log inconsistency, one of which is how we deal with SQL statements that fail mid-execution. (The other, we believe, has to do with savepoints, and we're still working on that one.)

For the statement rollback bug, there were two problems:

We were not correctly removing recorded row changes when a statement affecting multiple rows failed after changing the first row.
Because we can now segment a single statement's row changes across multiple Transaction messages (due to having a max message size), we had no way to rollback transaction log changes made by a failed statement once it had been split up into multiple messages.

For issue #1, consider this statement:

INSERT INTO t1 (not_null_column) VALUES ('abc'),('def'),(NULL);

As each row is inserted, the in-memory protobuf message recording the transaction row changes is updated to include the row inserts. As we insert into this column which does not allow NULLs, we'll get changes recorded for values 'abc' and 'def'. However, once we try to insert NULL, the statement fails and InnoDB rolls back the two inserted rows. These two rows were not being removed from the in-memory message structure, which later gets sent through the replication stream and written to the transaction log, thus the inconsistency. The fix is fairly straightforward in that we just recognize the error, then remove the entries before sending the protobuf message downstream.

For issue #2, the situation is similar to issue #1, except that once row changes were sent through the replication stream (i.e., recorded in the transaction log file), we had no way to undo them if the current statement was still active and ended up failing. We decided to resolve this by adding a new protobuf Statement message type: ROLLBACK_STATEMENT.

The implications of this new Statement type? The first is that we now need to be able to clearly identify statement boundaries so we know how much work to undo. That means we no longer "roll up" like SQL statements into the same protobuf Statement message as an optimization for space (this avoided repeating the header information for the Statement message). So now each row-changing SQL statement executed gets its very own protobuf Statement message.

The second implication is that readers of the transaction log file need to be able to either:

rollback statements (you could conceivably use savepoints for this), or
cache Statement messages until the end segment is seen (which it can then execute the changes) or discard the cached messages if the next read Statement is a ROLLBACK_STATEMENT type.

These changes are now in our trunk. And major props to Stewart who helped us identify and fix this bug using his storage_engine_api_tester plugin.

Drizzle Replication Progress Report

2010-10-21T14:21:00.000-04:00

In an attempt to keep folks up-to-date as to what is happening with Drizzle replication, I thought that I'd blog about what we are currently doing, and where we are heading.

Current Area of Focus

A building is only as solid as its foundation. For Drizzle replication, that foundation is the transaction log. Our focus, as of late, has been testing the transaction log heavily, ensuring that the messages within it are correct and that its contents can be successfully used to replicate to another machine and have an exact copy of our data. Myself, Patrick Crews (awesome QA guy), and Joe Daly (awesome contributor) have been focused on this testing and fixing bugs for the last several weeks. This Launchpad Blueprint tracks our progress and list of bugs. And Patrick has posted some blog entries about this work.

Initially we started out simple with a single database session. This helped to uncover many issues, most of which have been fixed. We have now moved on to testing our transaction log with many concurrent database connections. The results are looking very good so far!

Next Steps

In its simplest form, replication can be achieved by just shipping the transaction log file from master to slave and applying it to the slave. Drizzle obviously needs a solution for this since we do not yet have a native implementation of replication. So we've decided that as a first-phase replication solution, we'll use a 3rd party solution to help us.

I looked at a few options at how we could implement this:

RabbitMQ - This is a message exchange system that would allow us to simply send our Google Protobuf Messages (the basic unit of communication in our replication architecture) from the master's transaction log (or alternatively, as a plugin reading the replication stream directly from the Drizzle kernel) to a server running RabbitMQ. Slaves could then connect to the RabbitMQ server to receive the replication events. This has the benefit of moving much of the I/O associated with replication off to another machine. Also, we already have a plugin created by Marcus Eriksson that will send the replication events to a RabbitMQ server. He also has an applier that will run on the slave to receive the events and apply them to the database.
ZeroMQ - Someone suggested that I look at this. This is an asynchronous message-processing library that looks really interesting. However, this is a networking library and would be more appropriate to use this in a native replication solution, which is a possibility in the future.
Tungsten Replicator - Continuent supplies an open source solution for replication for multiple database platforms (notably MySQL and Postgres). It provides some wonderful features, like easy promotion of slave to master in case of failure, and filtering of transaction log events.

ZeroMQ doesn't really fit what I was looking for in this phase (to write as little code as possible), so I've put off looking at it for now. Though I am excited to look at using this in the future.

RabbitMQ was a good possibility. Marcus has already done quite a bit of work with this, but the code is a bit out of date currently. There have been several changes to the protobuf messages since Marcus originally wrote this code, and it's difficult to keep up when we change so quickly. Still, it wouldn't take much effort to get this code up-to-date and working.

Honestly, I didn't know much about Tungsten Replicator until recently. This really looks like an excellent, full-featured product. It is well thought out in its design, and seems to be fairly simple to get up and running. Then I got really excited about it when I found out that Marcus has already been working with Continuent on adding Drizzle support! Not only that, Marcus announced his creation of a set of Java tools that contain the common code for working with the Drizzle transaction log. This can be used in his Tungsten work, as well as re-incorporated into his RabbitMQ work eventually.

It seems like a real win to work with Marcus on helping him to add Drizzle support to Tungsten, so this, along with transaction log testing, is my focus going forward. I hope to write up a simple HOWTO on using Tungsten Replicator with Drizzle once it is ready to use.

Looking to the Future

While nothing is written in stone just yet, we have multiple potential ideas for replication that are pretty exciting. Just some of them are:

Replication streams separated on a per-catalog basis. This would help to eliminate some contention issues and reduce the headaches that a single user could cause to the entire replication system.
Ability to limit resource usage of the slave, such as limiting concurrent writes per device.
Optional compression of the replication stream.

Although we have a bit of work to do, I think we have the potential to deliver some pretty interesting stuff in the future.

What other ideas would you like to see implemented in Drizzle replication?

Drizzle Transaction Message Limit

2010-08-29T17:52:00.002-04:00

Some recent changes I made have recently been pushed to Drizzle trunk that affect the size of the Transaction protobuf message that any replication stream will see (e.g., the transaction log). This was necessary to fix bug 600795.

Without a Transaction message size limit, for any bulk operations, like LOAD DATA, we would have ended up with a Transaction message that could possibly contain a very large Statement message that contained all of the INSERT data for the bulk load. This obviously could eat up a large amount of memory if we kept allowing the Statement to grow without bounds. The Drizzle kernel, when it can, keeps appending the values to INSERT onto the same record.

To circumvent this, we now allow multiple Transaction records for a single database transaction. Each Transaction GPB message representing a single database transaction will all have the same transaction ID, and only the last Transaction message will have the Statement's end_segment attribute set to true.

Here is an example of this change that you might now see in the transaction log:

transaction_context {
  server_id: 1
  transaction_id: 3
  start_timestamp: 1283118092815781
  end_timestamp: 1283118092815869
}
statement {
  type: INSERT
  start_timestamp: 1283118092815782
  end_timestamp: 1283118092815868
  insert_header {
   table_metadata {
   schema_name: "test"
   table_name: "t"
   }
   field_metadata {
   type: INTEGER
   name: "id"
   }
   field_metadata {
   type: VARCHAR
   name: "a"
   }
  }
  insert_data {
   segment_id: 1
   end_segment: false
   record {
   insert_value: "2"
   insert_value: "abc"
   is_null: false
   is_null: false
   }
   record {
   insert_value: "3"
   insert_value: "def"
   is_null: false
   is_null: false
   }
  }
}

transaction_context {

server_id: 1

transaction_id: 3

start_timestamp: 1283118092816250

end_timestamp: 1283118092816725

}

statement {

type: INSERT

start_timestamp: 1283118092816251

end_timestamp: 1283118092816724

insert_header {

table_metadata {

schema_name: "test"

table_name: "t"

}

field_metadata {

type: INTEGER

name: "id"

}

field_metadata {

type: VARCHAR

name: "a"

}

insert_data {

segment_id: 1

end_segment: true

record {

insert_value: "4"

insert_value: "ghi"

is_null: false

}

record {

insert_value: "5"

insert_value: "jkl"

is_null: false

}

This example is a bit contrived as there is no need to split up such a small transaction, but you can see the basic changes here. We have two Transaction messages, both with the same transaction ID. You can see that the Statement's end_segment is set to false in the first message, while the Statement within the second Transaction message has end_segment set to true.

So, in case it isn't obvious, there are now two ways to determine when you should commit if you are a replication stream TransactionApplier, or if you are reading from the transaction log:

~~If the transaction ID changes, COMMIT.~~
Or, if the current Transaction has all Statement messages with end_segment set to true, COMMIT.

Choose which ever method of the two best suits your needs.

Currently, if a Transaction message crosses the 1M threshold, the kernel will create a new Transaction message. Why did I choose 1M? Well, the Google Protobuf documentation says:

Protocol Buffers are not designed to handle large messages. As a general rule of thumb, if you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy.

So 1M seemed to be a reasonable default. I'll change this in the near future to be a configurable value once we get some changes to our sys var stuff merged.

Adding a Drizzle Executable in Xcode

2010-08-25T11:23:00.000-04:00

In my last post, I explained how to setup Drizzle under an Xcode project. This allows you to take advantage of Xcode's features while developing on Drizzle (or any other project of your own choosing). The one thing we weren't able to do was debug the Drizzle executable. This post remedies that.

So it turns out that this is an easy fix. But for Drizzle, there are a few extra hoops you have to jump through in order to get it to work.

The basic steps we need to do for Drizzle are:

Add a custom executable in Xcode
Setup any arguments you want to pass to the executable
Setup any environment variables needed for the executable to run properly

For most other projects, you can probably just get away with #1 and possibly #2. For Drizzle, though, we need to do #3 so that it can find its libraries.

Step #1 is easy. In the Groups & Files window, right click on Executables and then Add -> New Custom Executable...

This will pop up a window where you define where the executable resides. Once that is done, you can define what arguments to pass it, and what environment variables should be set when it runs, among other things. These should be self explanatory, and you should be able to set this up for your particular project. For Drizzle, though, it isn't so intuitive.

In Drizzle, after you run configure and make, the executable lives in a hidden directory within your xcode-branch repo directory. It will actually be in:

$drizzle-repo/xcode-branch/drizzled/.libs/drizzled

The executable you see in the xcode-branch/drizzled subdirectory is actually a shell script that runs the real executable for you. Don't ask me why. So enter the path to the executable in the .libs subdirectory:

Once you enter in the path and click Finish, you are given the chance to edit executable working directory, arguments, environment variables etc. The important part here is the information under the Arguments tab. We need to set the arguments to the executable as well as set the DYLD_LIBRARY_PATH environment variable so that the executable can find its dynamic libraries (otherwise, it looks for them in the installation directory, which if doesn't exist yet, will cause the executable to not start). Here is an example:

Once you have this setup, you should now be able to run your executable (after you've built it, of course), and use the Xcode debugger to merrily do some bug hunting. Make sure that you have unselected the Load symbols lazily option in Xcode Debugging preferences so that your breakpoints will be recognized!

Happy debugging.

Developing Drizzle with Xcode

2010-06-30T10:23:00.003-04:00

Even with all of its shortcomings, Xcode has excellent integration with the GNU debugger. It also has some good features for understanding a large code base that you may be unfamiliar with. For these reasons, I like to use Xcode for my development work. Unfortunately, it takes a bit of magic to get it setup to work with an external project that was not created specifically with/for Xcode and that does not use CVS/Subversion/Perforce.

If you want to use Xcode to work on Drizzle, these are the steps you need to take to setup a project. Note that Xcode doesn't support Bazaar so you still need to do some things from the command line. (Come on, Apple. CVS and Subversion just plain suck)

These instructions assume that the Drizzle repo is located in $drizzle-repo, as described here in the Drizzle wiki.

First, we need to create a new branch (we'll call it xcode-branch) of Drizzle trunk to use for our yet to be created Xcode project.

cd $drizzle-repo
bzr branch trunk xcode-branch

Next, we'll go into Xcode and create a new project. Choose the External Build System project. With this type of project, we'll be able to click on Build to run make for us.

Click the Choose... button and save your project some place appropriate (anywhere but inside your $drizzle-repo directory). We'll tell the project where to find the source files in xcode-branch in the following steps.

After saving, your project window should appear. At this point, we have no source code files in our project, so we need to tell Xcode to use our xcode-branch as the location of our source. We do that by right-clicking on our project name in the Groups & Files section of the project window and selecting Add -> Existing Files...

Navigate to your xcode-branch directory, select the directory, and click the Add button. In the window that appears, make sure that Copy items into destination group's folder option is NOT selected. We don't want to copy the source files into the Xcode project directory, we want to use them from within the Bazaar branch. We do this so that after we make changes with Xcode, we can easily open a Terminal window and commit our changes from the command line. (Again, this is because Xcode does not have Bazaar direct integration.) Also, make sure that Recursively create groups for any added folders is selected. So you should have:

Click Add and a folder for your xcode-branch should appear under your project name in the Groups & Files section.

So what have we just done? We basically did two things here:

Created a Bazaar branch for our modifications we will make under Xcode
Created an Xcode project that points to our Bazaar branch.

Our working Xcode project is split up into two separate directories: one for the Xcode project files, and one for the Bazaar source files. This may seem a bit odd if you expected everything to be under a single directory, but we have to do it this way so that we can still use Bazaar commands.

There is one last thing to do now. Xcode is now setup to automatically just call 'make' when we click on our Build button, but it will attempt to run 'make' in the Xcode project directory, not in our xcode-branch directory. This is an easy fix. Double-click on the target underneath Targets in the Groups & Files section of the project window. This will open up the following window:

Set the Directory under Custom Build Command to the location of your xcode-branch. Once you do this, then clicking Build will run 'make' from within the xcode-branch.

That's it! You can now edit the source, use Xcode features to jump to various definitions, search project files, and build the project using Command-B or the Build menu option. Obviously, you'll first have to create and run the configure script in your xcode-branch directory by hand before you can build. And you will not be able to use the other build commands, like Build and Run, or use the debugger until you setup an executable for the project. I'll cover that in a future post.