Sharding Using Spockproxy: A Sharding-only Version of MySQL Proxy

Spock.com is a people search engine serving millions of queries per day. In order to grow our back-end to service these queries against a billion records (and growing) we developed Spockproxy.

Spockproxy is a modified version of MySQL proxy which does sharding inherently. It only does sharding; no Lua or anything else. It is very easy to set up; all configuration information is kept in 3 configuration tables. See http://spockproxy.sourceforge.net/ for documentation and downloads.

I propose to cover how to layout the databases shards, what is universal data as opposed to federated data and how we use replication to copy universal data into each shard. Also how to set up replication to provide an additional read slave proxy as well as how to use these replicants for high availability.

We’ll also cover the limitations of Spockproxy and the issues of Sharded databases that we’ve run into as well as how we’re got around them.

Our problems and the solutions to them should be interesting to anyone who is looking to introduce Sharding to their environment even if they choose not to use Spockproxy at all.