SkyTools
From PostgreSQL wiki
This is a package of tools in use in Skype for replication and failover. It includes a generic queuing mechanism, PgQ and utility library for Python scripts.
The source code and project page.
A general overview presentation can be found here: File:Moskva DB Tools.v3.pdf, and a PGQ detailed presentation is at http://www.pgcon.org/2008/schedule/events/79.en.html.
Contents |
PgQ
This is the queue mechanism we use. It consists of PL/pgsql, and C code in database, with Python framework on top of it. It is derived from snapshot-based event handling ideas from Slony-I, re-written for more general use.
Features
- A database can have many queues.
- Producers can insert into any queue.
- There can be several consumers for any given queue and all consumers see all events.
Documentation
Monitoring
A PGQ Munin Plugin is available.
See Also
PGQ can be the base solution to implement a Eventually Consistent system.
Kristo's blog host a detailed example of a python PGQ consumer which sends confirmation emails after user inscription: http://kaiv.wordpress.com/2007/10/19/skytools-database-scripting-framework-pgq/
Londiste
Replication tool written in Python, using PgQ as event transport.
Features
- Tables can be added one-by-one into the replication set.
- The initial COPY for one table does not block event replay for other tables.
- Can compare tables on both sides.
Documentation
- Londiste Tutorial
- Londiste script usage (also available as man 1 londiste)
- Londiste configuration (also available as man 5 londiste)
- Londiste reference
Monitoring
Londiste is "just" a PGQ Consumer, so see about how to monitor the queue itself, with the PGQ_Munin_Plugin e.g.
walmgr
This script sets up WAL archiving, does initial backup and runtime WAL archive and restore.
See its documentation.
Provided specific consumers
Some more tools based on Skytools Framework are available for handling specific jobs :
Each documentation page begins with a description section, but to ease your browsing experience, they are all in the same page here: Skytools Specific Consumers.
