Q4M, short from Queue for MySQL, is a message queue that works as a pluggable storage engine of MySQL 5.1.

Q4M, short from Queue for MySQL, is a message queue that works as a pluggable storage engine of MySQL 5.1, designed to be robust, fast, flexible. The development started in late December of 2007, and although it is very primitive, operates quite swiftly.

Robust

Q4M logs all message transfers to disk. A power failure or OS crash would never loose a message.

Fast

Although Q4M syncs every operation to disk, it runs fast by using group commits, and checksums for data recovery. Even with HDD write cache disabled, it is able to transfer >1,000 messages per second on an ordinally Pentium 4 linux box.

Flexible

Since Q4M works as a storage engine of MySQL, it is possible to JOIN queues against ordinally MySQL tables. Or you may use triggers for sending data into queue.

Requirements:

· MySQL 5.1.x

Installing the Source Distribution

For installing the source distribution, source code of MySQL is required.
download and build MySQL 5.1 (installation is not mandatory, but if you plan to install Q4M into a binary distribution of MySQL, make sure to use the same configuration options for building the MySQL source distribution. The configuration of MySQL can be found in bin/mysqlbug script.)
untar the Q4M distribution

run configure
run make
run make install
run support-files/install.sql

Below are the important options when configuring Q4M.

--with-mysql=mysql-source-dir

set the directory of MySQL source code

--libdir=mysql-lib-dir

set the library directory of your MySQL. It is in most cases either lib/mysql or lib subdirectory of MySQL

--with-debug

if your MySQL is built with --with-debug option, Q4M should be built with this option as well. Or your MySQL server will crash.

--with-sync=yes|no|fsync|fcntl

select disk flush method. The fcntl option only works on Mac OS X, which calls fcntl with F_FULLFSYNC option.

Installing the Binary Distribution

Follow the steps below to install the binary distribution.

untar the distribution
copy libqueue_engine.so to lib directory of MySQL 5.1
run support-files/install.sql

Testing the Installation

To test your installation, run the run_tests.pl. You can set your database location by using the DBI environment variable. DBI_USER and DBI_PASSWORD variables are also avialable. An output like below should appear.

$ DBI='dbi:mysql:database=test;host=kazdev;port=3307' ./run_tests.pl

t/01-base................ok
t/02-queue...............ok
t/03-queue-error-wait....ok
t/03-queue-error.........ok
t/04-blob................ok
t/05-multireader.........

Multireader benchmark result:

Number of messages: 6400
Number of readers: 32
Elapsed: 3.366 seconds
Throughput: 1901.245 mess./sec.

t/05-multireader.........ok

All tests successful.

Files=6, Tests=5370, 140 wallclock secs (115.80 cusr 5.92 csys = 121.72 CPU)
$

Using Q4M

To evaluate Q4M, download either a binary or source distribution from the install page, and follow the installation instructions. We also have a small tutorial. You may use SQL to access Q4M queues, or there is a wrapper module available for perl (Queue::Q4M).

Limitations:

· row size is limited to 1GB
· table size is limited to 2**63 bytes
· no support for UPDATE and REPLACE
· no support for indexes
· removal of multiple rows from a single DELETE statement is not atomic
· may cause server crash under extremely low memory condition
· may lose data if OS crashes while Q4M is performing table compaction (since the directory is not fsynced)

What's New in This Release:

· This release fixes bugs related to conditional subscription and multi-table subscription, as well as other bugs that might have caused memory corruption or a server crash.