Commit 2c7d6f12 authored by Michael Widenius's avatar Michael Widenius

Added HandlerSocket plugin

- Fixed compiler errors
- Modified Makefiles to be part of plugin directory
- Some minor changes in database.cpp to use the new MariaDB handler interface
parent 26aa83bf
......@@ -1943,3 +1943,4 @@ storage/pbxt/bin/xtstat
libmysqld/sql_expression_cache.cc
mysql-test/mtr_command
scripts/convert-debug-for-diff
plugin/handler_socket/client/hsclient
......@@ -1291,3 +1291,34 @@ Use of any of this software is governed by the terms of the license below:
*/
***************************************************************************
%%The following software may be included in this product:
HandlerSocket plugin for MySQL
Copyright (c) 2010 DeNA Co.,Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of DeNA Co.,Ltd. nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY DeNA Co.,Ltd. "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL DeNA Co.,Ltd. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
***************************************************************************
Akira Higuchi (https://github.com/ahiguti)
- developed HanderSocket plugin, libhsclient, and perl-Net-HandlerSocket
Yoshinori Matsunobu (https://github.com/yoshinorim)
- introduced autotools, added support for MySQL 5.5.6, added statistics
variables
Jeff Hodges (https://github.com/jmhodges)
- fixed some autotools scripts
Toru Yamaguchi (https://github.com/zigorou)
- ported to MacOS X
Moriyoshi Koizumi (https://github.com/moriyoshi)
- fixed some autotools scripts
takeda-at (https://github.com/takada-at)
- added simple authorization function
1.0.6 - For MariaDB
* Modifications to Makefiles to be part of plugin directory
* Compiled by default in max builds
* Some minor changes in database.cpp to use the new MariaDB handler
interface
o * Fixed compiler warnings
1.0.6 - 2010-10-29
* Changed build instruction (autoreconf/configure/make), removed auto-generated files (Contributed by jmhodges)
*
1.0.5 - 2010-10-18
* Changed build procedures (using typical configure/make)
* Supported 5.5.6
* Added status variables
1.0.4 - 2010-08-15
* Initial public release
ACLOCAL_AMFLAGS = -I m4
SUBDIRS = @HANDLERSOCKET_SUBDIRS@
EXTRA_DIST= plug.in
perl:
cd perl-Net-HandlerSocket && perl Makefile.PL && make
install_perl:
cd perl-Net-HandlerSocket && make install
rpms: rpm_cli rpm_perl rpm_c
rpm_dir:
- mkdir dist
- mkdir dist/BUILD dist/RPMS dist/SOURCES dist/SPECS dist/SRPMS
rpm_cli: clean_cli rpm_dir
sed -e "s/HANDLERSOCKET_VERSION/$(VERSION)/" \
libhsclient/libhsclient.spec.template \
> libhsclient/libhsclient.spec
tar cvfz dist/libhsclient.tar.gz libhsclient
rpmbuild --define "_topdir `pwd`/dist" -ta \
dist/libhsclient.tar.gz
rpm_perl: clean_perl rpm_dir
sed -e "s/HANDLERSOCKET_VERSION/$(VERSION)/" \
perl-Net-HandlerSocket/perl-Net-HandlerSocket.spec.template \
> perl-Net-HandlerSocket/perl-Net-HandlerSocket.spec
cd perl-Net-HandlerSocket && perl Makefile.PL && make clean && \
rm -f Makefile.old
tar cvfz dist/perl-Net-HandlerSocket.tar.gz perl-Net-HandlerSocket
rpmbuild --define "_topdir `pwd`/dist" -ta \
dist/perl-Net-HandlerSocket.tar.gz
rpm_c: clean_c rpm_dir
sed -e "s/HANDLERSOCKET_VERSION/$(VERSION)/" \
handlersocket/handlersocket.spec.template \
> handlersocket/handlersocket.spec
sed -e "s|HANDLERSOCKET_MYSQL_INC|$(MYSQL_CFLAGS) $(MYSQL_INC)|" \
-e "s|HANDLERSOCKET_MYSQL_LIB|$(MYSQL_LIB)|" \
handlersocket/Makefile.plain.template \
> handlersocket/Makefile.plain
tar cvfz dist/handlersocket.tar.gz handlersocket
rpmbuild --define "_topdir `pwd`/dist" -ta \
dist/handlersocket.tar.gz
install_rpm_pl:
- sudo rpm -e perl-Net-HandlerSocket
- sudo rpm -e perl-Net-HandlerSocket-debuginfo
make clean
make rpm_perl
- sudo rpm -U dist/RPMS/*/perl*.rpm
installrpms:
- sudo rpm -e handlersocket
- sudo rpm -e handlersocket-debuginfo
- sudo rpm -e perl-Net-HandlerSocket
- sudo rpm -e perl-Net-HandlerSocket-debuginfo
- sudo rpm -e libhsclient
- sudo rpm -e libhsclient-debuginfo
make clean
make rpm_cli
- sudo rpm -U dist/RPMS/*/libhsclient*.rpm
make clean
make rpm_perl
- sudo rpm -U dist/RPMS/*/perl*.rpm
make clean
make rpm_c
- sudo rpm -U dist/RPMS/*/handlersocket*.rpm
clean_cli:
cd libhsclient && make clean
cd client && make clean
clean_perl:
cd perl-Net-HandlerSocket && perl Makefile.PL && make clean && \
rm -f Makefile.old
clean_c:
cd handlersocket && make clean
clean_all: clean_cli clean_perl clean_c
cd regtest && make clean
rm -rf dist/*/*
rm -f dist/*.tar.gz
Notes added by Monty:
This is HandlerSocket version 1.0.6 (See ChangeLog file)
The original code can be found at:
https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL
-----------------------------------------------------------------------------
HandlerSocket plugin for MySQL
Copyright (c) 2010 DeNA Co.,Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of DeNA Co.,Ltd. nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY DeNA Co.,Ltd. "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL DeNA Co.,Ltd. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-----------------------------------------------------------------------------
About HandlerSocket
HandlerSocket is a NoSQL plugin for MySQL. It works as a daemon inside the
mysqld process, accept tcp connections, and execute requests from clients.
HandlerSocket does not support SQL queries. Instead, it supports simple CRUD
operations on tables.
Because of the following reasons, HandlerSocket is much faster than the
mysqld/libmysql pair in some circumstances:
- HandlerSocket manipulates data without parsing SQL, which causes less
CPU usage.
- HandlerSocket reads many requests from clients and executes their
requests in bulk, which causes less CPU and disk usage.
- HandlerSocket client/server protocol is more compact than the
mysql/libmysql pair, which causes less network usage.
The current version of HandlerSocket only works with GNU/Linux. The source
archive of HandlerSocket includes a C++ and a Perl client libraries.
Here is a list of client libraries for other languages:
- PHP
http://openpear.org/package/Net_HandlerSocket
http://github.com/tz-lom/HSPHP
http://code.google.com/p/php-handlersocket/
- Java
http://code.google.com/p/hs4j/
http://code.google.com/p/handlersocketforjava/
- Python
http://pypi.python.org/pypi/python-handler-socket
https://code.launchpad.net/~songofacandy/+junk/pyhandlersocket
- Ruby
https://github.com/winebarrel/ruby-handlersocket
https://github.com/miyucy/handlersocket
- JavaScript
https://github.com/koichik/node-handlersocket
- Scala
https://github.com/fujohnwang/hs2client
The home of HandlerSocket is here:
https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL
More documents are available in docs-en/ and docs-ja/ directories.
#!/bin/sh
warn() {
echo -e "\tWARNING: $@" 1>&2
}
# init
LIBTOOLIZE=libtoolize
ACLOCAL=aclocal
AUTOCONF=autoconf
AUTOHEADER=autoheader
AUTOMAKE=automake
case `uname -s` in
Darwin)
LIBTOOLIZE=glibtoolize
;;
FreeBSD)
ACLOCAL_ARGS="$ACLOCAL_ARGS -I /usr/local/share/aclocal/"
;;
esac
# libtoolize
echo "Searching libtoolize..."
if [ `which $LIBTOOLIZE` ] ; then
echo -e "\tFOUND: libtoolize -> $LIBTOOLIZE"
else
warn "Cannot Found libtoolize... input libtool command"
read LIBTOOLIZE
LIBTOOLIZE=`which $LIBTOOLIZE`
if [ `which $LIBTOOLIZE` ] ; then
echo -e "\tSET: libtoolize -> $LIBTOOLIZE"
else
warn "$LIBTOOLIZE: Command not found."
exit 1;
fi
fi
# aclocal
echo "Searching aclocal..."
if [ `which $ACLOCAL` ] ; then
echo -e "\tFOUND: aclocal -> $ACLOCAL"
else
warn "Cannot Found aclocal... input aclocal command"
read ACLOCAL
ACLOCAL=`which $ACLOCAL`
if [ `which $ACLOCAL` ] ; then
echo -e "\tSET: aclocal -> $ACLOCAL"
else
warn "$ACLOCAL: Command not found."
exit 1;
fi
fi
# automake
echo "Searching automake..."
if [ `which $AUTOMAKE` ] ; then
echo -e "\tFOUND: automake -> $AUTOMAKE"
else
warn "Cannot Found automake... input automake command"
read AUTOMAKE
ACLOCAL=`which $AUTOMAKE`
if [ `which $AUTOMAKE` ] ; then
echo -e "\tSET: automake -> $AUTOMAKE"
else
warn "$AUTOMAKE: Command not found."
exit 1;
fi
fi
# autoheader
echo "Searching autoheader..."
if [ `which $AUTOHEADER` ] ; then
echo -e "\tFOUND: autoheader -> $AUTOHEADER"
else
warn "Cannot Found autoheader... input autoheader command"
read AUTOHEADER
ACLOCAL=`which $AUTOHEADER`
if [ `which $AUTOHEADER` ] ; then
echo -e "\tSET: autoheader -> $AUTOHEADER"
else
warn "$AUTOHEADER: Command not found."
exit 1;
fi
fi
# autoconf
echo "Searching autoconf..."
if [ `which $AUTOCONF` ] ; then
echo -e "\tFOUND: autoconf -> $AUTOCONF"
else
warn "Cannot Found autoconf... input autoconf command"
read AUTOCONF
ACLOCAL=`which $AUTOCONF`
if [ `which $AUTOCONF` ] ; then
echo -e "\tSET: autoconf -> $AUTOCONF"
else
warn "$AUTOCONF: Command not found."
exit 1;
fi
fi
echo "Running libtoolize ..."
$LIBTOOLIZE --force --copy
echo "Running aclocal ..."
$ACLOCAL ${ACLOCAL_ARGS} -I .
echo "Running autoheader..."
$AUTOHEADER
echo "Running automake ..."
$AUTOMAKE --add-missing --copy
echo "Running autoconf ..."
$AUTOCONF
mkdir m4 2> /dev/null
CXXFLAGS += -fimplicit-templates
AM_INCLUDES= -I$(srcdir)/../libhsclient
bin_PROGRAMS=hsclient
hsclient_SOURCES= hsclient.cpp
hsclient_LDFLAGS= -static -L../libhsclient -lhsclient
hsclient_CXXFLAGS= $(AM_INCLUDES)
hstest: hstest.o
$(CXX) $(CXXFLAGS) $(LFLAGS) hstest.o \
-L../libhsclient/.libs -lhsclient $(MYSQL_LIB) -o hstest
hstest.o: hstest.cpp
$(CXX) $(CXXFLAGS) $(MYSQL_INC) $(AM_INCLUDES) -c hstest.cpp
// vim:sw=2:ai
#include "hstcpcli.hpp"
#include "string_util.hpp"
namespace dena {
int
hstcpcli_main(int argc, char **argv)
{
config conf;
parse_args(argc, argv, conf);
socket_args sockargs;
sockargs.set(conf);
hstcpcli_ptr cli = hstcpcli_i::create(sockargs);
const std::string dbname = conf.get_str("dbname", "hstest");
const std::string table = conf.get_str("table", "hstest_table1");
const std::string index = conf.get_str("index", "PRIMARY");
const std::string fields = conf.get_str("fields", "k,v");
const int limit = conf.get_int("limit", 0);
const int skip = conf.get_int("skip", 0);
std::vector<std::string> keys;
std::vector<string_ref> keyrefs;
size_t num_keys = 0;
while (true) {
const std::string conf_key = std::string("k") + to_stdstring(num_keys);
const std::string k = conf.get_str(conf_key, "");
const std::string kx = conf.get_str(conf_key, "x");
if (k.empty() && kx == "x") {
break;
}
++num_keys;
keys.push_back(k);
}
for (size_t i = 0; i < keys.size(); ++i) {
const string_ref ref(keys[i].data(), keys[i].size());
keyrefs.push_back(ref);
}
const std::string op = conf.get_str("op", "=");
const string_ref op_ref(op.data(), op.size());
cli->request_buf_open_index(0, dbname.c_str(), table.c_str(),
index.c_str(), fields.c_str());
cli->request_buf_exec_generic(0, op_ref, num_keys == 0 ? 0 : &keyrefs[0],
num_keys, limit, skip, string_ref(), 0, 0);
int code = 0;
size_t numflds = 0;
do {
if (cli->request_send() != 0) {
fprintf(stderr, "request_send: %s\n", cli->get_error().c_str());
break;
}
if ((code = cli->response_recv(numflds)) != 0) {
fprintf(stderr, "response_recv: %s\n", cli->get_error().c_str());
break;
}
} while (false);
cli->response_buf_remove();
do {
if ((code = cli->response_recv(numflds)) != 0) {
fprintf(stderr, "response_recv: %s\n", cli->get_error().c_str());
break;
}
while (true) {
const string_ref *const row = cli->get_next_row();
if (row == 0) {
break;
}
printf("REC:");
for (size_t i = 0; i < numflds; ++i) {
const std::string val(row[i].begin(), row[i].size());
printf(" %s", val.c_str());
}
printf("\n");
}
} while (false);
cli->response_buf_remove();
return 0;
}
};
int
main(int argc, char **argv)
{
return dena::hstcpcli_main(argc, argv);
}
#!/usr/bin/perl
use strict;
use warnings;
use DB::HandlerSocket::Pool;
use DBI;
my %conf = ();
for my $i (@ARGV) {
my ($k, $v) = split(/=/, $i);
$conf{$k} = $v;
}
my $verbose = get_conf("verbose", 0);
my $actions_str = get_conf("actions",
"create,insert,verify,verify2,verify3,verify4,clean");
my $tablesize = get_conf("tablesize", 1000);
my $db = get_conf("db", "hstestdb");
my $table = get_conf("table", "testtbl");
my $table_schema = get_conf("table_schema", undef);
my $engine = get_conf("engine", "innodb");
my $host = get_conf("host", "localhost");
my $mysqlport = get_conf("mysqlport", 3306);
my $hsport_rd = get_conf("hsport_rd", 9998);
my $hsport_wr = get_conf("hsport_wr", 9999);
my $loop = get_conf("loop", 10000);
my $op = get_conf("op", "=");
my $ssps = get_conf("ssps", 0);
my $num_moreflds = get_conf("moreflds", 0);
my $moreflds_prefix = get_conf("moreflds_prefix", "f");
my $mysql_user = 'root';
my $mysql_password = '';
my $dsn = "DBI:mysql:database=;host=$host;port=$mysqlport"
. ";mysql_server_prepare=$ssps";
my $dbh = DBI->connect($dsn, $mysql_user, $mysql_password,
{ RaiseError => 1 });
my $hsargs = { 'host' => $host, 'port' => $hsport_rd };
my $hspool = new DB::HandlerSocket::Pool({
hostmap => {
"$db.$table" => {
host => $host,
port => $hsport_rd,
},
},
resolve => undef,
error => undef,
});
$table_schema = "(k int primary key, fc30 varchar(30), ft text)"
if (!defined($table_schema));
my @actions = split(/,/, $actions_str);
for my $action (@actions) {
print "ACTION: $action\n";
eval "hstest_$action()";
if ($@) {
die $@;
}
print "ACTION: $action DONE\n";
}
sub get_conf {
my ($key, $def) = @_;
my $val = $conf{$key};
if ($val) {
print "$key=$val\n";
} else {
$val = $def;
my $defstr = $def || "(undef)";
print "$key=$defstr(default)\n";
}
return $val;
}
sub hstest_create {
$dbh->do("drop database if exists $db");
$dbh->do("create database $db");
$dbh->do("use $db");
$dbh->do("create table $table $table_schema engine=$engine");
}
sub hstest_dump {
$dbh->do("use $db");
my $sth = $dbh->prepare("select * from $table");
$sth->execute();
my $arr = $sth->fetchall_arrayref();
for my $rec (@$arr) {
print "REC:";
for my $row (@$rec) {
print " $row";
}
print "\n";
}
}
sub hstest_insert {
$dbh->do("use $db");
my $sth = $dbh->prepare("insert into $table values (?, ?, ?)");
for (my $k = 0; $k < $tablesize; ++$k) {
my $fc30 = "fc30_$k";
my $ft = "ft_$k";
$sth->execute($k, $fc30, $ft);
}
}
sub hstest_verify {
$dbh->do("use $db");
my $sth = $dbh->prepare("select * from $table order by k");
$sth->execute();
my $arr = $sth->fetchall_arrayref();
my $hsres = $hspool->index_find($db, $table, "PRIMARY", "k,fc30,ft",
">=", [ 0 ], $tablesize, 0);
for (my $i = 0; $i < $tablesize; ++$i) {
my $rec = $arr->[$i];
my $differ = 0;
print "REC:" if $verbose;
for (my $j = 0; $j < 3; ++$j) {
my $fld = $rec->[$j];
my $hsidx = $i * 3 + $j;
my $hsfld = $hsres->[$hsidx];
if ($hsfld ne $fld) {
$differ = 1;
}
if ($differ) {
print " $fld:$hsfld" if $verbose;
} else {
print " $hsfld" if $verbose;
}
}
print "\n" if $verbose;
if ($differ) {
die "verification failed";
}
}
}
sub hstest_verify2 {
$dbh->do("use $db");
my $sth = $dbh->prepare("select * from $table order by k");
$sth->execute();
my $arr = $sth->fetchall_arrayref();
my $hsresa = $hspool->index_find_multi($db, $table, "PRIMARY",
"k,fc30,ft", [ [ -1, ">=", [ 0 ], $tablesize, 0 ] ]);
my $hsres = $hsresa->[0];
for (my $i = 0; $i < $tablesize; ++$i) {
my $rec = $arr->[$i];
my $differ = 0;
print "REC:" if $verbose;
for (my $j = 0; $j < 3; ++$j) {
my $fld = $rec->[$j];
my $hsidx = $i * 3 + $j;
my $hsfld = $hsres->[$hsidx];
if ($hsfld ne $fld) {
$differ = 1;
}
if ($differ) {
print " $fld:$hsfld" if $verbose;
} else {
print " $hsfld" if $verbose;
}
}
print "\n" if $verbose;
if ($differ) {
die "verification failed";
}
}
}
sub hashref_to_str {
my $href = $_[0];
my $r = '';
for my $k (sort keys %$href) {
my $v = $href->{$k};
$r .= "($k=>$v)";
}
return $r;
}
sub hstest_verify3 {
$dbh->do("use $db");
my $sth = $dbh->prepare("select * from $table order by k");
$sth->execute();
my $hsres_t = $hspool->index_find($db, $table, "PRIMARY", "k,fc30,ft",
">=", [ 0 ], $tablesize, 0);
my $hsres = DB::HandlerSocket::Pool::result_single_to_hasharr(
[ 'k', 'fc30', 'ft' ], $hsres_t);
for (my $i = 0; $i < $tablesize; ++$i) {
my $mystr = hashref_to_str($sth->fetchrow_hashref());
my $hsstr = hashref_to_str($hsres->[$i]);
if ($mystr ne $hsstr) {
print "DIFF my=[$mystr] hs=[$hsstr]\n" if $verbose;
die "verification failed";
} else {
print "OK $hsstr\n" if $verbose;
}
}
}
sub hstest_verify4 {
$dbh->do("use $db");
my $sth = $dbh->prepare("select * from $table order by k");
$sth->execute();
my $hsres_t = $hspool->index_find($db, $table, "PRIMARY", "k,fc30,ft",
">=", [ 0 ], $tablesize, 0);
my $hsres = DB::HandlerSocket::Pool::result_single_to_hashhash(
[ 'k', 'fc30', 'ft' ], 'k', $hsres_t);
my $rechash = $sth->fetchall_hashref('k');
while (my ($k, $href) = each (%$rechash)) {
my $mystr = hashref_to_str($href);
my $hsstr = hashref_to_str($hsres->{$k});
if ($mystr ne $hsstr) {
print "DIFF my=[$mystr] hs=[$hsstr]\n" if $verbose;
die "verification failed";
} else {
print "OK $hsstr\n" if $verbose;
}
}
}
sub hstest_clean {
$hspool->clear_pool();
$dbh->do("drop database if exists $db");
}
This diff is collapsed.
#!/usr/bin/perl
# vim:sw=8:ai:ts=8
use strict;
use warnings;
use DBI;
use Net::HandlerSocket;
my %conf = ();
for my $i (@ARGV) {
my ($k, $v) = split(/=/, $i);
$conf{$k} = $v;
}
my $verbose = get_conf("verbose", 0);
my $actions_str = get_conf("actions", "hsread");
my $tablesize = get_conf("tablesize", 10000);
my $db = get_conf("db", "hstest");
my $table = get_conf("table", "hstest_table1");
my $engine = get_conf("engine", "innodb");
my $host = get_conf("host", "localhost");
my $mysqlport = get_conf("mysqlport", 3306);
my $mysqluser = get_conf("mysqluser", "root");
my $mysqlpass = get_conf("mysqlpass", "");
my $hsport = get_conf("hsport", 9999);
my $loop = get_conf("loop", 10000);
my $op = get_conf("op", "=");
my $ssps = get_conf("ssps", 0);
my $num_moreflds = get_conf("moreflds", 0);
my $moreflds_prefix = get_conf("moreflds_prefix", "column0123456789_");
my $keytype = get_conf("keytype", "varchar(32)");
my $file = get_conf("file", undef);
my $dsn = "DBI:mysql:database=;host=$host;port=$mysqlport"
. ";mysql_server_prepare=$ssps";
my $dbh = DBI->connect($dsn, $mysqluser, $mysqlpass, { RaiseError => 1 });
my $hsargs = { 'host' => $host, 'port' => $hsport };
my $cli = new Net::HandlerSocket($hsargs);
my @actions = split(/,/, $actions_str);
for my $action (@actions) {
if ($action eq "table") {
print("TABLE $db.$table\n");
$dbh->do("drop database if exists $db");
$dbh->do("create database $db");
$dbh->do("use $db");
my $moreflds = get_createtbl_moreflds_str();
$dbh->do(
"create table $table (" .
"k $keytype primary key" .
",v varchar(32) not null" .
$moreflds .
") character set utf8 collate utf8_bin " .
"engine = $engine");
} elsif ($action eq "insert") {
print("INSERT $db.$table tablesize=$tablesize\n");
$dbh->do("use $db");
my $moreflds = get_insert_moreflds_str();
for (my $i = 0; $i < $tablesize; $i += 100) {
my $qstr = "insert into $db.$table values";
for (my $j = 0; $j < 100; ++$j) {
if ($j != 0) {
$qstr .= ",";
}
my $k = "" . ($i + $j);
my $v = "v" . int(rand(1000)) . ($i + $j);
$qstr .= "('$k', '$v'";
for (my $j = 0; $j < $num_moreflds; ++$j) {
$qstr .= ",'$j'";
}
$qstr .= ")";
}
$dbh->do($qstr);
print "$i/$tablesize\n" if $i % 1000 == 0;
}
} elsif ($action eq "read") {
print("READ $db.$table op=$op loop=$loop\n");
$dbh->do("use $db");
my $moreflds = get_select_moreflds_str();
my $sth = $dbh->prepare(
"select k,v$moreflds from $db.$table where k = ?");
for (my $i = 0; $i < $loop; ++$i) {
my $k = "" . int(rand($tablesize));
# print "k=$k\n";
$sth->execute($k);
if ($verbose >= 10) {
print "RET:";
while (my $ref = $sth->fetchrow_arrayref()) {
my $rk = $ref->[0];
my $rv = $ref->[1];
print " $rk $rv";
}
print "\n";
}
print "$i/$loop\n" if $i % 1000 == 0;
}
} elsif ($action eq "hsinsert") {
print("HSINSERT $db.$table tablesize=$tablesize\n");
$cli->open_index(1, $db, $table, '', 'k,v');
for (my $i = 0; $i < $tablesize; ++$i) {
my $k = "" . $i;
my $v = "v" . int(rand(1000)) . $i;
my $r = $cli->execute_insert(1, [ $k, $v ]);
if ($r->[0] != 0) {
die;
}
print "$i/$tablesize\n" if $i % 1000 == 0;
}
} elsif ($action eq "hsread") {
print("HSREAD $db.$table op=$op loop=$loop\n");
my $moreflds = get_select_moreflds_str();
$cli->open_index(1, $db, $table, '', "k,v$moreflds");
for (my $i = 0; $i < $loop; ++$i) {
my $k = "" . int(rand($tablesize));
# print "k=$k\n";
my $r = $cli->execute_find(1, $op, [ $k ], 1, 0);
if ($verbose >= 10) {
my $len = scalar(@{$r});
print "LEN=$len";
for my $e (@{$r}) {
print " [$e]";
}
print "\n";
}
print "$i/$loop\n" if $i % 1000 == 0;
}
} elsif ($action eq "hsupdate") {
my $vbase = "v" . int(rand(1000));
print("HSUPDATE $db.$table op=$op loop=$loop vbase=$vbase\n");
$cli->open_index(1, $db, $table, '', 'v');
for (my $i = 0; $i < $loop; ++$i) {
my $k = "" . int(rand($tablesize));
my $v = $vbase . $i;
print "k=$k v=$v\n";
my $r = $cli->execute_update(1, $op, [ $k ], 1, 0,
[ $v ]);
if ($verbose >= 10) {
print "UP k=$k v=$v\n";
}
print "$i/$loop\n" if $i % 1000 == 0;
}
} elsif ($action eq "hsdelete") {
print("HSDELETE $db.$table op=$op loop=$loop\n");
$cli->open_index(1, $db, $table, '', '');
for (my $i = 0; $i < $loop; ++$i) {
my $k = "" . int(rand($tablesize));
print "k=$k\n";
my $r = $cli->execute_delete(1, $op, [ $k ], 1, 0);
if ($verbose >= 10) {
print "DEL k=$k\n";
}
print "$i/$loop\n" if $i % 1000 == 0;
}
} elsif ($action eq "verify") {
verify_do();
}
}
sub verify_do {
my ($fail_cnt, $ok_cnt) = (0, 0);
my $sth = $dbh->prepare("select v from $db.$table where k = ?");
use FileHandle;
my $fh = new FileHandle($file, "r");
while (my $line = <$fh>) {
chomp($line);
my @vec = split(/\t/, $line);
my $k = $vec[3];
my $v = $vec[7];
next if (!defined($k) || !defined($v));
# print "$k $v\n";
$sth->execute($k);
my $aref = $sth->fetchrow_arrayref();
if (!defined($aref)) {
print "FAILED: $k notfound\n";
++$fail_cnt;
} else {
my $gv = $aref->[0];
if ($gv ne $v) {
print "FAILED: $k got=$gv expected=$v\n";
++$fail_cnt;
} else {
print "OK: $k $v $gv\n" if $verbose >= 10;
++$ok_cnt;
}
}
}
print "OK=$ok_cnt FAIL=$fail_cnt\n";
}
sub get_conf {
my ($key, $def) = @_;
my $val = $conf{$key};
if ($val) {
print "$key=$val\n";
} else {
$val = $def;
$def ||= '';
print "$key=$def(default)\n";
}
return $val;
}
sub get_createtbl_moreflds_str {
my $s = "";
for (my $j = 0; $j < $num_moreflds; ++$j) {
$s .= ",$moreflds_prefix$j varchar(30)";
}
return $s;
}
sub get_select_moreflds_str {
my $s = "";
for (my $i = 0; $i < $num_moreflds; ++$i) {
$s .= ",$moreflds_prefix$i";
}
return $s;
}
sub get_insert_moreflds_str {
my $s = "";
for (my $i = 0; $i < $num_moreflds; ++$i) {
$s .= ",?";
}
return $s;
}
#!/bin/bash
exec ./hstest test=10 tablesize=10000 host=localhost hsport=9998 num=10000000 \
num_threads=100 timelimit=10 $@
#!/bin/bash
exec ./hstest test=10 key_mask=9999 host=localhost port=9998 num=10000000 \
num_threads=100 timelimit=10 moreflds=50 $@
#!/bin/bash
./hstest test=7 key_mask=9999 host=localhost port=11211 num=10000 \
num_threads=10 timelimit=10 op=R $@
./hstest test=7 key_mask=9999 host=localhost port=11211 num=1000000 \
num_threads=100 timelimit=10 op=G $@
#!/bin/bash
exec ./hstest test=9 tablesize=9999 host=localhost mysqlport=3306 num=1000000 \
num_threads=100 verbose=1 timelimit=10 $@
#!/bin/bash
exec ./hstest test=9 key_mask=9999 host=localhost port=3306 num=1000000 \
num_threads=100 verbose=1 timelimit=10 moreflds=50 $@
# -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.
#AC_PREREQ([2.63b])
AC_INIT([handlersocket-plugin], [1.0.6], [https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL/issues])
AC_CONFIG_HEADERS([config.h])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])
AC_CONFIG_SRCDIR([libhsclient/fatal.cpp])
AC_CONFIG_MACRO_DIR([m4])
AC_PROG_CC
AC_PROG_CXX
AC_PROG_CPP
AC_PROG_LIBTOOL
AC_DEFUN([CONFIG_OPTION_MYSQL],[
AC_MSG_CHECKING([mysql source])
MYSQL_SOURCE_VERSION=
MYSQL_INC=
ac_mysql_source_dir=
AC_ARG_WITH([mysql-source],
[AS_HELP_STRING([--with-mysql-source=PATH], [MySQL source directory PATH])],
[
ac_mysql_source_dir=`cd $withval && pwd`
if test -f "$ac_mysql_source_dir/sql/handler.h" ; then
MYSQL_INC="-I$ac_mysql_source_dir/sql"
MYSQL_INC="$MYSQL_INC -I$ac_mysql_source_dir/include"
MYSQL_INC="$MYSQL_INC -I$ac_mysql_source_dir/regex"
MYSQL_INC="$MYSQL_INC -I$ac_mysql_source_dir"
AC_SUBST(MYSQL_INC)
if test -f "$ac_mysql_source_dir/VERSION"; then
source "$ac_mysql_source_dir/VERSION"
MYSQL_SOURCE_VERSION="$MYSQL_VERSION_MAJOR.$MYSQL_VERSION_MINOR.$MYSQL_VERSION_PATCH"
else
if test -f "$ac_mysql_source_dir/configure.in"; then
MYSQL_SOURCE_VERSION=`cat $ac_mysql_source_dir/configure.in | grep "\[[MySQL Server\]]" | sed -e "s|.*\([[0-9]]\+\.[[0-9]]\+\.[[0-9]]\+[[0-9a-zA-Z\_\-]]*\).*|\1|"`
else
AC_MSG_ERROR([invalid MySQL source directory: $ac_mysql_source_dir])
fi
fi
AC_MSG_RESULT([yes: Using $ac_mysql_source_dir, version $MYSQL_SOURCE_VERSION])
else
AC_MSG_ERROR([invalid MySQL source directory: $ac_mysql_source_dir])
fi
],
[AC_MSG_ERROR([--with-mysql-source=PATH is required for standalone build])]
)
MYSQL_BIN_VERSION=
ac_mysql_config=
AC_ARG_WITH([mysql-bindir],
[AS_HELP_STRING([--with-mysql-bindir=PATH], [MySQL binary directory PATH. This should be the directory where mysql_config is located.])],
[
mysql_bin_dir=`cd $withval 2> /dev/null && pwd || echo ""`
ac_mysql_config="$mysql_bin_dir/mysql_config"
],
[
AC_PATH_PROG([ac_mysql_config], [mysql_config])
]
)
AC_MSG_CHECKING([mysql binary])
if test ! -x "$ac_mysql_config" ; then
AC_MSG_ERROR([mysql_config not found! You have to specify the directory where mysql_config resides to --with-mysql-bindir=PATH.])
fi
MYSQL_CFLAGS_ADD=`"$ac_mysql_config" --cflags`
MYSQL_CFLAGS="$MYSQL_CFLAGS $MYSQL_CFLAGS_ADD -DFORCE_DBUG_OFF"
# FIXME
AC_SUBST(MYSQL_CFLAGS)
MYSQL_BIN_VERSION=`"$ac_mysql_config" --version`
AC_MSG_RESULT([yes: Using $ac_mysql_config, version $MYSQL_BIN_VERSION])
MYSQL_LIB=`"$ac_mysql_config" --libs_r`
LIB_DIR=`echo $MYSQL_LIB | sed -e "s|.*-L/|/|" | sed -e "s| .*||"`
# FIXME
if test a`basename "$LIB_DIR"` = amysql ; then
MYSQL_LIB="-L`dirname $LIB_DIR` $MYSQL_LIB"
# FIXME
fi
AC_SUBST(MYSQL_LIB)
if test a$MYSQL_SOURCE_VERSION != a$MYSQL_BIN_VERSION ; then
AC_MSG_ERROR([MySQL source version does not match MySQL binary version])
fi
AC_MSG_CHECKING([mysql plugin dir])
ac_mysql_plugin_dir=
AC_ARG_WITH([mysql-plugindir],
[AS_HELP_STRING([--with-mysql-plugindir=PATH], [MySQL plugin directory where handlersocket.so to be copied])],
[
ac_mysql_plugin_dir=`cd $withval && pwd`
if test -d "$ac_mysql_plugin_dir/" ; then
PLUGIN_DIR="$ac_mysql_plugin_dir"
AC_SUBST(PLUGIN_DIR)
AC_MSG_RESULT([yes: Using $ac_mysql_plugin_dir])
else
AC_MSG_ERROR([invalid MySQL plugin directory : $ac_mysql_plugin_dir])
fi
],
[
LIB_DIR_TMP=`"$ac_mysql_config" --plugindir`
if test ! -d "$LIB_DIR_TMP"; then
LIB_DIR_TMP=`"$ac_mysql_config" --libs_r | sed -e "s|.*-L/|/|" | sed -e "s| .*||"`/plugin
# FIXME
fi
ac_mysql_plugin_dir=$LIB_DIR_TMP
PLUGIN_DIR="$ac_mysql_plugin_dir"
AC_SUBST(PLUGIN_DIR)
AC_MSG_RESULT([--with-mysql-plugindir was not set. Using $ac_mysql_plugin_dir])
]
)
])
HANDLERSOCKET_SUBDIRS="libhsclient"
AC_ARG_ENABLE(handlersocket_server,
[ --enable-handlersocket-server build HandlerSocket plugin (defalut=yes)])
if test "$enable_handlersocket_server" != "no"; then
CONFIG_OPTION_MYSQL
HANDLERSOCKET_SUBDIRS="libhsclient handlersocket client"
fi
AC_SUBST(HANDLERSOCKET_SUBDIRS)
CFLAGS="$CFLAGS -Werror"
CXXFLAGS="$CXXFLAGS -Wall -g -fno-rtti -fno-exceptions -fPIC -DPIC"
AC_CONFIG_FILES([Makefile
handlersocket/Makefile
libhsclient/Makefile
client/Makefile])
AC_OUTPUT
-----------------------------------------------------------------------------
HandlerSocket plugin for MySQL
Copyright (c) 2010 DeNA Co.,Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of DeNA Co.,Ltd. nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY DeNA Co.,Ltd. "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL DeNA Co.,Ltd. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-----------------------------------------------------------------------------
About HandlerSocket
HandlerSocket is a NoSQL plugin for MySQL. It works as a daemon inside the
mysqld process, accept tcp connections, and execute requests from clients.
HandlerSocket does not support SQL queries. Instead, it supports simple CRUD
operations on tables.
Because of the following reasons, HandlerSocket is much faster than the
mysqld/libmysql pair in some circumstances:
- HandlerSocket manipulates data without parsing SQL, which causes less
CPU usage.
- HandlerSocket reads many requests from clients and executes their
requests in bulk, which causes less CPU and disk usage.
- HandlerSocket client/server protocol is more compact than the
mysql/libmysql pair, which causes less network usage.
The current version of HandlerSocket only works with GNU/Linux. The source
archive of HandlerSocket includes a C++ and a Perl client libraries.
Here is a list of other language bindings:
- PHP
http://openpear.org/package/Net_HandlerSocket
http://github.com/tz-lom/HSPHP
http://code.google.com/p/php-handlersocket/
- Java
http://code.google.com/p/handlersocketforjava/
- Python
https://code.launchpad.net/~songofacandy/+junk/pyhandlersocket
- Ruby
https://github.com/winebarrel/ruby-handlersocket
https://github.com/miyucy/handlersocket
- JavaScript(Node.js)
https://github.com/koichik/node-handlersocket
The home of HandlerSocket is here:
https://github.com/ahiguti/HandlerSocket-Plugin-for-MySQL
More documents are available in docs-en/ and docs-ja/ directories.
-----------------------------------------------------------------
handlersocket_verbose (default = 10, min = 0, max = 10000)
Specify the logging verboseness.
-----------------------------------------------------------------
handlersocket_address (default = '')
Specify the address to bind. If empty, it binds to 0.0.0.0.
-----------------------------------------------------------------
handlersocket_port (default = '9998')
Specify the port to bind. This option is for the listener for
read requests. If empty, the listener is disabled.
-----------------------------------------------------------------
handlersocket_port_wr (default = '9999')
Specify the port to bind. This option is for the listener for
write requests. If empty, the listener is disabled.
-----------------------------------------------------------------
handlersocket_epoll (default = 1, min = 0, max = 1)
Specify whether handlersocket uses epoll for I/O multiplexing.
-----------------------------------------------------------------
handlersocket_threads (default = 16, min = 1, max = 3000)
Specify the number of handlersocket worker threads. This option
is for the listener for read requests. Recommended value is
(the number of CPU cores * 2).
-----------------------------------------------------------------
handlersocket_threads_wr (default = 1, min = 1, max = 3000)
Specify the number of handlersocket worker threads. This option
is for the listener for write requests. Recommended value is 1.
-----------------------------------------------------------------
handlersocket_timeout (default = 300, min = 30, max = 3600)
Specify the socket timeout in seconds.
-----------------------------------------------------------------
handlersocket_backlog (default = 32768, min = 5, max = 1000000)
Specify the length of the listen backlog.
-----------------------------------------------------------------
handlersocket_sndbuf (default = 0, min = 0, max = 1677216)
Specify the maximum socket send buffer in bytes. If 0, the
system-wide default value is set.
-----------------------------------------------------------------
handlersocket_rcvbuf (default = 0, min = 0, max = 1677216)
Specify the maximum socket receive buffer in bytes. If 0, the
system-wide default value is set.
-----------------------------------------------------------------
handlersocket_readsize (default = 0, min = 0, max = 1677216)
Specify the minimum length of the handlersocket request buffer.
Larger value can make handlersocket faster for large requests,
but can consume memory. The default value is possibly 4096.
-----------------------------------------------------------------
handlersocket_accept_balance (default = 0, min = 0, max = 10000)
When this option is set to non-zero, handlersocket tries to
balance accepted connections among threads. Non-zero is
recommended if you use persistent connections (i.e., connection
pooling on the client side).
-----------------------------------------------------------------
handlersocket_wrlock_timeout (default = 12, min = 0, max = 3600)
Specify the lock timeout in seconds. When a write request is
performed, handlersocket locks an advisory lock named
'handlersocket_wr'. This option sets the timeout for the
locking.
1. Building Handlersocket
Handlersocket mainly consists of libhsclient, handlersocket, and C++/Perl clients. libhsclient is a common library shared from both client and server(plugin). handlersocket is a MySQL daemon plugin.
To build Handlersocket, you need both MySQL source code and MySQL binary. It is not required to pre-build MySQL source code, but source itself is needed because Handlersocket depends on MySQL header files that only MySQL source distribution contains. MySQL binary is just a normal MySQL binary distribution. You can use official MySQL binaries provided by Oracle.
Since Handlersocket uses daemon plugin interface supported from MySQL 5.1,
MySQL 5.1 or higher version is required.
Please make sure that you use identical MySQL version between MySQL source
and MySQL binary. Otherwise you might encounter serious problems (i.e. server
crash, etc).
Here are steps to build Handlersocket.
* Get MySQL source code
* Get MySQL binary
* Build Handlersocket
$ ./autogen.sh
$ ./configure --with-mysql-source=/work/mysql-5.1.50 --with-mysql-bindir=/work/mysql-5.1.50-linux-x86_64-glibc23/bin --with-mysql-plugindir=/work/mysql-5.1.50-linux-x86_64-glibc23/lib/plugin
--with-mysql-source refers to the top of MySQL source directory,
--with-mysql-bindir refers to where MySQL binary executables (i.e.
mysql_config) are located, and --with-mysql-plugindir refers to a plugin
directory where plugin libraries (*.so) are installed.
$ make
$ sudo make install
Both libhsclient and the handlersocket plugin will be installed.
2. Using Handlersocket
Append configuration options for handlersocket to my.cnf.
[mysqld]
loose_handlersocket_port = 9998
# the port number to bind to (for read requests)
loose_handlersocket_port_wr = 9999
# the port number to bind to (for write requests)
loose_handlersocket_threads = 16
# the number of worker threads (for read requests)
loose_handlersocket_threads_wr = 1
# the number of worker threads (for write requests)
open_files_limit = 65535
# to allow handlersocket accept many concurrent
# connections, make open_files_limit as large as
# possible.
Log in to mysql as root, and execute the following query.
mysql> install plugin handlersocket soname 'handlersocket.so';
If handlersocket.so is successfully installed, it starts
accepting connections on port 9998 and 9999. Running
'show processlist' should show handlersocket worker threads.
-----------------------------------------------------------------
On the client side, you need to install libhsclient for c++ apps
and perl-Net-HandlerSocket for perl apps. They do not require
MySQL to compile.
$ ./autogen.sh
$ ./configure --disable-handlersocket-server
$ make
$ sudo make install
$ cd perl-Net-HandlerSocket
$ perl Makefile.PL
$ make
$ sudo make install
-----------------------------------------------------------------
Alternatively, you can use the rpm installation. If your OS
supports rpms, you can use the following commands to build and
install handlersocket rpm packages.
(Server side, installs HandlerSocket plugin)
$ ./autogen.sh
$ ./configure --with-mysql-source=/work/mysql-5.1.50 --with-mysql-bindir=/work/mysql-5.1.50-linux-x86_64-glibc23/bin --with-mysql-plugindir=/work/mysql-5.1.50-linux-x86_64-glibc23/lib/plugin
$ make rpm_cli
$ sudo rpm -U dist/RPMS/*/libhsclient*.rpm
$ make rpm_c
$ sudo rpm -U dist/RPMS/*/handlersocket*.rpm
(Client side, installs client libraries)
$ ./autogen.sh
$ ./configure --disable-handlersocket-server
$ make rpm_cli
$ sudo rpm -U dist/RPMS/*/libhsclient*.rpm
$ make rpm_perl
$ sudo rpm -U dist/RPMS/*/perl-Net-HandlerSocket*.rpm
-----------------------------------------------------------------
To open a connection to the handlersocket plugin, you need to
create a Net::HandlerSocket object.
use Net::HandlerSocket;
my $args = { host => 'localhost', port => 9998 };
my $hs = new Net::HandlerSocket($args);
-----------------------------------------------------------------
Before executing table operations, you need to open an index to
work with.
my $err = $hs->open_index(3, 'database1', 'table1', 'PRIMARY',
'f1,f2');
die $hs->get_error() if $res->[0] != 0;
The first argument for open_index is an integer value which is
used to identify an open table, which is only valid within the
same Net::HandlerSocket object. The 4th argument is the name of
index to open. If 'PRIMARY' is specified, the primary index is
open. The 5th argument is a comma-separated list of column names.
-----------------------------------------------------------------
To read a record from a table using an index, call the
execute_single method.
my $res = $hs->execute_single(3, '=', [ 'foo' ], 1, 0);
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
The first argument must be an integer which has specified as the
first argument for open_index on the same Net::HandlerSocket
object. The second argument specifies the search operation. The
current version of handlersocket supports '=', '>=', '<=', '>',
and '<'. The 3rd argument specifies the key to find, which must
an arrayref whose length is equal to or smaller than the number
of key columns of the index. The 4th and the 5th arguments
specify the maximum number of records to be retrieved, and the
number of records skipped before retrieving records. The columns
to be retrieved are specified by the 5th argument for the
corresponding open_index call.
The execute_single method always returns an arrayref. The first
element is the error code, which is 0 when no error is occured.
The remaining are the field values. If more than one record is
returned, it is flatten to an 1-dimensional array. For example,
when 5 records that have 3 columns are returned, you can retrieve
values using the following code.
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
for (my $row = 0; $row < 5; ++$row) {
for (my $col = 0; $col < 3; ++$col) {
my $value = $res->[$row * 5 + $col];
# ...
}
}
-----------------------------------------------------------------
To update or delete records, you need to specify more arguments
for the execute_single method. Note that the Net::HandlerSocket
object must be connected to a handlersocket worker for write
operations, which is port 9999 by default.
(For safety, the port 9998 only allows read operations, and the
port 9999 allows write operations also. The port 9999 allows
read operations too, but slower than 9998 because of record
locking etc.. Port numbers can be changed using the
'handlersocket_port' and the 'handlersocket_port_wr'
configuration options of mysqld.)
my $args = { host => 'localhost', port => 9999 };
my $hs = new Net::HandlerSocket($args);
my $res = $hs->execute_single(3, '=', [ 'bar' ], 1, 0, 'U',
[ 'fubar', 'hoge' ]);
die $hs->get_error() if $res->[0] != 0;
my $num_updated_rows = $res->[1];
my $res = $hs->execute_single(3, '=', [ 'baz' ], 1, 0, 'D');
die $hs->get_error() if $res->[0] != 0;
my $num_deleted_rows = $res->[1];
The 6th argument for execute_single specifies the modification
operation. The current version supports 'U' and 'D'. For the 'U'
operation, the 7th argument specifies the new value for the row.
The columns to be modified are specified by the 5th argument for
the corresponding open_index call. For the 'D' operation, the
7th argument can be omitted.
-----------------------------------------------------------------
The execute_single method can be used for inserting records also.
my $res = $hs->execute_single(3, '+', [ 'foo', 'bar', 'baz' ]);
die $hs->get_error() if $res->[0] != 0;
my $num_inserted_rows = $res->[1];
The 3rd argument must be an arrayref whose elements correspond to
the 5th argument for the corresponding open_index call. If there
is a column which is not appeared in the 5th argument for the
open_index, the default value for the column is set.
-----------------------------------------------------------------
Multiple operations can be executed in a single call. Executing
multiple operations in a single call is much faster than
executing them separatedly.
my $rarr = $hs->execute_multi([
[ 0, '>=', [ 'foo' ], 5, 0 ],
[ 2, '=', [ 'bar' ], 1, 0 ],
[ 4, '<', [ 'baz' ], 10, 5 ],
]);
for my $res (@$rarr) {
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
# ...
}
-----------------------------------------------------------------
When an error is occured, the first element of the returned
arrayref becomes a non-zero value. A negative value indicates
that an I/O error is occured and the Net::HandlerSocket object
should be disposed. A positive value means that the connection is
still active and the Net::HandlerSocket object can be reused
later.
----------------------------------------------------------------------------
The HandlerSocket protocol
----------------------------------------------------------------------------
Basic syntax
- The HandlerSocket protocol is line-based. Each line ends with LF(0x0a).
- Each line consists a concatenation of tokens separated by HT(0x09).
- A token is either NULL or an encoded string. Note that you need to
distinguish NULL from an empty string, as most DBMs does so.
- NULL is expressed as a single NUL(0x00).
- An encoded string is a string with the following encoding rules.
- Characters in the range [0x10 - 0xff] are not encoded.
- A character in the range [0x00 - 0x0f] is prefixed by 0x01 and
shifted by 0x40. For example, 0x03 is encoded as 0x01 0x43.
- Note that a string can be empty. A continuation of 0x09 0x09 means that
there is an empty string between them. A continuation of 0x09 0x0a means
that there is an empty string at the end of the line.
----------------------------------------------------------------------------
Request and Response
- The HandlerSocket protocol is a simple request/response protocol. After a
connection is established, the client side sends a request, and then the
server side sends a response.
- A request/response consists of a single line.
- Requests can be pipelined; That is, you can send multiple requests (ie.
lines) at one time, and receive responses for them at one time.
----------------------------------------------------------------------------
'open_index' request
The 'open_index' request has the following syntax.
P <indexid> <dbname> <tablename> <indexname> <columns>
- <indexid> is a number in decimal.
- <dbname>, <tablename>, and <indexname> are strings. To open the primary
key, use PRIMARY as <indexname>.
- <columns> is a comma-separated list of column names.
Once an 'open_index' request is issued, the HandlerSocket plugin opens the
specified index and keep it open until the client connection is closed. Each
open index is identified by <indexid>. If <indexid> is already open, the old
open index is closed. You can open the same combination of <dbname>
<tablename> <indexname> multple times, possibly with different <columns>.
For efficiency, keep <indexid> small as far as possible.
----------------------------------------------------------------------------
Getting data
The 'find' request has the following syntax.
<indexid> <op> <vlen> <v1> ... <vn> <limit> <offset>
- <indexid> is a number. This number must be an <indexid> specified by a
'open_index' request executed previously on the same connection.
- <op> specifies the comparison operation to use. The current version of
HandlerSocket supports '=', '>', '>=', '<', and '<='.
- <vlen> indicates the length of the trailing parameters <v1> ... <vn>. This
must be smaller than or equal to the number of index columns specified by
specified by the corresponding 'open_index' request.
- <v1> ... <vn> specify the index column values to fetch.
- <limit> and <offset> are numbers. These parameters can be omitted. When
omitted, it works as if 1 and 0 are specified.
----------------------------------------------------------------------------
Updating/Deleting data
The 'find_modify' request has the following syntax.
<indexid> <op> <vlen> <v1> ... <vn> <limit> <offset> <mop> <m1> ... <mk>
- <mop> is either 'U' (update) or 'D' (delete).
- <m1> ... <mk> specifies the column values to set. The length of <m1> ...
<mk> must be smaller than or equal to the length of <columns> specified by
the corresponding 'open_index' request. If <mop> is 'D', these parameters
are ignored.
----------------------------------------------------------------------------
Inserting data
The 'insert' request has the following syntax.
<indexid> '+' <vlen> <v1> ... <vn>
- <vlen> indicates the length of the trailing parameters <v1> ... <vn>. This
must be smaller than or equal to the length of <columns> specified by the
corresponding 'open_index' request.
- <v1> ... <vn> specify the column values to set. For columns not in
<columns>, the default values for each column are set.
----------------------------------------------------------------------------
Response syntax
HandlerSocket returns a response of the following syntax for each request.
<errorcode> <numcolumns> <r1> ... <rn>
- <errorcode> indicates whether the request has successfully executed or not.
'0' means success. Non-zero means an error.
- <numcolumns> indicates the number of columns of the result set.
- <r1> ... <rn> is the result set. The length of <r1> ... <rn> is always a
multiple of <numcolumns>. It is possible that <r1> ... <rn> is empty.
If <errorcode> is non-zero, <numcolumns> is always 1 and <r1> indicates a
human-readable error message, though sometimes <r1> is not provided.
----------------------------------------------------------------------------
Response for 'open_index'
If 'open_index' is succeeded, HandlerSocket returns a line of the following
syntax.
0 1
----------------------------------------------------------------------------
Response for 'find'
If 'find' is succeeded, HandlerSocket returns a line of the following
syntax.
0 <numcolumns> <r1> ... <rn>
- <numcolumns> always equals to the length of <columns> of the corresponding
'open_index' request.
- <r1> ... <rn> is the result set. If N rows are found, the length of <r1>
... <rn> becomes ( <numcolumns> * N ).
----------------------------------------------------------------------------
Response for 'find_modify'
If 'find_modify' is succeeded, HandlerSocket returns a line of the following
syntax.
0 1 <nummod>
- <nummod> is the number of modified rows.
----------------------------------------------------------------------------
Response for 'insert'
If 'insert' is succeeded, HanderSocket returns a line of the following
syntax.
0 1
-----------------------------------------------------------------
ソースコードの利用にあたっての免責事項
本ソフトウェアの開発者および株式会社ディー・エヌ・エーは、本フト
ウェアの不稼動、稼動不良を含む法律上の瑕疵担保責任、その他保証責
任を負わないものとします。また、本ソフトウエアの開発者および株式
会社ディー・エヌ・エーは、本ソフトウェアの商品性、またはお客様の
特定の目的に対する適合性について、いかなる保証も負わないものとし
ます。
-----------------------------------------------------------------
handlersocket pluginについて
mysqlサーバに常駐し、innodb等のストレージエンジンへの直接のアクセ
スを提供するプラグインです。handlersocketプラグインは自前のリス
ナーを持ち、専用のクライアントライブラリ(libhsclient)を使ってそれ
にアクセスします。
mysqlの標準クライアントライブラリ(libmysql)を使ったアクセスと比べ
て、以下のような利点があります。
・接続あたりに消費するリソースが少ないため、同時接続数が事実上無
制限。したがって接続数を気にせず持続接続を使えます。
・高速(単純な参照クエリで3倍〜10倍程度)。
・通信プロトコルがコンパクト。libmysqlを使うとデータ転送時にレ
コード名などが付随するために通信内容が冗長ですが、libhsclientで
はデータのみが転送されるため、帯域消費が少なくなります。場合に
よっては10倍以上libmysqlのほうが冗長になります。
現在のバージョンでは以下のような処理をサポートしています。
・指定された索引について、指定された値と完全一致するようなレコー
ドを取得。(SELECT ??? FROM tbl WHERE k1 = v1 AND k2 = v2...)。
索引を使わない検索はサポートしていません。
・指定された索引について、指定された値の位置の前後のレコードを取
得。(SELECT ??? FROM tbl WHERE k1 >= v1 LIMIT 100)
・前述のような手段で取得したレコードに対するUPDATEとDELETE
・レコードのINSERT
以下のような言語をサポートします。
・C++。libhsclientをリンクします。
・Perl。Net::HandlerSocketをuseします。
現在のバージョンではGNU/Linuxでのみ動作します。
-----------------------------------------------------------------
既知の問題
・killでhandlersocketスレッドを殺すと、スレッド数が減ったまま回復
しません。
-----------------------------------------------------------------
HandlerSocketプラグインのビルド方法(RPMを使わない方法)
以下のようにしてconfigureを実行します。
$ ./autogen.sh
$ ./configure --with-mysql-source=/work/mysql-5.1.50 --with-mysql-bindir=/work/mysql-5.1.50-linux-x86_64-glibc23/bin --with-mysql-plugindir=/work/mysql-5.1.50-linux-x86_64-glibc23/lib/plugin
ここで--with-mysql-sourceにはMySQLのソースコードのトップディレク
トリを指定します。--with-mysql-bindirにはインストール済みのMySQL
のmysql_configコマンドが有るディレクトリを指定します。
その後以下のようにビルド・インストールします。
$ make
$ sudo make install
-----------------------------------------------------------------
クライアントライブラリのビルド方法(RPMを使わない方法)
クライアントライブラリをビルドする際には、MySQLのソースコードは
必要ありません。またMySQLがインストールされている必要もありません。
$ ./autogen.sh
$ ./configure --disable-handlersocket-server
$ make
$ sudo make install
$ cd perl-Net-HandlerSocket
$ perl Makefile.PL
$ make
$ sudo make install
-----------------------------------------------------------------
ビルド方法(RPM)
以下のように実行すれば、rpmパッケージがビルド&インストールされま
す。
(MySQLサーバ側、HandlerSocketプラグインをインストールする)
$ ./autogen.sh
$ ./configure --with-mysql-source=/work/mysql-5.1.50 --with-mysql-bindir=/work/mysql-5.1.50-linux-x86_64-glibc23/bin --with-mysql-plugindir=/work/mysql-5.1.50-linux-x86_64-glibc23/lib/plugin
$ make rpm_cli
$ sudo rpm -U dist/RPMS/*/libhsclient*.rpm
$ make rpm_c
$ sudo rpm -U dist/RPMS/*/handlersocket*.rpm
(クライアント側、クライアントライブラリをインストールする)
$ ./autogen.sh
$ ./configure --disable-handlersocket-server
$ make rpm_cli
$ sudo rpm -U dist/RPMS/*/libhsclient*.rpm
$ make rpm_perl
$ sudo rpm -U dist/RPMS/*/perl-Net-HandlerSocket*.rpm
-----------------------------------------------------------------
起動
mysqlを起動した状態で、mysqlの設定ファイル(my.cnf等)に以下の内容を
追加します。
[mysqld]
handlersocket_port = 9998
# handlersocketが接続を受け付けるポート(参照系リクエスト用)
handlersocket_port_wr = 9999
# handlersocketが接続を受け付けるポート(更新系リクエスト用)
handlersocket_address =
# handlersocketがバインドするアドレス(空のままでOK)
handlersocket_verbose = 0
# デバッグ用
handlersocket_timeout = 300
# 通信タイムアウト(秒)
handlersocket_threads = 16
# handlersocketのワーカースレッド数
thread_concurrency = 128
# handlersocketが幾つかのスレッドを占有するため、大きめの
# 値を指定してください
open_files_limit = 65535
# ソケットを大量に開けるようにするため、大きめの値を指定し
# てください
以下のクエリを実行します。
mysql> install plugin handlersocket soname 'handlersocket.so';
Query OK, 0 rows affected (0.06 sec)
以上でhandlersocketへクライアントからアクセスできるようになります。
-----------------------------------------------------------------
handlersocketプラグインへの接続を開くには、Net::HandlerSocketオブ
ジェクトを作成します。
use Net::HandlerSocket;
my $args = { host => 'localhost', port => 9998 };
my $hs = new Net::HandlerSocket($args);
-----------------------------------------------------------------
検索などの命令を実行する前に、処理対象となる索引を開く必要があり
ます。
my $err = $hs->open_index(3, 'database1', 'table1', 'PRIMARY',
'f1,f2');
die $hs->get_error() if $res->[0] != 0;
最初の引数は開く索引に付ける番号です。付けた番号は同一の
Net::HandlerSocketオブジェクトについてのみ有効です。第4引数は開く
索引の名前で、「PRIMARY」が指定され場合はプライマリキーが開かれま
す。第5引数は「,」で区切られた列名のリストです。
-----------------------------------------------------------------
テーブルから索引を使って行を取得するには、execute_singleメソッド
を呼びます。
my $res = $hs->execute_single(3, '=', [ 'foo' ], 1, 0);
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
最初の引数は索引の番号で、同じNet::HandlerSocketオブジェクトへ
open_indexで付けたものでなければなりません。第2引数には操作を指定
します。現在のバージョンでは、「=」、「>=」、「<=」、「>」、「<」
の操作が利用可能です。第3引数は配列への参照で、これは探すべき行の
キー値を指定します。配列の長さは索引のキーの個数と同じか少ない数
でなければなりません。第4引数と第5引数はそれぞれ、取得する最大行
数、取得前に読み飛ばす行数を指定します。取得される列は対応する
open_index呼び出しの第5引数で指定されたものになります。
execute_singleメソッドは常に配列への参照を返します。最初の要素は
エラーコードで、これが0ならば成功を表します。残りの要素は列の値で
す。もし取得されたデータが複数行の場合は、それが一つの配列へ連結
された形で格納されています。例えば、5行3列のデータの場合、次のよ
うなコードによってその内容を取得できます。
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
for (my $row = 0; $row < 5; ++$row) {
for (my $col = 0; $col < 3; ++$col) {
my $value = $res->[$row * 5 + $col];
# ...
}
}
-----------------------------------------------------------------
行を更新または削除するには、更に多くの引数を指定して
execute_singleメソッドを呼び出します。書き込み処理をするには、
対象のNet::HandlerSocketオブジェクトは更新用handlersocketワーカ(既
定ではポート9999)へ接続されたものでなくてはなりません。
(安全のため、ポート9998は参照処理だけを受け付け、ポート9999は更新
処理も受け付けるようになっています。ポート9999は参照処理も受け付
けますが、レコードロック等の影響で遅くなります。ポート番号は
mysqldの「handlersocket_port」と「handlersocket_port_wr」の設定項
目で変更できます。)
my $args = { host => 'localhost', port => 9999 };
my $hs = new Net::HandlerSocket($args);
my $res = $hs->execute_single(3, '=', [ 'bar' ], 1, 0, 'U',
[ 'fubar', 'hoge' ]);
die $hs->get_error() if $res->[0] != 0;
my $num_updated_rows = $res->[1];
my $res = $hs->execute_single(3, '=', [ 'baz' ], 1, 0, 'D');
die $hs->get_error() if $res->[0] != 0;
my $num_deleted_rows = $res->[1];
execute_singleの第6引数は変更処理の種類を指定します。現在のバー
ジョンでは「U」と「D」が利用可能です。「U」については、第7引数で
新しい値を指定します。更新される列は、対応するopen_index呼び出し
の第5引数で指定されたものになります。「D」については第7引数は省
略できます。
-----------------------------------------------------------------
execute_singleメソッドは列の挿入にも使用できます。
my $res = $hs->execute_single(3, '+', [ 'foo', 'bar', 'baz' ]);
die $hs->get_error() if $res->[0] != 0;
my $num_inserted_rows = $res->[1];
第3引数は、対応するopen_index呼び出しの第5引数の列リストと同じだ
けの長さの配列への参照でなければなりません。open_index呼び出しの
第5引数に指定されていない列については、その列の既定値がセットされ
ます。
-----------------------------------------------------------------
execute_multiメソッドを使えば、複数のリクエストを一つの呼び出しで
実行することができます。これはリクエストを個別に実行するより高速
です。
my $rarr = $hs->execute_multi([
[ 0, '>=', [ 'foo' ], 5, 0 ],
[ 2, '=', [ 'bar' ], 1, 0 ],
[ 4, '<', [ 'baz' ], 10, 5 ],
]);
for my $res (@$rarr) {
die $hs->get_error() if $res->[0] != 0;
shift(@$res);
# ...
}
-----------------------------------------------------------------
エラーが起こると返値の配列参照の最初の要素が0以外になります。負の
数の場合はI/Oエラーが起こったことを示し、その場合はその
Net::HandlerSocketオブジェクトは破棄するべきです。正の値の場合は
接続は維持されているため、そのオブジェクトはそれ以後も再利用でき
ます。
-----------------------------------------------------------------
handlersocketの通信プロトコル
-----------------------------------------------------------------
構文
・コマンド行は改行(LF)で終わる。
・コマンド行は複数のトークンからなり、トークン間はTABで区切られる。
・トークンはNULLトークンか、文字列トークンのいずれか。
・NULLトークンは単一のNUL文字であらわされる。
・文字列トークンは、0バイト以上の文字列であらわされる。ただし0x10
未満の文字については0x01を前置し、0x40を加えたコードであらわさ
れる。それ以外の文字はその文字自身のコードであらわされる。
-----------------------------------------------------------------
リクエストとレスポンス
・接続が確立した直後の状態では、まずクライアントがコマンド行を送
る。(リクエスト)
・サーバはクライアントが送ったリクエストと丁度同じ数のコマンド行
を返す。(レスポンス)
・リクエストはパイプライン化してよい。つまりクライアントは前に
送ったリクエストに対する返事を待たずに次のリクエストを送っても
よい。
-----------------------------------------------------------------
リクエスト
・open_index命令は次のような構文を持つ。
'P' indexid dbname tablename indexname fieldlist
indexidは開いている索引に付けられる番号で、同一接続上で後に実行
する命令の、対象索引を指定するために使われる。dbname、tablename、
indexnameはそれぞれ開きたいDB、テーブル、索引の名前。索引の名前
として"PRIMARY"を指定するとプライマリキーが開かれる。fieldlist
はカンマ区切りの列名のリスト。
・find命令は次のような構文を持つ。
indexid op nflds v1 ... vn limit offset
indexidは実行対象の索引を指定する。opは索引検索の演算子(後述)。
v1からvnは可変長で、その個数はnflds。nfldsはindexidで指定された
open_index命令のindexnameの索引のfieldlistのフィールド数に等し
いか小さくなくてはならない。m2からmkは可変長で、その個数は
indexidで指定されたopen_index命令が発行された際のfieldlistに一
致しなければならない。コマンド行のlimit以降は省略できる。limit
とoffsetは、検索条件に合致する列のうちレスポンスに返す列数の上
限と、スキップする列数。limitとoffsetを省略した場合はそれぞれ1
と0が指定されたときと同じ動作をする。find命令はレスポンスとして、
条件に合致した列のリストを返す。opとして指定できる演算子は次の
とおり。
'=' - v1 ... vnと一致するものを取得
'>' - v1 ... vnより大きいものを昇順に取得
'>=' - v1 ... vnに一致するか大きいものを昇順に取得
'<' - v1 ... vnより小さいものを降順に取得
'<=' - v1 ... vnに一致するか等しいものを降順に取得
nfldsが1より大きい(v1 ... vnが2個以上)ときは辞書式順序で比較さ
れる。
・find_modify命令は次のような構文を持つ。
indexid op nflds v1 ... vn limit offset modop m1 ... mk
modopより前の部分はfind命令と同等で、これによって操作対象の行を
指定する。その操作対象の行に対しmodopで指定された変更処理を実行
する。m1 ... mkは可変長で、省略できる。modopは次いずれか。
'U' - indexidで指定されたopen_index命令のfieldlist列
の内容を、m1 ... mkの値で更新する。
'D' - 対象の行を削除する。m1 ... mkの値は無視される。
・insert命令はのような構文を持つ。
indexid '+' nflds v1 ... vn
indexidで指定されたテーブルに、列を挿入する。v1 ... vnは可変長
で、その個数はnflds。nfldsはindexidで指定されたopen_index命令の
indexnameの索引のfieldlistのフィールド数に等しいか小さくなくて
はならない。
-----------------------------------------------------------------
レスポンス
・open_index命令が成功したとき、レスポンスは次の構文を持つ。
'0' '1'
・find命令が成功したとき、レスポンスは次の構文を持つ。
'0' nflds v1 ... vn
nfldsは結果セットの列の数をあらわす。v1 ... vnは可変長で、その
長さはnfldsの整数倍。v1 ... vnは空のこともあり、それは条件に合
致するレコードが存在しなかったことをあらわす。結果セットが複数
行になったときはv1 ... vnの長さがnfldsの2倍以上となり、最初の
行から順にv1 ... vnにセットされる。
・modify命令が成功したとき、レスポンスは次の構文を持つ。
'0' '1' nummod
nummodは変更が施された行数。nummodが0のときは変更された行が無
かったことをあらわす。
・insert命令が成功したとき、レスポンスは次の構文を持つ。
'0' '1'
・命令が失敗したとき、レスポンスは命令に関わらず次の構文を持つ。
err '1' message
errは0以外の数値で、エラーコードをあらわす。messageは人間可読な
エラーメッセージ。ただしmessageが無いこともある。
Copyright (c) 2010 DeNA Co.,Ltd.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of DeNA Co.,Ltd. nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY DeNA Co.,Ltd. "AS IS" AND ANY EXPRESS OR
IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
EVENT SHALL DeNA Co.,Ltd. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
CXXFLAGS += -fimplicit-templates
pkgplugindir = $(PLUGIN_DIR)
noinst_HEADERS = database.hpp hstcpsvr.hpp hstcpsvr_worker.hpp mysql_incl.hpp
pkgplugin_LTLIBRARIES = handlersocket.la
handlersocket_la_LDFLAGS = -module ../libhsclient/libhsclient.la
handlersocket_la_CFLAGS = $(MYSQL_INC) $(MYSQL_CFLAGS) $(AM_CFLAGS) \
-I$(srcdir)/../libhsclient
handlersocket_la_CXXFLAGS = $(MYSQL_INC) $(MYSQL_CFLAGS) $(AM_CFLAGS) \
-I$(srcdir)/../libhsclient
handlersocket_la_SOURCES = database.cpp handlersocket.cpp \
hstcpsvr_worker.cpp hstcpsvr.cpp
MYSQL_INC = HANDLERSOCKET_MYSQL_INC
MYSQL_LIB = HANDLERSOCKET_MYSQL_LIB
CXX = g++ -Wall -g -fno-rtti -fno-exceptions -fPIC -DPIC
LIBS = $(MYSQL_LIB) -lhsclient -lpthread -lz
CXXFLAGS = -I/usr/include/handlersocket $(MYSQL_INC)
LDFLAGS =
CXXFLAGS += -O3 -DNDEBUG
HANDLERSOCKET_OBJS = database.o hstcpsvr.o hstcpsvr_worker.o
all: handlersocket.so
handlersocket.so: $(HANDLERSOCKET_OBJS) handlersocket.cpp
$(CXX) $(CXXFLAGS) -fno-strict-aliasing -shared $^ -o $@ $(LDFLAGS) \
-Wl,-soname -Wl,$@ $(LIBS)
clean:
rm -f *.a *.so *.o
LIBDIR = $(shell \
if [ -e /usr/lib64/mysql ]; then echo /usr/lib64; else echo /usr/lib; fi)
install: handlersocket.so
sudo sh -c 'ulimit -c unlimited ; /etc/init.d/mysql stop ; \
cp handlersocket.so handlersocket.so.cpy && \
mv handlersocket.so.cpy \
$(LIBDIR)/mysql/plugin/handlersocket.so && \
/etc/init.d/mysql start'
This diff is collapsed.
// vim:sw=2:ai
/*
* Copyright (C) 2010 DeNA Co.,Ltd.. All rights reserved.
* See COPYRIGHT.txt for details.
*/
#ifndef DENA_DATABASE_HPP
#define DENA_DATABASE_HPP
#include <string>
#include <memory>
#include <vector>
#include <stdint.h>
#include "string_buffer.hpp"
#include "string_ref.hpp"
#include "config.hpp"
namespace dena {
struct database_i;
typedef std::auto_ptr<volatile database_i> database_ptr;
struct dbcontext_i;
typedef std::auto_ptr<dbcontext_i> dbcontext_ptr;
struct database_i {
virtual ~database_i() { }
virtual dbcontext_ptr create_context(bool for_write) volatile = 0;
virtual void stop() volatile = 0;
virtual const config& get_conf() const volatile = 0;
static database_ptr create(const config& conf);
};
struct prep_stmt {
typedef std::vector<uint32_t> fields_type;
private:
dbcontext_i *dbctx; /* must be valid while *this is alive */
size_t table_id; /* a prep_stmt object holds a refcount of the table */
size_t idxnum;
fields_type ret_fields;
fields_type filter_fields;
public:
prep_stmt();
prep_stmt(dbcontext_i *c, size_t tbl, size_t idx, const fields_type& rf,
const fields_type& ff);
~prep_stmt();
prep_stmt(const prep_stmt& x);
prep_stmt& operator =(const prep_stmt& x);
public:
size_t get_table_id() const { return table_id; }
size_t get_idxnum() const { return idxnum; }
const fields_type& get_ret_fields() const { return ret_fields; }
const fields_type& get_filter_fields() const { return filter_fields; }
};
struct dbcallback_i {
virtual ~dbcallback_i () { }
virtual void dbcb_set_prep_stmt(size_t pst_id, const prep_stmt& v) = 0;
virtual const prep_stmt *dbcb_get_prep_stmt(size_t pst_id) const = 0;
virtual void dbcb_resp_short(uint32_t code, const char *msg) = 0;
virtual void dbcb_resp_short_num(uint32_t code, uint32_t value) = 0;
virtual void dbcb_resp_begin(size_t num_flds) = 0;
virtual void dbcb_resp_entry(const char *fld, size_t fldlen) = 0;
virtual void dbcb_resp_end() = 0;
virtual void dbcb_resp_cancel() = 0;
};
enum record_filter_type {
record_filter_type_skip = 0,
record_filter_type_break = 1,
};
struct record_filter {
record_filter_type filter_type;
string_ref op;
uint32_t ff_offset; /* offset in filter_fields */
string_ref val;
record_filter() : filter_type(record_filter_type_skip), ff_offset(0) { }
};
struct cmd_exec_args {
const prep_stmt *pst;
string_ref op;
const string_ref *kvals;
size_t kvalslen;
uint32_t limit;
uint32_t skip;
string_ref mod_op;
const string_ref *uvals; /* size must be pst->retfieelds.size() */
const record_filter *filters;
cmd_exec_args() : pst(0), kvals(0), kvalslen(0), limit(0), skip(0),
uvals(0), filters(0) { }
};
struct dbcontext_i {
virtual ~dbcontext_i() { }
virtual void init_thread(const void *stack_bottom,
volatile int& shutdown_flag) = 0;
virtual void term_thread() = 0;
virtual bool check_alive() = 0;
virtual void lock_tables_if() = 0;
virtual void unlock_tables_if() = 0;
virtual bool get_commit_error() = 0;
virtual void clear_error() = 0;
virtual void close_tables_if() = 0;
virtual void table_addref(size_t tbl_id) = 0; /* TODO: hide */
virtual void table_release(size_t tbl_id) = 0; /* TODO: hide */
virtual void cmd_open_index(dbcallback_i& cb, size_t pst_id, const char *dbn,
const char *tbl, const char *idx, const char *retflds,
const char *filflds) = 0;
virtual void cmd_exec_on_index(dbcallback_i& cb, const cmd_exec_args& args)
= 0;
virtual void set_statistics(size_t num_conns, size_t num_active) = 0;
};
};
extern unsigned long long int open_tables_count;
extern unsigned long long int close_tables_count;
extern unsigned long long int lock_tables_count;
extern unsigned long long int unlock_tables_count;
#if 0
extern unsigned long long int index_exec_count;
#endif
#endif
// vim:sw=2:ai
/*
* Copyright (C) 2010 DeNA Co.,Ltd.. All rights reserved.
* See COPYRIGHT.txt for details.
*/
#include <memory>
#include <string>
#include <stdio.h>
#include "config.hpp"
#include "hstcpsvr.hpp"
#include "string_util.hpp"
#include "mysql_incl.hpp"
#define DBG_LOG \
if (dena::verbose_level >= 100) { \
fprintf(stderr, "%s %p\n", __PRETTY_FUNCTION__, this); \
}
#define DBG_DO(x) if (dena::verbose_level >= 100) { x; }
#define DBG_DIR(x)
using namespace dena;
static char *handlersocket_address = 0;
static char *handlersocket_port = 0;
static char *handlersocket_port_wr = 0;
static unsigned int handlersocket_epoll = 1;
static unsigned int handlersocket_threads = 32;
static unsigned int handlersocket_threads_wr = 1;
static unsigned int handlersocket_timeout = 30;
static unsigned int handlersocket_backlog = 32768;
static unsigned int handlersocket_sndbuf = 0;
static unsigned int handlersocket_rcvbuf = 0;
static unsigned int handlersocket_readsize = 0;
static unsigned int handlersocket_accept_balance = 0;
static unsigned int handlersocket_wrlock_timeout = 0;
static char *handlersocket_plain_secret = 0;
static char *handlersocket_plain_secret_wr = 0;
struct daemon_handlersocket_data {
hstcpsvr_ptr hssvr_rd;
hstcpsvr_ptr hssvr_wr;
};
static int
daemon_handlersocket_init(void *p)
{
DENA_VERBOSE(10, fprintf(stderr, "handlersocket: initialized\n"));
config conf;
conf["use_epoll"] = handlersocket_epoll ? "1" : "0";
if (handlersocket_address) {
conf["host"] = handlersocket_address;
}
if (handlersocket_port) {
conf["port"] = handlersocket_port;
}
/*
* unix domain socket
* conf["host"] = "/";
* conf["port"] = "/tmp/handlersocket";
*/
if (handlersocket_threads > 0) {
conf["num_threads"] = to_stdstring(handlersocket_threads);
} else {
conf["num_threads"] = "1";
}
conf["timeout"] = to_stdstring(handlersocket_timeout);
conf["listen_backlog"] = to_stdstring(handlersocket_backlog);
conf["sndbuf"] = to_stdstring(handlersocket_sndbuf);
conf["rcvbuf"] = to_stdstring(handlersocket_rcvbuf);
conf["readsize"] = to_stdstring(handlersocket_readsize);
conf["accept_balance"] = to_stdstring(handlersocket_accept_balance);
conf["wrlock_timeout"] = to_stdstring(handlersocket_wrlock_timeout);
std::auto_ptr<daemon_handlersocket_data> ap(new daemon_handlersocket_data);
if (handlersocket_port != 0 && handlersocket_port_wr != handlersocket_port) {
conf["port"] = handlersocket_port;
if (handlersocket_plain_secret) {
conf["plain_secret"] = handlersocket_plain_secret;
}
ap->hssvr_rd = hstcpsvr_i::create(conf);
ap->hssvr_rd->start_listen();
}
if (handlersocket_port_wr != 0) {
if (handlersocket_threads_wr > 0) {
conf["num_threads"] = to_stdstring(handlersocket_threads_wr);
}
conf["port"] = handlersocket_port_wr;
conf["for_write"] = "1";
conf["plain_secret"] = "";
if (handlersocket_plain_secret_wr) {
conf["plain_secret"] = handlersocket_plain_secret_wr;
}
ap->hssvr_wr = hstcpsvr_i::create(conf);
ap->hssvr_wr->start_listen();
}
st_plugin_int *const plugin = static_cast<st_plugin_int *>(p);
plugin->data = ap.release();
return 0;
}
static int
daemon_handlersocket_deinit(void *p)
{
DENA_VERBOSE(10, fprintf(stderr, "handlersocket: terminated\n"));
st_plugin_int *const plugin = static_cast<st_plugin_int *>(p);
daemon_handlersocket_data *ptr =
static_cast<daemon_handlersocket_data *>(plugin->data);
delete ptr;
return 0;
}
static struct st_mysql_daemon daemon_handlersocket_plugin = {
MYSQL_DAEMON_INTERFACE_VERSION
};
static MYSQL_SYSVAR_UINT(verbose, dena::verbose_level, 0,
"0..10000", 0, 0, 10 /* default */, 0, 10000, 0);
static MYSQL_SYSVAR_UINT(epoll, handlersocket_epoll, PLUGIN_VAR_READONLY,
"0..1", 0, 0, 1 /* default */, 0, 1, 0);
static MYSQL_SYSVAR_STR(address, handlersocket_address,
PLUGIN_VAR_READONLY | PLUGIN_VAR_MEMALLOC, "", NULL, NULL, NULL);
static MYSQL_SYSVAR_STR(port, handlersocket_port,
PLUGIN_VAR_READONLY | PLUGIN_VAR_MEMALLOC, "", NULL, NULL, NULL);
static MYSQL_SYSVAR_STR(port_wr, handlersocket_port_wr,
PLUGIN_VAR_READONLY | PLUGIN_VAR_MEMALLOC, "", NULL, NULL, NULL);
static MYSQL_SYSVAR_UINT(threads, handlersocket_threads, PLUGIN_VAR_READONLY,
"1..3000", 0, 0, 16 /* default */, 1, 3000, 0);
static MYSQL_SYSVAR_UINT(threads_wr, handlersocket_threads_wr,
PLUGIN_VAR_READONLY, "1..3000", 0, 0, 1 /* default */, 1, 3000, 0);
static MYSQL_SYSVAR_UINT(timeout, handlersocket_timeout, PLUGIN_VAR_READONLY,
"30..3600", 0, 0, 300 /* default */, 30, 3600, 0);
static MYSQL_SYSVAR_UINT(backlog, handlersocket_backlog, PLUGIN_VAR_READONLY,
"5..1000000", 0, 0, 32768 /* default */, 5, 1000000, 0);
static MYSQL_SYSVAR_UINT(sndbuf, handlersocket_sndbuf, PLUGIN_VAR_READONLY,
"0..16777216", 0, 0, 0 /* default */, 0, 16777216, 0);
static MYSQL_SYSVAR_UINT(rcvbuf, handlersocket_rcvbuf, PLUGIN_VAR_READONLY,
"0..16777216", 0, 0, 0 /* default */, 0, 16777216, 0);
static MYSQL_SYSVAR_UINT(readsize, handlersocket_readsize, PLUGIN_VAR_READONLY,
"0..16777216", 0, 0, 0 /* default */, 0, 16777216, 0);
static MYSQL_SYSVAR_UINT(accept_balance, handlersocket_accept_balance,
PLUGIN_VAR_READONLY, "0..10000", 0, 0, 0 /* default */, 0, 10000, 0);
static MYSQL_SYSVAR_UINT(wrlock_timeout, handlersocket_wrlock_timeout,
PLUGIN_VAR_READONLY, "0..3600", 0, 0, 12 /* default */, 0, 3600, 0);
static MYSQL_SYSVAR_STR(plain_secret, handlersocket_plain_secret,
PLUGIN_VAR_READONLY | PLUGIN_VAR_MEMALLOC, "", NULL, NULL, NULL);
static MYSQL_SYSVAR_STR(plain_secret_wr, handlersocket_plain_secret_wr,
PLUGIN_VAR_READONLY | PLUGIN_VAR_MEMALLOC, "", NULL, NULL, NULL);
/* warning: type-punning to incomplete type might break strict-aliasing
* rules */
static struct st_mysql_sys_var *daemon_handlersocket_system_variables[] = {
MYSQL_SYSVAR(verbose),
MYSQL_SYSVAR(address),
MYSQL_SYSVAR(port),
MYSQL_SYSVAR(port_wr),
MYSQL_SYSVAR(epoll),
MYSQL_SYSVAR(threads),
MYSQL_SYSVAR(threads_wr),
MYSQL_SYSVAR(timeout),
MYSQL_SYSVAR(backlog),
MYSQL_SYSVAR(sndbuf),
MYSQL_SYSVAR(rcvbuf),
MYSQL_SYSVAR(readsize),
MYSQL_SYSVAR(accept_balance),
MYSQL_SYSVAR(wrlock_timeout),
MYSQL_SYSVAR(plain_secret),
MYSQL_SYSVAR(plain_secret_wr),
0
};
static SHOW_VAR hs_status_variables[] = {
{"table_open", (char*) &open_tables_count, SHOW_LONGLONG},
{"table_close", (char*) &close_tables_count, SHOW_LONGLONG},
{"table_lock", (char*) &lock_tables_count, SHOW_LONGLONG},
{"table_unlock", (char*) &unlock_tables_count, SHOW_LONGLONG},
#if 0
{"index_exec", (char*) &index_exec_count, SHOW_LONGLONG},
#endif
{NullS, NullS, SHOW_LONG}
};
static int show_hs_vars(THD *thd, SHOW_VAR *var, char *buff)
{
var->type= SHOW_ARRAY;
var->value= (char *) &hs_status_variables;
return 0;
}
static SHOW_VAR daemon_handlersocket_status_variables[] = {
{"Hs", (char*) show_hs_vars, SHOW_FUNC},
{NullS, NullS, SHOW_LONG}
};
mysql_declare_plugin(handlersocket)
{
MYSQL_DAEMON_PLUGIN,
&daemon_handlersocket_plugin,
"handlersocket",
"higuchi dot akira at dena dot jp",
"",
PLUGIN_LICENSE_BSD,
daemon_handlersocket_init,
daemon_handlersocket_deinit,
0x0100 /* 1.0 */,
daemon_handlersocket_status_variables,
daemon_handlersocket_system_variables,
0
}
mysql_declare_plugin_end;
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment