Addresses #307

Locktree now accepts max memory parameter.
This counts only 'toku_points' and payloads.
Whenever it goes over the allowed memory, it will return ENOMEM as
BDB does.

Tests were only updated to compile correctly (and return EINVAL
if max size is 0).
Tests do not yet test whether the max memory system works.

git-svn-id: file:///svn/tokudb@1954 c7de825b-a66e-492c-adef-691d508d4ae1