Thrudb - Document Oriented Database Services

The S3 backend provides an exteremly scalable and fast-ish (if you’re in EC2) persistent storage backend. Features

• Keeps statistics on the counts of service method calls, for logging &apm; monitoring.
• Simple and clean api.

See Also

• scripts/thrudoc_rrdpull.pl

Use & Configuration

    KEEP_STATS = 1

The S3 backend provides an extremely scalable and fast-ish (if you’re in EC2) persistent storage backend. Features

• Massively Scalable
• Simple setup, just provide your AWS credentials and bucket (tablename).

Future Direction & Development

• s3 client replacement/rewrite

Use & Configuration

    AWS_ACCESS_KEY = <aws_access_key>
    AWS_SECRET_ACCESS_KEY = <secret_access_key>
    S3_BUCKET_PREFIX = thrudoc_prefix_

The replication backend provides

Future Direction & Development

Use & Configuration

   REPLICATION_NAME = 4803@localhost
   REPLICATION_PRIVATE_NAME = ds_s1
   REPLICATION_GROUP = thrudoc
   REPLICATION_STATUS_FILE=replication_status
   REPLICATION_STATUS_FLUSH_FREQUENCY=30

The MySQL backend provides an exteremly fast and scalable persistent storage backend. Features

• Massively Scalable
  • >4000 TPS mixed read/write with a single modest desktop box running both the MySQL and Thrudoc servers
  • Partitioned data storage to allow near-infinite dataset size
  • Simple setup, just install MySQL and Thrudoc, run the tablename create script, and set a couple of configuration values
• Supports automatic and instant read-only failover and master switchback.
• Straightforward in DB storage, so you can easily write your own apps/scripts to work with the data
• Works with any MySQL setup: replication, backup, …

Suggested/Proposed Architecture

One or more Thrudoc servers fronting a pool of two or more MySQL servers in master-master pairs. datatables are assigned to each mysql server and it’s sibling will provide read-only failover. Once the downed host returns to service the Thrudoc servers will automatically switch back to it in read-write mode without any data loss. If for some reason the host can not be brought back up a manual read-write failover can take place, to the sibling if it can handle the load of both, or to a host replacing the downed one. At most replication delay information is lost, if the downed host can’t be brought back up. Future Direction & Development

• Directory Manager
  • Split and combine tables to grow/shrink the backend and balance load
• Multi-master (ring replication) fail-over support

Use & Configuration

mysql_backend_create_tablename.pl --master-hostname=<host> --master-port=<port>
--database=<db> --root-pass=<password> --tablename=<tablename>
--hosts=<host1,host2,...> --parts-per=<2> --with-circular-slaves

Sensible defaults are provided for all options above. If –with-circular-replication is provided there must be an even number of hosts and they will be paired up in the order given. To use the MySQL backend add the following lines to your thrudoc.conf replacing the right hand sides with values appropriate for your environment.

    MYSQL_MASTER_HOST = <host>
    MYSQL_MASTER_PORT = <port>
    MYSQL_SLAVE_HOST = <host>
    MYSQL_SLAVE_PORT = <port>
    MYSQL_DIRECTORY_DB = <db>
    MYSQL_USERNAME = <username>
    MYSQL_PASSWORD = <password>

The Memcached caching backend provides a caching layer on top of a persistent backend. It operates transparently intercepting calls to the primary backend and sets and gets data from a Memcached pool of servers lowering the load on the primary and increasing performance.

The memcached caching backend is powered by libmemcached a full featured C memcached client library. Features

• Transparent and simple caching of any persistent backend
• Single configuration value setup.

Future Direction & Development

• Expose as much of the underlying libmemcached features as possible/makes sense

Use & Configuration To use the memcached caching backend add the following line to your thrudoc.conf replacing the server names and ports with the list of hosts in your memcached pool.

    MEMCACHED_SERVERS = <server.one:port,server.two:port,...>