Slap.KV.Cluster (slap_kv v0.6.1)

Copy Markdown View Source

The shards that hold the rows, a Slap.Cluster. A partition's rows are on shard shard_for(partition). Each shard runs partition writers; each partition has one writer. Set child_options: [partition_writers: n] on the cluster (default 16).

Each shard limits the work waiting for durability. The limits must be positive integers. Defaults:

  • :max_inflight_bytes_per_writer - 64 MiB.
  • :max_inflight_bytes_per_shard - 256 MiB.
  • :max_inflight_requests_per_writer - 1,024.
  • :max_batch_ops - 1,000.
  • :max_batch_bytes - 8 MiB.

The shard's owner checks a request against them before it reads or writes any row. A request's size is its encoded keys (two length bytes, the partition and the key) and put values; a linearizable get counts as one request of zero bytes. A request counts until it is durable and answered. The limits do not cover mailboxes, HTTP decoding, response bodies or total BEAM and native memory. The README's "Limits on pending work" describes the errors.

Run one instance of this built-in cluster per VM. It can own many shards and participate in a cluster across VMs. The same VM can also run Slap.Streams.Cluster.

To configure a cluster under your application's name, define a module with use Slap.KV.Cluster, otp_app: :my_app, supervise that module in place of Slap.KV.Cluster, and pass cluster: MyApp.KVCluster to KV operations and Slap.KV.HTTP.Router. Run one KV cluster per VM; peer discovery is node-wide.

Configure it like any Slap.Cluster:

config :slap_kv, Slap.KV.Cluster,
  store: {:url, "s3://bucket/prefix", aws_endpoint: "https://rustfs:9000"},
  path: "kv",
  shards: 8

Durable writes wait for SlateDB's next WAL flush. This cluster sets flush_interval to 10 ms; SlateDB's default is 100 ms. Other settings leave the 10 ms default intact. The README describes the latency and object-store request trade-off.

Give the cluster its own :path when it shares a store with another cluster.

Summary

Functions

Defines a KV cluster under the given :otp_app.

Where every shard is. See Slap.Cluster.assignments/1.

Waits for every shard to finish starting. See Slap.Cluster.await_ready/2.

Runs a function with a shard's context. See Slap.Cluster.call/4.

Returns the accepted :child_options keys for a KV cluster.

The shards open on this node.

Where a shard is. See Slap.Cluster.lookup/2.

The shard for a placement key. See Slap.Cluster.shard_for/2.

Starts the cluster. opts override the application environment, which overrides the :defaults given to use Slap.Cluster.

Current shard lifecycle and readiness. See Slap.Cluster.status/2.

Stops the cluster on this node, closing its shards.

A name for a per-shard process. See Slap.Cluster.via/3.

Functions

__using__(opts)

(macro)

Defines a KV cluster under the given :otp_app.

assignments()

Where every shard is. See Slap.Cluster.assignments/1.

await_ready(timeout)

Waits for every shard to finish starting. See Slap.Cluster.await_ready/2.

call(shard, mfa, opts \\ [])

Runs a function with a shard's context. See Slap.Cluster.call/4.

child_option_keys()

@spec child_option_keys() :: [atom()]

Returns the accepted :child_options keys for a KV cluster.

local_shards()

The shards open on this node.

lookup(shard)

Where a shard is. See Slap.Cluster.lookup/2.

shard_for(key)

The shard for a placement key. See Slap.Cluster.shard_for/2.

start_link(opts \\ [])

Starts the cluster. opts override the application environment, which overrides the :defaults given to use Slap.Cluster.

status(timeout \\ 5000)

Current shard lifecycle and readiness. See Slap.Cluster.status/2.

stop(timeout \\ :infinity)

Stops the cluster on this node, closing its shards.

via(shard, name)

A name for a per-shard process. See Slap.Cluster.via/3.