This guide combines an overview of Welle with a quick tutorial that helps you to get started with it. It should take about 10 minutes to read and study the provided code examples. This guide covers:
This work is licensed under a Creative Commons Attribution 3.0 Unported License (including images & stylesheets). The source is available on Github.
This guide covers Welle 1.5.
Welle is an idiomatic Clojure client for Riak. It is simple and easy to use, strives to support every Riak 1.0+ feature, has next to no performance overhead compared to the official Java client and is well maintained.
Welle is not a replacement for the Riak Java client, instead, Welle is symbiotic with it. Welle does not try to offer object/document mapping functionality or introduce abstractions on top of what Clojure and Riak have. With Welle, you work with native Clojure and Java data structures like maps, vectors, strings, dates and so on. This approach has pros and cons but (we believe) closely follows Clojure's philosophy of reducing incidental complexity. It also fits key/value data model of Riak very well.
Welle is built from the ground up for Clojure 1.3 and later.
Welle currently uses Riak Java driver 1.0.x under the hood and thus supports Riak 1.0 and later versions. Please note that some features may be specific to Riak 1.1, 1.2 or later versions.
Welle 1.4.0 is compatible with Riak 1.3.0.
Welle artifacts are released to Clojars.
[com.novemberain/welle "1.5.0"]
Add Clojars repository definition to your pom.xml:
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
And then the dependency:
<dependency>
<groupId>com.novemberain</groupId>
<artifactId>welle</artifactId>
<version>1.5.0</version>
</dependency>
It is recommended to stay up-to-date with new versions. New releases and important changes are announced @ClojureWerkz.
Riak and Welle support two transports: HTTP and Protocol Buffers. They vary in performance characteristics and supported features (PBC API supports searcn and secondary indexes starting with Riak 1.2). For the purpose of this guide we will concentrate on the HTTP transport. This is what most applications use and it delivers pretty good performance most applications will be satisfied with.
clojurewerkz.welle.core/connect! function connects to Riak using HTTP transport and sets up the default client. You can invoke it in
three ways:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]))
;; connects to a Riak node at http://127.0.0.1:8098/riak
(wc/connect!)
;; connects to a Riak node at the given endpoint, client id will be generated
(wc/connect! "http://riak.data.megacorp.internal:8098/riak")
;; the same as the previous example but also uses provided client id
(wc/connect! "http://riak.data.megacorp.internal:8098/riak" "myapp-client.0001")
It is very common to use the 0-arity (first example) in development and the 1-arity (second example) for QA and production environments. Most of the time, relying on client id to be generated by Welle is sufficient. You can learn more about client ids and how they are used by Riak for conflict detection and resolution in the Riak documentation.
Riak stores data in buckets (similar to tables in relational databases and collections in MongoDB).
So, before storing data, it is a good idea to prepare a bucket. You use clojurewerkz.welle.buckets/update function to do it. In the simplest case,
you only pass it the name of the bucket:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]))
(wc/connect!)
;; creates a new bucket with default properties
(wb/create "things")
Riak buckets have properties. In the example above, we rely on all defaults. In many cases, you will want to tweak bucket properties.
To do so, pass additional arguments to clojurewerkz.welle.buckets/update, like so:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]))
(wc/connect!)
;; creates a new bucket with properties explicitly specified
(wb/create "accounts" :n-val 5)
in the example above we instruct Riak to replicate all objects stored in the bucket "accounts" to 5 nodes (because accounts are valuable information we absolutely don't want to lose).
Besides showing off impressive benchmarks and striving to handle Web Scale, the primary purpose of data stores like Riak
is to store data.
Riak has several components to it but it is a key/value store at heart. You interact with it using functions in the clojurewerkz.welle.kv namespace. For example,
clojurewerkz.welle.kv/store function stores objects in Riak. It takes a bucket name, a key and a value:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv]))
(wc/connect!)
(wb/create "things")
(kv/store "things" "a-key" (.getBytes "value"))
In the example above we store some data as bytes: Riak is content agnostic and will happily store whatever you throw at it. However, most applications work with data more structured than raw bytes, for example, as JSON documents or text in the CSV format. To make developer experience a bit nicer, Riak will store content type of the stored value and Welle will automatically serialize stored value if content type is one of
Content type is passed as one of the optional arguments to the same function, clojurewerkz.welle.kv/store:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv]))
(wc/connect!)
(wb/create "accounts")
(let [key "novemberain"
val {:name "Michael" :age 27 :username key}]
(kv/store "accounts" key val :content-type "application/json; charset=UTF-8"))
Often you will want to use constants provided by the Riak Java client (that Welle uses underneath):
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants))
(wc/connect!)
(wb/create "accounts")
(let [key "novemberain"
val {:name "Michael" :age 27 :username key}]
(kv/store "accounts" key val :content-type Constants/CTYPE_JSON_UTF8))
However, Riak Java client constants do not include Clojure data content type. To use it, pass "application/clojure" as the content type:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv]))
(wc/connect!)
(wb/create "accounts")
(let [key "novemberain"
val {:name "Michael" :age 27 :username key :created-at (java.util.Date.) :hacks #{"clojure" "java" "ruby" "scala" "erlang"}}]
;; stores data serialized to be read by the Clojure reader, so the set and the date will be
;; transparently read back when this value is fetched. Please note that to serialize and deserialize
;; dates as Clojure data you will have to use Clojure 1.4+. Previous Clojure versions do not have date/instanti
;; literals support in the reader.
(kv/store "accounts" key val :content-type "application/clojure"))
Serialized values will be deserialized automatically by Welle when you fetch them, as you will see later in this guide.
To fetch a stored value, use clojurewerkz.welle.kv/fetch function. In the simplest case it takes a bucket name and a key:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants))
(wc/connect!)
(wb/create "accounts")
(let [bucket "accounts"
key "novemberain"
val {:name "Michael" :age 27 :username key}]
;; stores data serialized as JSON
(kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8)
;; fetches it back
(kv/fetch bucket key))
It returns a list of values because in an eventually consistent system like Riak it is sometimes possible that multiple
versions of an object will be stored in a cluster. They are called siblings. We won't
get into siblings and conflicts resolution in this guide, just be aware of this possibility and that clojurewerkz.welle.kv/fetch returns
a list.
Granted, most of the time (in systems that are light on writes, almost always) the list will only contain one value. Fortunately,
Clojure has us covered here: it is possible use positional destructuring to get the value without additional calls to clojure.core/first:
;; fetch an object (possibly with siblings)
;; here we use positional destructuring to keep the code concise and idiomatic
(let [[val] (kv/fetch bucket key)]
val)
So if you are sure that there will be no conflicts, this practice is encouraged. Alternatively, you can use clojurewerkz.welle.kv/fetch-one to fetch only a single value.
So far we haven't passed any arguments to clojurewerkz.welle.kv/fetch. In case you need to do it, it is very similar to how you do
it when storing data:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants))
(wc/connect!)
;; replicate values 3 times in the cluster
(wb/create "accounts" :n-val 3)
(let [bucket "accounts"
key "novemberain"
val {:name "Michael" :age 27 :username key}]
;; stores data serialized as JSON to 2 vnodes
(kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :w 2)
;; fetches it back, 2 vnodes must respond
(kv/fetch bucket key :r 2))
For those familiar with the Riak Java client, quorum values can be passed as com.basho.riak.client.cap.Quora and com.basho.riak.client.cap.Quorum
instances, too:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants
com.basho.riak.client.cap.Quora))
(wc/connect!)
;; replicate values 3 times in the cluster
(wb/create "accounts" :n-val 3)
(let [bucket "accounts"
key "novemberain"
val {:name "Michael" :age 27 :username key}]
;; stores data serialized as JSON to 2 vnodes
(kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :w 2)
;; fetches it back, all vnodes must respond
(kv/fetch bucket key :r Quora/ALL))
Fetching data by key may get you quite far if you are smart about choose keys but often it is more convenient to use secondary indexes to query your data. Secondary indexes (often referred to as 2i) have two sides to them:
With Riak, you specify indexes and index values when storing values using the :indexes option:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants))
(wc/connect!)
(wb/create "accounts")
(let [bucket "accounts"
key "novemberain"
val {:name "Michael" :age 27 :username key}]
;; stores the value and adds it to two secondary indexes: username and age
(kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :indexes {:username #{username} :age 27}))
Each value may have its own set of indexes. There is no predefined set of indexes, like with some other data stores (for example, PostgreSQL). If index values are specified for an object, Riak will index the value and distribute information about the new index entry in the cluster. Please note that string indexes need to be passed as sets (and may contain multiple values).
Once you have some objects stored and indexed, lets take a look how to perform 2i queries with Welle. clojurewerkz.welle.kv namespace
provides a function to do it, index-query. It is used similarly to clojurewerkz.welle.kv/fetch:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants))
(wc/connect!)
(wb/create "accounts")
(let [bucket "accounts"
key "novemberain"
val {:name "Michael" :age 27 :username key}]
;; stores the value and adds it to two secondary indexes: username and age
(kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :indexes {:username #{username} :age 27})
;; perform a 2i query on the username
(kv/index-query bucket :username username))
You pass it the name of the index to use and a value. Welle will return you a list of keys that you can then fetch, combine with a list of keys returned by another index query or use with map/reduce.
Riak also supports range queries. In the example used throughout this guide we index a user's age. To get a list of keys of objects
with ages between 25 and 30, pass a pair (typically as a vector, so [25 30]) for index value:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants))
(wc/connect!)
(wb/create "accounts")
(let [bucket "accounts"
key "novemberain"
val {:name "Michael" :age 27 :username key}]
;; stores the value and adds it to two secondary indexes: username and age
(kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :indexes {:username #{username} :age 27})
;; perform a 2i query on the age index, as a range
(kv/index-query bucket :age [25 30]))
To delete an object, use clojurewerkz.welle.kv/delete function. In the simplest case it takes a bucket name and a key,
just like clojurewerkz.welle.kv/fetch:
(ns welle.docs.examples
(:require [clojurewerkz.welle.core :as wc]
[clojurewerkz.welle.buckets :as wb]
[clojurewerkz.welle.kv :as kv])
(:import com.basho.riak.client.http.util.Constants))
(wc/connect!)
(wb/create "accounts")
(let [bucket "accounts"
key "novemberain"
val {:name "Michael" :age 27 :username key}]
;; stores data serialized as JSON
(kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8)
;; and deletes it
(kv/delete bucket key))
The documentation is organized as a number of guides, covering all kinds of topics.
We recommend that you read the following guides first, if possible, in this order:
Please take a moment to tell us what you think about this guide on Twitter or the Welle mailing list
Let us know what was unclear or what has not been covered. Maybe you do not like the guide style or grammar or discover spelling mistakes. Reader feedback is key to making the documentation better.