About this guide

This guide combines an overview of Welle with a quick tutorial that helps you to get started with it. It should take about 10 minutes to read and study the provided code examples. This guide covers:

  • Features of Welle
  • Clojure and Riak version requirements
  • How to add Welle dependency to your project
  • Basic operations (creating buckets, storing and fetching objects, index queries, using map/reduce)
  • Overview of how Welle automatic serialization works

This work is licensed under a Creative Commons Attribution 3.0 Unported License (including images & stylesheets). The source is available on Github.

What version of Welle does this guide cover?

This guide covers Welle 2.0, including development releases.

Welle Overview

Welle is an idiomatic Clojure client for Riak. It is simple and easy to use, strives to support every Riak 1.0+ feature, has next to no performance overhead compared to the official Java client and is well maintained.

What Welle is not

Welle is not a replacement for the Riak Java client, instead, Welle is symbiotic with it. Welle does not try to offer object/document mapping functionality or introduce abstractions on top of what Clojure and Riak have. With Welle, you work with native Clojure and Java data structures like maps, vectors, strings, dates and so on. This approach has pros and cons but (we believe) closely follows Clojure's philosophy of reducing incidental complexity. It also fits key/value data model of Riak very well.

Supported Clojure versions

Welle is built from the ground up for Clojure 1.3 and later.

Supported Riak versions

Welle currently uses Riak Java driver 1.0.x under the hood and thus supports Riak 1.0 and later versions. Please note that some features may be specific to Riak 1.1, 1.2 or later versions.

Welle 1.4.0 is compatible with Riak 1.3.0.

Adding Welle Dependency To Your Project

Welle artifacts are released to Clojars.

With Leiningen

[com.novemberain/welle "2.0.0"]

With Maven

Add Clojars repository definition to your pom.xml:

<repository>
  <id>clojars.org</id>
  <url>http://clojars.org/repo</url>
</repository>

And then the dependency:

<dependency>
  <groupId>com.novemberain</groupId>
  <artifactId>welle</artifactId>
  <version>2.0.0</version>
</dependency>

It is recommended to stay up-to-date with new versions. New releases and important changes are announced @ClojureWerkz.

Connecting to Riak

Riak and Welle support two transports: HTTP and Protocol Buffers. They vary in performance characteristics and supported features (PBC API supports searcn and secondary indexes starting with Riak 1.2). For the purpose of this guide we will concentrate on the HTTP transport. This is what most applications use and it delivers pretty good performance most applications will be satisfied with.

Using HTTP transport

clojurewerkz.welle.core/connect! function connects to Riak using HTTP transport and sets up the default client. You can invoke it in three ways:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core :as wc]))

;; connects to a Riak node at http://127.0.0.1:8098/riak
(wc/connect!)

;; connects to a Riak node at the given endpoint, client id will be generated
(wc/connect! "http://riak.data.megacorp.internal:8098/riak")

;; the same as the previous example but also uses provided client id
(wc/connect! "http://riak.data.megacorp.internal:8098/riak" "myapp-client.0001")

It is very common to use the 0-arity (first example) in development and the 1-arity (second example) for QA and production environments. Most of the time, relying on client id to be generated by Welle is sufficient. You can learn more about client ids and how they are used by Riak for conflict detection and resolution in the Riak documentation.

Creating buckets

Riak stores data in buckets (similar to tables in relational databases and collections in MongoDB). So, before storing data, it is a good idea to prepare a bucket. You use clojurewerkz.welle.buckets/update function to do it. In the simplest case, you only pass it the name of the bucket:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core :as wc]
            [clojurewerkz.welle.buckets :as wb]))

(wc/connect!)

;; creates a new bucket with default properties
(wb/create "things")

Riak buckets have properties. In the example above, we rely on all defaults. In many cases, you will want to tweak bucket properties. To do so, pass additional arguments to clojurewerkz.welle.buckets/update, like so:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core :as wc]
            [clojurewerkz.welle.buckets :as wb]))

(wc/connect!)

;; creates a new bucket with properties explicitly specified
(wb/create "accounts" :n-val 5)

in the example above we instruct Riak to replicate all objects stored in the bucket "accounts" to 5 nodes (because accounts are valuable information we absolutely don't want to lose).

Storing Values

Besides showing off impressive benchmarks and striving to handle Web Scale, the primary purpose of data stores like Riak is to store data. Riak has several components to it but it is a key/value store at heart. You interact with it using functions in the clojurewerkz.welle.kv namespace. For example, clojurewerkz.welle.kv/store function stores objects in Riak. It takes a bucket name, a key and a value:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv]))

(wc/connect!)

(wb/create "things")
(kv/store "things" "a-key" (.getBytes "value"))

In the example above we store some data as bytes: Riak is content agnostic and will happily store whatever you throw at it. However, most applications work with data more structured than raw bytes, for example, as JSON documents or text in the CSV format. To make developer experience a bit nicer, Riak will store content type of the stored value and Welle will automatically serialize stored value if content type is one of

  • JSON
  • JSON in UTF-8
  • Clojure data (that can be read by the Clojure reader)
  • Text
  • Text in UTF-8

Content type is passed as one of the optional arguments to the same function, clojurewerkz.welle.kv/store:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv]))

(wc/connect!)
(wb/create "accounts")

(let [key "novemberain"
      val {:name "Michael" :age 27 :username key}]
  (kv/store "accounts" key val :content-type "application/json; charset=UTF-8"))

Often you will want to use constants provided by the Riak Java client (that Welle uses underneath):

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants))

(wc/connect!)
(wb/create "accounts")

(let [key "novemberain"
      val {:name "Michael" :age 27 :username key}]
  (kv/store "accounts" key val :content-type Constants/CTYPE_JSON_UTF8))

However, Riak Java client constants do not include Clojure data content type. To use it, pass "application/clojure" as the content type:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv]))

(wc/connect!)
(wb/create "accounts")

(let [key "novemberain"
      val {:name "Michael" :age 27 :username key :created-at (java.util.Date.) :hacks #{"clojure" "java" "ruby" "scala" "erlang"}}]
  ;; stores data serialized to be read by the Clojure reader, so the set and the date will be
  ;; transparently read back when this value is fetched. Please note that to serialize and deserialize
  ;; dates as Clojure data you will have to use Clojure 1.4+. Previous Clojure versions do not have date/instanti
  ;; literals support in the reader.
  (kv/store "accounts" key val :content-type "application/clojure"))

Serialized values will be deserialized automatically by Welle when you fetch them, as you will see later in this guide.

Fetching Values

To fetch a stored value, use clojurewerkz.welle.kv/fetch function. In the simplest case it takes a bucket name and a key:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants))

(wc/connect!)
(wb/create "accounts")

(let [bucket "accounts"
      key    "novemberain"
      val    {:name "Michael" :age 27 :username key}]
  ;; stores data serialized as JSON
  (kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8)
  ;; fetches it back
  (kv/fetch bucket key))

It returns an immutable response map with the following keys:

  • :result: one or more objects returned by Riak
  • :vclock: vector clock of the response
  • :has-siblings?: true if response has siblings
  • :has-value?: true if response is non-empty
  • :modified?: false when conditional GET returned a non-modified response
  • :deleted?: true if this object has been deleted but there is a vclock for it

To obtain a previously stored result, take :result from the response. Note that for clojurewerkz.welle.kv/fetch :result will contain a list of values. In an eventually consistent system like Riak it is sometimes possible that multiple versions of an object will be stored in a cluster. They are called siblings. We won't get into siblings and conflicts resolution in this guide, just be aware of this possibility and that clojurewerkz.welle.kv/fetch returns a response where :result is a list.

Granted, most of the time (in systems that are light on writes, almost always) the list will only contain one value. Fortunately, Clojure has us covered here: it is possible use positional destructuring to get the value without additional calls to clojure.core/first:

;; fetch an object (possibly with siblings)
;; here we use positional destructuring to keep the code concise and idiomatic
(let [{:keys [result]} (kv/fetch bucket key)
      [val]            result]
  val)

So if you are sure that there will be no conflicts, this practice is encouraged. Alternatively, you can use clojurewerkz.welle.kv/fetch-one to automatically get only the first value from the :result.

So far we haven't passed any arguments to clojurewerkz.welle.kv/fetch. In case you need to do it, it is very similar to how you do it when storing data:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants))

(wc/connect!)
;; replicate values 3 times in the cluster
(wb/create "accounts" :n-val 3)

(let [bucket "accounts"
      key    "novemberain"
      val    {:name "Michael" :age 27 :username key}]
  ;; stores data serialized as JSON to 2 vnodes
  (kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :w 2)
  ;; fetches it back, 2 vnodes must respond
  (kv/fetch bucket key :r 2))

For those familiar with the Riak Java client, quorum values can be passed as com.basho.riak.client.cap.Quora and com.basho.riak.client.cap.Quorum instances, too:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants
           com.basho.riak.client.cap.Quora))

(wc/connect!)
;; replicate values 3 times in the cluster
(wb/create "accounts" :n-val 3)

(let [bucket "accounts"
      key    "novemberain"
      val    {:name "Michael" :age 27 :username key}]
  ;; stores data serialized as JSON to 2 vnodes
  (kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :w 2)
  ;; fetches it back, all vnodes must respond
  (kv/fetch bucket key :r Quora/ALL))

Using Secondary Indexes (2i)

Fetching data by key may get you quite far if you are smart about choose keys but often it is more convenient to use secondary indexes to query your data. Secondary indexes (often referred to as 2i) have two sides to them:

  • Indexing data
  • Querying

With Riak, you specify indexes and index values when storing values using the :indexes option:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants))

(wc/connect!)
(wb/create "accounts")

(let [bucket "accounts"
      key    "novemberain"
      val    {:name "Michael" :age 27 :username key}]
  ;; stores the value and adds it to two secondary indexes: username and age
  (kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :indexes {:username #{username} :age 27}))

Each value may have its own set of indexes. There is no predefined set of indexes, like with some other data stores (for example, PostgreSQL). If index values are specified for an object, Riak will index the value and distribute information about the new index entry in the cluster. Please note that string indexes need to be passed as sets (and may contain multiple values).

Secondary Index (2i) Queries

Once you have some objects stored and indexed, lets take a look how to perform 2i queries with Welle. clojurewerkz.welle.kv namespace provides a function to do it, index-query. It is used similarly to clojurewerkz.welle.kv/fetch:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants))

(wc/connect!)
(wb/create "accounts")

(let [bucket "accounts"
      key    "novemberain"
      val    {:name "Michael" :age 27 :username key}]
  ;; stores the value and adds it to two secondary indexes: username and age
  (kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :indexes {:username #{username} :age 27})
  ;; perform a 2i query on the username
  (kv/index-query bucket :username username))

You pass it the name of the index to use and a value. Welle will return you a list of keys that you can then fetch, combine with a list of keys returned by another index query or use with map/reduce.

Riak also supports range queries. In the example used throughout this guide we index a user's age. To get a list of keys of objects with ages between 25 and 30, pass a pair (typically as a vector, so [25 30]) for index value:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants))

(wc/connect!)
(wb/create "accounts")

(let [bucket "accounts"
      key    "novemberain"
      val    {:name "Michael" :age 27 :username key}]
  ;; stores the value and adds it to two secondary indexes: username and age
  (kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8 :indexes {:username #{username} :age 27})
  ;; perform a 2i query on the age index, as a range
  (kv/index-query bucket :age [25 30]))

Deleting Values

To delete an object, use clojurewerkz.welle.kv/delete function. In the simplest case it takes a bucket name and a key, just like clojurewerkz.welle.kv/fetch:

(ns welle.docs.examples
  (:require [clojurewerkz.welle.core    :as wc]
            [clojurewerkz.welle.buckets :as wb]
            [clojurewerkz.welle.kv      :as kv])
  (:import com.basho.riak.client.http.util.Constants))

(wc/connect!)
(wb/create "accounts")

(let [bucket "accounts"
      key    "novemberain"
      val    {:name "Michael" :age 27 :username key}]
  ;; stores data serialized as JSON
  (kv/store bucket key val :content-type Constants/CTYPE_JSON_UTF8)
  ;; and deletes it
  (kv/delete bucket key))

What To Read Next

The documentation is organized as a number of guides, covering all kinds of topics.

We recommend that you read the following guides first, if possible, in this order:

Tell Us What You Think!

Please take a moment to tell us what you think about this guide on Twitter or the Welle mailing list

Let us know what was unclear or what has not been covered. Maybe you do not like the guide style or grammar or discover spelling mistakes. Reader feedback is key to making the documentation better.

comments powered by Disqus