all repos — h3rald @ b6d12a3d9a471671d2fedeb8a36651f808b455f7

The sources of https://h3rald.com

content/articles/litestore.md

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
 100
 101
 102
 103
 104
 105
 106
 107
 108
 109
 110
 111
 112
 113
 114
 115
 116
 117
 118
 119
 120
 121
 122
 123
 124
 125
 126
 127
 128
 129
 130
 131
 132
 133
 134
 135
 136
 137
 138
 139
 140
 141
 142
 143
 144
 145
 146
 147
 148
 149
 150
 151
 152
 153
 154
 155
 156
 157
 158
 159
 160
 161
 162
 163
 164
 165
 166
 167
 168
 169
 170
 171
 172
 173
 174
 175
 176
 177
 178
 179
 180
 181
 182
 183
 184
 185
 186
 187
 188
 189
 190
 191
 192
 193
 194
 195
 196
 197
 198
 199
 200
 201
 202
 203
 204
 205
 206
 207
 208
 209
 210
 211
 212
 213
 214
 215
 216
 217
 218
 219
 220
 221
 222
 223
 224
 225
 226
 227
 228
 229
 230
 231
 232
 233
 234
 235
 236
 237
 238
 239
 240
 241
 242
 243
 244
 245
 246
 247
 248
 249
 250
 251
 252
 253
 254
 255
 256
 257
 258
 259
 260
 261
 262
 263
 264
 265
 266
 267
 268
 269
 270
 271
 272
 273
 274
 275
 276
 277
 278
 279
 280
 281
 282
 283
 284
 285
 286
 287
 288
 289
 290
 291
 292
 293
 294
 295
 296
 297
 298
-----
:permalink: litestore
:title: Introducing LiteStore
:subtitle: A tiny, lightweight, self-contained, RESTful document store
:type: article
:intro: |
  Lately I have become more and more interested in client-side single-page applications. Nowadays you can write your web apps in Javascript using your favorite framework, without any server-side logic, but you obviously still need:
   
   * some web service to retrieve and persist your application data.
   * a web server to serve the source code and the static assets of your web application.
  
  NodeJS is probably one of the easiest backend to setup for prototyping SPAs. It is very easy to create a simple web server in Node and to implement a simple REST API using Express or a similar framework, but you still need to install node and write some code to wire up your backend.

  I wanted something even more lazy then that. I wanted a fully _self-contained_ program able to:

  * Serve static files
  * Act as a simple JSON document store
  * Provide a simple REST API to work with
  * (bonus!) provide a way to pack web apps for easy distribution

  ...and that's how I ended up developing _LiteStore_.

:tags:
- webdevelopment
- databases
- litestore
- opensource
:date: 2015-09-20 21:01:45.000000000 +01:00
-----
Lately I have become more and more interested in client-side single-page applications. Nowadays you can write your web apps in Javascript using your favorite framework, without any server-side logic, but you obviously still need:
   
   * some web service to retrieve and persist your application data.
   * a web server to serve the source code and the static assets of your web application.
  
NodeJS is probably one of the easiest backend to setup for prototyping SPAs. It is very easy to create a simple web server in Node and to implement a simple REST API using Express or a similar framework, but you still need to install node and write some code to wire up your backend.

I wanted something even more lazy then that. I wanted a fully _self-contained_ program able to:

  * Serve static files
  * Act as a simple JSON document store
  * Provide a simple REST API to work with
  * (bonus!) provide a way to pack web apps for easy distribution

...and that's how I ended up developing _LiteStore_.

[LiteStore](/litestore) is a lightweight, self-contained, RESTful, searchable, multi-format, NoSQL document store and web server. That sounds pretentious, but it is essentially an accurate description of what LiteStore is and does. 

It is built using the [Nim](http://nim-lang.org/) programming language and its Administration App (see below) is built using [Mithril](https://lhorie.github.io/mithril/), two relatively less-known projects, both of which I highly recommend if you want to take a break from more mainstream programming languages like C# or Java and Javascript frameworks like AngularJS or React.


### Downloading LiteStore

The easiest way to understand what LiteStore does, is to try it out. Simply download a release package from the [GitHub repo](https://github.com/h3rald/litestore) (or from [here](/litestore) for your convenience) that is suitable for your operating system and architecture.

The release package contains the following files:

* **litestore** / **litestore.exe** — The LiteStore executable.
* **LiteStore_UserGuide.html** — The official user guide describing the program, its usage, architecture, API, etc.
* **data.db** — The default data store file, containing the LiteStore Administration App.

That's all. No libraries to download, no installers, just an executable file. I love self-contained programs.

### Running LiteStore

To start a LiteStore server with the default settings:

1. Open a command prompt in the directory containing the LiteStore executable and the data.db data store file.
2. Enter **litestore** in the command prompt.

_or_ (on Windows):

1. Double-click **litestore.exe**

You should get a message similar to the following:

<samp>2015-09-20 @ 05:03:36    INFO: LiteStore v1.0.0 started on 127.0.0.1:9500.</samp>

...meaning that LiteStore is running on the specified address.

If you open a browser and type <http://localhost:9500> you should get the following JSON response:

```
{
    "version": "LiteStore v1.0.0",
    "datastore_version": 1,
    "size": "4.83 MB",
    "read_only": false,
    "log_level": "info",
    "directory": null,
    "mount": false,
    "total_documents": 77,
    "total_tags": 18,
    "tags": [
        { "$dir:admin": 77 },
        { "$format:binary": 14 },
        { "$format:text": 63 },
        { "$subtype:css": 3 },
        { "$subtype:html": 1 },
        { "$subtype:javascript": 36 },
        { "$subtype:octet-stream": 2 },
        { "$subtype:png": 8 },
        { "$subtype:svg+xml": 1 },
        { "$subtype:vnd.ms-fontobject": 1 },
        { "$subtype:x-font-otf": 1 },
        { "$subtype:x-font-ttf": 1 },
        { "$subtype:x-icon": 1 },
        { "$subtype:x-less": 10 },
        { "$subtype:x-markdown": 12 },
        { "$type:application": 41 },
        { "$type:image": 10 },
        { "$type:text": 26 }
    ]
}
```

Congrats, LiteStore is up and running!

### Digging Deeper...

LiteStore is, first and foremost, a RESTful document store. Think CouchDb or MongoDb, but smaller, much smaller. Here's basically what happens when you run the **litestore** command:

1. LiteStore checks for a file named **data.db** within the current directory and creates one if it doesn't exists.
2. LiteStore assumes that such file is a properly-formatted LiteStore data store file and tries to connect to it.
3. If all goes well, LiteStore starts a simple HTTP server on port 9500, ready to receive HTTP requests.

(The path of the data store file, the port, etc. are all configurable settings &mdash; run [litestore -h](class:cmd) for more details)

Now, if you access <http://localhost:9500> while LiteStore is running, you'll get JSON response listing some stats about the currently-loaded data store file, and some information on stored documents and tags, if any. The **data.db** file that ships with LiteStore already contains some documents. Well, actually it contains _a web application to administer LiteStore itself_.

### The LiteStore Administration App

Now try accessing the following URL:

<http://localhost:9500/docs/admin/index.html>

If all goes well, you should see the following page:

![Info](/images/litestore/litestore_info.png)

You are now viewing the LiteStore Administration App, a simple single-page application implemented on top of the [Mithril](https://lhorie.github.io/mithril/) Javascript framework. The cool part is that this application is stored within the **data.db** file and is being served by LiteStore itself.

The LiteStore Administration App is an example of single-page application developed for LiteStore: it uses pretty much all LiteStore functionalities that are exposes by the LiteStore REST API. 

On the home page you can see the same stats displayed on LiteStore root URL (<http://localhost:9500>), but you can also click any of the tag links to view documents by tag:

![Tag](/images/litestore/litestore_tag.png)

...and clicking a document title will display the document content and tags:

![Document](/images/litestore/litestore_doc.png)

Note that it is also possible to create, upload, edit or delete documents and manage tags:

![Document Operations](/images/litestore/litestore_doc_ops.png)

But perhaps the best feature of this web app (and of LiteStore itself) is the full-text search capability:

![Document Search](/images/litestore/litestore_search.png)

By leveraging LiteStore search API (which in turn leverages SQLite exceptional [FTS4 extension](https://www.sqlite.org/fts3.html)), it is pretty easy to build fast search-oriented web applications.

### Under the Hood

By now you are probably wonder how this whole thing can work. This is best explained with a simple exercise: let's export and re-import the Administration App!

First of all, stop the LiteStore instance by pressing <kbd>CTRL+C</kbd> within the prompt window.

Now execute the following command, from the same directory where you started LiteStore:

> %terminal%
>
> litestore -d:admin export

LiteStore will create a directory called [admin](class:dir). If you browse its contents, you will find the full source code and assets of the LiteStore Administration App (also available in the LiteStore [Github repo](https://github.com/h3rald/litestore/)).

Now delete the [data.db](class:file) file. Don't worry, you can always get a new one from a release package, at worst.

Run LiteStore again, with no parameters:

> %terminal%
>
> litestore 

If you access <http://localhost:9500> in your browser, you'll get the following response:

```
{
  version: "LiteStore v1.0.0",
  datastore_version: 1,
  size: "0.02 MB",
  read_only: false,
  log_level: "info",
  directory: null,
  mount: false,
  total_documents: 0,
  total_tags: 0,
  tags: [ ]
}
```

This is basically saying that the datastore file is empty. And indeed LiteStore just created a new data.db file. If you try accessing the URL of the Administration App (<http://localhost:9500/docs/admin/index.html>) you won't go very far:

```
{
  error: "Document 'admin/index.html' not found."
}
```

As expected, because the datastore file is empty.

Now stop LiteStore and execute the following command:

> %terminal%
>
> litestore -d:admin import

This command will import the contents of the [admin](class:dir) folder into the datastore file, creating documents and tags automatically.

If you run LiteStore again, you'll see that everything is back in order, and you can access the Administration App again.

### The HTTP REST API

The HTTP REST API exposed by LiteStore allows you to retrieve documents in their original content type, so for example:

* <http://localhost:9500/docs/admin/index.html> retrieves the contents of the [admin/index.html](class:kwd) document in HTML format.
* <http://localhost:9500/docs/admin/index.html?raw=true> retrieves the [admin/index.html](class:kwd) document in raw (JSON) format.

This is why when you access the first URL you are able to see the app.

Queries that return a list of documents, on the other hand, always return a JSON response. For example, <http://localhost:9500/docs?tags=$subtype:css&contents=false> returns all the documents whose subtype is set to [css](class:kwd) (without displaying their contents in this case):

```
{
    "tags": [
        "$subtype:css"
    ],
    "total": 3,
    "execution_time": 0.0005269999999999997,
    "results": [
        {
            "id": "admin/styles/bootstrap-theme.min.css",
            "created": "2015-09-20T06:23:10Z",
            "modified": null,
            "tags": [
                "$type:text",
                "$subtype:css",
                "$format:text",
                "$dir:admin"
            ]
        },
        {
            "id": "admin/styles/bootstrap.min.css",
            "created": "2015-09-20T06:23:10Z",
            "modified": null,
            "tags": [
                "$type:text",
                "$subtype:css",
                "$format:text",
                "$dir:admin"
            ]
        },
        {
            "id": "admin/styles/litestore.css",
            "created": "2015-09-20T06:23:10Z",
            "modified": null,
            "tags": [
                "$type:text",
                "$subtype:css",
                "$format:text",
                "$dir:admin"
            ]
        }
    ]
}
```

For all the details on the HTTP REST API, see [the guide](/litestore/LiteStore_UserGuide.htm#HTTP.API.Reference).

### Conclusion

LiteStore does not aim to be the _next big thing_ in the NoSQL document store landscape, but it shines when used as a simple and smart way to prototype single-page applications, or implement personal apps like a local Wiki. 

Personally, I use it whenever I have to prepare a quick demo for a proof-of-concept SPA, and I don't want to have to worry about the backend. The Administration App is a great starting point to see how to use the LiteStore API to build an application. Even if you don't have any experience with the Mithril framework (neither did I until I built this!), you can have a look at the [models.js](https://github.com/h3rald/litestore/blob/master/admin/js/models.js) file to see how to access pretty much every functionality exposed by the API. I also managed to wire up a quick and dirty AngularJS service to do something similar, and it wasn't too hard.

I hope some of you will find LiteStore useful and will give it a shot. If you want to contribute, feel free to fork the [repo](https://github.com/h3rald/litestore/) and send me a pull request. It would be great to have some [Mithril](https://lhorie.github.io/mithril/) or [Nim](http://nim-lang.org/) expert to have a look at this and improve it!


### Credits

Special thanks to the following individuals, communities and organizations that made LiteStore possible:

* Dwayne Richard Hipp and all the contributors to [SQLite](http://www.sqlite.org/), the true gem that powers LiteStore.
* Andreas Rumpf and all the contributors to the [Nim Programming Language](http://nim-lang.org/), used to develop LiteStore.
* Leo Horie, for creating the [Mithril Javascript Framework](https://lhorie.github.io/mithril/), used to develop the LiteStore Administration App.
* The creators and contributors to the [Bootstrap](http://getbootstrap.com/) CSS and Javascript framework, used by the LiteStore Administration App.
* The creators and contributors to the [Ace Editor](http://ace.c9.io/), used by the LiteStore Administration App.
* Cristopher Jeffrey and all the contributors to the [Marked Javascript Library](https://github.com/chjj/marked) used by the LiteStore Administration App.