Implemented PATCH support for data.
h3rald h3rald@h3rald.com
Sun, 25 Feb 2018 15:45:21 +0100
5 files changed,
268 insertions(+),
60 deletions(-)
M
LiteStore_UserGuide.htm
→
LiteStore_UserGuide.htm
@@ -4373,6 +4373,7 @@ <li><a href="#Key-Features">Key Features</a>
<ul> <li><a href="#-Multi-format-Documents"> Multi-format Documents</a></li> <li><a href="#-Document-Tagging"> Document Tagging</a></li> + <li><a href="#-Enhanced-Querying-of-JSON-documents"> Enhanced Querying of JSON documents</a></li> <li><a href="#-Full-text-Search"> Full-text Search</a></li> <li><a href="#-RESTful-HTTP-API"> RESTful HTTP API</a></li> <li><a href="#-Directory-Bulk-Import/Export"> Directory Bulk Import/Export</a></li>@@ -4560,7 +4561,8 @@ </ul>
</li> <li><a href="#PATCH-docs/:id">PATCH docs/:id</a> <ul> - <li><a href="#Example">Example</a></li> + <li><a href="#Example:-Patching-Tags">Example: Patching Tags</a></li> + <li><a href="#Example:-Patching-Daya">Example: Patching Daya</a></li> </ul> </li> <li><a href="#DELETE-docs/:id">DELETE docs/:id</a>@@ -4714,6 +4716,11 @@ <h4><span class="fa-tags"></span> Document Tagging<a href="#document-top" title="Go to top"></a></h4>
<p>You can add custom tags to documents to easily categorize them and retrieve them. Some system tags are also added automatically to identify the document content type, format and collection.</p> +<a name="-Enhanced-Querying-of-JSON-documents"></a> +<h4><span class="fa-tasks"></span> Enhanced Querying of JSON documents<a href="#document-top" title="Go to top"></a></h4> + +<p>By leveraging the <a href="https://www.sqlite.org/json1.html">SQLite JSON1 extension</a> and implementing custom query string parsing, LiteStore provides enhanced filtering, ordering, and custom field selection of JSON documents.</p> + <a name="-Full-text-Search"></a> <h4><span class="fa-search"></span> Full-text Search<a href="#document-top" title="Go to top"></a></h4>@@ -4868,12 +4875,12 @@
<p>The easiest way to get LiteStore is by downloading one of the prebuilt binaries from the [Github Release Page][release]:</p> <ul> -<li><a href="https://github.com/h3rald/litestore/releases/download/1.2.0litestore_1.2.0_macosx_x64.zip">LiteStore for Mac OS X (x64)</a> – Compiled on OS X El Capitan (LLVM CLANG 7.0.0)</li> -<li><a href="https://github.com/h3rald/litestore/releases/download/1.2.0/litestore_1.2.0_windows_x64.zip">LiteStore for Windows (x64)</a> – Cross-compiled on OS X El Capitan (MinGW-w64 GCC 4.8.2)</li> -<li><a href="https://github.com/h3rald/litestore/releases/download/1.2.0/litestore_1.2.0_windows_x86.zip">LiteStore for Windows (x86)</a> – Cross-compiled on OS X El Capitan (MinGW-w64 GCC 4.8.2)</li> -<li><a href="https://github.com/h3rald/litestore/releases/download/1.2.0/litestore_1.2.0_linux_x64.zip">LiteStore for Linux (x64)</a> – Cross-compiled on OS X El Capitan (GNU GCC 4.8.1)</li> -<li><a href="https://github.com/h3rald/litestore/releases/download/1.2.0/litestore_1.2.0_linux_x86.zip">LiteStore for Linux (x86)</a> – Cross-compiled on OS X El Capitan (GNU GCC 4.8.1)</li> -<li><a href="https://github.com/h3rald/litestore/releases/download/1.2.0/litestore_1.2.0_linux_arm.zip">LiteStore for Linux (ARM)</a> – Cross-compiled on OS X El Capitan (GNU GCC 4.8.2)</li> +<li><a href="https://github.com/h3rald/litestore/releases/download/1.3.0litestore_1.3.0_macosx_x64.zip">LiteStore for Mac OS X (x64)</a></li> +<li><a href="https://github.com/h3rald/litestore/releases/download/1.3.0/litestore_1.3.0_windows_x64.zip">LiteStore for Windows (x64)</a></li> +<li><a href="https://github.com/h3rald/litestore/releases/download/1.3.0/litestore_1.3.0_windows_x86.zip">LiteStore for Windows (x86)</a></li> +<li><a href="https://github.com/h3rald/litestore/releases/download/1.3.0/litestore_1.3.0_linux_x64.zip">LiteStore for Linux (x64)</a></li> +<li><a href="https://github.com/h3rald/litestore/releases/download/1.3.0/litestore_1.3.0_linux_x86.zip">LiteStore for Linux (x86)</a></li> +<li><a href="https://github.com/h3rald/litestore/releases/download/1.3.0/litestore_1.3.0_linux_arm.zip">LiteStore for Linux (ARM)</a></li> </ul>@@ -4904,7 +4911,7 @@
<p>To get the app up and running (assuming that you have the <span class="cmd">litestore</span> executable in your path):</p> <ol> -<li>Download the default <a href="https://github.com/h3rald/litestore/releases/download/1.2.0/data.db">data.db</a> file. This file is a LiteStore data store file containing the sample app.</li> +<li>Download the default <a href="https://github.com/h3rald/litestore/releases/download/1.3.0/data.db">data.db</a> file. This file is a LiteStore data store file containing the sample app.</li> <li>Go to the local directory in which you downloaded the <span class="cmd">data.db</span> file.</li> <li>Run <span class="cmd">litestore -s:data.db</span></li> <li>Go to <a href="http://localhost:9500/docs/admin/index.html">localhost:9500/docs/admin/index.html</a>.</li>@@ -5479,7 +5486,7 @@ <li>A value that can be a number, string, <strong>true</strong>, <strong>false</strong> or <strong>nil</strong></li>
</ul> -<div class="sidebar"><p>Limitations</p> +<div class="warning"><p>Limitations</p> <ul> <li>Parenthesis are not supported.</li>@@ -5701,20 +5708,23 @@
<a name="PATCH-docs/:id"></a> <h4>PATCH docs/:id<a href="#document-top" title="Go to top"></a></h4> -<p>Adds, removes, replaces or tests the specified document for tags. Operations must be specified using the <a href="http://jsonpatch.com/">JSONPatch</a> format.</p> +<p>Adds, removes, replaces or tests the specified document for tags or data. Operations must be specified using the <a href="http://jsonpatch.com/">JSONPatch</a> format.</p> -<p>Always retrieve document tags first before applying a patch, to know the order tags have been added to the document.</p> +<div class="tip"><p>Tip</p> + +<p>If you plan on patching tags, always retrieve document tags first before applying a patch, to know the order tags have been added to the document.</p></div> <div class="warning"><p>Limitations</p> <ul> <li>Only <strong>add</strong>, <strong>remove</strong>, <strong>replace</strong> and <strong>test</strong> operations are supported.</li> -<li>It is currently only possible to change tags, not other parts of a document.</li> +<li>It is only possible to patch <strong>data</strong> and/or <strong>tags</strong> of a document.</li> +<li>It is only possible to patch the data of JSON documents.</li> </ul> </div> -<a name="Example"></a> -<h5>Example<a href="#document-top" title="Go to top"></a></h5> +<a name="Example:-Patching-Tags"></a> +<h5>Example: Patching Tags<a href="#document-top" title="Go to top"></a></h5> <pre><code>$ curl -i -X PATCH 'http://localhost:9500/docs/test.json' --header "Content-Type:application/json" -d '[{"op":"add", "path":"/tags/3", "value":"test1"},{"op":"add", "path":"/tags/4", "value":"test2"},{"op":"add", "path":"/tags/5", "value":"test3"}]' HTTP/1.1 200 OK@@ -5727,6 +5737,43 @@
{"id": "test.json", "data": {"test": true}, "created": "2015-09-20T09:06:25Z", "modified": null, "tags": ["$type:application", "$subtype:json", "$format:text", "test1", "test2", "test3"]} </code></pre> +<a name="Example:-Patching-Daya"></a> +<h5>Example: Patching Daya<a href="#document-top" title="Go to top"></a></h5> + +<p>Given the following document:</p> + +<pre><code>{ + "id": "test", + "data": { + "name": { + "first": "Tom", + "last": "Paris" + }, + "rank": "lieutenant" + }, + "created": "2018-02-25T14:33:14Z", + "modified": null, + "tags": [ + "$type:application", + "$subtype:json", + "$format:text" + ] +} +</code></pre> + +<p>The following PATCH operation can be applied to change its data.</p> + +<pre><code>$ curl -i -X PATCH 'http://localhost:9500/docs/test' --header "Content-Type:application/json" -d '[{"op": "replace", "path": "/data/name", "value": "Seven of Nine"}, {"op": "remove", "path": "/data/rank"}]' +HTTP/1.1 200 OK +server: LiteStore/1.3.0 +access-control-allow-origin: * +content-type: application/json +access-control-allow-headers: Content-Type +Content-Length: 172 + +{"id":"test","data":{"name":"Seven of Nine"},"created":"2018-02-25T14:33:14Z","modified":"2018-02-25T14:35:52Z","tags":["$type:application","$subtype:json","$format:text"]} +</code></pre> + <a name="DELETE-docs/:id"></a> <h4>DELETE docs/:id<a href="#document-top" title="Go to top"></a></h4>@@ -6167,7 +6214,7 @@ </ul>
</div> <div id="footer"> - <p><span class="copy"></span> Fabio Cevasco – February 24, 2018</p> + <p><span class="copy"></span> Fabio Cevasco – February 25, 2018</p> <p><span>Powered by</span> <a href="https://h3rald.com/hastyscribe"><span class="hastyscribe"></span></a></p> </div> </div>
M
admin/md/api_docs.md
→
admin/md/api_docs.md
@@ -222,7 +222,7 @@ * A path expression indicating a field or array item within the JSON document.
* One operator among the following: eq, not eq, gt, gte, lt, lte, contains. * A value that can be a number, string, **true**, **false** or **nil** -> %sidebar% +> %warning% > Limitations > > * Parenthesis are not supported.@@ -431,17 +431,21 @@ ```
#### PATCH docs/:id -Adds, removes, replaces or tests the specified document for tags. Operations must be specified using the [JSONPatch](http://jsonpatch.com/) format. +Adds, removes, replaces or tests the specified document for tags or data. Operations must be specified using the [JSONPatch](http://jsonpatch.com/) format. -Always retrieve document tags first before applying a patch, to know the order tags have been added to the document. +> %tip% +> Tip +> +> If you plan on patching tags, always retrieve document tags first before applying a patch, to know the order tags have been added to the document. > %warning% > Limitations -> +> > * Only **add**, **remove**, **replace** and **test** operations are supported. -> * It is currently only possible to change tags, not other parts of a document. +> * It is only possible to patch **data** and/or **tags** of a document. +> * It is only possible to patch the data of JSON documents. -##### Example +##### Example: Patching Tags ``` $ curl -i -X PATCH 'http://localhost:9500/docs/test.json' --header "Content-Type:application/json" -d '[{"op":"add", "path":"/tags/3", "value":"test1"},{"op":"add", "path":"/tags/4", "value":"test2"},{"op":"add", "path":"/tags/5", "value":"test3"}]'@@ -453,6 +457,44 @@ Access-Control-Allow-Origin: *
Server: LiteStore/1.0.3 {"id": "test.json", "data": {"test": true}, "created": "2015-09-20T09:06:25Z", "modified": null, "tags": ["$type:application", "$subtype:json", "$format:text", "test1", "test2", "test3"]} +``` + +##### Example: Patching Daya + +Given the following document: + +``` +{ + "id": "test", + "data": { + "name": { + "first": "Tom", + "last": "Paris" + }, + "rank": "lieutenant" + }, + "created": "2018-02-25T14:33:14Z", + "modified": null, + "tags": [ + "$type:application", + "$subtype:json", + "$format:text" + ] +} +``` + +The following PATCH operation can be applied to change its data. + +``` +$ curl -i -X PATCH 'http://localhost:9500/docs/test' --header "Content-Type:application/json" -d '[{"op": "replace", "path": "/data/name", "value": "Seven of Nine"}, {"op": "remove", "path": "/data/rank"}]' +HTTP/1.1 200 OK +server: LiteStore/1.3.0 +access-control-allow-origin: * +content-type: application/json +access-control-allow-headers: Content-Type +Content-Length: 172 + +{"id":"test","data":{"name":"Seven of Nine"},"created":"2018-02-25T14:33:14Z","modified":"2018-02-25T14:35:52Z","tags":["$type:application","$subtype:json","$format:text"]} ``` #### DELETE docs/:id
M
admin/md/overview.md
→
admin/md/overview.md
@@ -27,6 +27,10 @@ #### [](class:fa-tags) Document Tagging
You can add custom tags to documents to easily categorize them and retrieve them. Some system tags are also added automatically to identify the document content type, format and collection. +#### [](class:fa-tasks) Enhanced Querying of JSON documents + +By leveraging the [SQLite JSON1 extension](https://www.sqlite.org/json1.html) and implementing custom query string parsing, LiteStore provides enhanced filtering, ordering, and custom field selection of JSON documents. + #### [](class:fa-search) Full-text Search By leveraging [SQLite FTS4 extension](http://www.sqlite.org/fts3.html) and implementing an enhanced algorithm for result rankings, LiteStore provides full-text search for all textual documents out-of-the-box.@@ -41,4 +45,4 @@ To make serving a single-page application _from LiteStore_ even easier and faster, you can automatically import (and export) the contents of a directory recursively.
#### [](class:fa-cloud-upload) Directory Mounting and Mirroring -After importing the contents of a directory into a LiteStore data store, you can _mount it_ on LiteStore and mirror all data store changes to the filesystem. Incidentally, that's how most of the LiteStore Admin test app was built [](class:fa-smile-o).+After importing the contents of a directory into a LiteStore data store, you can _mount it_ on LiteStore and mirror all data store changes to the filesystem. Incidentally, that's how most of the LiteStore Admin test app was built [](class:fa-smile-o).
M
lib/api_v3.nim
→
lib/api_v3.nim
@@ -42,7 +42,6 @@ """
for f in fragments: var matches = @["", ""] if f.find(clause, matches) != -1: - let direction = matches[0] let field = "json_extract(documents.data, '$1')" % matches[1] if matches[0] == "-": clauses.add("$1 DESC" % field)@@ -196,19 +195,16 @@ else:
discard return cb(req, LS, resource, id) -proc applyPatchOperation*(tags: var seq[string], op: string, path: string, value: string): bool = - var matches = @[""] - if path.find(peg"^\/tags\/{\d+}$", matches) == -1: - raise newException(EInvalidRequest, "cannot patch path '$1'" % path) - let index = matches[0].parseInt - LOG.debug("- PATCH -> $1 tag index '$2' - Total tags: $3." % [op, $index, $tags.len]) + +proc patchTag(tags: var seq[string], index: int, op, path, value: string): bool = + LOG.debug("- PATCH -> $1 tag['$2'] = \"$3\" - Total tags: $4." % [op, $index, $value, $tags.len]) case op: of "remove": let tag = tags[index] if not tag.startsWith("$"): tags[index] = "" # Not removing element, otherwise subsequent indexes won't work! else: - raise newException(EInvalidRequest, "Cannot remove system tag: $1" % tag) + raise newException(EInvalidRequest, "cannot remove system tag: $1" % tag) of "add": if value.match(PEG_USER_TAG): tags.insert(value, index)@@ -220,7 +216,7 @@ raise newException(EInvalidRequest, "invalid tag: $1" % value)
of "replace": if value.match(PEG_USER_TAG): if tags[index].startsWith("$"): - raise newException(EInvalidRequest, "Cannot replace system tag: $1" % tags[index]) + raise newException(EInvalidRequest, "cannot replace system tag: $1" % tags[index]) else: tags[index] = value else:@@ -232,8 +228,90 @@ of "test":
if tags[index] != value: return false else: - raise newException(EInvalidRequest, "invalid operation: $1" % op) + raise newException(EInvalidRequest, "invalid patch operation: $1" % op) + return true + +proc patchData*(data: var JsonNode, origData: JsonNode, op: string, path: string, value: JsonNode): bool = + LOG.debug("- PATCH -> $1 path $2 with $3" % [op, path, $value]) + var keys = path.replace(peg"^\/data\/", "").split("/") + if keys.len == 0: + raise newException(EInvalidRequest, "no valid path specified: $1" % path) + var d = data + var dorig = origData + var c = 1 + for key in keys: + if d.kind == JArray: + try: + var index = key.parseInt + if c >= keys.len: + d.elems[index] = value + case op: + of "remove": + d.elems.del(index) + of "add": + d.elems.insert(value, index) + of "replace": + d.elems[index] = value + of "test": + if d.elems[index] != value: + return false + else: + raise newException(EInvalidRequest, "invalid patch operation: $1" % op) + else: + d = d[index] + dorig = dorig[index] + except: + raise newException(EInvalidRequest, "invalid index key '$1' in path '$2'" % [key, path]) + else: + if c >= keys.len: + case op: + of "remove": + if d.hasKey(key): + d.delete(key) + else: + raise newException(EInvalidRequest, "key '$1' not found in path '$2'" % [key, path]) + of "add": + d[key] = value + of "replace": + if d.hasKey(key): + d[key] = value + else: + raise newException(EInvalidRequest, "key '$1' not found in path '$2'" % [key, path]) + of "test": + if dorig.hasKey(key): + if dorig[key] != value: + return false + else: + raise newException(EInvalidRequest, "key '$1' not found in path '$2'" % [key, path]) + else: + raise newException(EInvalidRequest, "invalid patch operation: $1" % op) + else: + d = d[key] + dorig = dorig[key] + c += 1 return true + + +proc applyPatchOperation*(data: var JsonNode, origData: JsonNode, tags: var seq[string], op: string, path: string, value: JsonNode): bool = + var matches = @[""] + let p = peg""" + path <- ^tagPath / fieldPath$ + tagPath <- '\/tags\/' {\d+} + fieldPath <- '\/data\/' ident ('\/' ident)* + ident <- [a-zA-Z0-9_]+ / '-' + """ + if path.find(p, matches) == -1: + raise newException(EInvalidRequest, "cannot patch path '$1'" % path) + if path.match(peg"^\/tags\/"): + let index = matches[0].parseInt + if value.kind != JString: + raise newException(EInvalidRequest, "tag '$1' is not a string." % $value) + let tag = value.getStr + return patchTag(tags, index, op, path, tag) + elif tags.contains("$subtype:json"): + return patchData(data, origData, op, path, value) + else: + raise newException(EInvalidRequest, "cannot patch data of a non-JSON document.") # Low level procs@@ -377,21 +455,28 @@ let jbody = body.parseJson
if jbody.kind != JArray: return resError(Http400, "Bad request: PATCH request body is not an array.") var options = newQueryOptions() - options.select = @["documents.id AS id", "created", "modified"] + options.select = @["documents.id AS id", "created", "modified", "data"] let doc = LS.store.retrieveRawDocument(id, options) if doc == "": return resDocumentNotFound(id) let jdoc = doc.parseJson - var tags = newSeq[string](0) + var tags = newSeq[string]() + var origTags = newSeq[string]() for tag in jdoc["tags"].items: tags.add(tag.str) + origTags.add(tag.str) + var data: JsonNode + var origData: JsonNode + if tags.contains("$subtype:json"): + origData = jdoc["data"].getStr.parseJson + data = origData.copy var c = 1 for item in jbody.items: if item.hasKey("op") and item.hasKey("path"): if not item.hasKey("value"): item["value"] = %"" try: - apply = applyPatchOperation(tags, item["op"].str, item["path"].str, item["value"].str) + apply = applyPatchOperation(data, origData, tags, item["op"].str, item["path"].str, item["value"]) if not apply: break except:@@ -400,14 +485,22 @@ else:
return resError(Http400, "Bad request: patch operation #$1 is malformed." % $c) c.inc if apply: - try: - for t1 in jdoc["tags"].items: - discard LS.store.destroyTag(t1.str, id, true) - for t2 in tags: - if t2 != "": - LS.store.createTag(t2, id, true) - except: - return resError(Http500, "Unable to patch document '$1' - $2" % [id, getCurrentExceptionMsg()]) + if not origData.isNil and origData != data: + try: + var doc = LS.store.updateDocument(id, data.pretty, "application/json") + if doc == "": + return resError(Http500, "Unable to patch document '$1'." % id) + except: + return resError(Http500, "Unable to patch document '$1' - $2" % id, getCurrentExceptionMsg()) + if origTags != tags: + try: + for t1 in jdoc["tags"].items: + discard LS.store.destroyTag(t1.str, id, true) + for t2 in tags: + if t2 != "": + LS.store.createTag(t2, id, true) + except: + return resError(Http500, "Unable to patch document '$1' - $2" % [id, getCurrentExceptionMsg()]) return LS.getRawDocument(id) # Main routing
M
test/http_api.nim
→
test/http_api.nim
@@ -45,7 +45,7 @@ ids.add(id)
var ops = """ [ {"op": "add", "path": "/tags/3", "value": "tag1$1"}, - {"op": "add", "path": "/tags/4", "value": "tag$2"}, + {"op": "add", "path": "/tags/4", "value": "tag$2"} ] """ % [$count, $(count mod 2)] discard jpatch("docs/" & ids[count], ops.parseJson)@@ -95,44 +95,66 @@
test "PATCH document tags": var rget = jget("docs?tags=t1") check(rget.status == "404 Not Found") - var ops = """ - [ + var ops = %*[ {"op": "add", "path": "/tags/3", "value": "t1"}, {"op": "add", "path": "/tags/4", "value": "t2"}, {"op": "add", "path": "/tags/5", "value": "t3"} ] - """ - var rpatch = jpatch("docs/" & ids[0], ops.parseJson) + var rpatch = jpatch("docs/" & ids[0], ops) check(rpatch.status == "200 OK") rget = jget("docs/?tags=t1") check(rget.body.parseJson["total"] == %1) - ops = """ - [ + ops = %*[ {"op": "add", "path": "/tags/3", "value": "t1"}, {"op": "add", "path": "/tags/4", "value": "t3"} ] - """ - rpatch = jpatch("docs/" & ids[1], ops.parseJson) + rpatch = jpatch("docs/" & ids[1], ops) check(rpatch.status == "200 OK") - ops = """ - [ + ops = %*[ {"op": "add", "path": "/tags/3", "value": "t2"}, {"op": "add", "path": "/tags/4", "value": "t3"} ] - """ - rpatch = jpatch("docs/" & ids[2], ops.parseJson) + rpatch = jpatch("docs/" & ids[2], ops) check(rpatch.status == "200 OK") - ops = """ - [ + ops = %*[ {"op": "replace", "path": "/tags/3", "value": "t4"}, {"op": "remove", "path": "/tags/4"} ] - """ - rpatch = jpatch("docs/" & ids[0], ops.parseJson) + rpatch = jpatch("docs/" & ids[0], ops) check(rpatch.status == "200 OK") rget = jget("docs/?tags=t2,t3") check(rget.body.parseJson["total"] == %1) check(info("total_documents") == %8) + + test "PATCH document data": + var ops = %*[ + {"op": "remove", "path": "/data/name/first"}, + {"op": "add", "path": "/data/test", "value": 111}, + {"op": "replace", "path": "/data/friends/0", "value": {"id": 11, "name": "Tom Paris"}} + ] + var rpatch = jpatch("docs/" & ids[0], ops) + var data = rpatch.body.parseJson["data"] + check(data["name"] == %*{"last": "Walters"}) + check(data["test"] == %111) + check(data["friends"][0] == %*{"id": 11, "name": "Tom Paris"}) + ops = %*[ + {"op": "add", "path": "/data/not_added", "value": "!!!"}, + {"op": "test", "path": "/data/test", "value": 222}, + {"op": "replace", "path": "/data/test", "value": "!!!"} + ] + rpatch = jpatch("docs/" & ids[0], ops) + data = rpatch.body.parseJson["data"] + check(data["test"] == %111) + check(data.hasKey("not_added") == false) + ops = %*[ + {"op": "replace", "path": "/data/test", "value": 222}, + {"op": "test", "path": "/data/test", "value": 222}, + {"op": "add", "path": "/data/not_added", "value": "!!!"} + ] + rpatch = jpatch("docs/" & ids[0], ops) + data = rpatch.body.parseJson["data"] + check(data["test"] == %111) + check(data.hasKey("not_added") == false) test "HEAD documents": var rhead = jhead("docs/invalid/")