Notes 2018-06-02: Universal (isomorphic) JavaScript databases, IPFS (OrbitDB), DAT (HyperDB/Hyperdrive)


#1

Universal JavaScript databases

Spent the week continuing to look into universal JavaScript databases for Indie Site.

Candidates:

Dexie

Dexie has an awesome API. It uses IndexedDB in the browser, SQLite under Node.js (via IndexedDBShim).

I got the test suite running under Node.js and discovered a number of issues when running under IndexedDBShim – some of which are IndexedDBShim issues. David, who’s been awesome, made quite a bit of progress in fixing some of those.

Live pull-request of my branch: https://github.com/dfahlander/Dexie.js/pull/707

I have real concerns of the weight/stability of IndexedDBShim, however.

JungleDB

Towards the end of the week, I stumbled on a relatively new isomorphic/universal JavaScript database called JungleDB. It’s being used in production by an in-browser blockchain project called Nimiq. It has a comprehensive test suite and benchmarks. I also built and ran the Nimiq client and it was rather performant.

JungleDB is a light abstraction on top on IndexedDB in the browser and on top of LevelDB on Node.js that is geared towards performance and very suitable for Directed Acyclic Graph/Merkle Tree use cases. It has secondary indices and transactions.

My concern is whether it is too low level.

NanoSQL

NanoSQL also uses IndexedDB in the browser and LevelDB on Node.js. It has a SQL interface, which I’m not the biggest fan of (but I wouldn’t dismiss it purely on that account by any means). The developer, Scott Lott is ver responsive and handled a couple of issues almost immediately (they were regarding write performance with secondary indices in under Node.js and an issue with auto-incremented primary keys being incorrectly formed).

However, it does seem like a young and less tested solution (I couldn’t find a test suite for the browser, only Node, for example) and the performance with multiple indices is still far lower than in Dexie.

Am I thinking too low level?

After my experience with Heartbeat, where we didn’t have control over the core data model and replication engine (Syncthing), have I gone too far in the opposite direction? Right now, I have a very good idea of the conflict-free replicated datatype and the directed acyclic graph that will form the data model. I also have a good idea of which database I would go with (JungleDB). However, I am going to have to recreate from scratch the work that’s taken the (very smart) folks on the DAT and IPFS projects years to implement (and both projects are still relatively in their infancy). (Shout outs to Secure Scuttlebutt also. Not mentioned here since it’s approach is a little different to my needs.)

Anyway, so before I commit to (re)inventing the wheel, I’m going to spend the next week taking a deep dive into the current state of IPFS and DAT. I want to see whether it might save me a lot of work (and also include us in a vibrant already-existing community) if we built higher-level abstractions on top of the base functionality provided by IPFS or DAT/Hyperdrive/HyperDB/Hypercore.

DAT or IPFS?

A cursory glance today showed me that while the DAT project has deprecated its in-browser implementation effort while that of IPFS is still going strong. They’re also researching CRDTs and have a great demo of using IPFS in conjunction with Y.js.

And they even have a decentralised database called OrbitDB that runs under both browsers and Node. So I think that’s where I’ll start my exploration… (See update, below.)


#2

Just had a look at the IPFS investors and cooled right off. Something always felt off about that effort and I now know what it is.

(Also, Y.js is Operational Transformations-based. So no go. I’m looking for a CRDT-based solution.)


#3

Also, from a purely technical standpoint, just did a quick test with OrbitDB. The browserified source of a basic example with minimum imports comes out at 6MB uncompressed/3.2MB minified. That’s a no go.


#4

Just did a quick test with DAT-JS as well and couldn’t get the basic example running in either Node or the browser:

[Error] Failed to load resource: The certificate for this server is invalid. You might be connecting to a server that is pretending to be “signalhub.mafintosh.com” which could put your confidential information at risk. (all, line 0)
[Error] Failed to load resource: The certificate for this server is invalid. You might be connecting to a server that is pretending to be “signalhub.mafintosh.com” which could put your confidential information at risk. (cjhxzidxt00003e4q0zbglrab, line 0)
…

Sample code from the readme:

var Dat = require('dat-js')

var dat = Dat()
dat.add(function (repo) {
  console.log('dat key is:', repo.key)
  var writer = repo.archive.createFileWriteStream('hello.txt')
  writer.write('world')
  writer.end(function () { replicate(repo.key) })
})

function replicate (key) {
  var clone = Dat()
  clone.add(key, function (repo) {
    var readStream = repo.archive.createFileReadStream('hello.txt')
    readStream.on('data', function (data) {
      console.log(data.toString()) // prints 'world'
    })
  })
}

#5

Right, I’ve decided to roll my own. That way we can design exactly what we need for our use case from the data type(s) up.

Update: 2018/06/05: Seriously looking into hypercore, hyperdb, hyperdrive and DAT. I was initially put off my the lack of documentation + what appeared to be a lack of interest in supporting plain vanilla browser (web) apps/PWAs but having played with hypercore today and having just stumbled on Jim Pick’s excellent decentralised shopping list PWA, I am going to devote some time into really trying to wrap my brain around this. It looks like Mathias and co. have built most of what we need for Indie Site.


#6

Following with extreme interest. I need to remove firebase from my vue.js web thing. Will look at DAT indeed though I think I need multi writer which looks very bleeding edge.


#7

Yes, multi-writer is what makes my use-case possible. So happy I don’t have to reinvent the wheel; especially given the geniuses working on DAT :slight_smile: