I got the test suite running under Node.js and discovered a number of issues when running under IndexedDBShim – some of which are IndexedDBShim issues. David, who’s been awesome, made quite a bit of progress in fixing some of those.
Live pull-request of my branch: https://github.com/dfahlander/Dexie.js/pull/707
I have real concerns of the weight/stability of IndexedDBShim, however.
JungleDB is a light abstraction on top on IndexedDB in the browser and on top of LevelDB on Node.js that is geared towards performance and very suitable for Directed Acyclic Graph/Merkle Tree use cases. It has secondary indices and transactions.
My concern is whether it is too low level.
NanoSQL also uses IndexedDB in the browser and LevelDB on Node.js. It has a SQL interface, which I’m not the biggest fan of (but I wouldn’t dismiss it purely on that account by any means). The developer, Scott Lott is ver responsive and handled a couple of issues almost immediately (they were regarding write performance with secondary indices in under Node.js and an issue with auto-incremented primary keys being incorrectly formed).
However, it does seem like a young and less tested solution (I couldn’t find a test suite for the browser, only Node, for example) and the performance with multiple indices is still far lower than in Dexie.
Am I thinking too low level?
After my experience with Heartbeat, where we didn’t have control over the core data model and replication engine (Syncthing), have I gone too far in the opposite direction? Right now, I have a very good idea of the conflict-free replicated datatype and the directed acyclic graph that will form the data model. I also have a good idea of which database I would go with (JungleDB). However, I am going to have to recreate from scratch the work that’s taken the (very smart) folks on the DAT and IPFS projects years to implement (and both projects are still relatively in their infancy). (Shout outs to Secure Scuttlebutt also. Not mentioned here since it’s approach is a little different to my needs.)
Anyway, so before I commit to (re)inventing the wheel, I’m going to spend the next week taking a deep dive into the current state of IPFS and DAT. I want to see whether it might save me a lot of work (and also include us in a vibrant already-existing community) if we built higher-level abstractions on top of the base functionality provided by IPFS or DAT/Hyperdrive/HyperDB/Hypercore.
DAT or IPFS?
A cursory glance today showed me that while the DAT project has deprecated its in-browser implementation effort while that of IPFS is still going strong. They’re also researching CRDTs and have a great demo of using IPFS in conjunction with Y.js.
And they even have a decentralised database called OrbitDB that runs under both browsers and Node.
So I think that’s where I’ll start my exploration… (See update, below.)