I'm more than a little worried about the following all being `Coming Soon`:
- Cursors & carets
- Two way sync to your DB
- Video presence & calls
- both the Group and BinaryCoStreams
All of these are the key reasons I would be evaluating this framework to handle my data. All of these are not fully implemented yet.
It is these key topics of live reloading/updating data that make or break an app. In my opinion, if you haven't concretely solved these problems, you haven't really built a viable state framework for 2024.
I really like the patterns they've implemented though, looks a lot like the same framework I just built on top of MobX, websockets, and React for a recent project. They're headed in the right direction, but I'm not sure they realize how much more work they have to go before this is fully fleshed out.
Which brings in even more questions. What is the performance of these multiplayer experiences? Can I have 1000+ users all connected to the same chat session? What about 10,000?
(these numbers might seem high, but they're what I'm expected to deliver in my day-job)
Eventually, yes! The origin story of Jazz is that I did a lot of experiments and research to convince myself of satisfying performance characteristics. Right now, the pure-TypeScript implementation is not very optimised and I'm very intentionally optimising it as needed for early adopter apps instead of predicting where bottlenecks are. So if you start building a chat app with 10s of thousands of users interacting, we'll make sure that's possible by when you want to launch.
This lets us more easily prioritise features vs performance.
Not to go off topic but isn't performance metrics your responsibility ? Those are small numbers in reality, but if you are not performing your own tests and trusting the maintainer/publisher instead then you're not doing your 'day-job' properly.
Yeah, I hear you. Which is why we assessed and designed our own platform for handling the above stated traffic (and much more) for multi-user real-time chat sessions.
The commenter seems to know a lot about Jazz, so I took the opportunity to ask further questions.
Good stuff, and I hope I didn’t come across too rude but 10k parallel is not a problem for even basic setups to manage, it’s a problem solved many different ways some out of the box some not. Maybe your question was phrased wrong. But hit into 10kk parallel, there’s an interesting metric to discuss optimization.
I’m reminded of the quasi meme “You have designed an infinitely scalable service for 10 clients “
What plans do you have for a third-party audit/review of your backend to verify the claims being made?
Its one thing to use this service on the basis of encryption claims - its another thing to have to clean up the mess from a forgotten API key being leaked somewhere... is there, therefore, a third party involved in an audit of Jazz?
First it's all open source so even without knowing that my paid service uses the same open source libraries, you can verify yourself that nothing leaves the client unencrypted.
Of course we will also do public audit(s) with security companies to make sure our cryptographic protocols are sound and implemented correctly.
Thanks for the info. Of course, "its all open source" is a fair answer, but its also a dangerous one, since it shifts culpability for verification to anyone who has the time/resources to audit the open source components that you're using.
But, it also has to be stated, there is no guarantee that you're using only the open source components you've declared, which is why an impartial audit by a third party would be necessary before this project can be used to build products in some industries.
Anyway, I see that you have other concerns, so no worries and thanks for the honest answer.
Seems like you have lots of real world mobx experience.
Have you ever read this article, basically saying that every front end state (including mobx) basically ends up being a worse version of a standard database?
I ended up finding that article after running into lots of the challenges with mobx State tree. I ended up trying to use watermelondb, a sqlite wrapped for react native, but gave up entirely on offline due to bugs and project abandonment
Yes, I have read and found similar articles to what you've posted!
It is an ongoing exploration for us to find the best way to store/consume reactive state data within our client sites/apps. So far, MobX has given us a lot of leeway since we can create the data store we want for our data.
Going forward, I foresee us switching to something closer to WatermelonDB or maybe even just SQLite with a thin wrapper in React to create observability. I'm not 100% sure where it's going to go, but I agree with you that flexible state management for large platforms evolves fast and the difficulty of creating fast look-ups rapidly approaches "reinventing the DB" client-side.
I've had to build similar on top of SQLite for various other mobile/desktop apps, the problem hasn't really changed, the only difference is we're now using React + JS instead of C++, C#, or Java.
I've done the same with mobx-state-tree, and realized I was essentially building a relational database with cache-ing in JavaScript.
The only thing SQLite is missed, which is important when integrating with React, is incremental view maintenance: how do we maintain as much immutable state on an update as possible to avoid wasteful re-renders?
i have built similar thing long time ago, using couchdb as replication-storage mechanism, and it's variants on non-server platforms - touchdb/android/ios <-> pouchdb/js-browser etc , but yes, except serverside, all other platforms still had sqlite underneath.
The analogy is more like an easier to implement version of a system(Custom Frontend Data Store) is worse than the standardized system (Database) rather than one system(database) being implemented on top of another system(filesystem).
The database has more features relative to filesystem, so we wouldn't miss the filesystem whereas a powerful indexing system is a feature in database missing in frontend data stores.
Some other examples below(Firefox plugins which are file browsers, FTP clients, C programs informally implementing subset of Common LISP features).
I would retort with, stop trying to chuck a SQL database everywhere.
The issue imho is that multiplayer synchronization (like Jazz) should rely on primitives(CRDT or similar) that you really can't hide under abstractions but should be part of the datamodel to allow for custom synchronization protocols for different parts of the datamodel or even exposing synchronization faults to the user.
An analogy would perhaps be how wrongly abstracted systems like CORBA and DCOM lost out (Even if you can run into them in crevices of enterprise systems) to things centered around HTTP calls or simpler message oriented systems like gRPC.
Plus the sync only works between clients AFAICT — literally all the listed backends are also “coming soon”. Which is troubling, I’d guess that almost all local-first applications need some sort of trusted verification server, at the very least. But maybe I’m biased/unimaginative; I suppose some kinds of collaboration apps (figma, google docs, etc) might work fine serverless? Certainly not games or AI agents, right?
This seems awesome, but my natural game-stopping questions are;
1. When will backends be supported?
2. Why doesn’t the site mention “local-first”?
Oh and just for fun;
3. Why “Collaborative Values” instead of what I learned in school, “Shared Memory”?
1. The sync is not just between clients, it by default works over a central backend infrastructure that syncs and persists. Plus you can create server workers (with jazz-nodejs) that also are clients to the same infrastructure and can do side-effects like API calls and interacting with “normal” databases in response to react state changing. And they can also mutate Jazz state.
2. The next version will. I like to think of Jazz as distributed state, of which local-first is a special case.
3. Shared memory to me sounds like concurrency is handled with locks, while Jazz uses CRDTs
Over the past few years, I've worked on and used a bunch of different frameworks to try to solve for distributed state. Jazz feels like it has the right "shape" - the right primitives to quickly build an app with magical powers like offline, real-time collab, etc.
If you can push the distributed state problem below the framework level, it becomes dead simple to build apps, which feels great.
Jess would know, DXOS is also amazing and in this space. I learned about DXOS at the local-first unconference after Strange Loop and was very impressed by Jess and his work.
anyone thinking about hanging their hat on writing decentralized, local first, crdt-driven software should also consider the complexity of changing your schema when there are potentially thousands of clients with different opinions of what that schema should be. some interesting research has been done[1] in this domain but i haven't seen any library of this ilk that supports real-world schema evolution in a way doesn't make me really, really wish i had a proper backend to lay down the law. the fact that migrations are in a "coming soon" state makes me wary of using jazz, and i wonder how they would approach this problem with the library as it is now.
It’s a tough problem! I think 90% of it is solved with GraphQL/ protobuf like rules (“only ever introduce new fields”)
There are some edge cases where you might loose easy access to some data if two clients migrate concurrently, but we’re hoping to provide patterns to mitigate these
Edit: Right now it all depends on you to implement migrations “the right way” but we hope to provide guardrails soon
I've been thinking about this as well and wondering if one possible approach to avoid the eventual messy database state—where fields are "append-only" and never deleted—might be to include the up/down migration logic as part of the payload.
This approach may require a central authority (with no access to user data) responsible solely for providing the schema and migration patterns as code transformations.
Since running arbitrary code from the payload introduces potential security risks, the migration code could be cryptographically signed, ensuring that only valid, trusted transformation code is executed.
A possible additional security layer would be to have the transformation code execute in a sandbox which can only output JSON data. (keeping a possible full before-migration version as backup in case something went wrong would always be a good idea)
Another option would be to use a transformation library for migrations, but in this case, the approach would only describe (as JSON) the functions and parameters needed to transition the schema from one version to another.
> I think 90% of it is solved with GraphQL/ protobuf like rules (“only ever introduce new fields”)
Agreed, that’s the only
sensible thing to do. Not sure it’s 90% though.
> but we’re hoping to provide patterns to mitigate these
Hope is not confidence inspiring for the most difficult problem at the heart of the system. That doesn’t mean it has to be impossible, but it needs to be taken seriously and not an afterthought.
Another thing you have to think about is what happens when data of a new schema is sent to a client on an older schema. Does the “merging” work with unknown fields? Does it ignore and drop them? Or do you enforce clients are up to date in some way so that you don’t have new-data-old-software?
Founder here - thanks for posting this! It’s getting late where I am so everyone who has questions please don’t be shy to ask, will elaborate in detail tomorrow!
How does the mesh (zero-server) work under-the-hood? What are some example of applications you see being possible with this library? Your site also mentions either using a central server or not having a server. The first idea that comes to mind (perhaps because of your chat example on the homepage) is a decentralised P2P messaging app - something like Messenger or WhatsApp but is completely free from central servers.
The underlying protocol is peer-to-peer sync, so the mesh is really just a bunch of geographically distributed servers that sync between each other as needed and act as "one giant syncing & persistence peer" for your app - mostly so you can sync stuff between devices even if they're not online at the same time, and to use it like cloud storage (I like to call it "S3 for collaborative data")
On a high level, it solves a similar problem: a full backend so you mostly only have to write a frontend.
But Jazz does so in a completely different (local-first) way: it implements shared state on top of CRDTs and implements user identity and permissions based on public-key cryptography. This means you can create, store and share data from the client (even while offline) while still benefitting from cloud storage and real-time sync between devices and users by default.
What happens to the mesh if this doesn't work out as a commercial enterprise and the company behind it goes away? Practically speaking how much of it is running on your computer's today?
Well the version of the Mesh that we offer would go away, but the framework is open source as is a simple but fully functional version of the sync server, so you could somewhat easily host your own mesh. Not having lock-in here is super important for us
Nice abstractions. Though I worry they’re going to be too rigid for some applications.
- how can I setup delivery receipts for messages in your chat app? I want to know if they got my message and when it was sent.
- how can I create a chat group where the entire list of access permissions is not transparently available to everyone? For example, customer support: I don’t want to give out a list of every employee because a customer opens a chat that any employee and respond to.
- how are you handling p2p sync to multiple devices with the same account? Are all of my devices made public to anyone I collaborate with? Or does one device always need to be online?
- add a “read” property to Message that the other person sets
- groups not only allow adding exact accounts, but also allow for “invites” which in your example only one employee could accept for a new support chat
- devices aren’t synced p2p by default but through the syncing server like everything else. Right now, all devices in one account use the same keypair for signing, so people you collaborate with can’t tell (except that they’re using different anonymous session ids)
I love building with Jazz. It's so refreshing to build apps without thinking about backends.
The number one thing I'm looking forward to is React Native support. Having local data that syncs is essential for many mobile apps, but there are no easy solutions yet. (Besides iCloud, which is limited to iOS/macOS)
With Jazz, I can see a future where building syncing, cross-platform apps becomes effortless.
Really interesting. Looking over their github, seems that at its core is a CRDT JSON structure with encryption and permissions built in. Storage, transport and auth are agnostic, but they offer some flavors (sqlite, websockets, clerk, react, nodejs). We're interested in svelte and rust support for now. Looks like the local-first space is gaining traction. Probably this is the most interesting execution so far.
We want to support as many environments as possible and are doing them in order of easiest & most requested - you are the first one to ask about Dart/Flutter
React Native is released (experimental)
After that will be really good support for Next.JS (Jazz for both SSR and client)
A bit later will probably be a Rust port, which then also makes bindings to Swift/Kotlin/Go/... easy
Edit: This IS being used to build a product, see reply below
One thing I always find weird about these sorts of projects is I get the feeling this is a framework built up from scratch. I've always had really bad luck doing that. Extracting and refining something from a running application always feels like you catch the cases that actually matter.
It's neat that it's quick to start, but I think that's mostly because the framework has only been evaluated by making tiny toy examples. Staying reaaaaally close to the framework code is sort of like painting with your nose against the canvas. At the scale of features you want to fit in:
- live sync
- authorization
- video chat
- an open connection to your DB, presumably meaning this is your main data fetching API
you're basically building 20% of someones else's app that currently doesn't exist. Once it does exist, it'll only be a simple toy example for a short while.
It's sensibly licensed though (MIT), and honestly, skimming the source, seems clean to work with! I hope people using this contribute back to it, to get over that starter-project hurdle.
Oh neat! I stand corrected then! That's really cool!
Definitely get that showcase up, I think it just helps to confirm this isn't one of those stealth-mode "solves everything" dev tools that disappears along with it's VC funding.
I guess it would be nice to see a comparison with other CRDTs (Automerge, Yjs, Loro) and syncing frameworks/databases (PowerSync, RxDB, SignalDB, InstantDB).
Thank you I really appreciate it.
As an European, I am trying to use only database/sync services based in Europe using European datacenter.
Is it the case for Jazz?
If you only share data with Europeans, it should only end up on the cache server in London and the main central server in Prague.
I was planning to add a second main server in the US that would also receive everything but will think about how to keep data separate for users/companies that need that.
In any case all the data is end-to-end encrypted, but I guess some of the metadata that is stored could still be considered sensitive for your scenario.
An interesting project and looks quite promising. I will be keeping an eye on it. The source code can be found here https://github.com/gardencmp/jazz (TypeScript).
Hi! So wondering how did you solve collaboration issues using CRDTs exclusively. So I suppose that for a db you have an encryption key and then you have users with different rights, some of which can remove the other ones (let's call them admins or whatever). If your db has a certain encryption key then it should be known to all current members. Also if you remove a person he/she should not be able do decrypt new messages in the db.
So suppose you have 4 people, Alice, Bob, Clark and David, where Alice and Clark are "admins". A network partitioning happens and Alice removes Bob, so Alice changes an encryption key to X and sends a message to everybody, concurrently without knowing about the Alice's removal Clark removes David and also changes and encryption key to Y and sends a message to everyone.
So in this case if we allow the generation of new keys to happen simultaneously (without consensus) one of the deletions will not be effective, because what we wanted is both Bob and David to be removed and not know the new encryption key for the db. But now Bob knows Y and David knows X. Thus neither X, nor Y can be the encryption key, because it is known by one of the deleted persons.
Sorry for the long explanation, but I am wondering what is your solution to such a problem.
Also Instant is a database (it has queries and so forth, more appropriate for bigger data). I would put Jazz at the document sync side of the spectrum:
With decentralised I always like to be very precise:
- Jazz is decentralised in a "source of truth" sense. You mutate data locally and sync edits between participants until you reach eventually consistent history (and thus state, as per the underlying CRDTs)
- the protocol is peer-to-peer
- the way you use it typically is in a very centralised way. You use a central sync and persistence peer (either your own, or my distributed Jazz Mesh service) and every device syncs to that. That way devices don't have to be online at the same time to sync
Not sure if this question belongs on this post but kinda interested in how validation and e.g data schema enforcement work in local first / distributed infrastructure.
Assume this is built into the way you think about the model (and my mental model definitely isn't there yet), but having read the permissions part of the jazz docs (on/off?) and thinking about how client-first platforms like firebase do it with private rules on a centralised server .. I don't get it. Is it possible to do complex validation against e.g writes? Is it a natural state of CRDTs, somehow handled in the PKC, or just a side effect of the data models demanded of the setup?
Any info or pointers to further reading would be ace!
Right now, we're not doing complex validation very well, we basically rely on all clients having the same schema or backwards-compatible schemas and prevent errors at the type level.
But because the schemas are runtime constructs as well, we might be able to do more complex stuff in the future. Basically any rule you can express as a function of (previous history, new transaction) -> is new transaction valid? can be a rule in theory, and each client will verify it in an eventually consistent way.
For the time being, the Group abstraction with it's reader, writer and admin roles, plus composition of CoValues in different groups is sufficient for most permission-like rules, which is what most apps need most of the time.
Hey, thanks for the response. This does make sense. The group roles do seem relatively powerful.
I guess what I'm probably getting at is how does it hold from a threat perspective? What is to stop an actor distributing something extra large or unscrupulous between nodes. TS is notoriously hard to use for validation within types (e.g. string of max length, number between), but clearly covalues go beyond this.
Anyway, I'm writing this having not yet tried the framework. So I'll do that first and then do some further reading, and then jump onto the discord if I have more questions. Cheers.
My first thoughts after visiting the webiste: People will be quick to create apps with this. But what happens if I want to implement something that is not supported by the framework yet? That is a common problem with frameworks in my experience.
Looks like it's not load that brought it down but someone posting a 340kb individual message that's triggering a bug because we didn't account for that sort of thing somewhere.
The good thing is we now have an easily reproducible case, thanks HN :)
Looks cool! a bit of crypto hygiene though, I'd recommend passing the ECDH output through a KDF before using it in xsalsa20-poly1305.
Also, if you're using xsalsa20 anyway, why go through the trouble of doing incremental n0nces? One of the main benefits of opting for an extended-n0nce construction is that it simplifies n0nce handling greatly. Why not randomly generate the 24 byte n0nce and forget about the tricky and error prone project of ensuring n0nce uniqueness?
> Jazz Mesh is currently free — and it's set up as the default sync & storage peer in Jazz
How difficult is it to run my own backend?
Edit:
> Completely DIY Mesh.
> Build your own network of sync and storage nodes. Handle networking, security and backups yourself.
> Costs:
> N × instance cost for your sync nodes.
> Very high self-hosted egress costs.
> High self-hosted storage costs.
Compared to Mesh Pro at $79 for 30,000 sync-minutes (“Sync-minutes are counted on a per-connected-device, per-minute basis.”) per month… I bet the cost of self-hosted storage and egress is very low, actually, of course assuming you have the competence and time to DIY it.
Storage seems reasonable, maybe even a bit too cheap, but the sync minutes look way too expensive. One problem is with end-user monetization: users are used to pay for storage, but not for how much time they can use your app.
For a small storage-heavy indie app I have, I'd pay $79 for base price, $45 for additional storage, and $1500 for sync minutes.
Random one, but how do you find using Cal - Im keen to try it out?? I saw it on Onestack.cloud hosted for A$10 per 10 users per month which is pretty decent - our startup has a few founders and are pre-revenue, so means we couldn't really afford per user per month.
I'm jazzed about Jazz, I think Anselm got the priorities right and I'm excited to use it soon. If you get the chance to hear him talk about it, I highly recommend that too!
Kudos to your work - I have been following the project for a while and it's impressive what you have build mostly by yourself and without any of the usual VC funding!
Neat work! Can the author provide any ideas around:
1) Differences from fire-svelte for core functionality?
2) For the beyond core aspects, like Group-WebRTC (great innovation btw), how will things like TURN/PEER servers work to manage the WebRTC connections?
Svelte-fire* allows you to basically sync state between frontend use (with optional chaining operators) and your firebase/firestore backend. You can basically call stuff as a variable and it gets it from the database and likewise updates it.
Specifically, once I load something (say a document), if I edit in on another tab, it will re-render on my original tab so long as the document it references (by key) is the same and firebase permissions allow me to read it as that firebase user.
It means all my permissions and schema goes in firebase/firebase rules (e.g. users can only access their own documents of type "Car" but can read all documents of type "Road"). Which is quite handy to do stuff quickly that syncs across users and works as a single source of truth.
It's an add on that works with Svelte (and Firebase). *(sorry, I got the wrong phrase)
I’m only showcasing the react bindings because that’s mainstream, but they are only a very thin wrapper over jazz-browser, which you can use in vanilla JS or to build your own framework bindings
Yes, plus it’s actually local-first (works offline) and you don’t have to trust the syncing & persistence infrastructure because only encrypted edits are synced!
So it if I already have an app using instantdb, how hard would it be to switch to jazz? Does it work in a very similar way? Or would it likely require a lot of code modification?
This is absolutely fantastic! I have been looking on and off for something like this for years (some of the things I have used are CouchDB, command/event sourcing, and ditto.live).
I have not been able to read the docs fully, but some questions:
* How do you handle state that is too big to send to the client fully? In a chat application, does the client need to have the whole channel history, or is syncing a subset of that supported?
* Does permissioning support partial CoValues? For example, "You can edit the contents but not the title of the 'blog post #11' object"
* Do you have resources about the suggested data modelling? Things like how granular should a CoValue be and what the trade-offs are.
* How do you handle deletion? Do you tombstone? Is there a way to fully scrub a value from history (to support, for example, GDPR's right to erasure)?
- Jazz is built with granular syncing in mind. CoValues are meant to be small (think one level of a JSON tree) and then reference each other. You only resolve references and sync as far down as you need / in lists you can do pagination.
- CoValues can reference CoValues that belong to a different group and thus have different permissions. In your example the title would need to live in its own CoValue and group compared to the content, which is a tiny bit more setup, but totally viable
- no resources for data modelling yet, but the guide + example apps should give you a very good idea. If you have any questions, please ask on Discord! https://discord.gg/sesjU2W5
- deletion is typically soft-deletion with tombstones. Total erasure of entire CoValues is coming soon but should only be needed in compliance situations (such as GDPR)
Partykit is mostly an abstraction on top of Cloudflare Durable Objects and offers simplified Websockets for room-like multiplayer.
Jazz is a framework for building apps around locally mutable collaborative values which are granularly synced (you don't have to worry about data transport or coordinating sync between collaborators, you just set permissions and edit data)
Meteor works on top of a traditional backend (typically MongoDB afaik?) to give you full-stack reactivity.
Jazz implements essentially a distributed database with permissions based on public-key cryptography, so you can meaningfully create, edit, share and store data on the client (even while offline), which gets synced to other users and devices (and stored in the cloud) whenever you're online
Is this synchronisation (theoretically) possible over p2p communication protocols like Bluetooth or wifi direct?
I would more than love to have local networks that can be emergent and growing and shrinking in this way.
Reminds me of my student days when I was traveling in the train, wondering why we couldn't have a p2p social network for people who are in this train _right now_
>I had to spam click back like 30 times to get back to this hacker news comment thread
Click-and-holding or right-clicking the back button will give you a list of last N URLs in your tab history. This page only generates one auto-redirect, so the HN URL will show up.
Just wanted to apologise to everyone for this, this kind of stuff drives me nuts and I'm not sure how I never noticed - it seems to be a result of how we use the iframe to render the chat example. Investigating!
Tip for next time this happens: hold down the back button for a menu of your history. It can help get where you want faster. Although not sure it helps too much if you literally had to click 30 times
I’m also experiencing issues with the website. I when to the docs page and accidentally pressed back in my browser, after which the forward button wouldn’t work to undo the back operation.
Seems like the website breaks basic browser navigation.
On Firefox 131.0 I clicked through the tabs with the demo code, then pressed my mouse's "back" button and it didn't work. So I manually clicked the back button and it directed me back to this page.
Then I opened it again and clicked the back button and it didn't work again.
Reproduced with Firefox 131.0 on Windows 11. Happens if I click to jazz.tools. After pressing back once, I am still on jazz.tools, but have a forward arrow. It does seem related to the "chat" because the "result" window changes when I click between those back/forward arrow controls of the browser.