Project N (subtitle: Where's this site headed?)

This is about the new community platform I've been working on, that will eventually power some future iteration of Libre/AG.

A major trend observed in internet communities over the past few years is the migration from traditional forums to "macro-scale" social media such as reddit and stackexchange that provide easier content discovery and aggregation across multiple, smaller social networks. It's not an understatement to say they have pretty much killed off most smaller general discussion boards that failed to accumulate a critical mass prior to early-2010s. However, file sharing remains to be one of the few niches that these corporate-backed sites fail to venture into beyond a certain point (mostly due to fuzzy legal details relevant to corporates based in US), and I think there's an opportunity for a massive (10-100x) expansion of the site if we're able to come up with a well-designed platform.

Take the reddit community /r/piracy for example. Despite most discussion threads only tangentially involving actual file sharing, the forum maintains around 3k concurrent users, which is 15x the scale of Libre.

A reddit clone isn't viable without the multi-million user base, but there's an alternative way of tapping into thousands of other social graphs: going decentralized. Recently ActivityPub-based decentralized social networks such as Mastodon started to gain traction (aided by Tumblr & Facebook blunders), and the Fediverse network proved itself to be a solid foundation to build new communities and rapidly gather an audience.

Enter Project N, a congenial cyberspace for 21st century hackers. This is my attempt at building our own platform on the Fediverse network, mostly designed to be a replacement of our current forum software. The following is an overview of the early draft I sketched out thus far.

Backend: "n-core"
Primarily written in Python 3, the core drives the main backend logic such as authentication, data management (ORM), and APIs.

1. Authentication is done through ed25519 cryptographic signatures. This eliminates the use of passwords, and logging in is as simple as calling auth/challenge?for=user, signing the message locally through an app or with the javascript frontend, and POSTing the response to auth/response. Multiple identities can be tied to multiple keys (many-to-many).

2. The database is powered by ArangoDB, a graph database written in C++. Graph databases are a relatively new technology that is distinguished from traditional RDBMS such as MySQL in that they can naturally model multi-tier relations commonly found in social graphs (such as followers/friends/mentions) and traverse them orders of magnitudes faster [1] by leveraging specialized data structures and semantic queries. Arango seems to be particularly well-designed and actively maintained compared to the usual alternatives such as neo4j and dgraph; and it's currently one of the best performing FOSS databases. The only downside is the limited language ecosystem; currently only Python, Go, and Javascript have fully-featured ORM drivers, which is why I went with Python as opposed to faster alternatives such as Elixir or Crystal.

3. The API provides /auth/* and /api endpoints. The latter is a single endpoint that implements the GraphQL standard.

4. Federation will not be implemented until much later on. After reaching some level of API stability, ActivityPub and Scuttlebutt protocols will be implemented for communication with Fediverse (Mastodon) and Scuttlebutt networks.

5. What sets it apart from existing engines (mastodon, misskey, etc.): better integration of threaded discussions, support for custom intra-instance protocols (games, streams, end-to-end encrypted messages), key-based identities, and integration of both SSBC and AP protocols.


Frontend: "n-ui-web"
This is the official web frontend implemented in ReactJS. Not much work has been here, other than investigating which technology to use. Current stack is as follows:
- ReactJS
- Typescript (TSX) for markup logic
- "Atomic CSS" for styling - in particular tachyons or tailwind-css

The frontend aims to be information-dense and media friendly, with design elements adapted from, Twitter, and imageboards.

This hopefully provides some insight as to what's coming next for Libre. It's a slow (and somewhat painful) process, as some of the techs are out of my comfort zone and I'm trying to approach this more carefully than I did the transition from AG to Libre. There may be an alpha prototype showcase around the end of 2019, and a beta launch sometime in 2020 if all things go well.

Further Reading