Document: WM-040 P. Webb
Category: Project 2019.09.08
Socii Dispatch 07
Abstract
It’s been a LONG road to 2.0. Old Socii was busted and didn’t feel
good to use. Even I didn’t want to use it so changes had to be
made. New Socii is super pleasant to use and I’m happy again. What
follows is a deep-dive into the history of the architecture and why I
made the changes I made.
Body
The previous version of Socii consisted of seven "microservices" and
two web apps that were eventually condensed to two APIs and
(hopefully) a sensible stack. This post will detail the process and
why the changes were made.
1. 1.0
1.1. Microservices
Horus: the notifier
Hu: the authenticator (user management)
Nemty: the emailer
Ra: the one who knows all (GraphQL)
Seshat: the keeper of records (post management)
Thoth: the media converter
1.2. Web Apps
Marketing: the pretty informational site
Network: the actual web app
1.3. The Rest
DigitalOcean: the server
Mailgun: backend for email delivery
MongoDB: the database
2. How it worked
User visited socii.network and liked what they saw so they
decided to sign up. Being a new user, they were logged in
immediately. On the backend, Hu created the user and pinged Nemty
with the user’s email address. Nemty created and sent a welcome
email to the user.
If the user decided to make a post, Seshat would get those details
and save it to the database. Thoth took care of uploading the
user’s profile picture. Horus facilitated details of post actions
(favorite, reply, &c) to Nemty. Ra was the glue that held
everything together.
3. What didn’t work
3.1. Microservices
Adding a feature to one microservice often meant updating nearly
all of them and that grew hella tedious. When Socii 1.0 was in
development the web industry was super buzzed about microservices,
everywhere! They would solve all your problems! In my case, they
added more than removed problems. It is fully possible that I
did not write them in the way microservices are supposed to be
written but when most of the tutorials on the subject came from
the Medium[1] hive mind, you can understand how good information
and examples (!!) are hard to come by.
3.2. Web Apps
Like my microservices, I did not enjoy doing a lot of work on the
Network and then realizing Marketing wasn’t up to date with things
ranging from a shade of blue to the modules powering the app. I
did a lot of copy/pasting and it felt like I was doing something
wrong. Tedium!
3.3. The Rest
DigitalOcean and Mailgun have been doing well for me. In fact, I
believe I’m grandfathered into Mailgun’s free tier due to the low
volume of Socii’s email output. I’ve experienced zero issues with
email deliverability so I’d gladly pay for it (make Socii
popular y’all).
MongoDB? Sigh
Like many front-end developers, I gravitated to it for my projects
because it was the "least scary" database available. Plenty of
ORMs[2] are available for it and I used mongoose[3] for a while
before transitioning to mongoist[4] during one of the phases of
the refactor (in total, there have been at least four phases).
The issue that stumped me for at least a month before urging me
to investigate other databases involved updating the schema of
posts and users. I forget the details exactly but I had a defined
schema and exported the data to perform a database migration. The
migration kept failing due to permission checks of some sort and
no amount of search engine kung fu rescued me from my woes. Mongo
is super unhelpful when things go wrong.
I ended up cursing myself for being so reliant on a database with
piss-poor error messages.
4. The Search
4.1. Microservices
I had been using ZEIT’s[5] micro[6]/micro-dev[7] modules as the
basis for my microservices but the lack of updates/interest from
the maintainers[8] urged me to look elsewhere.
I also read many articles about how companies like Airbnb and
Netflix managed their software, to better inform my own decisions
about the future of my microservices. In the realm of that
research I learned about "monorepos" with the most popular NodeJS
module being Lerna[9]. I decided not to use it but to implement
one of the core features, which was to put all of my microservices
into one API and go from there.
4.2. Web Apps
The original Socii made extensive use of websockets in order to
provide a responsive UI. That…worked…for a while. I soon realized
I was merely replicating what single-page applications do.
The web industry’s favorite framework for SPAs is (was?) React and
I despise React. It has always seemed too bulky and I’m allergic
to packages with a boatload of dependencies.
Far too many times in my web development career I have been forced
to stop using a useful package because of a dependency being out
of date or suffering from a vulnerability and the
author/maintainers of that dependency being unresponsive or plain
not caring. Bulky packages also become targets for misuse.
Companies like Snyk[10] exist to help mitigate this issue but the
first step, IMHO, is to limit the attack area yourself by choosing
smaller (often with full feature-parity) packages.
Anyhoo, I looked at Preact[11], Composi[12], and a lot of
interesting HyperScript mini-frameworks before settling on
Mithril[13]. I really thought this was IT. I found the perfect
front-end framework! Right?! Heh, NO.
The original codebase was server-side rendered which meant shared
links in Apple Messages and elsewhere online showed rich metadata;
title of the page, description, and Socii’s icon image. With
Mithril, I lost all that. Frustrated once more, it was back to the
drawing board.
4.3. The Rest
Your service is only as good as your database.
Your database of choice is the jewel, the Big Kahuna, the very
foundation of your service. So, I tested Apple’s FoundationDB.
Long story short, I found it relatively easy to setup but was
confused about how I’d put data into it’s Document Layer (document
data model on FoundationDB, implementing MongoDB’s wire protocol;
I’d be able to plug ’n play, basically).
However, I did find an awesome NodeJS module written by Steve
Korshakov, CEO of Openland[14] <@openland/foundationdb[15].
Unfortunately, it didn’t have Document Layer support so my
search continued.
No other database looked pleasant to use (to me). One of my
developer friends espoused the beauty of MariaDB but I was spoiled
by ORMs. I didn’t want to learn how to write weird syntax just to
interface with my database (I also didn’t realize at the time that
TypeORM[16] was a thing, which renders my previous sentence null).
5. 2.0
5.1. Microservices
Five of the original six microservices I created were condensed
into a single GraphQL API. The outlier was Thoth (media
converter/file management) because as great as GraphQL is, it
cannot handle file uploads as easy as a standard REST API could.
Instead, this is the process: when I need to upload something
(like a profile picture for instance), the upload is happening in
the background as soon as possible. The response from Thoth is the
resulting URL from DigitalOcean’s Spaces (basically AWS) that
becomes a hidden input’s value that will finally get sent to the
API to be inserted into the database, after a user clicks
"save"/"upload".
Kind of convoluted but it works for now.
So, I now have two APIs: a GraphQL-based one (primary) and a
REST-based one (file uploads). Both are built upon the restify[17]
framework (it’s like Express but focused on API creation).
5.2. Web Apps
Back at the drawing board but fully understanding what my
requirements were for a front-end framework, it didn’t take me
long to find another one of ZEIT’s open-source projects, Next[18].
Alas, it was built upon React but if I could have an SSR-powered
SPA I’d swallow that. And so I did until approximately a week and
a half later when I came across a blog post[19] comparing Next and
something called "Sapper". This section of the post blew my
mind (emphasis mine):
*The same ‘hello world’ app that took 204kb with React and Next
weighs just 7kb with Sapper.* That number is likely to fall
further in the future as we explore the space of optimisation
possibilities, such as not shipping any JavaScript at all for
pages that aren’t interactive, beyond the tiny Sapper runtime
that handles client-side routing.
Sapper has Express-compatible middleware, making it easy to
understand and customize for people who are used to using Express
(and Fastify, Restify, &c). It is built upon a framework that’s
not really a framework: Svelte. Like the name implies, Svelte
compiles code to tiny, framework-less vanilla JavaScript. Quite
honestly, this is everything I’ve been looking for; a lightweight
SPA with SSR support!
With the web app framework FINALLY figured out, I had to then
figure out what I wanted to do about having two front-ends to
manage. And…I didn’t want to manage two of them. I decided to
merge them instead and deprecate the use of a sub-domain for the
Network. So, if you were logged-in you would see the global feed
but if you were logged-out you would see the marketing homepage.
The navigation would also change accordingly.
5.3. The Rest
The database I settled upon is RethinkDB[20]. An interesting
choice considering that one of the most commented issues in one of
its repos on Github is titled, "is this project dead?[21]". Well,
the original company behind it is but there’s still a small,
vibrant community around it that’s been using Rethink in
production for years.
One of the great things about Rethink is the web interface[22] it
comes with (you can disable this if you want). This interface has
a feature called "Data Explorer" and like the name implies, it
allows you to test database commands (or, quickly query something
from your database). Coming from a database that effectively told
you nothing via opaque errors to a database that gave you the
tools to succeed was a breath of fresh air and made me question
why the company behind Rethink failed in the first place. The
founder of the company, Slava Akhmechet, wrote about their
failure[23] in depth and the TL;DR is: great product, wrong time.
6. The Future
I believe I have created a stable foundation for the future of
Socii’s development. So, what’s next? A laundry list of features:
images in posts, more profile customization, groups, iOS app,
re-enabled JSON Feed for profiles, end-to-end encryption, &c. I
don’t have experience with CI/CD (Continuous
Integration/Development) workflows so maybe that’ll be the next
thing major thing I tackle. Of course, I’ll share my experiences
along the way. 🕸