January Update (2021)
We're starting a new series of (monthly) blog posts to cover what goes on in the company. Usually, we announce significant changes and features by posting on the blog and sending out an e-mail newsletter, and the smaller user/client-facing changes are mentioned in the changelog. Still, there's a lot of work that's being done that needs to be surfaced. These posts are meant to surface that work. There'll be 12 such posts this year, one per month at the end of every month.
So, here's some of the (hidden) work we've been doing in January.
Deletion mechanisms on our ingestion servers
We've worked a lot on the way we delete recordings on the ingestion servers. These are Node powered media servers that handle all the recordings that come through our streaming desktop recorder, native mobile recorder and POST API. Once we receive a complete recording, it is pulled by a Transcoding server where it is processed. Deleting these recordings might seem like a simple task, but the decision mechanism behind each deletion is quite complex. Generally speaking, we need to delete recordings as soon as possible to make space for new ones and minimise any risk to users' privacy. At the same time, we need to delay and prevent deletions if something goes wrong to avoid data loss.
After a few iterations we've decided on two principles that govern how we'll perform deletions:
- Recordings for which we know the state will be deleted as soon as possible. These are recordings that have been copied by the Transcoding server and are not needed anymore for client playback (like streamed wss recordings are), uploaded recordings that are not in one of the accepted formats or bigger than the current limit of 1GB, recordings that have an missing/incomplete/nonexistent account hash or environment it, recordings for expired trials and expired paid subscriptions, incomplete uploads, etc.
- Recordings for which we do not know the state will be deleted as late as possible to allow for their recovery and prevent data loss. In theory, these will show up if there's an error on our side (like an SQL error) and their normal processing pipeline stops. We do not expect to see many of these.
These two principles also have the advantage of being technically possible to respect them.
On top of that whether or not a file is deleted from the disk is decided by the OS. If there's another process accessing the file that we're deleting, the OS will not re-allocate that space as free space. Thus we need to be extra careful around file deletions to make sure ghost files are not lingering around (keeping our disks full).
The 1st bach of deletion updates is ready, and it covers files received through the mobile recording client, desktop uploads and the POST API. We've tested these changes and we'll be rolling them out this week. In February we'll follow up with a batch of deletion updates for the (streaming) desktop recording client.
Removal of our legacy Flash recording client
Throughout 2020 our plan was to phase out the Flash recording client once it's not supported anymore by Adobe and the major browser makers, but kind of still keep it around for a while for users with old browsers.
That changed when we learned Adobe built a kill switch inside (the latest versions of) Flash Player. On the 12th of January that kill switch activated so now old browsers can still run Flash Player, but Flash Player itself will not run Flash content.
Combine that with the new browsers that can't run Flash Player at all (Safari 14, Chrome 88, Edge 88, Firefox 85), and we have nothing to keep around the old Flash recording client for.
These changes were immediately reflected in the number of recordings we received daily from the Flash recording client, a number that was already low (2.33% in November 2020). Here's the data for January 2021:
So today, as announced on the 1st of December, we're pulling the plug.
