May & June Update (2021)

Octavian -

Some of the stuff we've worked on during May and June:

Testing Ingestion Servers & Provisioning

A lot of our time has been spent testing our ingestion servers with more and more simultaneous users. We can now consistently take on 1200 users recording through our desktop recorder simultaneously on a single region.

This is important because, as a result, we can now accurately estimate and provision the resources needed for large volumes of simultaneous recordings.

The next milestone is 2400 users streaming simultaneously against one media server.

application/json webhooks

Our webhooks can finally POST data as Content-type: application/json. More details in the changelog. A tutorial for using our new webhook type with AWS Lambda is coming soon.

Migrating to Hetzner & New EU2 Region

As a result of the Schrems II decision from last year we're migrating most of our compute and temporary storage infrastructure in the EU from DigitalOcean Inc. to Hetzner Online GmbH.  This is an ongoing process.

Part of the process mentioned above consists of rolling out a new EU2 region hosted only with EEA providers. The compute and short term storage will be provided by Hetzner while Scaleway will be used for long term storage. This new region will be a strong contender for GDPR compatibility, and we expect most EU clients to migrate to it.

Hence our change of sub-processors notification from last week. A new DPA is also in the works.

Monitoring the platform

Adding Hetzner as a compute provider meant that we could not use DigitalOcean's monitoring & alerting tools for the migrated infrastructure. This accelerated a plan we had since last year: implement Prometheus and Grafana for deep internal monitoring of our platform.

It was a significant effort from our team.

We now have several layers of monitoring and alerts:

  • external monitoring with Pingdom
  • basic internal monitoring provided by DigitalOcean and Hetzner
  • deep internal monitoring of our Nginx instances with the lovely (but sadly unsupported/discontinued) Nginx Amplify
  • deep internal monitoring of our Node processes through PM2 Monit
  • deep internal monitoring with Prometheus now collecting hundreds of metrics from most of our machines; Grafana for viewing the data.
  • our own soft platform metrics; Grafana for viewing the data.