Scalable System Design

You are at the home page of project-based tutorials on distributed systems.

Project 1. Self-hosted Kubernetes cluster using Kubespray (DS101)

During this project we are going to install bere-metal Kubernetes cluster, with ElasticSearch/Kibana logging, Prometheus/Kibana monitoring and WireGuard VPN over public network. Also, we are going to explore Hetzner Cloud as cheap hosting for our cluster.

  1. Basics: Initial server setup, Server security (code: ScalableSystemDesign/ansible-initial-server-setup)
  2. Provisioning: Kubernetes 1.10 installation using kubespray
  3. Persistence: Kubernetes local volume
  4. Logging: Kubernetes logging using ElasticSearch, Kibana, Fluentd (EFK) (code: ScalableSystemDesign/ansible-kubernetes-elasticsearch-logging)
  5. Metrics: Monitoring Kubernetes cluster using Prometheus, Grafana Kubernetes App
  6. Security: Kubernetes cluster over public network using WireGuard VPN
  7. Automate: Deploy Kubernetes on Hetzner Cloud using Jenkins Pipelines (code: ScalableSystemDesign/deployer)

Project 2. Binance WebSocket listener using Akka (DS107)

During this project we are going to design and deploy reliable WebSocket client in distributed environment.

  1. Goals: Project Overview
  2. Basics: Scala Futures 101, Akka Streams 101
  3. Monitoring: Metrics
  4. Data model: Apache Cassandra for Time Series
  5. Programming: Designing reliable, fault-tolerant WebSocket client using Akka (code: ScalableSystemDesign/cryptocurrency-trades-storage)