आयुष 🌱

Evolution of system designs from an AI Engineer perspective

2 min read

Notes from Lecture 2 of this Open MOOC offered by Berkeley RDI

TLDR: Talk by an entrepreneur about the evolution of compute requirements for AI

  1. New algorithms
    1. timeline - 2022-gpt, 2023-MoE, 2024-Test time scaling, 2025-RL
    2. RL allows more sophisticated loss functions (as compared to just next-token prediction)
  2. Applications - parameters
    1. Consumer apps - thriving - landscape is highly fluid competetitive - chat, code, search. Prosumers are willing to pay e.g for productivity.
    2. Business apps - nascent - but enterprise apps are moving faster. started with closed source, and moved to train/tune their own as data built up.
    3. Perfect app experience is correlated but independent from models.
  3. AI Infra
    1. bitter lesson - generalised method that leverage computation are most effective
    2. Cloud history - scientific computing (1970s), Virtual Private servers (90s), Web service cloud (AWS) (OOs), data clouds (snowflake, databricks)(10s), AI cloud (requires exaflops of compute) (20s)
    3. AI is different than computational compute
      1. data compute - io >> compute, simple abstraction, very distributed systems (mapreduce etc)
      2. web services - io > compute, arbitrary code, parallel system
      3. AI Compute - compute >> IO, arbitrary code, very distributed systems (e.g distributed training) - slurm, MPI style
    4. Conventional cloud proposition no longer holds
      1. proposition (1. easy way to install and acquire software (e.g cloud database), 2. flexible supply chain (ec2 instances))
      2. For AI Cloud,
        1. the software is simple (few libraries, frameworks) as opposed to many applications and middleware that conventional cloud is designed for,
        2. Workload is unified (numerical computation) whereas conventional cloud is designed for varied workloads.
        3. Supply Chain flexibility - conventional cloud - supports virtualisation and migration of applications between VMs, AI Cloud - low flexibility as training job machines are occupied, if one gpu fails, restart the entire job
      3. NeoCLoud - semi-analysis (website) - lambda, nvidia's offerings etc
      4. Things to care about
        1. developer efficiency
        2. Infra efficiency - account of GPUs dying
      5. Best practices - multi cloud GPU supply chain, elasticity and utilisation management, ai native platform for ev, training and inference (ray, anyscale)
  4. Hardware - hw and sw design - using NV switches - each machinee is able to access another machines memory- all machines in a rack act like a single device "mainframe" - of GPUs
  5. make workloads - idempotent and interruptible - ideally

Graph View