George Holtz (geohot) just raised $5M for tinygrad and plans to sell a $15,000 Machine Learning AMD Epyc box with AMD 7900 XT video cards. At first glance, this seems like a really risky investment into a low-margin commodity hardware company. AMD can’t compete with NVidia, so how can we expect geohot to compete with NVidia on AMD hardware?

Well, I’ve been following geohot for a long while. He delivers what he promises, but does it years later and on different hardware than initially promised :) Here’s the timeline of my evolution alongside George that informs my conviction:

(Feel free to ignore personal details, but they help inform my perspective of George)

  • 2014: I joined Pure Storage
  • 2015: geohot gets into self-driving.
  • Comma built a very cheap model-training data-center on NVidia gamer video cards
  • 2018: Bought Prius Prime because it was best-supported comma.ai car :)
  • 2018: Convinced Pure to buy me a comma.ai kit to self-drive my Prius for a patent I helped file
  • 2019: Evaluated and tracked performance of Pure hardware some thought was too fast to benchmark economically
    • Discovered AMD platforms could run 2x as fast as same gen Intel due to vastly superior PCI/memory bandwidth
    • Making use of this hardware required partitioning containers by numa, pinning them to CPUs, assigning nic interrupts to same CPUs/containers, driving maximum user-space NFS traffic in parallel using fio
    • Only way to go faster is to run on accelerators that bypass the CPU. Did not purse this due to complexity.
  • October 2020: Geohot got interested in how to train models more minimally/cost-efficiently, leading to the creation of tinygrad, an alternative to PyTorch
  • 2022: Working on enterprise software while my homeland gets invades proves too much for me, I parted ways with Pure, sold Prius(for more than I paid), gifted comma.ai kit to a friend, moved to Ukraine.
  • LLMs prove to be the thing that can distract me from daily news in Ukraine
  • 2023: Most self-driving efforts went bankrupt. Successful ones with 2 exceptions are not profitable.
  • Comma.ai is now likely in the top-5 (probably top-2) in the self-driving space, but by far number #1 in terms of amount of self-driving for money invested
  • Geohot built a community around tinygrad, porting it to various hardware and porting various ML models to it
  • 3 weeks ago: Built an AMD box
  • 2 weeks ago: Got an in-depth look at how sucky AMD software is
  • Yesterday: Launched his company :)

The Future

World needs an ML platform that is sold at cost and that can link multiple GPUs, network cards, NVMe drives directly over PCIe DMA with CPU only being there to coordinate various pieces(NVidia knows this).

  • In the time of [economic] crisis small LLMs [mammals] will out-evolve the bigger [dinosaurs]
  • LLM inference will become multi-step based on multiple special-purpose models collaborating together (fixie, AutoGPT)
  • Centralized uber-expensive, high-latency(cos not running locally) OpenAI will prove to be too slow/expensive for most valuable use-cases.
  • Geohot will get llama LLM and (hopefully, pretty-please) whisper ported to tinygrad
  • Tinygrad will be the cheapest way to train/finetune for the limited number of models it supports, thanks to AMD hardware’s ability to bypass CPU bottlenecks
  • Models will mutate into GPU<->NVMe database frontends based on initial work like RETRO and SAIL
  • George will partner with an existing chip vendor for more minimalist hardware, making proprietary solutions like DOJO and Google TPU obsolete

Geohot will win by wiping the floor with his non-commodity competition. Tinygrad will pave the way there. Geohot may or may not get fabulously rich doing it, but we will all be better off for it.