Towards milli-joules per token? AI on the Apple Watch
Can an Apple Watch run a (small) LLM (SLM)? Yes, it seems so. The Apple Watch Ultra has an S9 or S10 System on a Chip (SoC) 64 bit multicore CPU with integrated Neural Engine supporting hardware accelerated AI.
A short video showing a demo:
It is running the Microsoft TinyStories 1M model (1M parameters, 16 bit per parameter), but the CPU/NPU of the Apple Watch could potentially run much larger (still small) models like HuggingFace’s SmolLM 135M, Google’s Gemma 270M or Liquid AI’s LFM2-350M - perhaps by allowing a burst memory mode for 10-30 seconds to run larger LLMs on watchOS?
What would you use AI on a smart watch for, perhaps together with all the biometric data?
Amund