PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device. Project Kanban https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/fe441a42-5fce-448b-a3e5-ea4abb43ba23 PowerInfer v.s.