LLM Engineering

A powerful local RAG (Retrieval Augmented Generation) application that lets you chat with your PDF documents using Ollama and LangChain. This project includes multiple interfaces: a modern Next.js web app, a Streamlit interface, and Jupyter notebooks for experimentation.

⚠️ Warning: Not Actively Maintained This repository is no longer actively maintained. For a more up-to-date alternative, please visit the LitGPT project: https://github.com/Lightning-AI/litgpt , which serves as the successor to this repository.

A Rust, Python and gRPC server for text generation inference. Used in production at Hugging Face to power Hugging Chat, the Inference API and Inference Endpoints.

This is a port of BlinkDL/RWKV-LM to ggerganov/ggml. Besides the usual FP32, it supports FP16, quantized INT4, INT5 and INT8 inference.

KoboldCpp is an easy-to-use AI text-generation software for GGML and GGUF models, inspired by the original KoboldAI. It's a single self-contained distributable that builds off llama.cpp and adds many additional powerful features.

We also introduce a diffusion-based value model that reduces variance and improves stability during optimization. Based on TraceRL, we derive a series of diffusion language models, TraDo, which achieve state-of-the-art performance on math and coding reasoning tasks.

I've started to work on reimplementation of the library here: FastTensors Please star it if you'd like to see GGML-compatible implementation in pure Go. Please check out my related project Booster We dream of a world where fellow ML hackers are grokking REALLY BIG GPT models in their homelabs without having GPU clusters consuming a shit tons of $$$.

This repo supports the paper "QLoRA: Efficient Finetuning of Quantized LLMs", an effort to democratize access to LLM research. QLoRA uses bitsandbytes for quantization and is integrated with Hugging Face's PEFT and transformers libraries.

A self-evolving librarian agent for document collections. AutoRAG searches your PDFs, wikis, notes, research papers, and knowledge bases — then curates the results into clean, numbered knowledge units.

⚡LLM Zoo is a project that provides data, models, and evaluation benchmark for large language models.⚡ [[Tech Report]](assets/llmzoo.pdf) technology gifted by the creator. For example, many pioneers have made great efforts to spread the use of light bulbs and vaccines to developing countries.

LLM Twin Course: Building Your Production-Ready AI Replica Learn to architect and implement a production-ready LLM & RAG system by building your LLM Twin From data gathering to productionizing LLMs using LLMOps good practices. by Decoding AI By finishing the "LLM Twin: Building Your Production-Ready AI Replica" free course, you will learn how to design, train, and deploy a production-ready LLM twin of yourself powered by LLMs, vector DBs, and LLMOps good practices.

USearch and FAISS both employ the same HNSW algorithm, but they differ significantly in their design principles. USearch is compact and broadly compatible without sacrificing performance, primarily focusing on user-defined metrics and fewer dependencies.

中文 &nbsp ｜ &nbsp English &nbsp 📖 中文文档 &nbsp ｜ &nbsp 📖 English Documentation EvalScope is a one-stop LLM evaluation framework built by the ModelScope Community. Just one command to start — it supports model capability evaluation, inference performance stress testing, and result visualization.

PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device. Project Kanban https://github.com/SJTU-IPADS/PowerInfer/assets/34213478/fe441a42-5fce-448b-a3e5-ea4abb43ba23 PowerInfer v.s.

A Swift client library for interacting with the Ollama API. Pass "json" to get back a JSON string, or specify a full JSON Schema: The format parameter works with both chat and generate methods.

𝕏 Follow me on X • 🤗 Hugging Face • 💻 Blog • 📙 LLM Engineer's Handbook The LLM course is divided into three parts: 1. 🧩 LLM Fundamentals is optional and covers fundamental knowledge about mathematics, Python, and neural networks.

Unsloth Studio lets you run and train models locally. Features • News • Quickstart • Notebooks • Documentation Unsloth Studio (Beta) lets you run and train text, audio, embedding, vision models on Windows, Linux and macOS.

Osmantic Deployment System Turn your PC, Mac, or Linux box into a private AI server. AI server and homelab setup is rapidly becoming a solved problem.

Run Stable Diffusion on Apple Silicon with Core ML [\[Blog Post\]](https://machinelearning.apple.com/research/stable-diffusion-coreml-apple-silicon) [\[BibTeX\]](#bibtex) This repository comprises: If you run into issues during installation or runtime, please refer to the FAQ section. Please refer to the System Requirements section before getting started.