Datus — Open-Source Data Engineering Agent

What is Datus?

Datus is an open-source data engineering agent that builds evolvable context for your data system — turning natural language into accurate SQL through domain-aware reasoning, semantic search, and continuous learning.

Data engineering is shifting from "building tables and pipelines" to "delivering scoped, domain-aware agents for analysts and business users." Datus makes that shift concrete.

Key Features

Build Evolvable Context, Not Static Pipelines

Traditional data engineering ends at data delivery. Datus goes further — it builds a living knowledge base that captures schema metadata, reference SQL, semantic models, metrics, and domain knowledge into a unified context layer. This context is what makes LLM-generated SQL accurate and trustworthy, and it improves with every interaction through a continuous learning loop. → Contextual Data Engineering

From Exploration to Domain-Specific Agents

Datus provides a complete journey for data engineers: start with a Claude-Code-like CLI to explore your data interactively, use Plan Mode to review before executing, and build up context over time. When a domain matures, package it into a Subagent — a scoped chatbot with curated context, tools, and business rules — and deliver it to analysts via web, API, or MCP. → Subagent docs

Metrics and Semantic Layer

Go beyond raw SQL with pluggable semantic adapters. Define business metrics in YAML via MetricFlow integration, and let Datus generate SQL from metric queries — bridging the gap between business language and database dialect. Use Dashboard Copilot to turn existing BI dashboards into conversational analytics. → Semantic Adapters docs

Measure and Improve

Built-in evaluation framework supporting BIRD and Spider 2.0-Snow datasets. Benchmark your agent's SQL accuracy, compare configurations, and track improvements as context evolves. → Benchmark docs

Open Platform

• 10+ LLM providers (OpenAI, Claude, Gemini, DeepSeek, Qwen, Kimi, OpenRouter, and more) with per-node model assignment — mix models within a single workflow
• 11 databases — Built-in SQLite & DuckDB, plus pluggable adapters for PostgreSQL, MySQL, Snowflake, StarRocks, ClickHouse, and more
• MCP Protocol — Both an MCP server (exposing Datus tools to Claude Desktop, Cursor, etc.) and an MCP client (consuming external tools via .mcp in the CLI). → MCP docs
• Skills — Extend Datus with agentskills.io-style packaged tools, configurable permissions, and marketplace support. → Skills docs

Getting Started

Install

Requirements: Linux or macOS. Python 3.12 is installed automatically when you use the one-liner.

One-liner (Linux / macOS)

Stable install from PyPI:

curl -fsSL https://raw.githubusercontent.com/datus-ai/datus-agent/main/install.sh | sh

This creates a dedicated venv at ~/.datus/venv, installs datus-agent from PyPI into it, and drops datus, datus-cli, datus-api, datus-mcp, datus-agent, datus-gateway, and datus-pip shims into ~/.local/bin. Open a new shell (or source ~/.zshrc) to pick up PATH, then run datus to launch the REPL — use /model to configure an LLM, /datasource to add a datasource, and (optionally) /init to generate AGENTS.md for the current project.

To install additional Python packages into the global venv later, use datus-pip install <package> (it is a shim for ~/.datus/venv/bin/pip).

Dev install from GitHub source (picks up unreleased changes):

curl -fsSL https://raw.githubusercontent.com/datus-ai/datus-agent/main/install-dev.sh | sh
# or pin to a branch / tag / commit
curl -fsSL https://raw.githubusercontent.com/datus-ai/datus-agent/main/install-dev.sh | DATUS_REF=feature/foo sh

Pin a PyPI version (stable installer only):

curl -fsSL https://raw.githubusercontent.com/datus-ai/datus-agent/main/install.sh | DATUS_VERSION=0.2.6 sh

Other variables supported by both installers: DATUS_HOME (default ~/.datus), DATUS_BIN_DIR (default ~/.local/bin), DATUS_FORCE=1 to recreate the venv, DATUS_NO_MODIFY_PATH=1 to skip shell rc edits.

Manual install

pip install datus-agent
datus

After the REPL starts, run /model to configure an LLM, /datasource to add a datasource, and (optionally) /init to generate AGENTS.md for the current project. For detailed guidance, see the Quickstart Guide.

Four Ways to Use Datus

Tip: Use datus-cli --print --datasource demo for JSON streaming to stdout — useful for piping into other tools.

Architecture

Workflow Engine

Datus uses a configurable node-based workflow engine. Each workflow is a plan of nodes executed in sequence, parallel, or as sub-workflows:

workflow:
  plan: planA
  planA:
    - schema_linking     # Find relevant tables
    - parallel:          # Run in parallel
      - gen_sql     # SQL generation
      - reasoning        # Chain-of-thought reasoning
    - selection          # Pick the best result
    - execute_sql        # Run the query
    - output             # Format and return

Node Types

RAG Knowledge Base

The knowledge base is powered by LanceDB and organizes context into multiple layers:

• Schema Metadata — Table and column descriptions, relationships
• Reference SQL — Curated query examples with summaries
• Reference Templates — Parameterized Jinja2 SQL templates for stable, reusable queries
• Semantic Models — Business logic and metric definitions
• Metrics — Executable business metrics via semantic layer integration
• Platform Docs — Ingested from GitHub repos, websites, or local files

Build the knowledge base with:

datus-agent bootstrap-kb --datasource demo --components metadata,reference_sql

Configuration

Datus is configured via agent.yml. Launch datus and use /model plus /datasource to populate it interactively, or copy conf/agent.yml.example and edit it by hand.

API keys are injected via environment variables using ${ENV_VAR} syntax.

Supported LLM Providers

Embedding models: OpenAI, Sentence-Transformers, FastEmbed, Hugging Face.

Per-node model assignment lets you use different providers for different workflow steps (e.g., a cheaper model for schema linking, a stronger model for SQL generation).

Supported Databases

See Database Adapters documentation for details.

How It Works

Explore — Chat with your database, test queries, and ground prompts with @table or @file references.

datus-cli --datasource demo
/Check the top 10 banks by assets lost @table duckdb-demo.main.bank_failures

Build Context — Generate semantic models, import SQL history, define metrics. Each piece becomes reusable context for future queries.

/gen_semantic_model xxx        # Generate semantic model from tables
/gen_sql_summary               # Index SQL history for retrieval

Create a Subagent — Package mature context into a scoped, domain-aware chatbot with curated tools and business rules.

.subagent add mychatbot        # Create a new subagent

Deliver — Serve the subagent to analysts via web (localhost:8501/?subagent=mychatbot), REST API, or MCP — with feedback collection (upvotes, issue reports) built in.

Measure — Run benchmarks against BIRD or Spider 2.0-Snow to track SQL accuracy as context evolves.

Iterate — Analyst feedback loops back: engineers fix SQL, add rules, refine semantic models, and extend with Skills or MCP tools. The agent gets more accurate over time.

→ End-to-end tutorial · CLI docs · Knowledge Base docs · Subagent docs

Development

uv sync                                           # Install dependencies
uv run pytest tests/unit_tests/ -q                # Run CI tests (no external deps)
uv run ruff format . && uv run ruff check --fix . # Lint & format

Enable --save_llm_trace on CLI commands or set save_llm_trace: true per model in agent.yml to persist LLM inputs/outputs for debugging. → LLM Trace docs

See CLAUDE.md for full development conventions, architecture patterns, and testing rules.

License

Apache 2.0