Moonshine Voice

Voice Interfaces for Everyone

• Quickstart
• When should you choose Moonshine over Whisper?
• Using the Library
  • Speech to Text
  • Text to Speech
  • Conversational Agents
• Models
• API Reference
• Support
• Roadmap
• Acknowledgements
• License

Moonshine Voice is an open source AI toolkit for developers building real-time voice agents and applications.

• Everything runs on-device, so it's fast, private, and you don't need an account, credit card, or API keys.
• The framework and models are optimized for live streaming applications, offering low latency responses by doing a lot of the work while the user is still talking.
• All speech to text models are based on our cutting edge research and trained from scratch, so we can offer higher accuracy than Whisper Large V3 at the top end, down to tiny 1MB models for constrained deployments.
• It's easy to integrate across platforms, with the same library running on Python, iOS, Android, MacOS, Linux, Windows, Raspberry Pis, IoT devices, microcontrollers, DSPs, and wearables.
• Batteries are included. Its high-level APIs offer complete solutions for common tasks like transcription, text to speech, voice cloning, speaker identification (diarization), command recognition, and conversational agents, so you can build your voice application with a single library.
• It supports multiple languages, including English, Spanish, Mandarin, Japanese, Korean, Vietnamese, Ukrainian, and Arabic for STT, and English, Spanish, Arabic, German, French, Hindi, Italian, Japanese, Korean, Dutch, Portuguese, Russian, Turkish, Ukrainian, Vietnamese, and Mandarin for TTS.

Quickstart

Join our community on Discord to get live support.

Example apps for iOS, Android, macOS, Windows, and Raspberry Pi are published on GitHub Releases as separate archives (mostly {platform}-{Project}.tar.gz, matching folder names under examples/; Windows also ships moonshine-voice-windows-x86_64.tar.gz for the C++ sample). See the Examples section for the full list of release downloads.

Python

pip install moonshine-voice
moonshine-voice mic --language en

Listens to the microphone and prints updates to the transcript as they come in.

moonshine-voice dialog

Runs a spoken wifi-setup conversation: it listens for a trigger phrase, asks questions, and confirms the answers. Matching is semantic, so natural language variations are recognized. For more, check out our "Getting Started" Colab notebook and video.

moonshine-voice tts --language en_us --text "Hello world"

Synthesizes and speaks the text.

iOS

Download github.com/moonshine-ai/moonshine/releases/latest/download/ios-Transcriber.tar.gz, extract it, and then open the Transcriber/Transcriber.xcodeproj project in Xcode.

Android

Download github.com/moonshine-ai/moonshine/releases/latest/download/android-Transcriber.tar.gz, extract it, and then open the Transcriber folder in Android Studio.

Linux

Moonshine Voice ships prebuilt shared libraries for both x86_64 and arm64 Linux. The quickest way to try it is with the portable C++ example, which downloads the library, an English speech to text model, and a sample recording, then builds and runs a transcriber:

curl -O -L https://github.com/moonshine-ai/moonshine/releases/download/v0.1.0/cpp-examples.tar.gz
tar xzf cpp-examples.tar.gz
cd c++
./download-library.sh
g++ transcriber.cpp -Imoonshine-voice/include -Lmoonshine-voice/lib -lmoonshine -Wl,-rpath,'$ORIGIN/moonshine-voice/lib' -o transcriber
./transcriber

MacOS

Moonshine Voice supports both Apple Silicon (arm64) and Intel (x86_64) Macs.

Download github.com/moonshine-ai/moonshine/releases/latest/download/macos-MicTranscription.tar.gz, extract it, and then open the MicTranscription/MicTranscription.xcodeproj project in Xcode.

Windows

Download github.com/moonshine-ai/moonshine/releases/latest/download/windows-cli-transcriber.tar.gz, extract it, and then open the cli-transcriber\cli-transcriber.vcxproj project in Visual Studio.

It's a self-contained archive that includes the library and model, so Ctrl+Shift+B or F7 will build the executable.

Raspberry Pi

You'll need a USB microphone plugged in to get audio input, but the Python pip package has been optimized for the Pi, so you can run:

 sudo pip install --break-system-packages moonshine-voice
 moonshine-voice mic --language en

I've recorded a screencast on YouTube to help you get started, and you can also download github.com/moonshine-ai/moonshine/releases/latest/download/raspberry-pi-my-dalek.tar.gz for some fun, Pi-specific examples. The README has information about using a virtual environment for the Python install if you don't want to use --break-system-packages.

You can look at github.com/moonshine-ai/pi-help-bot for a more advanced example.

When should you choose Moonshine over Whisper?

TL;DR - When you're working with live speech.

See benchmarks for how these numbers were measured.

OpenAI's release of their Whisper family of models was a massive step forward for open-source speech to text. They offered a range of sizes, allowing developers to trade off compute and storage space against accuracy to fit their applications. Their biggest models, like Large v3, also gave accuracy scores that were higher than anything available outside of large tech companies like Google or Apple. At Moonshine we were early and enthusiastic adopters of Whisper, and we still remain big fans of the models and the great frameworks like FasterWhisper and others that have been built around them.

However, as we built applications that needed a live voice interface we found we needed features that weren't available through Whisper:

• Whisper always operates on a 30-second input window. This isn't an issue when you're processing audio in large batches, you can usually just look ahead in the file and find a 30-second-ish chunk of speech to apply it to. Voice interfaces can't look ahead to create larger chunks from their input stream, and phrases are seldom longer than five to ten seconds. This means there's a lot of wasted computation encoding zero padding in the encoder and decoder, which means longer latency in returning results. Since one of the most important requirements for any interface is responsiveness, usually defined as latency below 200ms, this hurts the user experience even on platforms that have compute to spare, and makes it unusable on more constrained devices.
• Whisper doesn't cache anything. Another common requirement for voice interfaces is that they display feedback as the user is talking, so that they know the app is listening and understanding them. This means calling the speech to text model repeatedly over time as a sentence is spoken. Most of the audio input is the same, with only a short addition to the end. Even though a lot of the input is constant, Whisper starts from scratch every time, doing a lot of redundant work on audio that it has seen before. Like the fixed input window, this unnecessary latency impairs the user experience.
• Whisper supports a lot of languages poorly. Whisper's multilingual support is an incredible feat of engineering, and demonstrated a single model could handle many languages, and even offer translations. This chart from OpenAI (raw data in Appendix D-2.4) shows the drop-off in Word Error Rate (WER) with the very largest 1.5 billion parameter model.

82 languages are listed, but only 33 have sub-20% WER (what we consider usable). For the Base model size commonly used on edge devices, only 5 languages are under 20% WER. Asian languages like Korean and Japanese stand out as the native tongue of large markets with a lot of tech innovation, but Whisper doesn't offer good enough accuracy to use in most applications The proprietary in-house versions of Whisper that are available through OpenAI's cloud API seem to offer better accuracy, but aren't available as open models.

• Fragmented edge support. A fantastic ecosystem has grown up around Whisper, there are a lot of mature frameworks you can use to deploy the models. However these often tend to be focused on desktop-class machines and operating systems. There are projects you can use across edge platforms like iOS, Android, or Raspberry Pi OS, but they tend to have different interfaces, capabilities, and levels of optimization. This made building applications that need to run on a variety of devices unnecessarily difficult.

All these limitations drove us to create our own family of models that better meet the needs of live voice interfaces. It took us some time since the combined size of the open speech datasets available is tiny compared to the amount of web-derived text data, but after extensive data-gathering work, we were able to release the first generation of Moonshine models. These removed the fixed-input window limitation along with some other architectural improvements, and gave significantly lower latency than Whisper in live speech applications, often running 5x faster or more.

However we kept encountering applications that needed even lower latencies on even more constrained platforms. We also wanted to offer higher accuracy than the Base-equivalent that was the top end of the initial models. That led us to this second generation of Moonshine models, which offer:

• Flexible input windows. You can supply any length of audio (though we recommend staying below around 30 seconds) and the model will only spend compute on that input, no zero-padding required. This gives us a significant latency boost.
• Caching for streaming. Our models now support incremental addition of audio over time, and they cache the input encoding and part of the decoder's state so that we're able to skip even more of the compute, driving latency down dramatically.
• Language-specific models. We have gathered data and trained models for multiple languages, including Arabic, Japanese, Korean, Spanish, Ukrainian, Vietnamese, and Chinese. As we discuss in our Flavors of Moonshine paper, we've found that we can get much higher accuracy for the same size and compute if we restrict a model to focus on just one language, compared to training one model across many.
• Cross-platform library support. We're building applications ourselves, and needed to be able to deploy these models across Linux, MacOS, Windows, iOS, and Android, as well as use them from languages like Python, Swift, Java, and C++. To support this we architected a portable C++ core library that handles all of the processing, uses OnnxRuntime for good performance across systems, and then built native interfaces for all the required high-level languages. This allows developers to learn one API, and then deploy it almost anywhere they want to run.
• Better accuracy than Whisper V3 Large. On HuggingFace's OpenASR leaderboard, our newest streaming model for English, Medium Streaming, achieves a lower word-error rate than the most-accurate Whisper model from OpenAI. This is despite Moonshine's version using 250 million parameters, versus Large v3's 1.5 billion, making it much easier to deploy on the edge.

Hopefully this gives you a good idea of how Moonshine compares to Whisper. If you're working with GPUs in the cloud on data in bulk where throughput is most important then Whisper (or Nvidia alternatives like Parakeet) offer advantages like batch processing, but we believe we can't be beat for live speech. We've built the framework and models we wished we'd had when we first started building applications with voice interfaces, so if you're working with live voice inputs, give Moonshine a try.

Using the Library

The Moonshine API is designed to take care of the details around capturing and transcribing live speech, giving application developers a high-level API focused on actionable events. I'll use Python to illustrate how it works, but the API is consistent across all the supported languages.

• Architecture
• Concepts
• Getting Started with Transcription
  • Transcription Event Flow
• Getting Started with a Conversational Agent
  • Agent Setup
• Getting Started with Text to Speech
  • Voice Samples
    • ZipVoice
    • Kokoro
    • Piper TTS
  • Converting Graphemes to Phonemes
• Examples
• Adding the Library to your own App
• Python
• iOS or MacOS
• Android
• Windows
• Debugging
  • Console Logs
  • Input Saving
  • API Call Logging
• Building from Source
  • Cmake
  • Language Bindings
  • Porting
• Downloading Models
  • Speech to Text Models
  • Intent Recognition Models
  • Text to Speech Models
• Benchmarking

Architecture

Our goal is to build a framework that any developer can pick up and use, even with no previous experience of speech technologies. We've abstracted away a lot of the unnecessary details and provide a simple interface that lets you focus on building your application, and that's reflected in our system architecture.

The basic flow is:

• Create a Transcriber object if you want the text that's spoken, or a DialogFlow if you only need to know that a user has requested an action.
• Attach an EventListener that gets called when important things occur, like the end of a phrase or an action being triggered, so your application can respond.
• Use a TextToSpeech object to make it a two-way conversation.

Traditionally, adding a voice interface to an application or product required integrating a lot of different libraries to handle all the processing that's needed to capture audio and turn it into something actionable. The main steps involved are microphone capture, voice activity detection (to break a continuous stream of audio into sections of speech), speech to text, speaker identification, and intent recognition. Each of these steps typically involved a different framework, which greatly increased the complexity of integrating, optimizing, and maintaining these dependencies.

Moonshine Voice includes all of these stages in a single library, and abstracts away everything but the essential information your application needs to respond to user speech, whether you want to transcribe it or trigger actions.

Most developers should be able to treat the library as a black box that tells them when something interesting has happened, using our event-based classes to implement application logic. Of course the framework is fully open source, so speech experts can dive as deep under the hood as they'd like, but it's not necessary to use it.

Concepts

A Transcriber takes in audio input and turns any speech into text. This is the first object you'll need to create to use Moonshine, and you'll give it a path to the models you've downloaded.

A MicTranscriber is a helper class based on the general transcriber that takes care of connecting to a microphone using your platform's built-in support (for example sounddevice in Python) and then feeding the audio in as it's captured.

A Stream is a handler for audio input. The reason streams exist is because you may want to process multiple audio inputs at once, and a transcriber can support those through multiple streams, without duplicating the model resources. If you only have one input, the transcriber class includes the same methods (start/stop/add_audio) as a stream, and you can use that interface instead and forget about streams.

A TranscriptLine is a data structure holding information about one line in the transcript. When someone is speaking, the library waits for short pauses (where punctuation might go in written language) and starts a new line. These aren't exactly sentences, since a speech pause isn't a sure sign of the end of a sentence, but this does break the spoken audio into segments that can be considered phrases. A line includes state such as whether the line has just started, is still being spoken, or is complete, along with its start time and duration.

A Transcript is a list of lines in time order holding information about what text has already been recognized, along with other state like when it was captured.

A TranscriptEvent contains information about changes to the transcript. Events include a new line being started, the text in a line being updated, and a line being completed. The event object includes the transcript line it's referring to as a member, holding the latest state of that line.

A TranscriptEventListener is a protocol that allows app-defined functions to be called when transcript events happen. This is the main way that most applications interact with the results of the transcription. When live speech is happening, applications usually need to respond or display results as new speech is recognized, and this approach allows you to handle those changes in a similar way to events from traditional user interfaces like touch screen gestures or mouse clicks on buttons.

A TextToSpeech object synthesizes audio for playback to the user.

A DialogFlow object manages conversations between the user and an agent. It's also a TranscriptEventListener, so you can attach it to a transcriber and have it invoke a callback whenever someone says something close in meaning to a phrase you registered — the basis of voice command recognition.

A Dialog object is created for each conversational exchange, and allows the agent to hold a multi-step discussion with the user.

Getting Started with Transcription

We have examples for most platforms so as a first step I recommend checking out what we have for the systems you're targeting.

Next, you'll need to add the library to your project. We aim to provide pre-built binaries for all major platforms using their native package managers. On Python this means a pip install, for Android it's a Maven package, and for MacOS and iOS we provide a Swift package through SPM.

The transcriber needs access to the files for the model you're using, so after downloading them you'll need to place them somewhere the application can find them, and make a note of the path. This usually means adding them as resources in your IDE if you're planning to distribute the app, or you can use hard-wired paths if you're just experimenting. The download script gives you the location of the models and their architecture type on your drive after it completes.

Now you can try creating a transcriber. Here's what that looks like in Python:

transcriber = Transcriber(model_path=model_path, model_arch=model_arch)

If the model isn't found, or if there's any other error, this will throw an exception with information about the problem. You can also check the console for logs from the core library, these are printed to stderr or your system's equivalent.

Now we'll create a listener that contains the app logic that you want triggered when the transcript updates, and attach it to your transcriber:

class TestListener(TranscriptEventListener):
    def on_line_started(self, event):
        print(f"Line started: {event.line.text}")

    def on_line_text_changed(self, event):
        print(f"Line text changed: {event.line.text}")

    def on_line_completed(self, event):
        print(f"Line completed: {event.line.text}")

transcriber.add_listener(listener)

The transcriber needs some audio data to work with. If you want to try it with the microphone you can update your transcriber creation line to use a MicTranscriber instead, but if you want to start with a .wav file for testing purposes here's how you feed that in:

    audio_data, sample_rate = load_wav_file(wav_path)

    transcriber.start()

    # Loop through the audio data in chunks to simulate live streaming
    # from a microphone or other source.
    chunk_duration = 0.1
    chunk_size = int(chunk_duration * sample_rate)
    for i in range(0, len(audio_data), chunk_size):
        chunk = audio_data[i: i + chunk_size]
        transcriber.add_audio(chunk, sample_rate)

    transcriber.stop()

The important things to notice here are:

• We create an array of mono audio data from a wav file, using the convenience load_wav_file() function that's part of the Moonshine library.
• We start the transcriber to activate its processing code.
• The loop adds audio in chunks. These chunks can be any length and any sample rate, the library takes care of all the housekeeping.
• As audio is added, the event listener you added will be called, giving information about the latest speech.

In a real application you'd be calling add_audio() from an audio handler that's receiving it from your source. Since the library can handle arbitrary durations and sample rates, just make sure it's mono and otherwise feed it in as-is.

The transcriber analyses the speech at a default interval of every 500ms of input. You can change this with the update_interval argument to the transcriber constructor. For streaming models most of the work is done as the audio is being added, and it's automatically done at the end of a phrase, so changing this won't usually affect the workload or latency massively.

The key takeaway is that you usually don't need to worry about the transcript data structure itself, the event system tells you when something important happens. You can manually trigger a transcript update by calling update_transcription() which returns a transcript object with all of the information about the current session if you do need to examine the state.

By calling start() and stop() on a transcriber (or stream) we're beginning and ending a session. Each session has one transcript document associated with it, and it is started fresh on every start() call, so you should make copies of any data you need from the transcript object before that.

The transcriber class also offers a simpler transcribe_without_streaming() method, for when you have an array of data from the past that you just want to analyse, such as a file or recording.

We also offer a specialization of the base Transcriber class called MicTranscriber. How this is implemented will depend on the language and platform, but it should provide a transcriber that's automatically attached to the main microphone on the system. This makes it straightforward to start transcribing speech from that common source, since it supports all of the same listener callbacks as the base class.

Transcription Event Flow

The main communication channel between the library and your application is through events that are passed to any listener functions you have registered. There are five major event types:

• LineStarted. This is sent to listeners when the beginning of a new speech segment is detected. It may or may not contain any text, but since it's dispatched near the start of an utterance, that text is likely to change over time.
• LineUpdated. Called whenever any of the information about a line changes, including the duration, audio data, and text.
• LineTextChanged. Called only when the text associated with a line is updated. This is a subset of LineUpdated that focuses on the common need to refresh the text shown to users as often as possible to keep the experience interactive.
• LineSpeakersChanged. Only fired when the opt-in identify_speakers option is enabled. Called when the speaker spans attached to a line change. Unlike the other line events, this can fire for lines that are already complete, because the diarization algorithm keeps refining its speaker assignments as more audio arrives.
• LineCompleted. Sent when we detect that someone has paused speaking, and we've ended the current segment. The line data structure has the final values for the text and duration.

We offer some guarantees about these events:

• LineStarted is always called exactly once for any segment.
• LineCompleted is always called exactly once after LineStarted for any segment.
• LineUpdated and LineTextChanged will only ever be called after the LineStarted and before the LineCompleted events for a segment.
• Those update events are not guaranteed to be called (and in practice can be disabled by setting update_interval to a very large value).
• There will only be one line active at any one time for any given stream.
• Once LineCompleted has been called, the library will never alter that line's text, timing, or audio data again. The one exception is the line's speaker spans: when identify_speakers is enabled, those can be revised for recent audio (signaled by LineSpeakersChanged), since diarization re-clusters a sliding window of recent speech. Assignments for audio older than diarization_cluster_window_sec are frozen.
• If stop() is called on a transcriber or stream, any active lines will have LineCompleted called.
• Each line has a 64-bit lineId that is designed to be unique enough to avoid collisions.
• This lineId remains the same for the line over time, from the first LineStarted event onwards.

Getting Started with a Conversational Agent

Many applications need a voice agent that can understand what users are saying and respond appropriately. To make this as straightforward as possible, we let you define different conversational flows. A flow can be as simple as responding to a query, or be a multi-step, branching conversation that takes actions.

To define these flows, you used a DialogFlow object, with callbacks that take Dialog arguments. Here's an example of a simple flow, taken from the github.com/moonshine-ai/pi-help-bot sample code:

    def report_ip_address(d: Dialog):
        ip = _find_local_ip()
        if ip is None:
            yield d.say("Sorry, I couldn't find a local IP address.")
            return
        speech_ip = re.sub(r"(\d)", r"\1 ", ip.replace(".", " dot "))
        yield d.say([
            f"Okay. Your local IP address is {speech_ip}. ",
            f"To repeat, that's {speech_ip}."
        ])

    dialog_flow.listen_for("What is my IP address?", report_ip_address)

This registers the report_ip_address() function to be called whenever the user says anything similar to "What is my IP address?". The matching is done semantically, so alternative phrasings like "Tell me your IP address" or "Can you tell me the local IP address?" should trigger it too. You can register as many top-level conversation starters as you'd like, the system will listen out and route to the closest in meaning.

The function itself receives a Dialog argument that represents the current conversational exchange. In this simple case we don't need any additional input from the user so we just use it to say() the information that was requested. We break the IP address into separate words for each digit for clarity, and replace the connecting periods with explicit "dot"s, so that 192.178.4.72 becomes "1 9 2 dot 1 7 8 dot 4 dot 72", since that's the conventional way to articulate them in speech.

For more complex conversations, like setting up a new wifi network, you can define multiple steps and branch points directly in Python:

    def connect_to_wifi(d: Dialog):
        input_ssid = yield d.ask("What's the name of your Wi-Fi network? Say list if you want to pick from a list or spell if you want to spell out the start of the name")
        input_ssid = input_ssid.strip()

        networks = _scan_wifi_networks()

        if input_ssid.lower().strip(string.punctuation) == "list":
            yield d.say("Say yes to the network you want to connect to.")
            for network in networks:
                if (yield d.confirm(f"{network}?")):
                    input_ssid = network
                    break
        elif input_ssid.lower().strip(string.punctuation) == "spell":
            input_ssid = yield d.ask("Spell out the start of the network name.", mode=SPELLED)

        found_ssid = fuzzy_match_network(input_ssid, networks)
        if found_ssid is None:
            yield d.say(f"Sorry, I couldn't find a matching network for {input_ssid}.")
            return

        password = yield d.ask(
            f"Please spell the Wi-Fi password for {found_ssid} one character at a time, and say done when finished.",
            mode=SPELLED,
        )

        yield d.say(f"Connecting to {found_ssid}.")

        try:
            result = subprocess.run(
                ["sudo", "nmcli", "device", "wifi",
                    "connect", found_ssid, "password", password],
                capture_output=True, text=True, timeout=30,
            )
        except FileNotFoundError:
            yield d.say("Sorry, network manager was not found on this system.")
            return
        except subprocess.TimeoutExpired:
            yield d.say("Sorry, the connection attempt timed out.")
            return

        if result.returncode == 0:
            yield d.say(f"Connected to {found_ssid}.")
        else:
            print(f"[ERROR] nmcli stderr: {result.stderr}", file=sys.stderr)
            yield d.say(
                f"Sorry, I wasn't able to connect to {found_ssid}. "
                "Please check the network name and password and try again."
            )

    dialog_flow.listen_for("Connect to Wi-Fi", connect_to_wifi)

The first thing the function does is ask the user to give them the name of the network they want to join, through the call:

input_ssid = yield d.ask("What's the name of your Wi-Fi network?...")

The Dialog class lets you ask users questions and will return the string containing the what they said in response. The only unusual feature here, compared to regular Python code, is the yield keyword. Because it may take some time for the user to respond, we call yield to hand back control to the main script until their response has been received. This is a general pattern for DialogFlow and you'll see it wherever we're waiting for the user to say something, to avoid blocking.

        if input_ssid.lower().strip(string.punctuation) == "list":
            yield d.say("Say yes to the network you want to connect to.")
            for network in networks:
                if (yield d.confirm(f"{network}?")):
                    input_ssid = network
                    break

Our example application supports a few different input methods - running through a list of networks, spelling out the first few letters, or saying the name. Here we implement the list approach by looping through all the available networks and asking the user whether each is the one they want. Here you can see that regular loops and conditional statements work as you'd expect in Python.

For each network, we call confirm(), which asks a question and then waits for a positive or negative result. Like all matching in the system this is done semantically, so "okay", "affirmative", and "go ahead" will work as well as a straightforward "yes".

        password = yield d.ask(
            f"Please spell the Wi-Fi password for {found_ssid} one character at a time, and say done when finished.",
            mode=SPELLED,
        )

Password input is tricky, because they consist of arbitrary letters, digits, and symbols, and so they have to be spelled out by the user. Moonshine supports this through the mode=SPELLED argument. This asks the user to spell out each character, and uses a fine-tuned model to recognise what the user is saying for each. As well as supporting regular utterances like "aitch" or "capital zee", it also supports the NATO alphabet ("alpha", "bravo", etc) and even short descriptive phrases like "E as in elephant". It repeats back what it heard, and lets you delete mistakes.

        try:
            result = subprocess.run(
                ["sudo", "nmcli", "device", "wifi",
                    "connect", found_ssid, "password", password],
                capture_output=True, text=True, timeout=30,
            )
        except FileNotFoundError:
            yield d.say("Sorry, network manager was not found on this system.")
            return
        except subprocess.TimeoutExpired:
            yield d.say("Sorry, the connection attempt timed out.")
            return

The flow also works with other control structures like exception handlers, so you can specify your conversations using idiomatic code, even for error recovery.

To give this a try for yourself, run this built-in example:

python -m moonshine_voice.dialog_flow

Agent Setup

An agent needs a speech-to-text Transcriber object to receive input and a TextToSpeech object to respond. DialogFlow understands the input, downloading and loading the embedding model it needs on first use:

    tts = TextToSpeech(args.tts_language)

    model_path, model_arch = get_model_for_language(args.language)
    mic_transcriber = MicTranscriber(
        model_path=model_path, model_arch=model_arch
    )

    dialog_flow = DialogFlow(tts=tts)
    add_commands(dialog_flow, tts)

    mic_transcriber.add_listener(dialog_flow)

    mic_transcriber.start()

The add_commands() function calls listen_for() for all of the phrases the agent should recognize.

Getting Started with Text to Speech

Voice interfaces often need to talk back, and Moonshine's TextToSpeech is designed to make that easy, across multiple languages. It's also self-contained, so you can use it independently from the transcription and dialog modules.

At its simplest, you can just specify the output language to create a speech synthesizer object and then pass text into it to speak it on the default audio device:

from moonshine_voice import TextToSpeech

tts = TextToSpeech("fr")
tts.say("Bonjour, mon ami")
tts.wait()  # block until playback finishes

say() returns immediately and queues the text for background synthesis and playback. Calling say() multiple times queues each utterance in order, and the next utterance is pre-synthesized while the current one plays. You can also pass a list of strings, cancel everything with stop(), or poll with is_talking():

tts.say(["One.", "Two.", "Three."])
tts.stop()  # cancel remaining utterances and halt playback

If you're on a machine without an audio output, or want to do further processing, you can retrieve the audio samples using the synthesize() method:

from moonshine_voice import TextToSpeech

tts = TextToSpeech("en-us")
audio_data, sample_rate = tts.synthesize("Howdy, partner")

As you can see, text to speech supports multiple languages. To see which are available, run the list_tts_languages() function:

from moonshine_voice import list_tts_languages
list_tts_languages()

['ar-msa', 'de-de', 'en-gb', 'en-us', 'es-ar', 'es-es', 'es-mx', 'fr-fr', 'hi-in', 'it-it', 'ja-jp', 'ko-kr', 'nl-nl', 'pt-br', 'pt-pt', 'ru-ru', 'tr-tr', 'uk-ua', 'vi-vn', 'zh-hans']

For each language, you can list which voices are available:

from moonshine_voice import list_tts_voices

list_tts_voices("ru")

{'present': [], 'downloadable': ['piper_ru_RU-denis-medium', 'piper_ru_RU-dmitri-medium', 'piper_ru_RU-irina-medium', 'piper_ru_RU-ruslan-medium']}

If a voice is marked as downloadable that means if you pass it in to the TextToSpeech constructor then Moonshine will download it to a cache automatically (as long as the download argument is its default true) and will be available on your machine with no internet access required for subsequent calls.

Voice Cloning

The integrated ZipVoice model can imitate someone's voice, given a short audio clip. Pass the clip to the TextToSpeech constructor's clone argument, either as a path to a .wav file or as a (pcm, sample_rate) pair of mono float samples. You can also pass clone_transcript, the text spoken in the clip; when omitted, Moonshine auto-transcribes the clip with its ASR model before cloning (this takes a few extra seconds on first use):

from moonshine_voice import TextToSpeech
import importlib.resources;

clone_path = importlib.resources.files("moonshine_voice.assets").joinpath("clone-test.wav")
clone_transcript = "Ever tried. Ever failed. No matter. Try Again. Fail again. Fail better."

tts = TextToSpeech(
    "en-us",
    clone=clone_path,
    clone_transcript=clone_transcript,
)
tts.say("Ask not what your country can do for you, but what you can do for your country")
tts.wait()

The ZipVoice engine is selected automatically when clone is set, so no voice argument is needed (passing a voice together with clone raises an error).

You can also try cloning from the command line. Since you won't always have easy access to a clean transcript of the speech you want to clone from, you can leave it out and have Moonshine automatically generate one, in both the API and command line.

curl -O -L 'https://github.com/moonshine-ai/moonshine/raw/refs/heads/main/python/src/moonshine_voice/assets/clone-test.wav'

python3 -m moonshine_voice.tts \
  --clone clone-test.wav \
  --text "I am so excited about Moonshine Voice's text to speech"

Voice Samples

To help you choose a voice, here are sample clips of each one saying "Welcome to Moonshine Voice text to speech". Each entry is the voice name you can pass to the TextToSpeech constructor; click the ▶ next to it to hear it.

ZipVoice

These voices were created using the zero-shot voice cloning capabilities of ZipVoice, a high-quality flow-matching TTS model from the k2-fsa team. It takes significantly longer to generate than Kokoro or PiperTTS, but offers voice cloning and more realistic speech.

Kokoro

These voices come from the excellent Kokoro project, an 82-million-parameter open-weight TTS model that delivers quality comparable to much larger models.

Piper TTS

The Piper project provides over a hundred lightweight voices across all of the languages Moonshine supports, from many contributors — too many to sample here. You can listen to every Piper voice on the Piper voice samples page, and use any of them with Moonshine through the piper_ voice names returned by list_tts_voices().

Converting Graphemes to Phonemes

As you may notice from the voice names, Moonshine Voice uses models from the fantastic Kokoro and PiperTTS projects. You can find full details on all the model and data sources we use for text to speech at core/moonshine-tts/data/README.md.

Given that there are other great TTS projects out there, why does the world need yet another implementation? Moonshine tries to run on as many platforms as possible and supports commercial applications, and both Kokoro and Piper use espeak-ng to convert text strings into phonemes, representations of the noises associated with the sentence, in the International Pronunciation Alphabet. Espeak-ng is licensed under the GPL, and while I am a fan of free software, the terms do make it hard to incorporate into applications that don't also release their source code under a similar license.

In the cloud this isn't as much of an issue, as many uses of espeak-ng can be implemented by calling out to an external executable, so the dependency isn't as problematic. This isn't an option on many edge operating systems unfortunately, as the only way to include code on iOS or Android is to link it into the application, which requires open sourcing the calling code.

To allow wider usage, we developed our own "grapheme to phoneme" module that performs a similar role, but has been written from scratch. You'll find the implementation in core/moonshine-tts and it's released under the same MIT License as the rest of this code base.

Every language requires a different process to convert its written form into speech, and often it varies by dialect too. This is why espeak-ng is so widely used, it has had years of work put into it to encode linguistic knowledge into a complex set of rules, many of which are heuristics that require a lot of testing to get right. The Moonshine Voice G2P engine is still new, and will need similar tuning to handle all of the variations across languages, but I'm hoping the initial implementation is a good start and will benefit from community feedback and contributions over time. Here are the current results for intelligibility across languages, using scripts/tts_g2p_intelligibility.py:

If you want access to just the grapheme to phoneme capability, without the speech synthesis, you can all it directly:

from moonshine_voice import GraphemeToPhonemizer

g2p = GraphemeToPhonemizer("en-us")
g2p.to_ipa("Hello world")

'həlˈoʊ wˈɝld'

Examples

The examples folder has code samples organized by platform. We use the usual tooling per stack (Android Studio and Gradle, Xcode and Swift on Apple platforms, Visual Studio on Windows). GitHub Releases currently ship the downloadable assets below (example trees are mostly named {platform}-{Project}.tar.gz; Windows and C++ also include prebuilt native library bundles).

• Android
  • TextToSpeech
  • Transcriber
• Portable C++
  • cpp-examples.tar.gz (sources plus the download-library.sh helper)
  • transcriber.cpp
  • text-to-speech.cpp
• iOS
  • TextToSpeech
  • Transcriber
• MacOS
  • BasicTranscription
  • DialogFlow
  • MicTranscription
  • TextToSpeech
• Windows
  • cli-transcriber
• Python
  • basic_transcription.py
  • mic_transcription.py
  • dialog_flow.py
  • ollama-voice/ollama_voice.py
• Raspberry Pi
  • my-dalek
  • Pi Help Bot

The examples usually include one minimal project that just creates a transcriber and then feeds it data from a WAV file, and another that's pulling audio from a microphone using the platform's default framework for accessing audio devices. Each one is a self-contained project you can copy out of the tree: the Android samples depend on ai.moonshine:moonshine-voice:0.1.0 from Maven Central, and the Apple ones pull MoonshineVoice from the Swift package.

None of them bundle model weights. Every engine downloads what it needs on first use — the speech model for Transcriber, the voice and G2P assets for TextToSpeech, the embedding model for DialogFlow — from https://download.moonshine.ai/, reporting progress through the onProgress callback the examples wire up to a label. Downloads are cached (under filesDir on Android, Caches/MoonshineModels on Apple platforms), so later launches run offline. Switching to a different voice triggers the same on-demand download for whatever that voice needs.

If you want a fully offline build with no first-run download, fetch the assets ahead of time and point the engine at them with modelsFrom(path); see docs/design/api-comparison.md for the tradeoff.

Adding the Library to your own App

We distribute the library through the most widely-used package managers for each platform. Here's how you can use these to add the framework to an existing project on different systems.

Python

The Python package is hosted on PyPi, so all you should need to do to install it is pip install moonshine-voice, and then import moonshine_voice in your project.

Command-line tools

Installing the pip package adds a moonshine-voice command (with a shorter moonshine alias) that groups the built-in tools as subcommands:

moonshine-voice --help

Run moonshine-voice <command> --help for the options each one accepts. Every subcommand is equivalent to running the underlying module directly, so moonshine-voice mic --language en and python -m moonshine_voice.mic_transcriber --language en do exactly the same thing.

iOS or MacOS

For iOS we use the Swift Package Manager, with an auto-updated GitHub repository holding each version. To use this right-click on the file view sidebar in Xcode and choose "Add Package Dependencies..." from the menu. A dialog should open up, paste https://github.com/moonshine-ai/moonshine-swift/ into the top search box and you should see moonshine-swift. Select it and choose "Add Package", and it should be added to your project. You should now be able to import MoonshineVoice and use the library. You will need to add any model files you use to your app bundle and ensure they're copied during the deployment phase, so they can be accessed on-device.

For reference purposes you can find Xcode projects with these changes applied in examples/ios/Transcriber and examples/macos/BasicTranscription.

Android