Open-LLM-VTuber / Open-LLM-VTuber

Open-LLM-VTuber 📢 v2.0 Development: We are focusing on Open-LLM-VTuber v2.0 — a complete rewrite of the codebase. v2.0 is currently in its early discussion and planning phase. We kindly ask you to refrain from opening new issues or pull requests for feature requests on v1. To participate in the v2 discussions or contribute, join our developer community on Zulip. Weekly meeting schedules will be announced on Zulip. We will continue fixing bugs for v1 and work through existing pull requests. Open-LLM-VTuber 📢 v2.0 开发： 我们目前正专注于 Open-LLM-VTuber v2.0 的开发，这是一次对代码库的彻底重写。v2.0 目前处于早期的讨论和规划阶段。我们恳请您不要再为 v1 版本提交新的功能请求（Issue 或 Pull Request）。若要参与 v2 的讨论或做出贡献，请加入我们的 Zulip 开发者社区。每周的会议安排将在 Zulip 上公布。我们将继续修复 v1 的 Bug 并处理现有的 Pull Request。

ENGLISH README | 中文 README | 한국어 README | 日本語 README Documentation | 常见问题 Common Issues doc (Written in Chinese): https://docs.qq.com/pdf/DTFZGQXdTUXhIYWRq User Survey: https://forms.gle/w6Y6PiHTZr1nzbtWA 调查问卷(中文): https://wj.qq.com/s2/16150415/f50a/ ENGLISH README | 中文 README | 한국어 README | 日本語 README 文档 | 常见问题文档 (中文): https://docs.qq.com/pdf/DTFZGQXdTUXhIYWRq 用户调查问卷: https://forms.gle/w6Y6PiHTZr1nzbtWA 调查问卷(中文): https://wj.qq.com/s2/16150415/f50a/

⚠️ This project is in its early stages and is currently under active development. ⚠️ If you want to run the server remotely and access it on a different machine, such as running the server on your computer and access it on your phone, you will need to configure https, because the microphone on the front end will only launch in a secure context (a.k.a. https or localhost). See MDN Web Doc. Therefore, you should configure https with a reverse proxy to access the page on a remote machine (non-localhost). ⚠️ 本项目处于早期阶段，目前正在积极开发中。⚠️ 如果您想远程运行服务器并在其他设备上访问（例如在电脑上运行服务器，在手机上访问），您需要配置 https，因为前端的麦克风功能仅在安全上下文（即 https 或 localhost）下才能启动。请参阅 MDN Web 文档。因此，您需要通过反向代理配置 https，以便在远程机器（非 localhost）上访问页面。

⭐️ What is this project? Open-LLM-VTuber is a unique voice-interactive AI companion that not only supports real-time voice conversations and visual perception but also features a lively Live2D avatar. All functionalities can run completely offline on your computer! You can treat it as your personal AI companion — whether you want a virtual girlfriend, boyfriend, cute pet, or any other character, it can meet your expectations. ⭐️ 这是什么项目？ Open-LLM-VTuber 是一款独特的语音交互式 AI 伴侣，不仅支持实时语音对话和视觉感知，还拥有生动的 Live2D 形象。所有功能均可完全在您的电脑上离线运行！您可以将其视为您的个人 AI 伴侣——无论是虚拟女友、男友、可爱宠物还是任何其他角色，它都能满足您的期待。

The project fully supports Windows, macOS, and Linux, and offers two usage modes: web version and desktop client (with special support for transparent background desktop pet mode, allowing the AI companion to accompany you anywhere on your screen). Although the long-term memory feature is temporarily removed (coming back soon), thanks to the persistent storage of chat logs, you can always continue your previous unfinished conversations without losing any precious interactive moments. 该项目全面支持 Windows、macOS 和 Linux，并提供两种使用模式：网页版和桌面客户端（特别支持透明背景的桌面宠物模式，让 AI 伴侣可以陪伴在您屏幕的任何位置）。尽管长期记忆功能暂时移除（即将回归），但得益于聊天记录的持久化存储，您可以随时继续之前未完成的对话，不会丢失任何珍贵的互动瞬间。

In terms of backend support, we have integrated a rich variety of LLM inference, text-to-speech, and speech recognition solutions. If you want to customize your AI companion, you can refer to the Character Customization Guide to customize your AI companion’s appearance and persona. The reason it’s called Open-LLM-Vtuber instead of Open-LLM-Companion or Open-LLM-Waifu is because the project’s initial development goal was to use open-source solutions that can run offline on platforms other than Windows to recreate the closed-source AI Vtuber neuro-sama. 在后端支持方面，我们集成了多种 LLM 推理、语音合成（TTS）和语音识别（ASR）解决方案。如果您想定制您的 AI 伴侣，可以参考《角色定制指南》来调整其外观和人设。之所以命名为 Open-LLM-Vtuber 而不是 Open-LLM-Companion 或 Open-LLM-Waifu，是因为该项目最初的开发目标是使用可以在 Windows 以外的平台上离线运行的开源方案，来复刻闭源的 AI Vtuber neuro-sama。

👀 Demo 👀 演示

✨ Features & Highlights ✨ 功能与亮点

🖥️ Cross-platform support: Perfect compatibility with macOS, Linux, and Windows. We support NVIDIA and non-NVIDIA GPUs, with options to run on CPU or use cloud APIs for resource-intensive tasks. Some components support GPU acceleration on macOS. 🖥️ 跨平台支持： 与 macOS、Linux 和 Windows 完美兼容。我们支持 NVIDIA 和非 NVIDIA GPU，并提供在 CPU 上运行或使用云端 API 处理资源密集型任务的选项。部分组件支持在 macOS 上进行 GPU 加速。

🔒 Offline mode support: Run completely offline using local models - no internet required. Your conversations stay on your device, ensuring privacy and security. 🔒 离线模式支持： 使用本地模型完全离线运行——无需联网。您的对话保留在您的设备上，确保隐私和安全。

💻 Attractive and powerful web and desktop clients: Offers both web version and desktop client usage modes, supporting rich interactive features and personalization settings. The desktop client can switch freely between window mode and desktop pet mode, allowing the AI companion to be by your side at all times. 💻 美观且强大的网页与桌面客户端： 提供网页版和桌面客户端两种使用模式，支持丰富的交互功能和个性化设置。桌面客户端可在窗口模式和桌面宠物模式之间自由切换，让 AI 伴侣时刻陪伴在您身边。

🎯 Advanced interaction features: 🎯 高级交互功能：

👁️ Visual perception: supporting camera, screen recording and screenshots, allowing your AI companion to see you and your screen. 👁️ 视觉感知： 支持摄像头、屏幕录制和截图，让您的 AI 伴侣能够“看到”您和您的屏幕。
🎤 Voice interruption without headphones: (AI won’t hear its own voice). 🎤 无需耳机的语音打断：（AI 不会听到自己的声音）。
🫱 Touch feedback: interact with your AI companion through clicks or drags. 🫱 触摸反馈： 通过点击或拖拽与您的 AI 伴侣互动。
😊 Live2D expressions: set emotion mapping to control model expressions from the backend. 😊 Live2D 表情： 设置情感映射，从后端控制模型表情。
🐱 Pet mode: supporting transparent background, global top-most, and mouse click-through - drag your AI companion anywhere on the screen. 🐱 宠物模式： 支持透明背景、全局置顶和鼠标穿透——可将您的 AI 伴侣拖动到屏幕的任何位置。
💭 Display AI’s inner thoughts: allowing you to see AI’s expressions, thoughts and actions without them being spoken. 💭 显示 AI 内心想法： 让您在不通过语音播报的情况下，看到 AI 的表情、想法和动作。
🗣️ AI proactive speaking feature. 🗣️ AI 主动发言功能。
💾 Chat log persistence: switch to previous conversations anytime. 💾 聊天记录持久化： 随时切换回之前的对话。
🌍 TTS translation support: (e.g., chat in Chinese while AI uses Japanese voice). 🌍 TTS 翻译支持：（例如：用中文聊天，但 AI 使用日语语音）。

🧠 Extensive model support: 🧠 广泛的模型支持：

🤖 Large Language Models (LLM): Ollama, OpenAI (and any OpenAI-compatible API), Gemini, Claude, Mistral, DeepSeek, Zhipu AI, GGUF, LM Studio, vLLM, etc. 🤖 大语言模型 (LLM)： Ollama、OpenAI（及任何兼容 OpenAI 的 API）、Gemini、Claude、Mistral、DeepSeek、智谱 AI、GGUF、LM Studio、vLLM 等。
🎙️ Automatic Speech Recognition (ASR): sherpa-onnx, FunASR, Faster-Whisper, Whisper.cpp, Whisper, Groq Whisper, Azure ASR, etc. 🎙️ 自动语音识别 (ASR)： sherpa-onnx、FunASR、Faster-Whisper、Whisper.cpp、Whisper、Groq Whisper、Azure ASR 等。
🔊 Text-to-Speech (TTS): sherpa-onnx, pyttsx3, MeloTTS, Coqui-TTS, GPTSoVITS, Bark, CosyVoice, Edge TTS, Fish Audio, Azure TTS, etc. 🔊 语音合成 (TTS)： sherpa-onnx、pyttsx3、MeloTTS、Coqui-TTS、GPTSoVITS、Bark、CosyVoice、Edge TTS、Fish Audio、Azure TTS 等。

🔧 Highly customizable: 🔧 高度可定制：

⚙️ Simple module configuration: Switch various functional modules through simple configuration file modifications, without delving into the code. ⚙️ 简单的模块配置： 通过修改简单的配置文件即可切换各种功能模块，无需深入代码。
🎨 Character customization: Import custom Live2D models to give your AI companion a unique appearance. Shape your AI companion’s persona by modifying the Prompt. Perform voice cloning to give your AI companion the voice you desire. 🎨 角色定制： 导入自定义 Live2D 模型，赋予您的 AI 伴侣独特的外观。通过修改 Prompt 来塑造 AI 伴侣的人设。进行语音克隆，让您的 AI 伴侣拥有您想要的音色。
🧩 Flexible Agent implementation: Inherit and implement the Agent interface to integrate any Agent architecture, such as HumeAI EVI, OpenAI Her, Mem0, etc. 🧩 灵活的 Agent 实现： 继承并实现 Agent 接口，以集成任何 Agent 架构，例如 HumeAI EVI、OpenAI Her、Mem0 等。
🔌 Good extensibility: Modular design allows you to easily add your own LLM, ASR, TTS, and other module implementations, extending new features at any time. 🔌 良好的扩展性： 模块化设计让您可以轻松添加自己的 LLM、ASR、TTS 及其他模块实现，随时扩展新功能。

👥 User Reviews Thanks to the developer for open-sourcing and sharing the girlfriend for everyone to use. This girlfriend has been used over 100,000 times. 👥 用户评价 感谢开发者开源并分享了这个“女朋友”供大家使用。这个“女朋友”已经被使用了超过 10 万次。

🚀 Quick Start Please refer to the Quick Start section in our documentation for installation. 🚀 快速开始 请参阅我们文档中的“快速开始”部分进行安装。

☝ Update ⚠️ v1.0.0 has breaking changes and requires re-deployment. You may still update via the method below, but the conf.yaml file is incompatible and most of the dependencies needs to be reinstalled with uv. For those who came from versions before v1.0.0, I recommend deploy. ☝ 更新 ⚠️ v1.0.0 包含重大变更，需要重新部署。您仍然可以通过以下方法更新，但 conf.yaml 文件不兼容，且大多数依赖项需要使用 uv 重新安装。对于从 v1.0.0 之前版本升级的用户，我建议重新部署。