diff --git a/docs/linux-voice-assistant-xvf3800.md b/docs/linux-voice-assistant-xvf3800.md index 38d4154..3604ddf 100644 --- a/docs/linux-voice-assistant-xvf3800.md +++ b/docs/linux-voice-assistant-xvf3800.md @@ -1,22 +1,14 @@ # Linux Voice Assistant with ReSpeaker XVF3800 4‑Mic Array — Installation & Configuration Guide -> Drafted with ChatGPT based on existing `linux-voice-assistant-2mic-install.md` and -> community test results from the XVF3800 integration Discussion. +> Drafted with ChatGPT based on existing `linux-voice-assistant-2mic-install.md` and community test results from the XVF3800 integration Discussion. -This guide shows how to use the **Seeed ReSpeaker XVF3800 USB 4‑Mic Array** with the -**linux-voice-assistant** (LVA) project on Raspberry Pi or other Linux hosts. +> Special thanks to @JenniferHatches and @cknic for making the recommendation and doing so much work already for support of the reSpeaker XVF3800. (https://github.com/cknic/linux-voice-assistant) The addition of XVF3800 support would never have happened without their troubleshooting of the code and fantastic suggestions when working through bugs. -Unlike the ReSpeaker 2‑Mic HAT, the XVF3800 is a **USB Audio Class 2.0** device: -it acts as a “smart microphone + speaker” front end with built‑in **AEC, beamforming, -noise suppression, AGC, and echo‑canceling speakerphone** processing. +This guide shows how to use the **Seeed ReSpeaker XVF3800 USB 4‑Mic Array** with the **Linux Voice Assistant** (LVA) project on Raspberry Pi or other Linux hosts. -LVA treats the XVF3800 as a **standard USB mic/speaker**: -- Audio in: 16 kHz, 16‑bit, 2‑channel PCM (downmixed to mono by the audio stack) -- Audio out: 16 kHz, 16‑bit, 2‑channel PCM via the XVF3800 headphone jack / speaker +Unlike the ReSpeaker 2‑Mic HAT, the XVF3800 is a **USB Audio Class 2.0** device: it acts as a “smart microphone + speaker” front end with built‑in **AEC, beamforming, noise suppression, AGC, and echo‑canceling speakerphone** processing. -No custom kernel drivers are required. Plug it in, select it as your audio device in -LVA’s `config.json`, and it will “just work” as a high‑quality mic for wake word and -ESPHome streaming. +No custom kernel drivers are required. Plug it in, setup the device in LVA’s `config.json`, and it will “just work” as a high‑quality mic and speaker (if connected). --- @@ -28,27 +20,65 @@ ESPHome streaming. - Other modern Linux with PipeWire or PulseAudio - Python 3.11+ recommended - **ReSpeaker XVF3800 4‑Mic Array**, connected via its **XMOS USB‑C port** -- Network access to your Home Assistant instance -- The `linux-voice-assistant` repository checked out (optionally on the `XVF3800` branch) +- SSH access to your LVA instance +- Network access to the Home Assistant for registration + +## Recommended Configuration + +- To use the advanced LED effects, upgrade to the latest XVF3800 firmware (respeaker_xvf3800_usb_dfu_firmware_v2.0.7.bin as of this writing). +-- https://github.com/respeaker/reSpeaker_XVF3800_USB_4MIC_ARRAY/blob/master/xmos_firmwares/dfu_guide.md +- Tested and works on a Raspberry Pi Zero 2W, but I don't recommend attempting to flash the firmware with this device. +- Pipewire (or PluseAudio) installed and tested. I haven't tested the XVF3800 with ALSA. --- -## 1. Get the code (XVF3800 branch) +## 1. Install system packages and Pipewire (Recommended) -If you don’t already have the project cloned, you can pull the dedicated XVF3800 branch: +```bash +sudo apt update +sudo apt upgrade +sudo apt install build-essential git \ + libmpv-dev mpv python3-dev python3-venv \ + pulseaudio-utils pipewire wireplumber \ + pipewire-audio libspa-0.2-modules libusb-1.0-0 +sudo reboot +``` + +--- + +## 2. Get the code (XVF3800 branch) + +If you don’t already have the project cloned: ```bash cd ~ -git clone --branch XVF3800 --single-branch https://github.com/imonlinux/linux-voice-assistant.git +git clone https://github.com/imonlinux/linux-voice-assistant.git cd linux-voice-assistant ``` -If you already have the repository, you can switch branches: +If you already have the repository, you can: ```bash +cp ~/linux-voice-assistant/linux_voice_assistant/config.json ~/ cd ~/linux-voice-assistant -git fetch origin -git checkout XVF3800 +git pull +cp ~/config.json ~/linux-voice-assistant/linux_voice_assistant/config.json +``` + +Implement the UDEV rule to prevent the XVF3800 from taking naps and to allow user access to the device + +```bash +cp ~/linux-voice-assistant/XVF3800/99-respeaker-xvf3800.rules /etc/udev/rules.d/ +``` + + +Add user to the plugdev group for access to the XVF3800 and reboot + +> Tip: If you use a different username than `pi`, adjust user accordingly. + +```bash +sudo usermod -aG plugdev pi +sudo reboot ``` Set up the project (creates a virtualenv, installs dependencies, etc.): @@ -58,206 +88,84 @@ cd ~/linux-voice-assistant script/setup ``` -> Tip: If you use a different username than `pi`, adjust paths accordingly. +1. Connect the XVF3800 to your Pi/host using the **XMOS USB‑C port** (the one next to the 3.5 mm audio jack). +2. If you want local audio output, connect the XVF3800’s headphone jack or JST-PH 2.0mm 2 Pin to an external speaker. --- -## 2. Connect and verify the XVF3800 +## 3. Update the config.json file for XVF3800 support and options -### 2.1 Plug in the board - -1. Connect the XVF3800 to your Pi/host using the **XMOS USB‑C port** (the one next to - the 3.5 mm audio jack). -2. If you want local audio output, connect the XVF3800’s headphone jack to an external - powered speaker. - -### 2.2 Confirm ALSA sees the device - -```bash -arecord -l -aplay -l -``` - -You should see entries similar to: - -```text -**** List of CAPTURE Hardware Devices **** -card 3: Array [reSpeaker XVF3800 4-Mic Array], device 0: USB Audio [USB Audio] - Subdevices: 0/1 - Subdevice #0: subdevice #0 - -**** List of PLAYBACK Hardware Devices **** -card 3: Array [reSpeaker XVF3800 4-Mic Array], device 0: USB Audio [USB Audio] - Subdevices: 1/1 - Subdevice #0: subdevice #0 -``` - -The exact card index (`3` above) may differ on your system. - -### 2.3 Quick `arecord` test (optional) - -Replace `3` with your card number: - -```bash -arecord -D plughw:3,0 -c 2 -r 16000 -f S16_LE -d 5 -vv /tmp/xvf3800-test.wav -aplay /tmp/xvf3800-test.wav -``` - -You should hear your voice through your default playback device, confirming that: -- The XVF3800 is streaming at **16 kHz**, **16‑bit**, **stereo**, and -- ALSA can capture from it reliably. - ---- - -## 3. PipeWire / PulseAudio device mapping - -On a PipeWire or PulseAudio system, check how the XVF3800 shows up at the graph level: - -```bash -pactl list short sources -pactl list short sinks -``` - -Typical output: - -```text -# Sources -309 alsa_input.usb-Seeed_Studio_reSpeaker_XVF3800_4-Mic_Array_...-00.analog-stereo PipeWire s16le 2ch 16000Hz RUNNING - -# Sinks -308 alsa_output.usb-Seeed_Studio_reSpeaker_XVF3800_4-Mic_Array_...-00.analog-stereo PipeWire s16le 2ch 16000Hz SUSPENDED -``` - -Key points: - -- The XVF3800 appears as an **analog stereo** source/sink (`2ch 16000 Hz`). -- LVA will use these devices through the `soundcard` library. - ---- - -## 4. LVA’s view of the XVF3800 (input/output devices) - -From the project root, use LVA’s helper flags to inspect devices: - -```bash -cd ~/linux-voice-assistant -script/run -- --list-input-devices -script/run -- --list-output-devices -``` - -Example output: - -```text -Input devices - -[0] reSpeaker XVF3800 4-Mic Array Analog Stereo -``` - -```text -Output devices - -auto: Autoselect device -pipewire: Default (pipewire) -pipewire/alsa_output.usb-Seeed_Studio_reSpeaker_XVF3800_4-Mic_Array_...analog-stereo: reSpeaker XVF3800 4-Mic Array Analog Stereo -... -``` - -Take note of: - -- The **input device name** for the XVF3800 (e.g. `reSpeaker XVF3800 4-Mic Array Analog Stereo`) -- The output device you want to use (often `pipewire` or the specific XVF “analog‑stereo” sink) - ---- - -## 5. Configure LVA audio for XVF3800 - -Edit the main config: +Edit the config.json file: ```bash nano ~/linux-voice-assistant/linux_voice_assistant/config.json ``` -Add or update the `audio` section: +Add or update the following sections (Recommended settings): + +> Tip: Be sure to change the name if you have more than one LVA! ```json -"audio": { - "input_device": "reSpeaker XVF3800 4-Mic Array Analog Stereo", - "output_device": "pipewire", - "input_block_size": 1024 +{ + "app": { + "name": "Linux Voice Assistant" + }, + "led": { + "enabled": true, + "led_type": "xvf3800", + "interface": "usb" + }, + "button": { + "enabled": true, + "mode": "xvf3800", + "poll_interval_seconds": 0.15 + }, + "mqtt": { + "host": "192.168.0.100", + "port": 1883, + "username": "mqtt", + "password": "Ss1552157" + }, + "audio": { + "volume_sync": true, + "max_volume_percent": 150 + } } ``` -Examples: - -- If you want both input and output on the XVF3800: - - ```json - "audio": { - "input_device": "reSpeaker XVF3800 4-Mic Array Analog Stereo", - "output_device": "pipewire/alsa_output.usb-Seeed_Studio_reSpeaker_XVF3800_4-Mic_Array_101991441253700076-00.analog-stereo", - "input_block_size": 1024 - } - ``` - -- If you only want to use the XVF3800 as a **mic** (and another device for playback), set - `output_device` to your preferred sink from `--list-output-devices`. - -### 5.1 How channels are handled (important detail) - -The XVF3800 exposes itself as a **2‑channel** source (stereo, 16 kHz). Internally, the -LVA `AudioEngine` uses the `soundcard` library and **always requests 1 channel** from the -selected input device: - -```python -with self.mic.recorder( - samplerate=16000, - channels=1, - blocksize=self.block_size, -) as mic_in: - audio_chunk_array = mic_in.record(self.block_size).reshape(-1) - ... -``` - -This has two important implications: - -1. The audio stack (PipeWire/PulseAudio/ALSA) takes care of converting from the XVF3800’s - **2‑channel** stream to a **mono** stream before LVA sees it. -2. LVA’s wake‑word engines and ESPHome streaming always receive 16 kHz **mono** PCM - (`int16`), regardless of whether the underlying hardware is mono or stereo. - -In practice: - -> You do **not** need any XVF3800‑specific code changes in the audio engine. -> Selecting the XVF3800 as the `audio.input_device` is sufficient; LVA will process it -> as a mono mic just like other devices. +> Take a look at ~/linux-voice-assistant/linux_voice_assistant/config.json.example for details on these settings as well as all available options. --- -## 6. Start LVA as a user-mode service +## 4. Start LVA as a user-mode service The service setup is identical to other configurations (2‑Mic, etc.). Assuming you want LVA to run as a user‑mode systemd service: -### 6.1 Enable linger for your user (e.g. `pi`) +Enable linger for your user (e.g. `pi`) ```bash sudo loginctl enable-linger pi ``` -### 6.2 Install the systemd user service +Install the systemd user service + +> Tip: If you use a different username than `pi`, adjust user accordingly. ```bash mkdir -p ~/.config/systemd/user -cp ~/linux-voice-assistant/service/linux-voice-assistant.service ~/.config/systemd/user/linux-voice-assistant.service +cp ~/linux-voice-assistant/service/linux-voice-assistant_xvf3800.service ~/.config/systemd/user/linux-voice-assistant.service ``` -### 6.3 Enable & start +Enable & start ```bash systemctl --user daemon-reload systemctl --user enable --now linux-voice-assistant.service ``` +> Note: The LVA initiate a XVF3800 reboot at start, this is needed to resolve a "mic not detecting audio" issue. As a result, the LVA will take a little longer to start up and be ready. + Check status: ```bash @@ -272,69 +180,22 @@ Look for log lines indicating: --- -## 7. Acoustic Echo Cancellation (AEC) considerations +## 5. Acoustic Echo Cancellation (AEC) The XVF3800 already implements **hardware AEC / echo‑canceling speakerphone** processing. -In many cases, you can rely **only** on the XVF3800’s built‑in AEC and **skip** the -PipeWire/PulseAudio `module-echo-cancel` setup. - -You have three practical options: - -### 7.1 Rely on XVF3800 AEC only (recommended starting point) - -- Do **not** load any user‑mode echo‑cancel module. -- Set `input_device` to the XVF3800 source (or an echo‑cancel source that wraps it, if your - system has one by default). -- Play media (music, games, etc.) through the XVF3800 or another output and test: - - Wake word recognition reliability - - STT quality in noisy playback conditions - -If it works well, this is the simplest and most CPU‑efficient setup. - -### 7.2 Combine XVF3800 with software AEC - -If you already have a PipeWire/PulseAudio echo‑cancel device configured, you can route -XVF3800 through it: - -1. Create an echo‑cancel source/sink pair (see `install_pipewire.md` or - `install_pulseaudio.md`). -2. Use `script/run --list-input-devices` / `--list-output-devices` to find names like - `Echo-Cancel Source` and `pipewire/echo-cancel-sink`. -3. Point LVA at those devices in `config.json`: - - ```json - "audio": { - "input_device": "Echo-Cancel Source", - "input_block_size": 1024, - "output_device": "pipewire/echo-cancel-sink" - } - ``` - -This stacks XVF3800’s hardware processing with additional software AEC. It may help in -particularly difficult acoustic environments, at the cost of some complexity and CPU. - -### 7.3 No AEC at all (not recommended for open‑air speakers) - -You can also run without any AEC (hardware or software) if: - -- You use **headphones** instead of speakers, or -- The device isn’t playing audio while listening (e.g. strictly microphone‑only role). - -In that case, you can treat XVF3800 as “just another USB mic” and ignore its speakerphone -capabilities. --- -## 8. Troubleshooting & debug tips +## 6. Troubleshooting & debug tips -### 8.1 Check that LVA sees the XVF3800 +### Check that LVA sees the XVF3800 -Re‑run: +Make sure the XVF3800 is available to the LVA code: ```bash cd ~/linux-voice-assistant -script/run -- --list-input-devices -script/run -- --list-output-devices +script/run --list-input-devices +script/run --list-output-devices ``` If `reSpeaker XVF3800 4-Mic Array Analog Stereo` doesn’t appear: @@ -347,7 +208,7 @@ If `reSpeaker XVF3800 4-Mic Array Analog Stereo` doesn’t appear: lsusb | grep -i 2886 ``` -### 8.2 Tail the LVA logs +### Tail the LVA logs ```bash journalctl --user -u linux-voice-assistant.service -f @@ -358,46 +219,3 @@ Look for: - “Audio engine started.” - “Opening audio input device: reSpeaker XVF3800 4-Mic Array Analog Stereo” - Wake‑word detection lines (MicroWakeWord / OpenWakeWord) - -### 8.3 Optional: XVF3800 probe script - -On the `XVF3800` branch, there is a small probe script in `tests/xvf3800_probe.py` that -can be used to inspect how the audio backend opens the device. - -For example: - -```bash -cd ~/linux-voice-assistant -python tests/xvf3800_probe.py --device "reSpeaker XVF3800 4-Mic Array Analog Stereo" -``` - -This will print: - -- The device list from the `soundcard` backend -- The samplerate and channel count used when opening the stream -- The shape and dtype of the recorded buffer - -It’s mainly a debug tool for development and Discussion threads; you don’t need it for -normal operation. - ---- - -## 9. Next steps: LEDs, button, and DOA integration - -This document covers **audio‑only** integration for the XVF3800. The board also has: - -- A 12‑LED WS2812 ring -- A hardware mute button and mute LED -- Direction‑of‑arrival (DOA) and voice activity indicators - -Future work in the LVA project may add an optional `Xvf3800Controller` that: - -- Talks to the XVF3800 over USB using a Python SDK (pyusb or HID) -- Publishes DOA, VAD, and mute button events to LVA’s internal event bus -- Synchronizes mute state and LED behavior between XVF3800, LVA, and Home Assistant - -For now, the important takeaway is: - -> As soon as the XVF3800 is selected as the input device in `config.json`, the Linux -> Voice Assistant can use it as a high‑quality microphone front end with no additional -> code changes.