Getting a Webcam and a Microphone Into a Model, Through a Proxmox LXC

Device passthrough that silently drops the mic, auto-exposure that halves your frame rate, an OpenCV flag that halves it again, and two audio pipelines that disagree about sample rates.
Author

Benedict Thekkel

Published

July 24, 2026

The goal was simple: point a camera at something and have a model tell me what it is, live, in a notebook. The machine is an unprivileged LXC container on Proxmox, and the camera is a USB webcam plugged into the host. Every number here was measured on that box, not guessed.

Nothing in this chain is hard. There are just five separate places it quietly does the wrong thing instead of failing.

1. The camera does not bring its microphone

A USB webcam presents as two unrelated kinds of device node, and passing one through does not pass the other:

Half Node Notes
Camera /dev/video0 UVC capture, MJPEG and YUYV, up to 2592x1520
Camera metadata /dev/video1 Not a capture node. Opening it fails with VIDIOC_G_INPUT
Microphone /dev/snd/pcmC0D0c + controlC0 ALSA card 0

They are separate entries in the Terraform device_passthrough list. Adding the video nodes and forgetting the sound nodes is the single most likely reason a demo that “should work” does not, and the error you get is about the microphone being absent rather than about passthrough.

Two more things worth knowing before you start debugging the wrong layer:

  • Device numbers are not stable across replug. Re-seat the camera, or attach a second one first, and video0 can become video2. Check on the host before applying.
  • /proc/asound/cards lies. procfs is not namespaced for sound, so inside the container it cheerfully lists the host’s sound cards even when the container has no /dev/snd at all. The only honest test is ls -l /dev/snd/.
ls -l /dev/video* /dev/snd/     # the actual test
cat /proc/asound/cards          # shows the HOST's cards. Ignore it.

2. Auto-exposure is a frame-rate control

The camera reports 30 FPS. In a dim room it delivers 15, because the sensor lengthens exposure time to get a usable image and the frame rate is what pays for it.

You can pin it, at the cost of doing your own exposure:

cap.set(cv2.CAP_PROP_AUTO_EXPOSURE, 1)   # 1 = manual, 3 = aperture priority (default)
cap.set(cv2.CAP_PROP_EXPOSURE, 150)

The sharp edge: this is a device-level setting that persists across processes. It outlives the Python process that set it, so a debugging session that leaves the camera in manual mode will confuse you tomorrow. Set it back to 3 when you are done.

3. CAP_PROP_BUFFERSIZE halves your frame rate and buys nothing

This is the one I would not have guessed. The usual advice for “my frames are stale” is to shrink the capture buffer to 1. On the V4L2 backend that halves the delivered frame rate and does not make frames any fresher.

Measured at 640x480 MJPG:

Both settings compose, and between them they span a 4x range in frame rate on identical hardware and identical code. Manual exposure with no buffer tweak is 30 FPS; auto-exposure with BUFFERSIZE=1 is 7.5.

Resolution, meanwhile, barely matters: at 1280x720 the read still costs about 67 ms. JPEG-encoding a 640x480 side-by-side pair for display costs 1.4 ms. The camera is the bottleneck, not your code, which is a useful thing to establish early because it stops you optimising the wrong half.

4. There is no GUI, so cv2.imshow is out

The container has no display, and you are looking at JupyterLab over HTTP anyway. Live video goes through IPython.display handles that you update in place:

view   = display(IPyImage(data=_jpeg(blank)), display_id=True)
status = display(Pretty("starting..."), display_id=True)

while ...:
    view.update(IPyImage(data=_jpeg(pair_view(raw, annotated))))
    status.update(Pretty(f"frame {n}  {fps:.1f} FPS  {info}"))

Measured end to end, camera through annotation through JPEG through display: 30 FPS. The display path is not what limits a live demo.

Four details that turn a demo into something usable:

  • Raw on the left, model output on the right. The model’s contribution should be visible rather than asserted.
  • Use Pretty(...), not a bare string. IPython renders a plain string through repr(), so you get your status line wrapped in quotes.
  • Catch KeyboardInterrupt so the stop button ends the loop cleanly, and release the camera in finally. A demo that raises mid-loop and keeps /dev/video0 open means the next run cannot start.
  • Bound the loop with a time budget. An unbounded while True in a notebook is a cell nobody can stop safely.

And warm up about ten frames before timing or inferring anything. The first frames come back dark while auto-exposure settles, so without a warm-up both your model output and your FPS number are measured on garbage.

5. The audio side disagrees with itself about sample rates

Two traps here, and the second one is silent.

The PortAudio device name is not the ALSA card id. Card id U2K shows up to PortAudio as UGREEN camera 2K: USB Audio (hw:0,0). Match on the card id and you fall through to “first input device”, which may be sysdefault, which may not be the mic you meant.

Record at the device’s native rate and resample yourself. This mic runs at 44.1 kHz; the speech models want 16 kHz. Asking ALSA for a rate the hardware does not offer is the usual cause of audio that transcribes like speech played too fast or too slow.

Then the part that produces no error at all:

Pipeline Rate Accepts a dict?
automatic-speech-recognition 16 kHz yes
audio-classification (AST) 16 kHz yes, and it resamples for you
zero-shot-audio-classification (CLAP) 48 kHz no - rejects dicts, and assumes a bare array is already at 48 kHz

Hand CLAP 16 kHz audio and it raises nothing. It scores a signal playing at a third of its true speed and returns confident nonsense. If a cell feeds two models, keep a copy at each rate.

The general form of the rule: do not assume, ask the feature extractor.

AutoFeatureExtractor.from_pretrained(model_id).sampling_rate

Fail loudly, and never fake the stream

Whisper hallucinates fluent sentences out of digital silence, so a muted capture does not fail. It produces a confident transcript of nothing. So every take reports its level, in dBFS, and both ends are checked:

Peak Meaning Action
< 1e-4 muted capture raise
< 0.05 quiet, degrades every model warn and scale up
> 1.0 overloaded, float32 overshoots full scale warn and normalise

Observed peaks on this mic swing between -36 and +1.5 dBFS between takes, so both ends are real cases rather than theoretical ones.

The broader rule I settled on: if the device is missing, raise. Do not fall back to a static image, a dataset sample, or simulated noise. A silent fallback makes a broken capture path render as a successful run, which is strictly worse than an error.

The chain, end to end

flowchart LR
  subgraph Host[Proxmox host]
    CAM[/dev/video0/]
    SND[/dev/snd/pcmC0D0c/]
  end
  subgraph CT[Unprivileged LXC]
    CV[OpenCV V4L2<br/>MJPEG + warm-up]
    SD[sounddevice<br/>native rate]
    RS[librosa resample]
    M[transformers pipeline]
    D[IPython display handle]
  end
  CAM -->|device_passthrough| CV --> M --> D
  SND -->|separate passthrough| SD --> RS --> M

Takeaway

Every one of these failed by doing something reasonable instead of raising: passthrough that half worked, a sensor trading frame rate for exposure, a buffer flag that made things worse, a display call with no display, and a model quietly accepting the wrong sample rate.

If you take three things: video and audio are separate passthroughs, do not set CAP_PROP_BUFFERSIZE, and ask the feature extractor for its sample rate rather than assuming 16 kHz. The working code lives in the Audio and Computer Vision notebooks of DL_tasks, and the container that hosts it is built by the Terraform and Ansible setup.

Back to top