VisionBridge: an MIT proxy that gives text-only LLMs vision capabilities

VisionBridge is an open-source, MIT-licensed proxy that sits between chat UIs and models, giving text-only reasoning LLMs (served via LM Studio, Ollama, vLLM, Z.ai, OpenRouter, or any OpenAI-compatible backend) the ability to interpret images. When an incoming request contains images, VisionBridge stores them, rewrites the prompt, and lets the reasoning model call tools such as look, ocr, scan, crop_and_look, and compare to query a separate vision model (LLaVA, Qwen-VL, GPT-4o-mini, etc.). The vision model handles seeing while the reasoning model handles thinking.

Before the tool loop starts, each image gets a one-shot scene description so the model begins with global context; multiple tool calls can run concurrently, and repeated images are served from a content-addressed cache. Tool calling defaults to native OpenAI function calling and transparently falls back to a prompt-JSON protocol when unsupported, letting the system work with both strong hosted models and weaker local ones. Errors like hallucinated image IDs or malformed JSON are fed back as observations so the model can self-correct.

Deployable via a single Docker command, VisionBridge supports popular backends like LM Studio, Ollama, and vLLM, and a single instance can serve multiple reasoning+vision model pairs under different model names. Built-in security includes SSRF protections, size limits, and optional Bearer key auth. For engineers, it offers a low-cost way to bolt vision capability onto existing open-source or hosted LLM stacks without retraining or custom weights, with full tool-call traceability via a /v1/traces endpoint.