» Tag

local-llm

11 posts

Jul 27, 20269.21
397B MoE Model Runs on a Single RTX PRO 6000 96GB GPU
Krasis runtime runs a 397B-parameter MoE model on one RTX PRO 6000 96GB GPU, hitting 2,354 tok/s prefill via CPU-VRAM expert streaming.
gpu llm-inference local-llm moe
Jul 12, 20268.43
Tesla P100's silent FP16 precision bug in llama.cpp fixed
A 3-line CUDA patch in llama.cpp fixes a years-old silent FP16 precision bug on Tesla P100 GPUs, with zero performance cost.
cuda gpu-inference llama.cpp local-llm
Jul 11, 20268.30
GLM 5.2 Local Deployment Guide: Quantization and Hardware Choices
A practical breakdown of running the 753B-parameter MoE model GLM 5.2 locally, comparing Unsloth dynamic quantizations, RAM needs, and hardware options.
glm-5.2 gpu-hardware llm-inference local-llm
Ad
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.com
Jul 8, 20268.10
VisionBridge: an MIT proxy that gives text-only LLMs vision capabilities
VisionBridge is a lightweight OpenAI-compatible proxy that lets any text-only LLM inspect images via a separate vision model, with no training or weights required.
llm local-llm mit-license open-source
Jul 7, 20268.10
antirez's DS4 engine: DeepSeek V4 Flash tops local coding on a MacBook Pro
DeepSeek V4 Flash via antirez's DS4 engine one-shots complex coding prompts on a MacBook Pro, outperforming other local models in real-world testing.
deepseek llm-engineering local-llm macbook-pro
Jul 13, 20268.00
Small Models by Choice: Running an Autonomous Agent on a 16GB M1 Mac
Why one engineer deliberately runs an autonomous agent on a 16GB M1 Mac with small models instead of scaling up to cloud LLMs — a design case study.
agent-design ai edge-computing llm
Jul 12, 20268.00
Kira Project: a tamper-evident evidence layer for AI agents
Kira Project's tamper-evident evidence architecture separates AI agent logs from reports, cryptographically proving what happened and what didn't.
agents ai local-llm observability
Jul 9, 20267.90
Ghost Vessel Gives a Local AI Agent a Face Instead of a Chat Box
A developer swapped his local LLM chat interface for a video-call-style avatar. Ghost Vessel uses pre-rendered clips and emotion tags to stay GPU-free, and is open source.
ai ai-agents llm local-llm
Jul 21, 20267.80
Building a Fully Local Voice Assistant: No Cloud, Just Hardware
I built a fully local voice assistant without cloud reliance. Open-source, 5 seconds response time.
local-llm machine-learning open-source speech-recognition
Aug 10, 20267.31
1.5B Model Trained to Write Shell Commands on a Laptop CPU
User trains a 1.5B Qwen2.5-Coder model to generate shell commands. It operates on a laptop.
ai local-llm machine-learning nlp