01Developer toolv0.5.0 on PyPI
LLMDyno
Benchmark local and cloud LLMs as coding agents, on your own hardware.
LLMDyno runs real multi-file coding tasks through open-source agent harnesses in offline Docker sandboxes, grades them with hidden tests and blind LLM judges, and compares correctness, code quality, speed, VRAM and cost. Drive it from a web dashboard, from Python or from the command line.
- agent harnesses, each in an offline sandbox
- 7agent harnesses, each in an offline sandbox
- worst VRAM estimate error across 84 configurations
- 66 MiBworst VRAM estimate error across 84 configurations
- API operations, each with a Python call and a CLI command
- 116API operations, each with a Python call and a CLI command
- cloud providers, plus any OpenAI-compatible endpoint
- 15+cloud providers, plus any OpenAI-compatible endpoint
pip install llmdyno