Guides

LLM API testing and evals, without the hosted stack.

Short, practical pages for developers who call LLM APIs — how to evaluate models on macOS, debug provider requests locally, and when a desktop client beats Postman or Promptfoo for day-to-day model work.

6 guides

• An LLM evaluation tool that runs on your Mac For developers picking models: run the same prompt across providers on your Mac, compare outputs, and score them with an LLM judge — no hosted eval stack. Updated August 6, 2026
• A local LLM API client for macOS Compose, stream, and inspect LLM API calls on your Mac — keys never leave through a product backend, unlike a shared Postman cloud workspace. Updated August 6, 2026
• Roshi vs Promptfoo New to both? Roshi is a local macOS app for exploratory evals; Promptfoo is a CI framework for large, repeatable prompt suites — use them at different stages. Updated August 6, 2026
• Roshi vs Postman for LLM APIs If you already use Postman for REST, keep it — Roshi is the specialist macOS client for streaming chat APIs, multi-model runs, and local LLM evals. Updated August 6, 2026
• Compare OpenAI, Claude, and Gemini on one prompt Fair bake-offs need one frozen prompt, parallel runs, and metrics beside the prose — not three playground tabs started minutes apart. Updated August 6, 2026
• Provider API testing and LLM-as-a-judge First prove the provider request path, then grade candidate outputs with a judge model and an explicit rubric — all on your Mac. Updated August 6, 2026