Guides

LLM API testing and evals, without the hosted stack.

Short, practical pages for developers who call LLM APIs — how to evaluate models on macOS, debug provider requests locally, and when a desktop client beats Postman or Promptfoo for day-to-day model work.

6 guides

• An LLM evaluation tool that runs on your MacFor developers picking models: run the same prompt across providers on your Mac, compare outputs, and score them with an LLM judge — no hosted eval stack.Updated August 6, 2026
• A local LLM API client for macOSCompose, stream, and inspect LLM API calls on your Mac — keys never leave through a product backend, unlike a shared Postman cloud workspace.Updated August 6, 2026
• Roshi vs PromptfooNew to both? Roshi is a local macOS app for exploratory evals; Promptfoo is a CI framework for large, repeatable prompt suites — use them at different stages.Updated August 6, 2026
• Roshi vs Postman for LLM APIsIf you already use Postman for REST, keep it — Roshi is the specialist macOS client for streaming chat APIs, multi-model runs, and local LLM evals.Updated August 6, 2026
• Compare OpenAI, Claude, and Gemini on one promptFair bake-offs need one frozen prompt, parallel runs, and metrics beside the prose — not three playground tabs started minutes apart.Updated August 6, 2026
• Provider API testing and LLM-as-a-judgeFirst prove the provider request path, then grade candidate outputs with a judge model and an explicit rubric — all on your Mac.Updated August 6, 2026