eSlams

OpenAI Evals is an open-source framework for evaluating large language models and systems built with them, including tool-using agents and prompt chains. It includes a registry of benchmark evals, supports custom model-graded and private evals based on a user's data, and uses a completion-function protocol for advanced workflows. The package can be installed with pip and run locally with an OpenAI API key; the repository also documents optional result logging to Snowflake and configuration through the OpenAI Dashboard.

DeepEval is an open-source LLM evaluation framework for testing and benchmarking LLM applications. It provides more than 50 plug-and-play metrics for evaluating AI agents, multi-turn chatbots, voice agents, retrieval-augmented generation systems, and other AI applications, and supports building evaluation pipelines for application-specific testing.

An open-source research simulation for computational agents that model believable human behavior in an interactive game environment. The repository includes a Django-based environment server and a separate agent simulation server; the two run concurrently, with the simulation using an OpenAI API key and the browser-based environment providing the interactive view and replayable demo animations. The project was tested with Python 3.9.12 and includes setup instructions, required packages, and predefined simulation scenarios.

AI Engineering from Scratch is a free, open-source curriculum developed by rohitg00 for learning to build and ship AI systems end to end. Its 20-phase sequence covers development tooling, mathematical and machine-learning foundations, deep learning, NLP, computer vision, generative AI, LLM applications, agent engineering, Model Context Protocol (MCP), agent skills, reinforcement learning, and related topics, with implementations in Python, TypeScript, Rust, and Julia. The repository describes 511 lessons and approximately 329 hours of material; each lesson produces a reusable artifact such as a prompt, skill, agent, or MCP server. Learners are instructed to read the lesson documentation, type and run the code from the repository root, record command evidence and outputs, and make a small change before continuing. It provides routes for complete foundations, mathematics and machine learning, production LLM applications, agent engineering, MCP, agent skills, and Claude certification preparation. The repository is distributed under the MIT license and is accompanied by a website containing the same lesson content.

Agent S is an open-source computer-use agent framework from Simular. It accepts natural-language tasks, observes a real desktop through screen and accessibility signals, and operates ordinary applications by clicking, typing, scrolling, and executing optional local code rather than relying on per-application APIs or scripts. The project has progressed through three generations: accessibility-tree grounding with an LLM, a vision-based manager-worker architecture using multiple foundation models, and Agent S3, which scales computer-use attempts through parallel rollouts and selection. Its workflow can combine model-driven exploration with reusable automation code, replay saved skills, and recover when the interface changes. The framework supports macOS, Windows, and Linux, works with models from OpenAI, Anthropic, Gemini, OpenRouter, vLLM, and open-weight providers, and is distributed through the `gui-agents` Python package. An optional local coding environment can execute Python and Bash with the current user's permissions, so the repository warns that this capability should be used only in trusted environments.