Hamel Husain’s Blog

DeepEval is an open-source LLM evaluation framework for testing and benchmarking LLM applications. It provides more than 50 plug-and-play metrics for evaluating AI agents, multi-turn chatbots, voice agents, retrieval-augmented generation systems, and other AI applications, and supports building evaluation pipelines for application-specific testing.

AI Engineering from Scratch is a free, open-source curriculum developed by rohitg00 for learning to build and ship AI systems end to end. Its 20-phase sequence covers development tooling, mathematical and machine-learning foundations, deep learning, NLP, computer vision, generative AI, LLM applications, agent engineering, Model Context Protocol (MCP), agent skills, reinforcement learning, and related topics, with implementations in Python, TypeScript, Rust, and Julia. The repository describes 511 lessons and approximately 329 hours of material; each lesson produces a reusable artifact such as a prompt, skill, agent, or MCP server. Learners are instructed to read the lesson documentation, type and run the code from the repository root, record command evidence and outputs, and make a small change before continuing. It provides routes for complete foundations, mathematics and machine learning, production LLM applications, agent engineering, MCP, agent skills, and Claude certification preparation. The repository is distributed under the MIT license and is accompanied by a website containing the same lesson content.

AI Engineering Lab is a free, self-paced 24-week AI engineering course developed by Zorost Intelligence AI Lab. It takes learners from Python and machine learning through deep learning, LLM internals, prompt and context engineering, retrieval-augmented generation with vector search, quantization, LoRA and DPO fine-tuning, evaluation harnesses, coding-agent harnesses, AI agents, MCP, and cloud and Databricks platforms including Azure AI Foundry, Google Vertex AI, AWS Bedrock, and Databricks. The curriculum uses one continuous fictional freight-company case study. Each week pairs a concept with a runnable Python, SQL, or PySpark artifact; from Week 3 onward, the work includes a metric and an error note. The repository contains 43 runnable notebooks, publishes the full curriculum online, requires no signup or paid API key, and states that the first eight weeks require no GPU. It is MIT licensed.

LLM Course is a free educational course and roadmap for learning large language models, published as a GitHub repository with roadmaps, articles, and Google Colab notebooks. It is organized into LLM Fundamentals, LLM Scientist, and LLM Engineer sections: the first covers mathematics, Python, neural networks, and natural language processing; the second covers transformer architecture, pre-training, post-training datasets, supervised fine-tuning, preference alignment, evaluation, quantization, and emerging techniques; and the third covers running models, vector storage, retrieval-augmented generation, agents, inference optimization, deployment, and security. Its practical materials include notebooks for fine-tuning, quantization, model merging, evaluation, dataset deduplication, and creating model interfaces.

GuideLLM is an open-source platform for benchmarking and evaluating real-world LLM inference deployments. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, using real or synthetic text and multimodal datasets and configurable execution profiles such as synchronous, concurrent, throughput, constant-rate, Poisson, and sweep workloads. It measures SLO-oriented latency and token statistics, including time to first token, inter-token latency, and end-to-end behavior, then produces console, JSON, CSV, or HTML reports for deployment tuning, capacity planning, and regression tracking. The tool provides both a CLI and API, with multiprocessing, threading, asynchronous execution, and Hugging Face, file, synthetic, or custom data sources.