Multi-Agent Document Understanding

A supervisor agent routes documents to specialized sub-agents — layout parsing, entity extraction, cross-reference resolution, and fact validation. Each agent has access to different tools and retrieval sources, allowing the system to handle multi-modal documents (text + tables + figures) at scale.

Domain-Specific Code Generation with Llama 3

Base LLMs generate generic code. Fine-tuning Llama 3 on ~50K internal code samples using QLoRA + Unsloth to teach it proprietary SDKs, API patterns, and coding conventions. Early results: 73% pass rate on internal API tests vs 12% for the base model.

Open-Source RAG Evaluation Framework

Moving beyond vibes-based RAG evaluation. A CI-friendly framework measuring retrieval precision/recall, answer faithfulness, hallucination detection, and end-to-end correctness — works with any retriever and any LLM, tracks regressions over time.

Reinforcement Learning from Human Feedback

Studying the full alignment pipeline — reward modeling from pairwise preferences, PPO vs DPO tradeoffs, constitutional AI, and open problems around reward hacking and scalable oversight. DPO is simpler and more stable for most cases, but PPO gives finer control. The real bottleneck is always preference data quality.

Mixture of Experts Architectures

How sparse activation scales model capacity without proportional compute cost. Working through Switch Transformers, ST-MoE, Mixtral, and DeepSeek-MoE. Key insight: expert load balancing is the critical implementation challenge, and sparse models need fundamentally different serving infrastructure.

Rust for High-Performance Inference

Python is ML’s lingua franca, but inference serving is a systems problem. Building lightweight servers with Tokio async runtime, ONNX Runtime Rust bindings, and zero-copy tensor handling — targeting sub-millisecond overhead where Python’s GIL is the bottleneck.

Designing Machine Learning Systems — Chip Huyen

The definitive guide to production ML. Covers data engineering, feature stores, training pipelines, model serving, monitoring, and the organizational patterns that make ML teams effective. Required reading for anyone moving models from notebooks to production.

KV-Cache Optimization Papers

Efficient KV-cache management is the key bottleneck for long-context LLM inference. Studying PagedAttention (vLLM), multi-query attention, grouped-query attention, and sliding window approaches. Understanding these tradeoffs directly informs inference infrastructure decisions.

The Staff Engineer’s Path — Tanya Reilly

Technical leadership beyond writing code. Architecture decisions that compound, mentoring that scales, and organizational influence through technical judgment. Rethinking what “impact” means at senior levels.

Designing an agentic pipeline where specialized agents handle different document understanding tasks — layout parsing, entity extraction, cross-reference resolution, and fact validation.

Architecture

The system uses a supervisor agent that routes documents to specialized sub-agents based on document type and complexity. Each agent has access to different tools and retrieval sources.

Key Challenges

• Handling multi-modal documents (text + tables + figures)
• Maintaining context across long documents without losing precision
• Balancing latency vs. accuracy in the agent orchestration layer

A hands-on workshop for engineers moving from ChatGPT-style prompting to production-grade prompt engineering.

Topics Covered

• Structured outputs — JSON mode, function calling, and Pydantic validation
• Prompt versioning — treating prompts as code with version control and A/B testing
• Guardrails — input validation, output filtering, and hallucination detection
• Testing prompts — unit tests for LLM outputs using semantic similarity and rubric grading
• Cost optimization — prompt caching, token budgeting, and model routing

Key Takeaway

The gap between a demo prompt and a production prompt is the same as the gap between a script and a service — error handling, testing, monitoring, and versioning.

Documenting my learning journey through RLHF and alignment techniques. Covering the full pipeline from preference data collection to reward model training to policy optimization.

Topics Covered

• Reward modeling from pairwise human preferences
• PPO vs DPO — tradeoffs in practice
• Constitutional AI and self-supervised alignment
• Open questions: reward hacking, goodharting, scalable oversight

Key Takeaways So Far

DPO is simpler and more stable than PPO for most use cases, but PPO gives more fine-grained control when you need it. The real bottleneck is always preference data quality.

MoE is how the industry is scaling LLMs beyond dense transformer limits. Studying the key architectures and implementation details.

Reading List

• Switch Transformers (Fedus et al., 2021)
• ST-MoE (Zoph et al., 2022)
• Mixtral (Jiang et al., 2024)
• DeepSeek-MoE and fine-grained expert design

Key Insights

• Expert load balancing is the critical implementation challenge
• Token-choice vs expert-choice routing has major throughput implications
• Sparse models need different serving infrastructure than dense models

Contributing to DeepAgents, a research framework exploring how multiple LLM-powered agents can collaborate on complex tasks through hierarchical planning and shared memory.

Contributions

• Memory module — implemented persistent vector memory for cross-session agent context
• Tool registry — built a dynamic tool discovery and registration system
• Evaluation harness — added benchmarks for multi-agent task completion on SWE-bench

Research Questions

• How do agents decompose ambiguous tasks into sub-plans?
• When should agents delegate vs. execute directly?
• What memory architectures minimize hallucination in long-horizon tasks?

Learnings

The biggest insight: agent reliability scales better with structured state machines (like LangGraph) than with pure prompt-driven autonomy. Explicit control flow + LLM reasoning at decision nodes beats end-to-end agent prompting.

Base LLMs generate generic code. When your team has internal SDKs, specific API patterns, and coding conventions, the model needs domain knowledge it was never trained on.

Approach

• Curated training dataset from internal repos (~50K code samples with docstrings)
• QLoRA fine-tuning with Unsloth for memory-efficient training on a single A100
• Custom evaluation harness testing function correctness, API usage accuracy, and style compliance

Results So Far

• 73% pass rate on internal API usage tests (vs 12% for base Llama 3)
• 3x faster inference with vLLM serving + speculative decoding

Ran a production-realistic benchmark of the three most popular vector databases for RAG workloads — not just synthetic benchmarks, but real embedding distributions from enterprise documents.

Methodology

• Dataset — 2M embeddings from real enterprise documents (1536-dim, OpenAI ada-002)
• Queries — 10K real user queries from production RAG system
• Metrics — recall@10, p95 latency, cost/1M queries, operational burden

Results Summary

Database	Recall@10	p95 Latency	Cost/Month (2M vectors)
Qdrant (self-hosted)	98.2%	12ms	$150
Pinecone (managed)	97.8%	18ms	$420
Weaviate (self-hosted)	97.5%	22ms	$180

Recommendation

Qdrant wins on performance and cost for teams comfortable with self-hosting. Pinecone wins on operational simplicity. Weaviate’s multi-tenancy support is best for SaaS use cases.

Python is the lingua franca of ML, but inference serving is a systems problem. Rust gives you the performance of C++ with memory safety guarantees.

Learning Path

1. Rust fundamentals — ownership, borrowing, lifetimes
2. Async with Tokio — building concurrent HTTP/gRPC servers
3. ONNX Runtime Rust bindings — running models without Python overhead
4. Zero-copy tensor handling — minimizing allocations in the hot path

Goal

Build a lightweight inference server that can serve ONNX models with sub-millisecond overhead, suitable for real-time applications where Python’s GIL is the bottleneck.

Built an end-to-end agentic RAG system that goes beyond simple retrieval — the LangGraph agent reasons over multiple sources, self-corrects, and provides grounded citations.

Architecture

• Retriever Agent — hybrid search (dense + sparse) across Qdrant vector store
• Reasoner Agent — multi-step chain-of-thought with GPT-4 / Claude
• Validator Agent — fact-checks responses against source documents
• Orchestrator — LangGraph state machine managing agent handoffs and retries

Key Decisions

• Chose LangGraph over vanilla LangChain for explicit control over agent state transitions
• Qdrant over Pinecone for self-hosted deployment and cost control
• Streaming responses via Server-Sent Events for perceived latency improvement

Evaluation

Built a custom evaluation harness using RAGAS metrics — faithfulness, answer relevancy, and context precision tracked per-query in MLflow.

Most RAG pipelines are evaluated vibes-only. This framework brings structured, repeatable evaluation to retrieval-augmented generation.

What It Measures

• Retrieval quality — precision, recall, and MRR of retrieved chunks
• Answer faithfulness — does the answer actually follow from the retrieved context?
• Hallucination detection — claims in the answer that aren’t grounded in any source
• End-to-end correctness — compared against golden test sets

Design Principles

• Works with any retriever and any LLM
• CI-friendly — runs as part of your test suite
• Tracks metrics over time to catch regressions

A production postmortem on migrating from naive HuggingFace pipeline() inference to vLLM — and the 3x latency improvement that came with it.

The Problem

Our Llama 3 8B model was serving at 800ms p95 with HuggingFace’s default inference. At 500 concurrent users, GPU utilization was only 40% — most time was spent in memory allocation and batch scheduling.

The Fix

• vLLM’s PagedAttention — eliminated KV cache fragmentation, GPU memory utilization jumped to 90%+
• Continuous batching — no more waiting for the slowest request in a batch
• AWQ quantization — 4-bit quantized model with negligible quality loss, 2x throughput
• Tensor parallelism — split model across 2x A10G for headroom

Results

Metric	Before	After
p95 Latency	800ms	250ms
Throughput	15 req/s	48 req/s
GPU Utilization	40%	92%
Cost/1K requests	$0.12	$0.04

Takeaway

vLLM is production-ready. The continuous batching alone is worth the migration. If you’re still using transformers.pipeline() for serving, you’re leaving 3x performance on the table.

Fine-tuned open-weight LLMs for enterprise use cases — legal document summarization, code review, and customer support triage.

Approach

• Base Models — Llama 3 8B, Mistral 7B, Phi-3 Mini
• Method — QLoRA (4-bit quantization + Low-Rank Adaptation) via Unsloth
• Data — curated instruction datasets (5K-20K examples per domain)
• Evaluation — custom benchmarks + human preference ranking

Why Unsloth

Unsloth’s fused kernels and memory optimizations let us fine-tune 8B models on a single A100 in under 2 hours — compared to 5+ hours with vanilla PEFT. The 4-bit training path kept VRAM under 24GB.

Results

Model	Task	Accuracy	vs Base
Llama 3 8B	Legal Summarization	91.3%	+18.7%
Mistral 7B	Code Review	87.5%	+22.1%
Phi-3 Mini	Support Triage	94.0%	+15.3%

Deployment

Models exported to GGUF format for llama.cpp inference and served via vLLM behind a FastAPI gateway with streaming support.

Most RAG systems are evaluated by “vibes” — someone reads a few outputs and says “looks good.” This talk presents a systematic evaluation framework used in production.

Key Points

• RAGAS metrics — faithfulness, answer relevancy, context precision, context recall
• Human preference ranking — A/B testing with domain experts using blind evaluation
• Regression testing — golden dataset of 200+ question-answer pairs, run on every deployment
• Failure taxonomy — categorizing RAG failures (retrieval miss, context overflow, hallucination, wrong attribution)

The Framework

1. Build a golden dataset with verified answers and source citations
2. Run RAGAS metrics on every PR that touches the RAG pipeline
3. Flag regressions >2% on any metric for human review
4. Weekly manual evaluation of edge cases and new failure modes

Audience

AI Engineers building production RAG systems who want to move beyond “it seems to work” to measurable, tracked quality.

Contributed to HuggingFace Transformers, the most widely-used library for state-of-the-art NLP and LLM inference.

Contributions

• Flash Attention integration — added FlashAttention-2 support for Mistral and Phi model families
• Quantization improvements — optimized GPTQ and AWQ quantization paths for faster loading
• Training utilities — improved gradient checkpointing for multi-GPU fine-tuning workflows
• Documentation — rewrote fine-tuning guides for the PEFT + Transformers integration

Impact

These optimizations reduced inference latency by 30-40% for affected model families and are now part of the default pipeline for millions of daily API calls on HuggingFace Hub.

A deep-dive into using DVC (Data Version Control) for production ML workflows — going beyond basic file tracking to full pipeline orchestration.

What’s Covered

• Dataset versioning — track large datasets in S3/GCS without bloating Git
• Pipeline DAGs — define training pipelines as reproducible dvc.yaml stages
• Experiment tracking — dvc exp for hyperparameter sweeps without branch pollution
• CI integration — automated retraining triggers on data drift detection

Key Insight

The biggest win from DVC isn’t version control — it’s reproducibility. When a model degrades in production, you can dvc checkout the exact data + code + params that produced the last good model and diff against current state.

Code Examples

# dvc.yaml — define a training pipeline
stages:
  prepare:
    cmd: python src/prepare.py
    deps: [data/raw]
    outs: [data/processed]
  train:
    cmd: python src/train.py --lr ${lr} --epochs ${epochs}
    deps: [data/processed, src/train.py]
    params: [lr, epochs]
    outs: [models/latest]
    metrics: [metrics.json]

Who This Is For

ML engineers tired of model_v2_final_FINAL.pkl and data scientists who want git bisect for their training data.

A zero-config embedding cache that sits between your code and the embedding API. Every embedding is stored in a local SQLite database — identical inputs return cached results instantly.

Why

During RAG development, you re-embed the same documents hundreds of times while iterating on chunking strategies, metadata, and retrieval logic. Each call costs money and adds latency.

Usage

from embed_cache import CachedEmbeddings

embedder = CachedEmbeddings(model="text-embedding-3-small")
vectors = embedder.embed(["document chunk 1", "document chunk 2"])
# Second call: instant, free
vectors = embedder.embed(["document chunk 1", "document chunk 2"])

Features

• Drop-in replacement for OpenAI and Cohere embedding clients
• SQLite backend — no infrastructure needed
• Cache hit rate tracking and cost savings reporting
• TTL support for cache invalidation

Designed and deployed a centralized ML experiment tracking platform that brought reproducibility and visibility to the team’s ML workflows.

Architecture

• MLflow Tracking Server — ECS-hosted with PostgreSQL metadata store
• Artifact Store — S3 with lifecycle policies for cost management
• Model Registry — versioned model artifacts with stage transitions (Staging → Production)
• Auth — OIDC integration with company SSO

Key Features

• Auto-logging for PyTorch, sklearn, and HuggingFace training runs
• Custom MLflow plugins for GPU utilization and cost tracking
• Slack notifications on model promotion events
• Grafana dashboards for experiment trends and compute usage

Designed a structured program to lower the barrier to open source contribution, specifically targeting ML/AI repositories.

Program Structure

• Week 1 — Git workflows, finding good first issues, reading codebases
• Week 2 — Paired PR sessions on real HuggingFace/LangChain issues
• Week 3 — Writing tests, documentation, and review etiquette
• Week 4 — Solo contributions with async mentor support

Results

• 20+ participants, 15 shipped their first merged PR
• 6 continued contributing after the program ended
• 3 became regular contributors to HuggingFace ecosystem projects

Takeaway

The biggest blocker for first-time contributors isn’t skill — it’s confidence. Pairing someone with a mentor for their first PR changes the trajectory.

AWS certification focused on designing, implementing, and maintaining ML solutions on AWS infrastructure.

Topics Covered

• Data engineering and feature stores on AWS
• SageMaker training, tuning, and deployment
• Model monitoring and A/B testing patterns
• Cost optimization for ML workloads
• Security and governance for ML pipelines

Built a comprehensive LLM observability platform on top of Weights & Biases to track production model health.

What We Track

• Token economics — cost per request, budget burn rate, model routing efficiency
• Quality metrics — hallucination rate, RAGAS scores, user feedback signals
• Performance — latency p50/p95/p99, throughput, GPU utilization
• Drift detection — embedding drift on input distributions, output length anomalies

Architecture

• W&B Tables for structured experiment comparison
• Custom W&B Weave evaluations for automated quality checks
• Slack alerts on metric threshold breaches
• Weekly auto-generated reports for stakeholder visibility

Impact

Caught a 15% accuracy regression within 2 hours of a model update — previously would have gone unnoticed for days.

A candid retrospective on our six-month Kubernetes migration — from “let’s just containerize everything” to a working platform.

What Went Right

• Incremental migration — moved services one at a time, not big-bang
• Helm charts as contracts — standardized deployment configs across teams
• Canary deploys from day one — caught 3 production regressions before full rollout

What Broke

• DNS resolution — CoreDNS under load caused 5xx spikes for a week
• Resource limits — OOMKilled pods everywhere until we profiled actual memory usage
• Secrets management — migrating from env files to Vault took longer than the k8s migration itself

Key Numbers

• 50 services migrated in 6 months
• Deploy time: 25 min → 4 min (GitHub Actions + ArgoCD)
• Incident rate: -35% in the first quarter post-migration

Built the team’s first feature store to solve the train-serve skew problem — ensuring ML models see the same features in training and production.

Architecture

• Offline store — Parquet files on S3, computed via Airflow batch jobs
• Online store — Redis cluster with sub-10ms reads for real-time serving
• Feature registry — centralized catalog with lineage, ownership, and freshness SLAs
• SDK — Python client for consistent feature retrieval in notebooks, training, and serving

Key Design Decisions

• Chose Redis over DynamoDB for online store — 3x lower p99 latency at our scale
• Parquet over Delta Lake for offline — simpler, team already familiar, good enough for batch
• Built custom registry instead of adopting Feast — our schema requirements didn’t fit

Impact

Eliminated train-serve skew for all production models. Feature reuse across teams went from 0% to 60%, reducing duplicate computation by ~$2K/month.

Abstract

We propose a parameter-efficient fine-tuning approach that reduces compute requirements by 4x while maintaining 97% of full fine-tuning performance on domain-specific benchmarks. Our method combines adaptive rank selection with gradient-aware layer freezing.

Key Contributions

• Adaptive LoRA rank selection — dynamically adjusts rank per layer based on gradient magnitude during training
• Layer-wise freezing scheduler — progressively freezes converged layers to redirect compute to under-trained parameters
• Domain benchmark suite — released evaluation suite covering legal, medical, and financial domains

Takeaways

The conference provided excellent networking with teams from DeepMind, Meta FAIR, and several university labs working on similar efficiency problems. Led to two follow-up collaborations.

Talk: “Beyond Accuracy — Monitoring LLMs in Production”

Most teams ship LLMs with basic latency/error monitoring and call it done. This talk covered the monitoring patterns that actually catch problems before users complain.

Topics Covered

• Semantic drift detection — embedding-based monitoring to catch when input distributions shift
• Response quality scoring — lightweight LLM-as-judge pipelines that run async on sampled outputs
• Cost attribution — tracing token usage back to features and user segments
• Alert fatigue prevention — adaptive thresholds that account for natural usage pattern changes

Organizer Role

This was the 24th edition of our monthly meetup. As co-organizer I handle venue logistics, speaker sourcing, and post-event content publishing. We’ve grown from 20 to 85 regular attendees over two years.

A CLI tool born from frustration with slow CI builds in large monorepos.

Features

• Parallel execution — runs independent tasks concurrently with configurable concurrency limits
• Dependency graph — DAG-based task ordering, only runs what’s needed
• Smart caching — content-addressable cache skips tasks when inputs haven’t changed
• Simple config — YAML task definitions, no DSL to learn

Why Go

Single binary distribution, fast startup, excellent concurrency primitives. Users download one binary — no runtime dependencies.

Usage

# taskr.yaml
tasks:
  lint:
    cmd: eslint src/
    inputs: ["src/**/*.ts"]
  test:
    cmd: pytest tests/
    deps: [lint]
    inputs: ["src/**/*.py", "tests/**/*.py"]
  build:
    cmd: docker build -t app .
    deps: [test]

taskr run build  # runs lint → test → build, skips cached steps

The Challenge

Build an AI agent that solves a real developer workflow problem. Judged on innovation, technical execution, and practical usefulness.

Our Approach

We built ReviewBot — a multi-agent system that:

1. Analyzes PR diffs — understands code changes in context of the full repository
2. Identifies issues — security vulnerabilities, performance problems, style violations
3. Suggests fixes — generates corrected code with explanations
4. Writes tests — auto-generates unit tests covering the changed code paths

Architecture

The system uses a LangGraph state machine with four specialized agents:

• Analyst — parses diffs and builds dependency graphs
• Reviewer — identifies potential issues using RAG over best practices
• Fixer — generates code suggestions using few-shot examples
• Tester — creates test cases using mutation testing principles

What I Learned

Building under pressure forces you to make ruthless scope decisions. We cut three planned features on Day 1 evening to focus on making the core flow bulletproof — that focus is what won us the judges.

Hands-on Kubernetes certification focused on application-level concerns — pod design, configuration management, multi-container patterns, and observability.

Topics Covered

• Core concepts: Pods, Deployments, Services, ConfigMaps, Secrets
• Multi-container patterns: sidecar, ambassador, adapter
• Observability: probes, logging, monitoring
• Networking: NetworkPolicies, Ingress, DNS
• State persistence: PVs, PVCs, StorageClasses

Built as a self-hosted alternative to expensive analytics SaaS — runs on a single VPS for small teams.

Tech Stack

• Frontend — React, Recharts, TanStack Query
• Backend — FastAPI, WebSocket streaming, Celery for background jobs
• Database — PostgreSQL with TimescaleDB extension for time-series
• Deployment — Docker Compose, single docker compose up

Features

• Real-time WebSocket event feed
• Custom SQL queries with chart visualization
• Shared dashboards with role-based access
• Lightweight — runs on a $5/month VPS for small teams

Battle-tested Terraform modules for building secure, scalable AWS environments with minimal configuration drift.

Modules Included

• VPC & Networking — multi-AZ setups with public/private subnets
• Compute — ECS, Lambda, and EC2 templates
• Security — IAM policies, Security Groups, KMS encryption with least-privilege defaults
• Monitoring — CloudWatch alarms, SNS notifications, cost alerts

Design Philosophy

• Security-first: no public subnets by default, encryption everywhere
• Cost-aware: right-sizing recommendations, budget alerts built in
• Composable: modules work independently or together
• Documented: every variable has a description and sensible default

As AI Engineer at Acme AI, I work across the full LLM application lifecycle — from model fine-tuning to production deployment.

Key Achievements

• RAG platform — built enterprise RAG system serving 10K+ queries/day with 94% accuracy
• Fine-tuning pipeline — established QLoRA fine-tuning workflow that ships 3 domain models/quarter
• Inference optimization — migrated to vLLM, reducing serving costs by 65%
• Agent framework — designed LangGraph-based agent orchestration adopted across 3 product teams

Tech Stack

Python, PyTorch, LangGraph, LangChain, HuggingFace, Unsloth, vLLM, MLflow, Qdrant, FastAPI, AWS

Led the redesign of a monolithic API layer into a scalable gateway supporting 200+ downstream microservices.

Key Components

• Rate limiting — token bucket per client with Redis backend
• Auth delegation — JWT validation + RBAC at the edge, zero auth in downstream services
• Circuit breakers — per-service circuit breakers with configurable thresholds
• Request routing — path-based routing with canary and blue-green support

Results

Reduced inter-service auth overhead by 80%, improved p99 latency from 45ms to 15ms, and eliminated cascading failures during partial outages.