Guides
Tokenizers v1: Encode, Decode and Scaling, Measured
Tokenizers v1 turns the encode–decode pipeline into a measurable component: one library, consistent normalization, and benchmarks that exp...
Sep 22, 202610 min
Articles tagged: Tokenization
6 articles
AI research
Benchmarking LLM Inference at Scale with AIPerf
NVIDIA's AIPerf gives teams a repeatable way to benchmark LLM inference at scale, capturing throughput, latency, and concurrency under rea...
Sep 21, 202611 min
Local models
Hugging Face and Cerebras bring Gemma 4 to real-time voice AI
Hugging Face and Cerebras collaborate to run Gemma 4 models for real-time voice AI on local hardware, enabling low-latency speech processi...
Jul 2, 20267 min
AI tools
Prompt Engineering Fails Quietly
Prompt regression causes AI outputs to degrade over time without warning. Learn why it happens, how to detect it, and practical strategies...
Jul 1, 20268 min
Guides
LLMs Inside the Product: A Practical Field Guide
A hands-on guide to integrating large language models into products, covering architecture patterns, prompt engineering, cost optimization...
Jun 17, 20266 min
AI tools
I Built a C++ Backend So My GPU Would Stop Eating Air
A clear and practical article about artificial intelligence for a professional audience.
Jun 6, 20268 min