https://ai-blog.noorshomelab.dev/tags/tokenization/Recent content in Tokenization on AI VOIDHugoenTue, 17 Mar 2026 00:00:00 +0000https://ai-blog.noorshomelab.dev/mermaid-lint-guide/chapter-2-designing-the-lexer/Tue, 17 Mar 2026 00:00:00 +0000https://ai-blog.noorshomelab.dev/mermaid-lint-guide/chapter-2-designing-the-lexer/<h2 id="chapter-2-designing-the-lexer-tokenization-of-mermaid-syntax">Chapter 2: Designing the Lexer: Tokenization of Mermaid Syntax</h2> <p>Welcome to Chapter 2 of our journey to build a robust Mermaid code analyzer and fixer in Rust! In the previous chapter, we laid the foundational project structure and set up our development environment. With the groundwork complete, we&rsquo;re now ready to dive into the core components of our compiler-like tool. This chapter focuses on the very first stage of any compiler pipeline: the <strong>Lexer</strong>.</p>https://ai-blog.noorshomelab.dev/tunix-mastery-2026/05-data-preparation/Fri, 30 Jan 2026 00:00:00 +0000https://ai-blog.noorshomelab.dev/tunix-mastery-2026/05-data-preparation/<h2 id="chapter-5-data-preparation-and-loading-for-tunix">Chapter 5: Data Preparation and Loading for Tunix</h2> <p>Welcome back, future LLM master! In the previous chapters, we laid the groundwork by understanding Tunix&rsquo;s architecture and setting up our development environment. Now, it&rsquo;s time to talk about the fuel that powers any Large Language Model: data!</p> <p>This chapter is all about getting your data ready for Tunix. We&rsquo;ll dive deep into the crucial steps of preparing your text-based datasets, understanding how to tokenize them, and setting up efficient data loading pipelines that play nicely with JAX and Tunix. Think of this as preparing a delicious meal – you need to carefully select, clean, and chop your ingredients before you can even think about cooking!</p>