LLMs-Journey

The LLMs-Journey repository covers topics including:

• Large Language Models (LLMs)
• Agentic Systems and Workflows
• Fine-Tuning (e.g., LoRA, QLoRa)
• Retrieval-Augmented Generation (RAG)
• Vision-Language Models (VLMs)
• Complementary Resources and Research

Blog Posts

Fine-Tuning:

• Fine-Tuning LLMs with LoRA and MLX-LM

Agents:

• Fine-Tuning LLMs for Function-Calling with MLX-LM
• A Job Postings Tool: A Guide to MLX-LM Server and Tool Use with the OpenAI Client
• Running Local Hugging Face Models with MLX-LM and the Agno Agentic Framework
• Agno + MLX: Human-in-the-Loop User Control Flow for a Job-Search Agent
• Agno + MLX + Agent Lightning: Automatic Prompt Optimization for a Job-Search Agent

VLMs:

• Qwen2.5-VL with MLX-VLM
• Image Segmentation with PaliGemma 2 Mix and MLX
• Image Segmentation with PaliGemma 2 mix, Transformers, Docker, FastAPI, and GitHub Actions
• Nanonets OCR: A Small Gem for Handwritten Notes

On-Device AI:

• Chat with Qwen3 on your iPhone: A Step-by-Step Guide

Misc:

• Thinking Backwards: The “Reversal Blessing” in LLM Multiple-Choice Reasoning

What to read?

Books:

• AI Engineering by Chip Huyen
• Hands-On Large Language Models by Jay Alammar, Maarten Grootendorst, GitHub
• THE HUNDRED-PAGE LANGUAGE MODELS BOOK by Andriy Burkov, GitHub
• Foundations of Large Language Models
• How to Scale Your Model
• The Ultra-Scale Playbook: Training LLMs on GPU Clusters

Agents:

• Hugging Face Agents Course
• ai-agents-for-beginners (Microsoft)
• Agents (Google’s whitepaper)
• Agency Is Frame-Dependent
• Agent Lightning: Train ANY AI Agents with Reinforcement Learning

Promptning:

• Prompt Engineering (Google’s whitepaper)
• GPT-4.1 Prompting Guide

Fine-Tuning:

• 🤗 PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware
• LoRA: Low-Rank Adaptation of Large Language Models
• QLORA: Efficient Finetuning of Quantized LLMs

RAG related:

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Chain-of-Retrieval Augmented Generation
• Introducing Contextual Retrieval

Evaluation:

• A Survey on LLM-as-a-Judge
• ARC Prize 2024: Technical Report
• FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
• MATH-Perturb: Benchmarking LLMs’ Math Reasoning Abilities against Hard Perturbations

Datasets:

• NuminaMath 1.5

Models:

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding and Round and Round We Go! What makes Rotary Positional Encodings useful?
• Qwen2.5-Math and paper
• Kimi K2: Open Agentic Intelligence
• LFM2 Technical Report

Vision:

• Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution
• Qwen2.5-VL and paper
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
• Magma: A Foundation Model for Multimodal AI Agents
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• PaliGemma 2: A Family of Versatile VLMs for Transfer
• Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks
• Kimi-VL Technical Report
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• FastVLM: Efficient Vision Encoding for Vision Language Models
• DINOv3
• Qwen3-VL Technical Report
• SAM 3: Segment Anything with Concepts
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
• VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
• DeepSeek-OCR: Contexts Optical Compression
• olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models
• olmOCR 2: Unit Test Rewards for Document OCR
• dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model
• Can Large Vision Language Models Read Maps Like a Human?
• Falcon Perception (Falcon OCR), GitHub
• GLM-OCR Technical Report, GitHub

Chain-of-Thought

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Self-Consistency Improves Chain of Thought Reasoning in Language Models

Visualisation-of-Thought:

• Imagine while Reasoning in Space: Multimodal Visualization-of-Thought

Test-Time Scaling

• s1: Simple test-time scaling and paper

Test-Time Compute

• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

AlphaGeometry:

• Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2

Apple:

• Machine Learning Research at Apple
• Distillation Scaling Laws
• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
• Reversal Blessing: Thinking Backward May Outpace Thinking Forward in Multi-choice Questions
• FastVLM: Efficient Vision Encoding for Vision Language Models
• Embarrassingly Simple Self-Distillation Improves Code Generation, GitHub

Misc:

• Scaling Pre-training to One Hundred Billion Data for Vision Language Models
• Competitive Programming with Large Reasoning Models
• MoBA: Mixture of Block Attention for Long-Context LLMs
• Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model, repo
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• Deepseek Papers
• InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
• Leveraging the true depth of LLMs
• NOLIMA: Long-Context Evaluation Beyond Literal Matching
• Memory Layers at Scale
• Towards an AI co-scientist
• On the consistent reasoning paradox of intelligence and optimal trust in AI: The power of ‘I don’t know’
• LIMO: Less is More for Reasoning
• How new data permeates LLM knowledge and how to dilute it
• Attention Is All You Need
• mHC: Manifold-Constrained Hyper-Connections
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization, GitHub
• Attention Residuals, GitHub
• TurboQuant, TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate, QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead, PolarQuant: Quantizing KV Caches with Polar Transformation

Resources

MLX:

• mlx-examples
• ml-explore
• mlx-vlm

LangChain:

• Build a Retrieval Augmented Generation (RAG) App: Part 1

Vector Databases:

• Chroma
• Pinecone
• Weaviate

HuggingFace:

• MLX Community
• Using MLX at Hugging Face

Leonie Notebooks:

• Fine-tuning Gemma 2 JPN for Yomigana with LoRA
• Advanced RAG with Gemma, Weaviate, and LlamaIndex
• RAG with Gemma on HF 🤗 and Weaviate in DSPy

GitHub Repos:

• rag-cookbooks
• Hands-On-Large-Language-Models
• Multimodal-RAG-Implementation
• Data_Science_Learning_Material
• RAG-To-Know
• Docling

Blogs/Posts:

• The Illustrated DeepSeek-R1
• Transformers from Scratch
• Chain of Thought
• Deekshith Dade

Frameworks:

• Agno, and github page
• Agent Lightning