GLM-4 Technical Report

The Problem

By mid-2024, strong LLMs were commonplace, but they were often isolated text generators. To solve real-world problems, a model needed to search the web for real-time information, execute Python code for math/data analysis, and interact with specialized APIs (like image generation or calendar systems). Integrating these tools often required fragile, external software frameworks (like LangChain) that had to constantly prompt the model on how to format its output.

The Idea

The team at Zhipu AI developed GLM-4 with a native, built-in "All Tools" framework. Instead of treating tool use as an afterthought, GLM-4 was trained extensively to natively understand, plan, and execute tool calls in a unified environment. Furthermore, it aggressively expanded context lengths up to 128k (and later 1M) tokens to allow for document-heavy analytical tasks.

How It Works

The All Tools Framework During the RLHF and alignment phases, GLM-4 was heavily fine-tuned on complex tool-use trajectories. It supports three native tool environments:

1. Web Browser: The model can issue search queries, retrieve page contents, and synthesize answers with citations.
2. Code Interpreter: The model can write Python code, execute it in a sandboxed environment, read the outputs (including generated charts and dataframes), and debug errors iteratively.
3. Function Calling (APIs): The model can correctly format JSON payloads to trigger external APIs based on user requests.

Long Context Processing To handle massive contexts, GLM-4 utilized RoPE (Rotary Positional Embedding) scaling techniques and was trained on deliberately long, complex documents to avoid the "lost in the middle" phenomenon (where models forget information in the center of a long prompt).

Multimodal Variants (GLM-4V) The architecture seamlessly integrates a vision encoder, allowing the model to perform OCR on complex charts, read documents, and visually ground its tool calls (e.g., searching the web for an item seen in an uploaded image).

Why It Mattered

GLM-4 proved that agentic capabilities (tool use and long-context synthesis) needed to be baked into the foundational training recipe, rather than patched on top. It stood as a highly capable bilingual (English/Chinese) model that offered an "out-of-the-box" autonomous agent experience.

What Came After

The success of the "All Tools" integration accelerated the industry trend toward treating LLMs as operating systems (LLM-as-OS), where the model acts as the central router executing tasks across various software tools seamlessly.