[Introducing Unsloth Studio: a new web UI for local AI\\
\\
🦥](https://unsloth.ai/docs/new/studio)

For the complete documentation index, see [llms.txt](https://unsloth.ai/docs/llms.txt). This page is also available as [Markdown](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms.md).

Tool calling is when a LLM is allowed to trigger specific functions (like “search my files,” “run a calculator,” or “call an API”) by emitting a structured request instead of guessing the answer in text. You use tool calls because they make outputs **more reliable and up-to-date**, and they let the model **take real actions** (query systems, validate facts, enforce schemas) rather than hallucinating.

In this tutorial, you will learn how to use local LLMs via Tool Calling with Mathematical, story, Python code and terminal function examples. Inference is done locally via llama.cpp, llama-server and OpenAI endpoints.

Tool calling is automatically setup when you use [Unsloth Studio](https://unsloth.ai/docs/new/studio/chat#auto-healing-tool-calling). Just select your model and toggle on or off tool-calling.

See right for an example of tool-calling being automatically applied for [Gemma 4](https://unsloth.ai/docs/models/gemma-4). Unsloth also has self-healing tool-calling ensuring you always have working tool calls.

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FstfdTMsoBMmsbQsgQ1Ma%252Flandscape%2520clip%2520gemma.gif%3Falt%3Dmedia%26token%3Deec5f2f7-b97a-4c1c-ad01-5a041c3e4013&width=768&dpr=3&quality=100&sign=e4b21b2d&sv=2)

Our guide should work for nearly [any model](https://unsloth.ai/docs/get-started/unsloth-model-catalog) including:

- [**Qwen3**-Coder-Next](https://unsloth.ai/docs/models/qwen3-coder-next), [Qwen3-Coder](https://unsloth.ai/docs/models/qwen3-coder-how-to-run-locally), and other **Qwen** models

- [**GLM**-4.7](https://unsloth.ai/docs/models/glm-4.7), 4.6, [GLM-4.7-Flash](https://unsloth.ai/docs/models/glm-4.7-flash) and [**Kimi K2.5**](https://unsloth.ai/docs/models/kimi-k2.5), [Kimi K2 Thinking](https://unsloth.ai/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally)

- [**DeepSeek**-V3.1](https://unsloth.ai/docs/models/tutorials/deepseek-v3.1-how-to-run-locally), DeepSeek-V3.2 and **MiniMax**

- [**gpt-oss**](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune)and [**NVIDIA Nemotron** 3 Nano](https://unsloth.ai/docs/models/tutorials/nemotron-3) and [**Devstral** 2](https://unsloth.ai/docs/models/tutorials/devstral-2)


[Qwen3-Coder-Next Tutorial](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms#qwen3-coder-next-tool-calling) [GLM-4.7-Flash Tutorial](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms#glm-4.7-flash--glm-4.7-calling)

### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#tool-calling-setup)    🔨Tool Calling Setup

Our first step is to Obtain the latest `llama.cpp` on [GitHub here](https://github.com/ggml-org/llama.cpp). You can follow the build instructions below as well. Change `-DGGML_CUDA=ON` to `-DGGML_CUDA=OFF` if you don't have a GPU or just want CPU inference. **For Apple Mac / Metal devices**, set `-DGGML_CUDA=OFF` then continue as usual - Metal support is on by default.

Copy

```
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

In a new terminal (if using tmux, use CTRL+B+D), we create some tools like adding 2 numbers, executing Python code, executing Linux functions and much more:

Copy

```
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def subtract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([\
        "A long time ago in a galaxy far far away...",\
        "There were 2 friends who loved sloths and code...",\
        "The world was ending because every sloth evolved to have superhuman intelligence...",\
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",\
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "subtract_number": subtract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [\
    {\
        "type": "function",\
        "function": {\
            "name": "add_number",\
            "description": "Add two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "multiply_number",\
            "description": "Multiply two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "subtract_number",\
            "description": "Subtract two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "write_a_story",\
            "description": "Writes a random story.",\
            "parameters": {\
                "type": "object",\
                "properties": {},\
                "required": [],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "terminal",\
            "description": "Perform operations from the terminal.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "command": {\
                        "type": "string",\
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",\
                    },\
                },\
                "required": ["command"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "python",\
            "description": "Call a Python interpreter with some Python code that will be ran.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "code": {\
                        "type": "string",\
                        "description": "The Python code to run",\
                    },\
                },\
                "required": ["code"],\
            },\
        },\
    },\
]
```

Show all 148 lines

We then use the below functions (copy and paste and execute) which will parse the function calls automatically and call the OpenAI endpoint for any model:

In this example we're using Devstral 2, when switching a model, ensure you use the correct sampling parameters. You can view all of them in our [guides here](https://unsloth.ai/docs/models/tutorials).

Copy

```
from openai import OpenAI
def unsloth_inference(
    messages,
    temperature = 0.7,
    top_p = 0.95,
    top_k = 40,
    min_p = 0.01,
    repetition_penalty = 1.0,
):
    messages = messages.copy()
    openai_client = OpenAI(
        base_url = "http://127.0.0.1:8001/v1",
        api_key = "sk-no-key-required",
    )
    model_name = next(iter(openai_client.models.list())).id
    print(f"Using model = {model_name}")
    has_tool_calls = True
    original_messages_len = len(messages)
    while has_tool_calls:
        print(f"Current messages = {messages}")
        response = openai_client.chat.completions.create(
            model = model_name,
            messages = messages,
            temperature = temperature,
            top_p = top_p,
            tools = tools if tools else None,
            tool_choice = "auto" if tools else None,
            extra_body = {"top_k": top_k, "min_p": min_p, "repetition_penalty" :repetition_penalty,}
        )
        tool_calls = response.choices[0].message.tool_calls or []
        content = response.choices[0].message.content or ""
        tool_calls_dict = [tc.to_dict() for tc in tool_calls] if tool_calls else tool_calls
        messages.append({"role": "assistant", "tool_calls": tool_calls_dict, "content": content,})
        for tool_call in tool_calls:
            fx, args, _id = tool_call.function.name, tool_call.function.arguments, tool_call.id
            out = MAP_FN[fx](**json.loads(args))
            messages.append({"role": "tool", "tool_call_id": _id, "name": fx, "content": str(out),})
        else:
            has_tool_calls = False
    return messages
```

Show all 40 lines

Now we'll showcase multiple methods of running tool-calling for many different use-cases below:

### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#writing-a-story)    Writing a story:

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "Could you write me a story ?"}],\
}]
unsloth_inference(messages, temperature = 0.15, top_p = 1.0, top_k = -1, min_p = 0.00)
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F04clxomfFQjhIKiS5FAY%252Fimage.png%3Falt%3Dmedia%26token%3D299279c9-cca6-48d6-ab74-523edef04160&width=768&dpr=3&quality=100&sign=50fc3493&sv=2)

### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#mathematical-operations)    Mathematical operations:

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "What is today's date plus 3 days?"}],\
}]
unsloth_inference(messages, temperature = 0.15, top_p = 1.0, top_k = -1, min_p = 0.00)
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fi15WEhfSIPuFeZUWjrG1%252Fimage.png%3Falt%3Dmedia%26token%3D74818449-637b-442a-b1d6-aa72b09fb6b5&width=768&dpr=3&quality=100&sign=27b38eef&sv=2)

### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#execute-generated-python-code)    Execute generated Python code

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "Create a Fibonacci function in Python and find fib(20)."}],\
}]
unsloth_inference(messages, temperature = 0.15, top_p = 1.0, top_k = -1, min_p = 0.00)
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FDsfcIrunKtsZ2RXRrjBo%252Fimage.png%3Falt%3Dmedia%26token%3D6aa2ab38-7def-4792-a2dc-c663329a41ff&width=768&dpr=3&quality=100&sign=f6404ac5&sv=2)

### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#execute-arbitrary-terminal-functions)    Execute arbitrary terminal functions

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "Write 'I'm a happy Sloth' to a file, then print it back to me."}],\
}]
messages = unsloth_inference(messages, temperature = 0.15, top_p = 1.0, top_k = -1, min_p = 0.00)
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FMhmXE34IBuR8H8SnuSMe%252Fimage%2520%283%29.png%3Falt%3Dmedia%26token%3D689114b7-5b50-41c9-8678-88f8f5da87a7&width=768&dpr=3&quality=100&sign=b4dc4527&sv=2)

## [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#qwen3-coder-next-tool-calling)    🌠 Qwen3-Coder-Next Tool Calling

In a new terminal, we create some tools like adding 2 numbers, executing Python code, executing Linux functions and much more:

Copy

```
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def subtract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([\
        "A long time ago in a galaxy far far away...",\
        "There were 2 friends who loved sloths and code...",\
        "The world was ending because every sloth evolved to have superhuman intelligence...",\
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",\
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "subtract_number": subtract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [\
    {\
        "type": "function",\
        "function": {\
            "name": "add_number",\
            "description": "Add two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "multiply_number",\
            "description": "Multiply two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "subtract_number",\
            "description": "Subtract two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "write_a_story",\
            "description": "Writes a random story.",\
            "parameters": {\
                "type": "object",\
                "properties": {},\
                "required": [],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "terminal",\
            "description": "Perform operations from the terminal.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "command": {\
                        "type": "string",\
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",\
                    },\
                },\
                "required": ["command"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "python",\
            "description": "Call a Python interpreter with some Python code that will be ran.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "code": {\
                        "type": "string",\
                        "description": "The Python code to run",\
                    },\
                },\
                "required": ["code"],\
            },\
        },\
    },\
]
```

Show all 148 lines

We then use the below functions which will parse the function calls automatically and call OpenAI endpoint for any LLM:

Copy

```
from openai import OpenAI
def unsloth_inference(
    messages,
    temperature = 1.0,
    top_p = 0.95,
    top_k = 40,
    min_p = 0.01,
    repetition_penalty = 1.0,
):
    messages = messages.copy()
    openai_client = OpenAI(
        base_url = "http://127.0.0.1:8001/v1",
        api_key = "sk-no-key-required",
    )
    model_name = next(iter(openai_client.models.list())).id
    print(f"Using model = {model_name}")
    has_tool_calls = True
    original_messages_len = len(messages)
    while has_tool_calls:
        print(f"Current messages = {messages}")
        response = openai_client.chat.completions.create(
            model = model_name,
            messages = messages,
            temperature = temperature,
            top_p = top_p,
            tools = tools if tools else None,
            tool_choice = "auto" if tools else None,
            extra_body = {"top_k": top_k, "min_p": min_p, "repetition_penalty" :repetition_penalty,}
        )
        tool_calls = response.choices[0].message.tool_calls or []
        content = response.choices[0].message.content or ""
        tool_calls_dict = [tc.to_dict() for tc in tool_calls] if tool_calls else tool_calls
        messages.append({"role": "assistant", "tool_calls": tool_calls_dict, "content": content,})
        for tool_call in tool_calls:
            fx, args, _id = tool_call.function.name, tool_call.function.arguments, tool_call.id
            out = MAP_FN[fx](**json.loads(args))
            messages.append({"role": "tool", "tool_call_id": _id, "name": fx, "content": str(out),})
        else:
            has_tool_calls = False
    return messages
```

Show all 40 lines

Now we'll showcase multiple methods of running tool-calling for many different use-cases below:

#### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#execute-generated-python-code-1)    Execute generated Python code

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "Create a Fibonacci function in Python and find fib(20)."}],\
}]
unsloth_inference(messages, temperature = 1.0, top_p = 0.95, top_k = 40, min_p = 0.00)
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F7fY3LSeNCjHXNjBwQkbI%252Fimage.png%3Falt%3Dmedia%26token%3D50eba62e-f8b2-424a-833b-be56696b4710&width=768&dpr=3&quality=100&sign=d097f8f2&sv=2)

#### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#execute-arbitrary-terminal-functions-1)    Execute arbitrary terminal functions

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "Write 'I'm a happy Sloth' to a file, then print it back to me."}],\
}]
messages = unsloth_inference(messages, temperature = 1.0, top_p = 1.0, top_k = 40, min_p = 0.00)
```

We confirm the file was created and it was!

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FabplwVbEMlsCEJTmxzSA%252Fimage.png%3Falt%3Dmedia%26token%3Deb27f30a-c91e-4aec-8fb0-f4a35921d3db&width=768&dpr=3&quality=100&sign=2536c4c3&sv=2)

## [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#glm-4.7-flash--glm-4.7-calling)    ⚡ GLM-4.7-Flash + GLM 4.7 Calling

We first download [GLM-4.7](https://unsloth.ai/docs/models/tutorials/glm-4.7) or [GLM-4.7-Flash](https://unsloth.ai/docs/models/tutorials/glm-4.7-flash) via some Python code, then launch it via llama-server in a separate terminal (like using tmux). In this example we download the large GLM-4.7 model:

Copy

```
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/GLM-4.7-GGUF",
    local_dir = "unsloth/GLM-4.7-GGUF",
    allow_patterns = ["*UD-Q2_K_XL*",], # For Q2_K_XL
)
```

If you ran it successfully, you should see:

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FqZCvBkQPi7GZj50pPHsJ%252Fimage.png%3Falt%3Dmedia%26token%3D2e97888a-fafe-4477-b99c-c7f3e2e316bf&width=768&dpr=3&quality=100&sign=ffcbbfb&sv=2)

Now launch it via llama-server in a new terminal. Use tmux if you want:

Copy

```
./llama.cpp/llama-server \
    --model unsloth/GLM-4.7-GGUF/UD-Q2_K_XL/GLM-4.7-UD-Q2_K_XL-00001-of-00003.gguf \
    --alias "unsloth/GLM-4.7" \
    --threads -1 \
    --fit on \
    --prio 3 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

And you will get:

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FMIqGMOXIMvl68jzDfyWy%252Fimage.png%3Falt%3Dmedia%26token%3Dc5739da8-b9eb-4a31-9878-3f0d5f51416e&width=768&dpr=3&quality=100&sign=ea574e6a&sv=2)

Now in a new terminal and executing Python code, reminder to run [Tool Calling Setup](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms#tool-calling-setup) We use GLM 4.7's optimal parameters of temperature = 0.7 and top\_p = 1.0

#### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#tool-call-for-mathematical-operations-for-glm-4.7)    Tool Call for mathematical operations for GLM 4.7

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "What is today's date plus 3 days?"}],\
}]
unsloth_inference(messages, temperature = 0.7, top_p = 1.0, top_k = -1, min_p = 0.00)
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FoFkZ20QOSGdzT4iz2SOB%252Fimage.png%3Falt%3Dmedia%26token%3De4ca30b0-dcec-4a26-b019-dd33f0600949&width=768&dpr=3&quality=100&sign=4da4ef65&sv=2)

#### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#tool-call-to-execute-generated-python-code-for-glm-4.7)    Tool Call to execute generated Python code for GLM 4.7

Copy

```
messages = [{\
    "role": "user",\
    "content": [{"type": "text", "text": "Create a Fibonacci function in Python and find fib(20)."}],\
}]
unsloth_inference(messages, temperature = 0.7, top_p = 1.0, top_k = -1, min_p = 0.00)
```

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FhS8sWtZwjwerElezCc2C%252Fimage.png%3Falt%3Dmedia%26token%3D39032ef8-386e-4837-8dd2-c552c80a3ee3&width=768&dpr=3&quality=100&sign=a13d9870&sv=2)

Copy

```
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def subtract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([\
        "A long time ago in a galaxy far far away...",\
        "There were 2 friends who loved sloths and code...",\
        "The world was ending because every sloth evolved to have superhuman intelligence...",\
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",\
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "subtract_number": subtract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [\
    {\
        "type": "function",\
        "function": {\
            "name": "add_number",\
            "description": "Add two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "multiply_number",\
            "description": "Multiply two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "subtract_number",\
            "description": "Subtract two numbers.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "a": {\
                        "type": "string",\
                        "description": "The first number.",\
                    },\
                    "b": {\
                        "type": "string",\
                        "description": "The second number.",\
                    },\
                },\
                "required": ["a", "b"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "write_a_story",\
            "description": "Writes a random story.",\
            "parameters": {\
                "type": "object",\
                "properties": {},\
                "required": [],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "terminal",\
            "description": "Perform operations from the terminal.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "command": {\
                        "type": "string",\
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",\
                    },\
                },\
                "required": ["command"],\
            },\
        },\
    },\
    {\
        "type": "function",\
        "function": {\
            "name": "python",\
            "description": "Call a Python interpreter with some Python code that will be ran.",\
            "parameters": {\
                "type": "object",\
                "properties": {\
                    "code": {\
                        "type": "string",\
                        "description": "The Python code to run",\
                    },\
                },\
                "required": ["code"],\
            },\
        },\
    },\
]
```

Show all 148 lines

### [Direct link to heading](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms\#devstral-2-tool-calling)    📙 Devstral 2 Tool Calling

We first download [Devstral 2](https://unsloth.ai/docs/models/tutorials/devstral-2) via some Python code, then launch it via llama-server in a separate terminal (like using tmux):

Copy

```
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF",
    local_dir = "unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*", "*mmproj-F16*"], # For Q4_K_XL
)
```

If you ran it successfully, you should see:

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252FC9M5eNGefCpi3bLe0Kbw%252Fimage.png%3Falt%3Dmedia%26token%3D727c22d5-368f-45ad-b698-ccb84e3bbbbf&width=768&dpr=3&quality=100&sign=21629b3f&sv=2)

Now launch it via llama-server in a new terminal. Use tmux if you want:

Copy

```
./llama.cpp/llama-server \
    --model unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/mmproj-F16.gguf \
    --alias "unsloth/Devstral-Small-2-24B-Instruct-2512" \
    --threads -1 \
    --fit on \
    --prio 3 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

You will see the below if it succeeded:

![](https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252Fo0VjjsMTBQCsfMNXcbYG%252Fimage.png%3Falt%3Dmedia%26token%3D3f0d9c51-1fd7-4d8d-9532-1a190f1b5830&width=768&dpr=3&quality=100&sign=edfd58b1&sv=2)

We then call the model with the following message and with Devstral's suggested parameters of temperature = 0.15 only. Reminder to run [Tool Calling Setup](https://unsloth.ai/docs/basics/tool-calling-guide-for-local-llms#tool-calling-setup)

[PreviousAider Polyglot Benchmarks](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs/unsloth-dynamic-ggufs-on-aider-polyglot) [NextVision Fine-tuning](https://unsloth.ai/docs/basics/vision-fine-tuning)

Last updated 1 month ago

Was this helpful?