llm_cmdline_tool

Overview

llm_cmdline_tool is a built-in command-line tool for running LLM inference on Genio Yocto platforms. It is pre-installed in the Yocto image at /usr/share/llm/llm_cmdline_tool.

The tool reads a YAML configuration file that specifies the model architecture, DLA paths, and tokenizer paths, then runs inference against a plain-text prompt file.

Prerequisites

Before running inference:

Download the model package for your platform from Pre-built Model Packages in the Model Zoo.
Extract the package on your host PC:
```
unzip <ModelName>.zip
```

Package Contents

After extraction, each model package contains the following structure:

<ModelName>/
├── 2048c/
│   ├── <ModelVariant>_128t2048c_0.dla   # Prompt-mode DLA
│   └── <ModelVariant>_1t2048c_0.dla     # Generative-mode DLA
├── scripts/
│   ├── prompts/
│   │   ├── sample_prompt-introduction.txt
│   │   ├── p01.txt
│   │   ├── ...
│   │   └── p10.txt
│   ├── 00_detect_os.bat
│   ├── 01_push-models.bat
│   ├── 02_push-tokenizer.bat
│   ├── 03_set-performance.bat
│   ├── 04_run-inference.bat
│   ├── config-android_np8-<ModelName>.yaml
│   ├── config-yocto_np8-<ModelName>.yaml
│   ├── model_config.bat
│   ├── run_all.bat
│   └── set_performance.sh
└── tokenizer/
    ├── added_tokens.yaml
    ├── embedding_int16.bin
    ├── merges.txt
    └── vocab.txt

The 2048c folder name indicates the KV cache size (2048 tokens); some models may use 1024c or other sizes. The scripts/ folder contains helper scripts for pushing files to the device and running inference via ADB.

Run Inference

Step 1: Push Model DLAs

Push the DLA files to the device:

adb shell "mkdir -p /usr/share/llm/<ModelName>"
adb push <ModelName>/2048c /usr/share/llm/<ModelName>/2048c/

Step 2: Push Tokenizer

Push the tokenizer files to the device:

adb push <ModelName>/tokenizer /usr/share/llm/<ModelName>/

Step 3: Run llm_cmdline_tool

Push the configuration file and a prompt, then run inference:

adb push <ModelName>/scripts/config-yocto_np8-<ModelName>.yaml /usr/share/llm/<ModelName>/
adb push <ModelName>/scripts/prompts/sample_prompt-introduction.txt /usr/share/llm/<ModelName>/prompts/
adb shell "cd /usr/share/llm; \
    llm_cmdline_tool /usr/share/llm/<ModelName>/config-yocto_np8-<ModelName>.yaml \
    -i /usr/share/llm/<ModelName>/prompts/sample_prompt-introduction.txt \
    -m 512"

Parameters Reference

llm_cmdline_tool Parameters
Parameter	Required	Description
`<config_yaml>`	Yes	Path to the YAML configuration file for the model. Contains architecture settings, DLA paths, and tokenizer paths.
`-i <prompt_file>`	Yes	Path to a plain-text file containing the input prompt.
`-m <max_tokens>`	No	Maximum number of tokens to generate. Defaults to 512 if not specified.

Configuration File

The YAML configuration file specifies the model architecture and runtime paths. The scripts/ folder in each model package includes a ready-to-use configuration file. Update the paths under runtimeOptions if you place files in a different location on the device.

modelOptions:
  promptTokenBatchSize: 128   # Tokens processed per batch in prompt mode
  cacheSize: 2048             # KV cache size (context length in tokens)
  hiddenSize: 2048
  numHead: 16
  numLayer: 28
  headDim: 128
  maxTokenLength: 32768

runtimeOptions:
  tokenizerPath:
    - /usr/share/llm/<ModelName>/tokenizer/vocab.txt
    - /usr/share/llm/<ModelName>/tokenizer/merges.txt
    - /usr/share/llm/<ModelName>/tokenizer/added_tokens.yaml
  tokenEmbPath: /usr/share/llm/<ModelName>/tokenizer/embedding_int16.bin
  dlaPromptPaths:
    - /usr/share/llm/<ModelName>/2048c/<ModelVariant>_128t2048c_0.dla
  dlaGenPaths:
    - /usr/share/llm/<ModelName>/2048c/<ModelVariant>_1t2048c_0.dla

Expected Output

After running llm_cmdline_tool, the tool prints per-prompt performance statistics followed by a summary. Look for the [Average Performance among the given 1 prompts] section to verify inference results:

>>>>>>>>>>> Current yaml config: qwen3-1.7b/config-yocto_np8-qwen3-1.7b.yaml <<<<<<<<<<<
Begin model init...
...
Done model init. (Time taken: 0.531881s)
Initialized HuggingFace tokenizer.
Vocab size: 151669
=========== Processing the 0-th input. ===========
Num prompt tokens: 12

[Prompt]
<|im_start|>user
tell me about you<|im_end|>
<|im_start|>assistant

Done analyzing prompt in 0.505058s (253.436 tok/s)
Hot swapping to 1t model...
Done model hot swapping. (Time taken: 0.38178s)

Response [Max Length = 128]:
<think>
Okay, the user asked me to tell them about me. I need to explain who I am as an AI assistant.
...
</think>
...

[Latency]
      Prompt Mode: 253.436 tok/s
  Generative Mode: 11.6558 tok/s

[Average Performance among the given 1 prompts]
      Prompt Mode: 253.436 tok/s
  Generative Mode: 11.6558 tok/s

The Prompt Mode and Generative Mode values should be within 5% of the reference performance data listed on the platform performance pages in the Model Zoo.