vlm_cmdline_tool

Overview

mllm_llava is a built-in command-line tool for running VLM (Vision Language Model) inference on Genio Yocto platforms.

The tool reads a YAML configuration file that specifies the model architecture, DLA paths, and tokenizer paths, then runs inference against a plain-text prompt file and an input image.

Prerequisites

Before running inference:

Download the model package for your platform from Pre-built Model Packages in the Model Zoo.
Extract the package on your host PC:
```
unzip <ModelName>.zip
```

Package Contents

After extraction, each model package contains the following structure:

<ModelName>/
├── 2048c/
│   ├── <ModelVariant>_128t2048c_0.dla   # Prompt-mode DLA
│   └── <ModelVariant>_1t2048c_0.dla     # Generative-mode DLA
├── encoder/
│   └── <ModelVariant>_0.dla             # Image encoder DLA
├── scripts/
│   ├── 00_detect_os.bat
│   ├── 01_push-models.bat
│   ├── 02_push-tokenizer.bat
│   ├── 03_set-performance.bat
│   ├── 04_run-inference.bat
│   ├── config-android_np8-<ModelName>.yaml
│   ├── config-yocto_np8-<ModelName>.yaml
│   ├── demo.jpeg
│   ├── model_config.bat
│   ├── run_all_vl.bat
│   ├── sample_prompt.txt
│   └── set_performance.sh
└── tokenizer/
    ├── added_tokens.yaml
    ├── embedding_fp16.bin
    ├── merges.txt
    └── vocab.txt

The encoder/ DLA handles vision encoding of the input image. The 2048c/ folder contains DLAs for the language model: prompt mode and generative mode. The scripts/ folder contains the YAML configuration and sample inputs for quick testing.

Run Inference

Step 1: Push DLA Files

Push both the image encoder DLA and the language model DLAs to the device:

adb shell "mkdir -p /usr/share/llm/<ModelName>/encoder"
adb shell "mkdir -p /usr/share/llm/<ModelName>/2048c"
adb push <ModelName>/encoder/ /usr/share/llm/<ModelName>/encoder/
adb push <ModelName>/2048c/ /usr/share/llm/<ModelName>/2048c

Step 2: Push Tokenizer

Push the tokenizer files to the device:

adb push <ModelName>/tokenizer/ /usr/share/llm/<ModelName>/tokenizer/

Step 3: Run mllm_llava

Push the configuration file, sample prompt, and sample image, then run inference:

adb push <ModelName>/scripts/config-yocto_np8-<ModelName>.yaml /usr/share/llm/<ModelName>/
adb push <ModelName>/scripts/sample_prompt.txt /usr/share/llm/<ModelName>/
adb push <ModelName>/scripts/demo.jpeg /usr/share/llm/<ModelName>/
adb shell "cd /usr/share/llm; \
    mllm_llava /usr/share/llm/<ModelName>/config-yocto_np8-<ModelName>.yaml \
    --preformatter Qwen3VLNoInput \
    -i /usr/share/llm/<ModelName>/sample_prompt.txt \
    -im /usr/share/llm/<ModelName>/demo.jpeg \
    -m 128"

Parameters Reference

mllm_llava Parameters
Parameter	Required	Description
`<config_yaml>`	Yes	Path to the YAML configuration file for the model. Contains architecture settings, DLA paths, and tokenizer paths.
`--preformatter <name>`	Yes	Preformatter mode for input formatting. Use `Qwen3VLNoInput` for Qwen3-VL models.
`-i <prompt_file>`	Yes	Path to a plain-text file containing the input text prompt.
`-im <image_file>`	Yes	Path to the input image file (JPEG or PNG).
`-m <max_tokens>`	No	Maximum number of tokens to generate. Defaults to 512 if not specified.

Configuration File

The YAML configuration file specifies the model architecture and runtime paths. The scripts/ folder in each model package includes a ready-to-use configuration file. Update the paths under runtimeOptions if you place files in a different location on the device.

modelOptions:
  promptTokenBatchSize: 128   # Tokens processed per batch in prompt mode
  cacheSize: 2048             # KV cache size (context length in tokens)
  hiddenSize: 2048
  numHead: 16
  numLayer: 28
  maxTokenLength: 8192
  imagePatchSize: 16          # Vision transformer patch size
  imageWidth: 384             # Input image width (pixels)
  imageHeight: 384            # Input image height (pixels)

runtimeOptions:
  tokenizerPath:
    - /usr/share/llm/<ModelName>/tokenizer/merges.txt
    - /usr/share/llm/<ModelName>/tokenizer/vocab.txt
    - /usr/share/llm/<ModelName>/tokenizer/added_tokens.yaml
  tokenEmbPath: /usr/share/llm/<ModelName>/tokenizer/embedding_fp16.bin
  dlaPromptPaths:
    - /usr/share/llm/<ModelName>/2048c/<ModelVariant>_128t2048c_0.dla
  dlaGenPaths:
    - /usr/share/llm/<ModelName>/2048c/<ModelVariant>_1t2048c_0.dla
  clipPath: /usr/share/llm/<ModelName>/encoder/<ModelVariant>_0.dla
  imageTokenSize: 144         # Number of visual tokens per image

Expected Output

After running mllm_llava, the tool prints per-prompt performance statistics followed by a summary. Look for the [Average Performance among the given 1 prompts] section to verify inference results:

INFO: Begin LLaVA model init...
...
INFO: Loading CLIP DLA: <FileSource: /usr/share/llm/<ModelName>/encoder/<ModelVariant>_0.dla>
INFO: Done LLaVA init. (Time taken: 1.32799s)
INFO: Initialized HuggingFace tokenizer.
INFO: Vocab size: 151669
=========== Processing the 0-th input. ===========
INFO: Done CLIP dla inference in: 0.420855s
INFO: Done analyzing prompt (Total 184 tokens) in 1.16786s (285.639 tok/s)
INFO: Hot swapping to 1t model...
INFO: Done model hot swapping. (Time taken: 0.618979s)

Response [Max Length = 128]:
 This is a photograph of a woman and a dog sitting on a sandy beach at sunset.
The woman is on the right, wearing a plaid shirt and dark pants, and is smiling
as she looks at the dog. The dog is on the left, wearing a blue harness, and is
sitting and reaching out to shake hands with the woman. ...

[Latency]
      Prompt Mode: 285.639 tok/s
  Generative Mode: 9.38503 tok/s

[Average Performance among the given 1 prompts]
      Prompt Mode: 285.639 tok/s
  Generative Mode: 9.38503 tok/s

The Prompt Mode and Generative Mode values should be within 5% of the reference performance data listed on the platform performance pages in the Model Zoo.