Llama 4 をオンプレミス環境で動かしてみた

pillow
transformers
accelerate
huggingface_hub[hf_xet]
torch
torchvision

uv venv -p 3.11
. .venv/bin/activate
uv pip install -r requirements.txt

Installed 42 packages in 3.06s
 + accelerate==1.6.0
 + certifi==2025.1.31
 + charset-normalizer==3.4.1
 + filelock==3.18.0
 + fsspec==2025.3.2
 + hf-xet==1.0.0
 + huggingface-hub==0.30.1
 + idna==3.10
 + jinja2==3.1.6
 + markupsafe==3.0.2
 + mpmath==1.3.0
 + networkx==3.4.2
 + numpy==2.2.4
 + nvidia-cublas-cu12==12.4.5.8
 + nvidia-cuda-cupti-cu12==12.4.127
 + nvidia-cuda-nvrtc-cu12==12.4.127
 + nvidia-cuda-runtime-cu12==12.4.127
 + nvidia-cudnn-cu12==9.1.0.70
 + nvidia-cufft-cu12==11.2.1.3
 + nvidia-curand-cu12==10.3.5.147
 + nvidia-cusolver-cu12==11.6.1.9
 + nvidia-cusparse-cu12==12.3.1.170
 + nvidia-cusparselt-cu12==0.6.2
 + nvidia-nccl-cu12==2.21.5
 + nvidia-nvjitlink-cu12==12.4.127
 + nvidia-nvtx-cu12==12.4.127
 + packaging==24.2
 + pillow==11.1.0
 + psutil==7.0.0
 + pyyaml==6.0.2
 + regex==2024.11.6
 + requests==2.32.3
 + safetensors==0.5.3
 + sympy==1.13.1
 + tokenizers==0.21.1
 + torch==2.6.0
 + torchvision==0.21.0
 + tqdm==4.67.1
 + transformers==4.51.0
 + triton==3.2.0
 + typing-extensions==4.13.1
 + urllib3==2.3.0

huggingface-cli login

huggingface-cli download meta-llama/Llama-4-Scout-17B-16E-Instruct

from transformers import AutoProcessor, Llama4ForConditionalGeneration
import torch

# model_id = "meta-llama/Llama-4-Maverick-17B-128E-Instruct"
model_id = "meta-llama/Llama-4-Scout-17B-16E-Instruct"

processor = AutoProcessor.from_pretrained(model_id)
model = Llama4ForConditionalGeneration.from_pretrained(
    model_id,
    # attn_implementation="flex_attention",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

url1 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
url2 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/datasets/cat_style_layout.png"
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": url1},
            {"type": "image", "url": url2},
            {"type": "text", "text": "Can you describe how these two images are similar, and how they differ?"},
        ]
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
)

response = processor.batch_decode(outputs[:, inputs["input_ids"].shape[-1]:])[0]
print(response)
print(outputs[0])

CUDA_VISIBLE_DEVICES=0,1,2,3 python llama4.py

The two images depict anthropomorphic animals, a rabbit and a cat, dressed in similar attire. The rabbit is standing on a dirt path with flowers and houses in the background, while the cat is sitting on a stone path surrounded by greenery.

**Similarities:**

* Both characters are wearing blue jackets, brown vests, and bow ties.
* They have similar facial expressions, with large eyes and endearing features.
* The backgrounds of both images feature natural settings, including paths, flowers, and houses.

**Differences:**

* **Species:** The most obvious difference is that one character is a rabbit and the other is a cat.
* **Posture:** The rabbit is standing upright on two legs, while the cat is sitting on all fours.
* **Background:** Although both backgrounds feature natural settings, they differ in terms of composition. The rabbit's background includes a house and mountains, whereas the cat's background features a more intimate garden scene with a wooden fence.
* **Coloring:** The rabbit has lighter-colored fur than the cat, which has darker orange tones.

Overall, while the two images share some similarities in terms of character design and setting, they also exhibit distinct differences in terms of species, posture, and background composition.

CUDA_VISIBLE_DEVICES=0,1,2,3 nsys profile python llama4.py