vLLM で Llama 4 をデプロイする際の最適なコンテキスト長を検証する

. .venv/bin/activate
uv pip install vllm

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --quantization fp8 --max-model-len 200000 --override-generation-config '{"max_new_tokens": 2048}'

INFO 04-07 00:16:33 [kv_cache_utils.py:578] GPU KV cache size: 538,512 tokens
INFO 04-07 00:16:33 [kv_cache_utils.py:581] Maximum concurrency for 200,000 tokens per request: 2.69x

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --quantization fp8 --max-model-len 500000 --override-generation-config '{"max_new_tokens": 2048}'

import openai
from transformers import AutoTokenizer
import argparse
import random
import string

def generate_random_string(length):
    return ''.join(random.choices(string.ascii_letters + string.digits, k=length))

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("model")
    parser.add_argument("text_length", type=int)
    parser.add_argument("--port", type=int, default=8000)
    args = parser.parse_args()
    
    tokenizer = AutoTokenizer.from_pretrained(args.model)
    prompt = generate_random_string(args.text_length)
    current_tokens = len(tokenizer.encode(prompt))
    
    while current_tokens < args.text_length:
        prompt += generate_random_string(args.text_length - current_tokens)
        current_tokens = len(tokenizer.encode(prompt))
    
    client = openai.OpenAI(api_key="none", base_url=f"http://localhost:{args.port}/v1")
    response = client.chat.completions.create(model=args.model, messages=[{"role": "user", "content": prompt}])
    print(response.choices[0].message.content)

if __name__ == "__main__":
    main()

$ python check.py meta-llama/Llama-4-Scout-17B-16E-Instruct 490000
The provided text appears to be a jumbled collection of letters and words without clear context or structure. It seems like a mix of random characters, possibly from a text file or a document that has been scrambled. 

However, if we try to decode or extract information from this text, we can see that there are some English words and phrases present:

- Various words like "The", "A", "Of", "And", "To", "Is", "In", "It", "For", "With", "As", etc.
- Some phrases like "Yt", "Xx", "Qc", "Kj", "Wb", "Yg", etc.

There are also some sentences or phrases that could be meaningful:

- "Yt Xx Kq"
- "Wb 0.5"
- "Is 7"
- "Kj Yg"

Without further context or information about what this text represents or what kind of decoding or analysis is required, it's challenging to provide a specific solution or interpretation. 

If you could provide more details about what you're trying to achieve or decode, I'll be happy to assist further.

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 8 --max-model-len 800000 --override-generation-config '{"max_new_tokens": 2048}'
INFO 04-07 01:53:08 [kv_cache_utils.py:578] GPU KV cache size: 1,312,400 tokens
INFO 04-07 01:53:08 [kv_cache_utils.py:581] Maximum concurrency for 800,000 tokens per request: 1.64x

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 8 --quantization fp8 --max-model-len 500000 --override-generation-config '{"max_new_tokens": 2048}'

INFO 04-07 01:17:41 [kv_cache_utils.py:578] GPU KV cache size: 1,621,792 tokens
INFO 04-07 01:17:41 [kv_cache_utils.py:581] Maximum concurrency for 500,000 tokens per request: 3.24x

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 --tensor-parallel-size 8 --max-model-len 300000 --override-generation-config '{"max_new_tokens": 2048}'
INFO 04-07 01:41:32 [kv_cache_utils.py:578] GPU KV cache size: 395,744 tokens
INFO 04-07 01:41:32 [kv_cache_utils.py:581] Maximum concurrency for 300,000 tokens per request: 1.32x