Please rotate your device

To continue exploring this website, please rotate your device to a vertical orientation

Qwen3 8B Reranker

Qwen/Qwen3 8B Reranker

A performant reranker model

embedding

Qwen-3-embeddings is prediction model that has two outputs: "no" and "yes", which indicate the match between query and a document.

This model is quantized to FP8 for deployment, which is supported by Nvidia's newest GPUs e.g. H100, H100_40GB, B200 or L4. Quantization is optional, but leads to higher efficiency.

The client code can be installed via pip.
https://github.com/basetenlabs/truss/tree/main/baseten-performance-client

Alternatively, you may use also your own client code.

Input

python
import os
from baseten_performance_client import (
    PerformanceClient, ClassificationResponse
)

api_key = os.environ["BASETEN_API_KEY"]
model_id = "xxxxxxx"
base_url = f"https://model-{model_id}.api.baseten.co/environments/production/sync"

client = PerformanceClient(base_url=base_url, api_key=api_key)

prefix = "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n"
suffix = "<|im_end|>\n"

def format_instruction(instruction, query, doc):
    if instruction is None:
        instruction = 'Given a web search query, retrieve relevant passages that answer the query'
    output = "{prefix}<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {doc}{suffix}"
    return output

texts_to_classify = [
    format_instruction(task=None, query="What is the capital of China?", doc="The capital of China is Beijing."),
    format_instruction(task=None, query="What is the capital of China?", doc="The capital of France is Paris.")
]

response: ClassificationResponse = client.classify(
    input=texts,
    model="my_model",
    truncate=True,
    batch_size=16,
    max_concurrent_requests=32,
)

Output

json
[
  {
    "score": 0.9861514,
    "label": "yes"
  },
  {
    "score": 0.01384861,
    "label": "no"
  }
]

Related models

See all models