How Hugging Face Jobs, Buckets, and Inference Endpoints Support Papers with Code Search

Papers with Code was revived to make open AI research easier to discover and use. The service helps people locate papers and their associated artifacts, compare state-of-the-art results across AI domains, share research, and build on existing work.

That mission requires a search engine that can handle more than exact text matching. Users may search for an exact title or arXiv identifier, but they may also enter a conceptual request such as “small language models for code generation,” refer to “the original BERT paper,” submit an incomplete title, or make spelling mistakes. The system must return useful results quickly, including when its embedding service is cold or temporarily unavailable.

![Papers with Code search results for the query 'DINO']

Search results on Papers with Code for the query DINO.

Hybrid search architecture

Papers with Code uses hybrid search, combining keyword retrieval with vector retrieval. Keyword search is effective for exact mentions, identifiers, and rare terms. Vector search finds semantically related content even when the query and document use different wording. A reranker, also known as a cross-encoder, can improve results further, although it adds latency and compute overhead.

The system uses PostgreSQL as its primary database. PostgreSQL full-text search provides the lexical retrieval baseline, while pgvector adds dense-vector retrieval. Reciprocal rank fusion, or RRF, combines the results from both branches.

Three Hugging Face services support the embedding workflow:

Hugging Face Jobs provides burstable GPU compute for embedding the paper corpus.
Hugging Face Storage Buckets provide durable storage between the database, experiments, and Jobs.
Hugging Face Inference Endpoints serve low-latency embeddings for live queries and incremental updates.

The system currently maintains embeddings for more than 110,000 papers sourced from arXiv and Daily Papers.

Architecture overview

The search system is divided into offline corpus processing and online retrieval:

![Architecture diagram of the offline corpus build and online hybrid search pipeline]

Architecture of the offline corpus build and online hybrid search pipeline.

The expensive, throughput-oriented work runs in Jobs. Durable input and output artifacts are stored in a Bucket. Only query embedding runs on the request path, through a protected Inference Endpoint. If the endpoint is cold, busy, or unhealthy, the service immediately falls back to full-text retrieval.

A strict embedding contract

Embedding pipelines can fail when a model revision changes, query and document prompts are mixed up, vectors are truncated inconsistently, or a paper's abstract changes after its vector was generated.

To avoid these problems, the embedding format is treated as a versioned API. Each paper is encoded as:

normalized title + "\n\n" + normalized abstract

Each vector-generation run records:

the model repository and exact revision;
the output dimension;
the input-format version;
whether the input is a query or document;
the normalization method; and
a content hash for the source title and abstract.

Production generation uses Qwen/Qwen3-Embedding-0.6B, pinned to an exact revision. The resulting vectors are 256-dimensional and L2-normalized. Model selection used the MTEB leaderboard, a benchmark for comparing embedding models.

Qwen3 embedding models provide two relevant capabilities:

Dynamic embedding size: Matryoshka Representation Learning, or MRL, allows the embedding size to be adjusted to balance quality, speed, and storage. The system uses 256 dimensions to keep search fast.
Instruction prompts: the model supports a document prompt for paper embeddings and a query prompt for live user searches.

The same contract is applied from export, through GPU inference, into PostgreSQL and online retrieval.

Jobs convert database snapshots into vector corpora

Embedding the full corpus is a batch workload. It needs a GPU for a limited period, benefits from high throughput, and should not consume compute resources between runs. Hugging Face Jobs is designed for this pattern. A Job can specify a command, a hardware flavor, and optionally a Docker image. It can also run uv scripts with dependencies declared inline.

The corpus build begins by exporting the latest version of every paper from a repeatable-read PostgreSQL snapshot. The exporter streams rows instead of loading the entire catalog into memory, writes bounded JSONL shards, and creates a manifest with row counts and SHA-256 checksums.

The immutable run directory is synchronized to a private Storage Bucket. The Bucket is then mounted with hf-mount into an l4x1 Job, which uses an NVIDIA L4 GPU with 24 GB of VRAM:

hf jobs uv run \
  --flavor l4x1 \
  --timeout 6h \
  --volume hf://buckets/OWNER/pwc-paper-embeddings:/bucket \
  embed_papers_job.py \
  --input /bucket/runs/RUN_ID/input \
  --output /bucket/runs/RUN_ID/output \
  --model Qwen/Qwen3-Embedding-0.6B \
  --revision MODEL_REVISION \
  --dimensions 256 \
  --allow-matryoshka

The worker performs the following steps:

Verifies the input manifest and every shard checksum.
Loads the pinned model revision.
Sorts texts by length to reduce padding.
Calls encode_document in batches, as described in the model card.
Reduces the batch size automatically if the GPU runs out of memory.
Truncates the Matryoshka representation to 256 dimensions and normalizes it.
Writes float16 Parquet shards atomically.
Records throughput, package versions, hardware, peak VRAM, row counts, and output checksums.

Each completed shard receives its own marker. If a Job restarts, it can skip verified work and resume from the remaining shards rather than overwriting existing embeddings.

In a 5,000-paper pilot, the Qwen Job encoded about 75 papers per second at 1024 dimensions on an L4 GPU. The same pass could be materialized deterministically at 512 and 256 dimensions, allowing storage and retrieval trade-offs to be compared without additional inference.

Buckets provide the handoff between systems

Storage Buckets are mutable, S3-like object storage on the Hub, designed for AI workloads. They support hf://buckets/... paths and can be mounted read-write in Jobs without a separate storage integration.

In this architecture, the Bucket is the boundary between systems with different lifecycles:

The production database exports source records.
Ephemeral Jobs consume those records and produce vectors.
An importer validates the results before updating the search index.

Artifacts are organized under immutable run prefixes:

runs/<run-id>/
├── input/
│   ├── manifest.json
│   └── papers-*.jsonl
└── output/
    ├── manifest.json
    ├── embeddings-*.parquet
    └── embeddings-*.complete.json

The Buckets themselves are mutable, so immutability is enforced by application rules. A run ID is never overwritten, and every artifact is covered by a manifest and checksum.

This structure provides:

Reproducibility: each database generation can be traced to a specific corpus snapshot, model revision, and artifact set.
Safe retries: Jobs can resume from completed shards in the same run prefix.
Lower-cost experiments: multiple models or dimensions can reuse one verified input snapshot.
Controlled rollout: importing a generation does not activate it. The generation is validated and indexed first.
Simple rollback: the previous generation and its artifacts remain available until the new generation is stable.

Before vectors are loaded into PostgreSQL, the importer rechecks schemas, checksums, dimensions, normalization, unique paper IDs, and current content hashes. A separate HNSW index is then built for the new generation. It is marked active only after every eligible current paper is covered.

Inference Endpoints serve live query embeddings

Batch embeddings handle the document side of retrieval. User queries must still be embedded at request time with the same model contract.

The pinned model is deployed as an authenticated Inference Endpoint backed by Text Embeddings Inference. The endpoint accepts query text and returns a normalized 256-dimensional vector using the model's query prompt. vLLM or SGLang could also be used for this role.

![Hugging Face Inference Endpoint overview for the Papers with Code query embedding model]

Hugging Face Inference Endpoint for the Papers with Code query embedding model.

The API searches the active pgvector generation using cosine distance:

SELECT paper_id,
       embedding <=> CAST(:query_vector AS halfvec(256)) AS distance
FROM paper_embeddings
WHERE generation_id = :active_generation
ORDER BY embedding <=> CAST(:query_vector AS halfvec(256))
LIMIT 50;

The HNSW index keeps the lookup fast. In the 5,000-paper pilot, the 256-dimensional Qwen index achieved 0.9955 Recall@20 against exact search, with 1.31 ms p50 and 2.21 ms p95 HNSW lookup latency. The table and index used about 27% of the storage required by the 1024-dimensional version while retaining essentially the same approximate-nearest-neighbor recall in that test.

The Endpoint is configured with a maximum of one replica and can scale to zero when idle. Cold starts are therefore treated as part of normal application behavior. The query client uses:

a one-second production timeout;
a non-blocking concurrency limit;
validation of response dimensions, finiteness, and norm;
a short cache keyed by the query and embedding generation;
a circuit breaker after repeated failures; and
logs containing only a normalized query fingerprint, not raw query text.

If the endpoint is scaling up, times out, returns a malformed vector, or has no available concurrency, the semantic branch is skipped immediately. The user still receives lexical results instead of waiting for an unavailable dependency.

![Hugging Face Inference Endpoint analytics dashboard showing request volume, errors, latency, and replica state]

Inference Endpoint analytics showing request volume, errors, latency, and replica state.

Combining lexical and semantic retrieval

For each query, the lexical branch retrieves up to 50 candidates using weighted PostgreSQL full-text search. The semantic branch retrieves up to 50 candidates from pgvector.

Their ranks are combined with weighted reciprocal rank fusion:

score(d) = Σr∈{lexical, semantic} wr / (k + rankr(d))

RRF combines ranks rather than raw scores, which avoids scale differences between the two retrieval systems. A document ranked highly by both branches receives a stronger combined position. The current configuration uses equal branch weights and k = 60, the rank-constant hyperparameter.

Dense retrieval improves recall for conceptual queries, while full-text search remains effective for exact terminology, identifiers, and rare names. Deterministic identity behavior is preserved on top of the fused ranking:

exact titles and arXiv IDs remain at the top;
the method taxonomy recognizes navigational searches such as “the original BERT paper”;
incomplete titles and bounded spelling mistakes use conservative trigram candidates; and
ambiguous fuzzy matches are rejected instead of being forced into a result.

Keyword search remains a useful, inexpensive baseline. Semantic or hybrid retrieval can be added when it provides a measurable improvement in retrieval quality. A reranker such as Qwen3-Reranker could be placed after keyword, semantic, or hybrid retrieval.

One Endpoint supports two update paths

The initial corpus is embedded with Jobs, but Papers with Code changes continuously. New papers arrive, abstracts are corrected, and new arXiv versions become current.

Starting a GPU Job for a small number of changed rows would introduce unnecessary startup and orchestration overhead. Instead, an hourly incremental process identifies missing or content-changed papers and sends a bounded delta to the same TEI Endpoint, this time with the document prompt.

Each run processes no more than 500 papers in batches of 16. Before writing an embedding, the source row is locked and its content hash is checked again. If the paper changed during inference, the vector is discarded and processed during a later run.

The resulting division of labor is:

Jobs: full rebuilds, new model generations, and large backfills.
Inference Endpoints: interactive query embeddings and small incremental document updates.
Buckets: large-build artifacts, resumability, and auditability.

The hourly process keeps the active index close to the live catalog without turning the online endpoint into an unbounded batch processor.

Related-paper retrieval

The same document embeddings support related-paper recommendations on each paper page.

![Related papers feature]

Lessons from the system

1. Separate throughput work from latency-sensitive work

Corpus and query embeddings use the same model but represent different infrastructure problems. Jobs optimize for throughput and bounded cost, while Inference Endpoints address availability and request latency.

2. Make storage the contract between compute and production

Buckets provide a clear handoff between computation and production. Checksummed artifacts create a reviewable boundary before data reaches the production index.

3. Pin more than the model name

The model revision, dimension, prompt, normalization, and input formatter all affect retrieval. These values should be stored together and validated throughout the pipeline.

4. Design for cold starts

Scale-to-zero is useful for intermittent traffic, but it requires a fast fallback. Hybrid search provides one naturally because lexical retrieval remains useful by itself.

5. Treat smaller vectors as a systems feature

Matryoshka embeddings make it possible to evaluate quality, memory use, index size, and latency together. In the pilot, 256 dimensions preserved approximate-nearest-neighbor recall while substantially reducing storage compared with 1024 dimensions.

6. Keep activation routine

New generations are imported beside the current generation, indexed independently, checked for complete and current coverage, and activated atomically. Rollback becomes a configuration change rather than an emergency recomputation.

How Hugging Face Jobs, Buckets, and Inference Endpoints Support Papers with Code Search