đ Hypencoder Model
This model, Hypencoder, is an innovative solution for information retrieval. It combines a text encoder and a Hypencoder to generate relevance scores, offering high - efficiency retrieval capabilities.
đ Quick Start
Using the pretrained Hypencoders as stand - alone models
from hypencoder_cb.modeling.hypencoder import Hypencoder, HypencoderDualEncoder, TextEncoder
from transformers import AutoTokenizer
dual_encoder = HypencoderDualEncoder.from_pretrained("jfkback/hypencoder.6_layer")
tokenizer = AutoTokenizer.from_pretrained("jfkback/hypencoder.6_layer")
query_encoder: Hypencoder = dual_encoder.query_encoder
passage_encoder: TextEncoder = dual_encoder.passage_encoder
queries = [
"how many states are there in india",
"when do concussion symptoms appear",
]
passages = [
"India has 28 states and 8 union territories.",
"Concussion symptoms can appear immediately or up to 72 hours after the injury.",
]
query_inputs = tokenizer(queries, return_tensors="pt", padding=True, truncation=True)
passage_inputs = tokenizer(passages, return_tensors="pt", padding=True, truncation=True)
q_nets = query_encoder(input_ids=query_inputs["input_ids"], attention_mask=query_inputs["attention_mask"]).representation
passage_embeddings = passage_encoder(input_ids=passage_inputs["input_ids"], attention_mask=passage_inputs["attention_mask"]).representation
passage_embeddings_single = passage_embeddings.unsqueeze(1)
scores = q_nets(passage_embeddings_single)
passage_embeddings_double = passage_embeddings.repeat(2, 1).reshape(2, 2, -1)
scores = q_nets(passage_embeddings_double)
⨠Features
Model Details
This is a Hypencoder Dual Encoder. It contains two trunks: the text encoder and Hypencoder. The text encoder converts items into 768 - dimension vectors, while the Hypencoder converts text into a small neural network which takes the 768 - dimension vector from the text encoder as input. This small network is then used to output a relevance score. To use this model, please take a look at the [Github](https://github.com/jfkback/hypencoder - paper) page, which contains the required code and details on how to run the model.
Model Variants
We released the four models used in the paper. Each model is identical except the small neural networks, which we refer to as q - nets, have different numbers of hidden layers.
Property |
Details |
Model Type |
Hypencoder Dual Encoder |
Base Model |
google - bert/bert - base - uncased |
Training Data |
microsoft/ms_marco |
Library Name |
transformers |
Pipeline Tag |
feature - extraction |
đ License
This model is released under the apache - 2.0 license.
đ Documentation
This is the official model from the paper Hypencoder: Hypernetworks for Information Retrieval.
đ Citation
BibTeX:
@misc{killingback2025hypencoderhypernetworksinformationretrieval,
title={Hypencoder: Hypernetworks for Information Retrieval},
author={Julian Killingback and Hansi Zeng and Hamed Zamani},
year={2025},
eprint={2502.05364},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2502.05364},
}