Synthetic Relevance Assessment (reanimator.relevance)#

Domain-independent synthetic 4-level UMBRELA relevance assessment engine.

RelevanceEvaluator#

class RelevanceEvaluator(
    backend: Optional[Any] = None,
    model_name: str = "synthetic-llm",
    domain_instruction: str = "general information retrieval and document understanding",
    concurrency: int = 10,
)

Methods#

evaluate_document(doc: Document, topics: Union[str, List[str], List[Dict]], modality: str = "tables") -> List[Judgement]#

Evaluate all resources extracted from a Document against given topics across the specified modality.

ModalityExtractor#

class ModalityExtractor:
    @staticmethod
    def extract(
        doc: Document,
        modality: str = "tables",
        chunk_size: int = 512,
        chunk_overlap: int = 100,
    ) -> List[Dict[str, Any]]

Supported Modalities#

  • 'full_document' / 'full_text': Full parsed document including text, HTML tables, figure captions, and formulas.

  • 'text_only': Body text excluding HTML tables and display math.

  • 'tables': Table objects with content DataFrames, captions, and references.

  • 'figures': Figure objects with captions and references.

  • 'formulas' / 'equations': Formula objects with surrounding text.

  • 'chunks' / 'passages': Text chunks created for RAG.

export_qrels#

def export_qrels(
    judgements: List[Judgement],
    file_path: Union[str, Path],
    binary: bool = False,
) -> None

Export relevance judgements in standard TREC qrels format (<query_id> 0 <resource_id> <score>).