One corpus, multiple representations

What industrial corpora have taught us about designing document representations in our RAG pipeline.

By Joanne Breitfelder, Lead Data Scientist at Ask for the moon.

‍

When building a RAG pipeline, it is quite natural to assume that a document follows a relatively linear processing chain: a parser extracts content from a PDF, converts it to Markdown, and then that text is split into chunks, transformed into embeddings, and finally used to build the context sent to the model.

In other words, the same artifact moves through the entire pipeline. This approach is simple and works very well in many situations. In fact, we started with this architecture, just like many other RAG systems.

In practice, this view works best when the information is primarily textual. However, the corpora we handle at Ask for the Moon are rarely like that. Industrial documents such as standards, technical manuals, or engineering files contain text, but also tables, charts, blueprints, pictograms, nomograms, and more.

Above all, these elements do not all play the same role. Some carry the sought-after information directly, while others serve more to illustrate or contextualize a point.

In this context, it becomes difficult to assume that a single representation can suit every stage of the pipeline.

‍

Preserving important information

This observation quickly raises a new question: what should actually be preserved from each element of a document?

For us, this question emerged gradually through user feedback. A misinterpreted table, an unreadable blueprint, or a chart that was difficult for the LLM to process made us realize that we needed to understand what information each element actually carried in order to determine what should be preserved within the pipeline.

To help reason about these different elements, you can ask yourself questions such as:

‍

In practice, this can lead to representations such as:

‍

¹ The caption generated by the OCR engine is usually embedded in the document text (e.g., "Icon indicating an electrical hazard"). It is generally not a particularly relevant element for retrieval, but it benefits "for free" from the indexing of the chunk to which it belongs.

² Depending on the context. For example, an icon or a photograph can be inherently meaningful, but this is not always the case.

This table is intentionally simplified. An annotated photograph of equipment in a factory, for instance, should not necessarily be treated the same way as an illustrative photograph of employees chatting. This incidentally shows how important relevant content categorization becomes.

However, this approach leads to a new question: if each element can result in multiple representations, why not systematically produce them all?

‍

Every representation has a cost

First, because every additional indexed representation becomes a potential candidate during retrieval. Yet, a vector search engine generally returns only a limited number of results. Describing logos, icons, or purely decorative elements therefore increases the risk of pushing aside truly relevant content.

Next, automatically generating a description for every image in a document can quickly become very expensive, even though a large portion of them provides no useful information. You risk increasing processing time, inference costs, and the amount of noise in the data, without any real benefit for RAG.

Finally, multimodal models generally impose a limit on the number of images they can process effectively. Beyond a certain number, they are often downscaled in resolution or even ignored. Here again, it is clearly preferable to reserve this budget for elements where visual information is truly important.

Thus, every representation consumes a resource. The challenge is therefore not to produce as many as possible, but only those that are truly needed, at the right time.

An impact on architecture

If we apply this line of reasoning to a table, for example, several representations naturally become necessary.

For retrieval, we generate an enriched description based on the table and its context within the document. Its goal is not to be a faithful reproduction of the original table, but to provide the vector search engine with the semantic signal it needs, particularly the concepts and vocabulary likely to appear in a user query.

Once the table is retrieved, this representation is no longer of much interest. The LLM then receives other artifacts: the table in Markdown to reason about the values, as well as its image, which helps better preserve certain elements (such as merged cells, annotations, colors, etc.).

This separation between retrieval representations and generation representations naturally modifies the pipeline architecture: the vector search engine no longer directly returns the context sent to the LLM, but rather a document element identifier, which then allows us to retrieve the representations adapted to each use case.

Description generated
        │
        ▼
   Embedding
        │
        ▼
 Vector Search
        │
        ▼
  Identifier
        │
        ▼
Artefacts repositary
    ├── Markdown
    ├── Image
    ├── Metadata
    └── ...

The role of retrieval is therefore no longer to build the context sent to the model. It serves primarily to identify relevant document elements, and the representations adapted for generation are then resolved in a second step.

Conclusion

This approach naturally makes ingestion longer and more costly than a simple "OCR → chunking → embeddings" chain, and it is a trade-off we accept. In fact, this way of designing our pipeline has proven particularly useful for our clients' industrial corpora, where information is rarely purely textual. It has allowed us to improve retrieval quality and provide the LLM with the most targeted and relevant information possible. Ultimately, this results in more accurate answers, even for particularly complex documents.

‍