#
Document Analysis Graph

Alfred at your service. As Mr. Wayne’s trusted butler, I’ve taken the liberty of documenting how I assist Mr Wayne with his various documentary needs. While he’s out attending to his… nighttime activities, I ensure all his paperwork, training schedules, and nutritional plans are properly analyzed and organized.

Before leaving, he left a note with his week training program. I then took the responsability to come up with a menu for tomorrow’s meals.

For future such event, let’s create a document analysis system using LangGraph to serve Mister Wayne’s needs. This system can:

1. Process images document
2. Extract text using vision models (Vision Language Model)
3. Perform calculations when needed (to demonstrate normal tools)
4. Analyze content and provide concise summaries
5. Execute specific instructions related to documents

##
The Butler’s Workflow

The workflow we’ll build, follows a structured this schema:

You can follow the code in this notebook that you can run using Google Colab.

##
Setting Up the environment

    %pip install langgraph langchain_openai Pillow base64 langchain_core

and imports :

    import base64
    from typing import List, TypedDict, Annotated, Optional
    from langchain.schema import HumanMessage
    from langchain_openai import ChatOpenAI
    from langchain_core.messages import AnyMessage, SystemMessage
    from langgraph.graph.message import add_messages
    from langgraph.graph import START, StateGraph
    from langgraph.prebuilt import tools_condition
    from langgraph.prebuilt import ToolNode
    from IPython.display import Image, display

##
Defining Agent’s State

This state is a little more complex than the previous ones we have seen. AnyMessage is a class from langchain that define messages and the add_messages is an operator that add the latest message rather than overwritting it with the latest state.

This is a new concept in LangGraph, where you can add operators in your state to define the way they should interact together.

    class AgentState(TypedDict):
        # The document provided
        input_file: Optional[str]  # Contains file path (PDF/PNG)
        messages: Annotated[list[AnyMessage], add_messages]

##
Preparing Tools

    vision_llm = ChatOpenAI(model="gpt-4o")

    def extract_text(img_path: str) -> str:
        """
        Extract text from an image file using a multimodal model.

        Master Wayne often leaves notes with his training regimen or meal plans.
        This allows me to properly analyze the contents.
        """
        all_text = ""
        try:
            # Read image and encode as base64
            with open(img_path, "rb") as image_file:
                image_bytes = image_file.read()

            image_base64 = base64.b64encode(image_bytes).decode("utf-8")

            # Prepare the prompt including the base64 image data
            message = [
                HumanMessage(
                    content=[
                        {
                            "type": "text",
                            "text": (
                                "Extract all the text from this image. "
                                "Return only the extracted text, no explanations."
                            ),
                        },
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": f"data:image/png;base64,{image_base64}"
                            },
                        },
                    ]
                )
            ]

            # Call the vision-capable model
            response = vision_llm.invoke(message)

            # Append extracted text
            all_text += response.content + "\n\n"

            return all_text.strip()
        except Exception as e:
            # A butler should handle errors gracefully
            error_msg = f"Error extracting text: {str(e)}"
            print(error_msg)
            return ""

    def divide(a: int, b: int) -> float:
        """Divide a and b - for Master Wayne's occasional calculations."""
        return a / b

    # Equip the butler with tools
    tools = [
        divide,
        extract_text
    ]

    llm = ChatOpenAI(model="gpt-4o")
    llm_with_tools = llm.bind_tools(tools, parallel_tool_calls=False)

##
The nodes

    def assistant(state: AgentState):
        # System message
        textual_description_of_tool="""
    extract_text(img_path: str) -> str:
        Extract text from an image file using a multimodal model.

        Args:
            img_path: A local image file path (strings).

        Returns:
            A single string containing the concatenated text extracted from each image.
    divide(a: int, b: int) -> float:
        Divide a and b
    """
        image=state["input_file"]
        sys_msg = SystemMessage(content=f"You are an helpful butler named Alfred that serves Mr. Wayne and Batman. You can analyse documents and run computations with provided tools:\n{textual_description_of_tool} \n You have access to some optional images. Currently the loaded image is: {image}")

        return {
            "messages": [llm_with_tools.invoke([sys_msg] + state["messages"])],
            "input_file": state["input_file"]
        }

##
The ReAct Pattern: How I Assist Mr. Wayne

Allow me to explain the approach in this agent. The agent follows what’s known as the ReAct pattern (Reason-Act-Observe)

1. Reason about his documents and requests
2. Act by using appropriate tools
3. Observe the results
4. Repeat as necessary until I’ve fully addressed his needs

This is a simple implementation of an agent using langGraph.

    ## The graph
    builder = StateGraph(AgentState)

    # Define nodes: these do the work
    builder.add_node("assistant", assistant)
    builder.add_node("tools", ToolNode(tools))

    # Define edges: these determine how the control flow moves
    builder.add_edge(START, "assistant")
    builder.add_conditional_edges(
        "assistant",
        # If the latest message requires a tool, route to tools
        # Otherwise, provide a direct response
        tools_condition,
    )
    builder.add_edge("tools", "assistant")
    react_graph = builder.compile()

    # Show the butler's thought process
    display(Image(react_graph.get_graph(xray=True).draw_mermaid_png()))

##
The Butler in Action

###
Example 1: Simple Calculations

In the following example, we added this divide example just as a

    messages = [HumanMessage(content="Divide 6790 by 5")]
    messages = react_graph.invoke({"messages": messages, "input_file": None})

The conversation would proceed:

    Human: Divide 6790 by 5

    AI Tool Call: divide(a=6790, b=5)

    Tool Response: 1358.0

    Alfred: The result of dividing 6790 by 5 is 1358.0.

###
Example 2: Analyzing Master Wayne’s Training Documents

When Master Wayne leaves his training and meal notes:

    messages = [HumanMessage(content="According to the note provided by Mr. Wayne in the provided images. What's the list of items I should buy for the dinner menu?")]
    messages = react_graph.invoke({"messages": messages, "input_file": "Batman_training_and_meals.png"})

The interaction would proceed:

    Human: According to the note provided by Mr. Wayne in the provided images. What's the list of items I should buy for the dinner menu?

    AI Tool Call: extract_text(img_path="Batman_training_and_meals.png")

    Tool Response: [Extracted text with training schedule and menu details]

    Alfred: For the dinner menu, you should buy the following items:

    1. Grass-fed local sirloin steak
    2. Organic spinach
    3. Piquillo peppers
    4. Potatoes (for oven-baked golden herb potato)
    5. Fish oil (2 grams)

    Ensure the steak is grass-fed and the spinach and peppers are organic for the best quality meal.

##
Key Takeaways

Should you wish to create your own document analysis butler, here are key considerations:

1. Define clear tools for specific document-related tasks
2. Create a robust state tracker to maintain context between tool calls
3. Consider error handling for tools fails
4. Maintain contextual awareness of previous interactions (ensured by the operator add_messages)

With these principles, you too can provide exemplary document analysis service worthy of Wayne Manor.

I trust this explanation has been satisfactory. Now, if you’ll excuse me, Master Wayne’s cape requires pressing before tonight’s activities.

< > Update on GitHub