Application Engineering: Agent, RAG and Memory

LLM Application development includes not only invoking model API, but also involves tool invocation, task orchestration, knowledge retrieval, memory management, evaluation, security and reliability.

1. Core architecture

LLM Application
│
├── Tool Calling
│   ├── Tool Schema / Arguments
│   ├── Validation / Authorization
│   ├── Parallel / Sequential Calls
│   └── Error Handling / Idempotency
│
├── Agent / Workflow
│   ├── ReAct / Plan-and-Execute
│   ├── Router / State Machine
│   ├── State / Checkpointing
│   └── Human-in-the-loop
│
├── RAG
│   ├── Parsing / Chunking / Embedding
│   ├── Hybrid Retrieval / Reranking
│   └── Context Assembly / Citation
│
├── Vector Database
│   ├── Similarity / ANN
│   ├── HNSW / IVF
│   └── Metadata Filtering
│
├── Memory
│   ├── Short-term / Long-term
│   ├── Retrieval / Updating
│   └── Invalidation
│
├── Evaluation / Observability
│   ├── Task Success / Groundedness
│   └── Trace / Logging / Metrics
│
└── Security / Reliability
    ├── Prompt Injection
    ├── Authorization / Sandboxing
    └── Retry / Rate Limit / Audit

2. Tool Calling / Function Calling

The model does not directly perform external operations, but generates tool names and structured parameters, and the actual execution is completed by the application backend.

User
  ↓
LLM
  ↓
Tool Name + Arguments
  ↓
Validate / Authorize
  ↓
Backend Executes Tool
  ↓
Tool Result
  ↓
LLM Continues

Tool Schema / JSON Schema

A tool description usually contains a name, description, and parameters Schema.

{
  "name": "get_weather",
  "description": "Get weather for a city",
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string"
      },
      "unit": {
        "type": "string",
        "enum": ["celsius", "fahrenheit"]
      }
    },
    "required": ["city"],
    "additionalProperties": false
  }
}

JSON Schema describes the shape of parameters, but cannot replace back-end business verification.

Tool Choice

Controls whether and which tool the model calls.

Common strategies:

Specific parameter names and support depend on model API.

Multiple tool calls

Parallel Tool Calling: Multiple tools are independent of each other and can be executed concurrently.

                 ┌── get_weather()
LLM Tool Calls ──┤
                 └── get_exchange_rate()
                         ↓
                  Aggregate Results

Sequential Tool Calling: Subsequent tools depend on the return value of the previous tool.

search_flight()
      ↓
get_flight_details()
      ↓
check_availability()
      ↓
book_flight()

Note: The model generates multiple Tool Call at once, which does not mean that the backend must execute in parallel. When resource dependencies or side effects are involved, the program should control the order of execution.

Tool Validation / Authorization

The parameters generated by the model are untrusted inputs.

The backend checks at least:

1. Are the JSON structure and field types correct?
2. Whether the parameters meet business constraints.
3. Whether the current user has the calling permission.
4. Whether the user has the authority to operate the specified resource.
5. Whether high-risk operations have been clearly identified.

For example:

def delete_document(user, document_id):
    document = get_document(document_id)

    if document.owner_id != user.id:
        raise PermissionError("Forbidden")

    document.delete()

Just because the model calls delete_document, it does not automatically mean that it has permission to delete documents.

Timeout / Retry / Idempotency

External tools may time out or fail.

• Timeout: Limit the waiting time.
• Retry: Retry on recoverable errors.
• Fallback: Switch to the alternative plan after the main plan fails.
• Idempotency: Repeated submission of the same logical operation will not produce unexpected side effects repeatedly.

For operations such as payment, booking, and sending notifications, duplicate requests can be identified through Idempotency Key.

Retry must consider idempotence, otherwise retry after network timeout may lead to repeated execution.

Tool Result

Tool results are also untrusted content.

Text in external web pages, documents, and API responses can be used as data, but cannot be automatically upgraded to system commands or user authorizations.

---

3. Agent Loop / Workflow

ReAct

ReAct adopts loop execution:

Reason
   ↓
Action
   ↓
Observation
   ↓
Reason
   ↓
Action
   ↓
...

The model continuously adjusts the next step based on the results returned by the tool.

Suitable for open-ended, multi-step tasks where the execution path is difficult to completely determine in advance.

Plan-and-Execute

Develop the overall plan first and then execute the individual steps.

User Task
    ↓
Planner
    ↓
Step 1 → Step 2 → Step 3
    ↓
Executor
    ↓
Result

Plans can also be re-planned during execution, which does not mean that plans are never allowed to be modified.

Router

Choose an execution path based on the task type.

User
  ↓
Router
  ├── General QA → LLM
  ├── Document QA → RAG
  ├── Weather → Tool
  └── Data Analysis → Data Agent

Router can be implemented using rules, classification models, or LLM.

State Machine / Workflow

State Machine restricts the process to explicit state transitions.

START
  ↓
SEARCH
  ↓
VALIDATE
  ↓
CONFIRM
  ↓
EXECUTE
  ↓
DONE

Workflow combines deterministic code, LLM nodes, and tool nodes.

There is no need to let LLM determine all the steps in producing Agent.

For example:

Upload Document
       ↓
Parse（deterministic code）
       ↓
Retrieve（Search）
       ↓
Summarize（LLM）
       ↓
Human Approval
       ↓
Save（deterministic code）

For business tasks with well-defined processes, Workflow + a handful of Agent decisions are often easier to control and test than completely free Agent Loop decisions.

Agent State / Checkpointing

This is a very important concept in Agent projects.

State saves the information required for current task execution.

For example:

{
    "messages": [...],
    "current_step": "validate",
    "tool_results": {...},
    "retry_count": 1,
    "approved": False
}

Checkpointing persists execution state so that:

• Recovery after error.
• Execution continues across requests.
• Pause waiting for manual confirmation.
• Check historical execution status.

Checkpointer of LangGraph can hold Graph State, while Store can be used for long-term data across Thread.

Note: Resume execution does not mean that each line of code is executed only once. Some nodes may be re-run, so external side effects still require idempotence protection.

Loop Termination / Budget

Agent must have a clear stopping condition.

Task Completed
Max Steps Reached
Timeout
Token / Cost Budget Exhausted
Unrecoverable Error
User Cancelled

In addition to max_steps, it is recommended to also limit:

• Maximum number of tool calls.
• Maximum total time spent.
• Maximum Token consumption.
• maximum cost.
• The number of retries for a single tool.

Simply setting Max Steps will not necessarily prevent a single tool call from blocking for a long time.

Human-in-the-loop

High-risk operations need to be suspended and wait for user confirmation.

Agent Proposes Action
          ↓
     Pause / Checkpoint
          ↓
     Human Approval
       ├── Reject → Stop
       └── Approve → Execute

For example:

• Delete data.
• Send mail.
• Execute payment.
• Modify production configuration.
• Submit an irreversible operation.

LangGraph's interrupt() supports saving state and restoring after waiting for external input.

---

4. RAG：Retrieval-Augmented Generation

RAG provides relevant context to LLM by retrieving external knowledge.

Indexing Pipeline

Documents
    ↓
Parsing
    ↓
Chunking
    ↓
Embedding
    ↓
Vector Database / Search Index

Query Pipeline

User Query
     ↓
Query Processing
     ↓
Retrieval
     ↓
Reranking
     ↓
Context Assembly
     ↓
LLM
     ↓
Answer + Citation

Document Parsing

Convert PDF, web pages, Markdown, Office documents, etc. into processable content.

Need to pay attention to:

• Title level.
• Tables.
• page number.
• Document structure.
• Images and multimodal content.

If the table structure is lost during parsing, the quality of subsequent Retrieval may also be affected.

Chunking

Break long documents into smaller retrieval units.

Common strategies:

Overlap can alleviate boundary information loss, but an excessively large Overlap will increase storage costs and duplicate retrieval results.

Embedding / Semantic Search

Embedding represents text as a vector.

Query → Query Encoder → Query Vector
                              ↓
                          Similarity
                              ↑
Document → Document Encoder → Doc Vector

The common Bi-Encoder can be encoded respectively as Query and Document.

It is not required that both sides must use exactly the same encoding process, but they must use vector spaces that are compatible with each other and have consistent training goals.

BM25 / Hybrid Search

BM25 Calculates relevance based on keyword matching, term frequency, inverse document frequency, and document length.

Semantic Search places more emphasis on semantic similarity.

Hybrid Search combines the two:

Query
  ├── BM25 Retrieval
  └── Vector Retrieval
          ↓
      Merge / Fusion
          ↓
        Rerank

RRF：Reciprocal Rank Fusion

RRF is a common sorting and fusion method in Hybrid Search.

Its core is to give documents a cumulative score based on their rankings in different search lists, rather than directly comparing the raw scores of BM25 and Cosine Similarity.

For the result (d), a common form is:

$$RRF(d)=\sum_i\frac{1}{k+\operatorname{rank}\_i(d)}$$

where (k) is the ranking smoothing constant.

Reranking

Phase 1 Retrieval focuses on speed and recall.

Reranking Re-evaluate candidate results using more complex models.

Common Cross-Encoder:

Query + Document
        ↓
Joint Encoding
        ↓
Relevance Score

Unlike Bi-Encoder, Cross-Encoder directly jointly processes Query and Document, which is generally more suitable for fine sorting but is more computationally expensive.

Query Rewriting / Multi-query Retrieval

Query Rewriting Reword user questions that are unclear or lack context.

Multi-query Retrieval generates multiple queries from different perspectives to improve recall coverage.

But increasing the number of queries also increases cost, latency, and noise.

Context Assembly

Construct the final search results into the LLM context.

Need to consider:

• Token Budget。
• Deduplicate.
• Source and page number.
• Relevance ranking.
• Context truncation.
• Retrieve content permissions.

Citation

Citation allows answers to be traced back to the original source.

However, generating Citation does not mean that the reference is necessarily correct. You should also verify whether the referenced material actually supports the corresponding statement.

RAG permission isolation

This is an important safety issue in production RAG.

For example, different users or organizations have different access rights to documents.

Resource permission filtering must be enforced by the backend when retrieving:

filters = {
    "tenant_id": authenticated_user.tenant_id
}

where tenant_id must come from an authenticated identity, rather than directly trusting the value generated by LLM.

You cannot retrieve all user profiles first and then let LLM decide on its own what content can be displayed.

---

5. Vector Database

Vector Database is used to store and retrieve Embedding, and ANN Index and Metadata Filtering are generally supported.

Embedding
    ↓
Vector Storage
    ↓
ANN Index
    ├── HNSW
    └── IVF
    ↓
Similarity Search
    ↓
Top-k

Similarity Metrics

If the vector passes through L2 Normalization, there is a definite correspondence between the orderings of Cosine Similarity, Dot Product, and Euclidean Distance.

Without normalization, they may be ranked differently.

ANN

Approximate Nearest Neighbor uses approximate indexing to find high similarity vectors at low computational cost.

It requires a trade-off between retrieval speed, memory, and Recall.

HNSW

Hierarchical Navigable Small World Search using multi-level neighbor graphs.

Common parameters:

• M: Control graph connection number.
• ef_construction: Affects index construction quality and cost.
• ef_search: Affects query speed and Recall.

IVF

Inverted File Index divides the vector into different Cluster.

When querying, only search part Cluster to reduce calculations.

Common parameters:

• nlist: Cluster quantity.
• nprobe: The number of Cluster searched during query.

Vector Database selection

A separate vector database is not necessarily required.

If your application already uses PostgreSQL, you can also use pgvector for vector storage and retrieval.

For small and medium-sized RAG:

PostgreSQL + pgvector

Usually an option worth considering.

When vector size, retrieval throughput, or distribution requirements increase, then evaluate specialized vector databases.

---

6. Memory

Memory allows Agent to exploit the current context or long-term saved information.

Short-term Memory

Conversation History
       ↓
Summarization
       ↓
Context Compression
       ↓
Current LLM Context

Mainly handles the current dialogue and task execution status.

Long-term Memory

Episodic Memory

For example:

2026-10-01:
Encountered when deploying a model CUDA OOM。

2026-10-03:
Decided to switch to another inference engine。

Semantic Memory

Distill historical events into more stable knowledge:

User frequently develops Django applications.

Episodic Memory emphasizes what happened, Semantic Memory emphasizes facts or knowledge that can be used in the long term.

Memory Retrieval

Long-term memory cannot put all Context in every time.

Can be based on:

• Semantic Similarity。
• Keywords。
• Recency。
• Importance。
• User / Task Scope。

Select relevant memories.

Memory Updating / Invalidation

Need to solve:

• What should I do if my user information changes?
• What should I do if new memories conflict with old memories?
• When will expired memories be deleted?
• What should I do if a user requests to delete information?
• How long should memories be kept?

For example:

Old:
User prefers Framework A.

New:
User now prefers Framework B.

The system should update the currently valid facts rather than holding on to two conflicting preferences forever.

At the same time, Prompt Injection contents or unverified tool return values ​​cannot be written directly to long-term memory.

---

7. Evaluation / Observability

Agent Applications need to observe both the final result and the internal execution process.

Evaluation

Groundedness is not equal to Correctness. The retrieved document itself may be inaccurate and may produce incorrect answers even if the model faithfully references it.

Trace

Trace records the complete calling path in a task.

Request
   ↓
LLM Call
   ↓
Tool Call
   ↓
Tool Result
   ↓
Retriever
   ↓
Reranker
   ↓
LLM Call
   ↓
Final Answer

Usually each step forms Span and records:

• Start and end time.
• Input and output summary.
• Token Usage。
• Model Name。
• Error。
• Tool Name。
• Parent / Child Relationship。

Logging and Metrics

Production environments usually require a combination of all three.

Take special care to avoid uncontrolled logging of user privacy, tool credentials, or sensitive file content in logs and Trace.

---

8. Security、Authorization & Reliability

Prompt Injection

Prompt Injection affects the model through input content, causing it to deviate from expected instructions or safety boundaries.

Direct Prompt Injection：

The attacker directly enters malicious instructions.

Indirect Prompt Injection：

Malicious instructions are hidden in external web pages, emails, PDF, RAG documents or tool return results.

External Document
       ↓
RAG Retrieval
       ↓
LLM Reads Document
       ↓
Malicious Instruction
       ↓
Unauthorized Tool Call

RAG and model fine-tuning cannot fundamentally eliminate Prompt Injection risk.

Trust Boundary

It is necessary to distinguish:

Trusted:
Application Rules
Authorization Policy
Explicit User Approval

Untrusted:
User-controlled Data
Retrieved Documents
Web Pages
Tool Results
LLM-generated Arguments

LLM should not be the final judge of authority.

OWASP also recommends treating external content as untrusted and performing tool parameter validation and access control outside of the model.

Least Privilege

Agent should only have the minimum permissions required to complete the current task.

For example:

Read-only Research Agent
    ↓
Search / Read Permissions

Booking Agent
    ↓
Search Permissions
    ↓
Explicit Confirmation
    ↓
Booking Permission

Confirmation / Human Approval

The following operations typically require additional confirmation or authorization:

• Delete important resources.
• Send sensitive information externally.
• Pay or book.
• Modify the production environment.
• Perform irreversible operations.

When confirming, users should be able to see the actual operation objects and key parameters, rather than generally approving all subsequent operations.

Input / Output Validation

Input validation:

Schema
Type
Range
Resource Existence
Business Rules
Authorization

Output verification:

Expected Format
Required Fields
Output Safety
Groundedness
Downstream Compatibility

For example, when the model generates SQL, it cannot be directly allowed to be executed in the production database just because it passes the JSON Schema verification.

Sandboxing

A restricted runtime environment should be used when executing code, Shell, or other high-risk tools.

Common controls:

• File system access scope.
• Network access rights.
• CPU / Memory limit.
• Execution timed out.
• Disallow unnecessary privileges.
• Temporary environment cleanup.

Reliability

In addition to input and output verification, you also need:

• Timeout。
• Retry / Backoff。
• Idempotency。
• Circuit Breaker。
• Rate Limiting。
• Audit Logs。
• Secrets Management。

Circuit Breaker is used to temporarily stop calling external services when they continue to fail, to avoid repeated requests that continuously consume resources.

Audit Logs should record who initiated what important operations and when, as well as the authorization and execution results.

Summary

The most important engineering principles:

1. LLM is responsible for proposing decisions, and the backend is responsible for verification and execution.
2. The deterministic business logic of uses ordinary program code first and does not have to be handed over to Agent.
3. The retrieval quality and data permission control of RAG are as important as the generation quality.
4. Agent must have state management, stop conditions and recovery mechanisms.
5. External content, tool results, and model output cannot automatically obtain trusted permissions.
6. evaluation Agent Priority should be paid to the final task success rate, not just the quality of the model answer.

---

Reference documentation

• LangGraph Documentation
• LangGraph Persistence
• Ragas Evaluation
• OWASP LLM Prompt Injection
• OWASP Prompt Injection Prevention
• pgvector