More and more companies are turning to RAG (Retrieval-Augmented Generation) to put their internal documents and business data to work with generative AI. But as soon as a project gets going, the same questions come up:
- How much does it cost to build a RAG system?
- How many months does it take to go from PoC to production?
- What data do we need to prepare?
- Can we use Excel files and databases, not just documents?
- How do we know whether the PoC has succeeded?
The short answer
What you will learn
- How RAG works, and typical timelines and costs
- Six factors that drive cost — and how prototypes, PoCs and production systems differ
- A six-step process for building a RAG system
- An 8-week SQL + RAG PoC case study, and how to judge PoC success
What Is RAG?
RAG stands for Retrieval-Augmented Generation. When a user asks a question, the system searches internal documents and databases for relevant information, passes it to a large language model (LLM), and generates an answer grounded in that information.
STEP 1
Retrieve
Find information relevant to the question in documents and databases
STEP 2
Augment
Pass the search results to the LLM as context
STEP 3
Generate
Produce an answer based on the retrieved information
A general-purpose generative AI answers only from what it learned during training. With RAG, answers can be grounded in your own information — internal policies, product manuals, technical documents, past support tickets — and sources can be shown alongside each answer.
Keep in mind
How Long Does It Take to Build a RAG System?
RAG projects usually move from technical validation to PoC, then to production development and company-wide rollout.
| Phase | Typical duration | Main purpose |
|---|---|---|
| Technical validation / prototype | 2–4 weeks | Check retrieval and answers with limited data |
| Small-scale PoC | 1–2 months | Evaluate accuracy and value with real data |
| PoC with mixed data | 2–3 months | Validate documents and databases together |
| Departmental production system | 2–4 months | Implement authentication, permissions, logging and integrations |
| Company-wide or customer-facing rollout | 4–8+ months | High availability, auditing, multi-department rollout |
A 2–4 week technical validation narrows the documents and questions in scope to prove basic feasibility. A business PoC uses the questions real users would ask to evaluate retrieval accuracy, answer quality and business impact. If data preparation or integration with several systems is needed, expect 2–3 months.
How Much Does a RAG System Cost?
There is no standard price for a RAG system. Based on publicly available information on RAG development in the Japanese market, typical budgets look like this:
| Scope | Typical cost | What is usually included |
|---|---|---|
| Prototype | ¥0.5M–¥2M | Limited data, basic search and answers |
| Small-scale PoC | ¥1.5M–¥5M | Requirements, data processing, accuracy evaluation |
| PoC with mixed data | ¥3M–¥8M+ | Document and database integration, business-fit evaluation |
| Departmental production system | ¥5M–¥15M+ | Authentication, permissions, logging, external integrations |
| Company-wide or customer-facing system | Custom quote | High availability, auditing, load handling |
*Rough market estimates, not BAP’s pricing. Actual costs are calculated after reviewing the target process, data, required accuracy, integrations and security requirements.
Six Factors That Drive the Cost of RAG
| Factor | Impact on cost and timeline |
|---|---|
| Data types | Document-only RAG needs a different scope from a system that also uses Excel and databases. Design must fit the data — e.g. SQL for structured data, vector search for documents |
| Data condition | Outdated documents, duplicates and multiple versions take time to clean up. Scanned PDFs, images and complex tables also need OCR or layout analysis |
| Required accuracy | General internal search and work such as contracts, finance or quality control demand very different accuracy. High-stakes use cases need more evaluation data and human approval |
| Access control | Design is far more complex when visibility depends on department or role. Confirm permission requirements from the PoC stage |
| Integrations | Connecting to SharePoint, Google Drive, Box, CRM, ERP or document management systems requires designing APIs, authentication and data sync |
| Operations after go-live | Document updates, answer-quality monitoring, API cost control, incident response and model updates — billed as monthly maintenance on top of the initial build |
Prototype vs. PoC vs. Production
| Phase | Goal | Typical deliverables |
|---|---|---|
| Prototype | Confirm that the basic RAG mechanism works | Simple UI, limited search index, basic answering (business impact and permissions often not evaluated) |
| PoC | Verify that RAG fits the business, using real data and representative questions | Validation system, evaluation questions and expected answers, accuracy and quality results, issues and improvements, proposed production architecture, rough production schedule |
| Production | Make the system safe for everyday use | User authentication, access control, activity and audit logs, data updates, quality and API cost monitoring, backups, incident response, security controls |
Key point
How to Build a RAG System: Six Steps
Define the target use case
“Make internal information searchable” is far too broad. Make it specific — and keep the first PoC to one process, department or product so it is easy to evaluate.
Internal HR policy questions Support answers from product manuals Searching past equipment incidents Variance analysis of management figures Finding sales proposal materials
Inventory your data
Check file formats, document counts and data volume, database structure, storage locations, update frequency, old versions and duplicates, personal and confidential data, and access rights. Excel figures and PDF documents need different retrieval methods, so understand your data before designing retrieval.
Build an evaluation set
Collect questions real users will ask. For each, define the expected answer, the documents to reference, the figures to use, unacceptable answers, and when a human must review. Without an evaluation set, you cannot tell whether fluent-sounding output is actually fit for work.
Build the retrieval layer
For documents: text extraction, chunking, metadata tagging and embedding generation, stored in a vector database. For structured data: check data types, field names, table relationships and aggregation logic, then load it into a relational database.
Route each question to the right method
Classify each question: document questions go to RAG search, numeric questions to SQL, mixed questions to both, and out-of-scope questions are declined. Use predefined workflows where the procedure is fixed, and let the AI plan its own steps where the path depends on the question.
Evaluate retrieval and generation separately
When an answer is wrong, pin down the cause: relevant documents not retrieved, right documents but poor generation, outdated data, calculation errors, unsupported content added, or data retrieved outside the user’s permissions. That tells you whether to fix the data, retrieval logic, prompts or the LLM. Microsoft’s guide “Design and develop a RAG solution” covers this approach in detail.
Case Study: An 8-Week RAG PoC for Management Accounting
BAP Solution Japan supported a Japanese enterprise in running a PoC that applied RAG and generative AI to management accounting.
The challenge
Collecting management figures from multiple sources, formatting them and aggregating them took significant effort. Staff were also doing the following by hand:
- Checking variances between actuals and plan
- Investigating causes in historical data
- Exploring improvement measures
- Preparing materials for management meetings
PoC objectives
- Technical: retrieve historical figures and related documents accurately
- Business: confirm that AI-generated explanations and proposals are sound
- End goal: output good enough to serve as a first draft of management reports, always reviewed by a person
The 8-week plan and team
The project ran for two months (8 weeks). After the client walked the team through its data structures and how the data is used, BAP organized the requirements and built and tested the system feature by feature. Each cycle was checked against real data, and the feedback was used to refine the answering approach and processing logic. In the final stage, the team prepared the environment and documentation and delivered on schedule.
Project manager Bridge SE (Japanese-speaking) AI engineer Full-stack engineer Infrastructure engineer
The team covered not only the AI model but also business requirements, databases, the web application and the cloud environment end to end. Read more about our setup in “Our AI development team for Japanese companies.”
Technology stack
| Area | Technology |
|---|---|
| AI platform | Amazon Bedrock |
| LLM | Claude Sonnet 4.5 |
| RAG / vector search | PostgreSQL, pgvector |
| Backend | Python, FastAPI, Uvicorn, Gunicorn |
| Frontend | React, TypeScript |
| Data | Structured data such as Excel; knowledge data such as Office documents |
*Chosen for this PoC’s requirements and data structure. In real projects, the LLM and infrastructure are selected after reviewing data volume, security, the existing environment and required accuracy.
Architecture: combining SQL and RAG
Rather than sending every question through vector search, the PoC used an architecture that switches between SQL search and RAG search depending on the question.
STEP 1
Data ingestion
Office documents are parsed and chunked, with embeddings and metadata stored in a vector DB. Excel data is validated and loaded into PostgreSQL
STEP 2
Intent Router
Classifies each question. Routine questions use predefined workflows; open-ended ones get an AI-generated execution plan
STEP 3
SQL search
The SQL Tool pulls exact figures — sales, plan, actuals — from PostgreSQL
STEP 4
RAG search
The RAG Tool retrieves business rules and past documents via vector search, keeping source references
STEP 5
Combine & answer
Results are merged into one context; Claude Sonnet 4.5 generates the answer and explanation, shown with sources
For a question like “Show the variance between plan and actuals and explain the causes based on past reports,” the system pulls the figures from PostgreSQL, finds related past documents with RAG, combines both and generates the answer. This produced responses far better suited to management accounting than document-only RAG.
What we evaluated — and the results
The question was not “can the AI write text?” but “can it be used for management accounting work?” We checked whether it could retrieve historical figures correctly, find relevant documents, explain variances reasonably, make proposals based on statistical data, explain persuasively, produce usable report drafts, and output results staff could easily review and edit. AI output was never used directly for management decisions; staff verified the sources and edited as needed.
8 weeks
PoC completed on schedule
SQL + RAG
Structured data and documents unified
Traceable
Every answer linked to its sources
Delivered
PoC environment and documentation handed over
The team validated the use case of drafting management reports and received positive feedback from the client.
*Unverified metrics, such as effort reduction or answer accuracy rates, are not included in these results.
How to Judge Whether a RAG PoC Has Succeeded
“The AI works” is not a success criterion. Beyond accuracy, check these conditions with production in mind:
Quality and risk
- Required documents and data can be retrieved
- Answers are grounded and sources are visible to users
- The conditions that cause wrong answers are understood
- Unanswerable questions are handled properly
- The scope of human review is clearly defined
Business and operations
- A clear benefit over current work (e.g. time saved) is expected
- Production costs can be estimated and are acceptable
- Data update methods and an operations owner are in place
- Security requirements can be met
For high-risk use cases
What to Prepare Before Requesting a Quote
Having the following ready before you talk to a RAG development partner makes it much easier to scope the PoC and get an accurate estimate:
Business problem to solve Target users Documents and databases in scope Data formats and locations Real example questions Expected answers Access requirements Systems to integrate Security requirements PoC success metrics Target timeline Post-launch operating model
You do not need every answer up front. A clear business problem, where the data lives and a few real questions are enough to scope the first validation.
Frequently Asked Questions
Q. Can a RAG PoC be done in 2–4 weeks?
A. Yes, a 2–4 week technical validation is possible when the use case, data and questions are clear. Evaluation with real business data, multiple integrations or detailed accuracy tuning usually takes 1–3 months.
Q. Can RAG use Excel data?
A. Yes. When figures must be aggregated precisely, it is best to load Excel data into a database and query it with SQL. Combined with RAG for documents, the system can answer using both figures and explanatory material.
Q. Will RAG eliminate wrong answers?
A. Not entirely. It is combined with safeguards such as limiting source data, citing sources in answers, handling unanswerable questions, rule-based checks and human review.
Q. Which LLM should we choose?
A. Look beyond accuracy to data handling, pricing, latency, language support and fit with your cloud environment. BAP’s management accounting PoC used Claude Sonnet 4.5 on Amazon Bedrock.
Q. Is ongoing maintenance needed after launch?
A. Yes. Documents and data need updating, answer quality needs checking, retrieval settings need tuning, and API costs and model changes need managing on an ongoing basis.
Summary
- RAG timelines and costs are driven by the use case, data types, accuracy, permissions, integrations and operations — not by document count
- Expect 2–4 weeks for technical validation and 1–3 months for a business PoC
- In BAP’s case study, a PoC combining SQL search and RAG search was completed in 8 weeks
Start by defining the use case and evaluation criteria, run a PoC on a limited data set, and use the results to decide on the features, budget and operating model needed for production. That is the most practical path to RAG in production.
References
Talk to BAP About RAG and Generative AI PoCs
BAP supports you end to end — from clarifying the business problem and running a RAG PoC to production development, integration with existing systems and post-launch improvement. We also build AI solutions that combine internal documents with Excel, databases and existing business systems. Reach out if you want to:
- Decide which process to start RAG with
- Use internal documents and numerical data together
- Confirm the timeline and team for a PoC
- Embed generative AI into existing systems
- Move a generative AI project stuck at PoC into production
Even without a defined use case, we can help you scope the process, data and PoC criteria. See BAP’s generative AI and RAG development services for details.
