Fine-Tuning vs RAG vs Prompting: Choosing the Cheapest Path

Before comparing these three approaches, you need to answer one question: what specifically do you want the LLM to do differently? The answer determines which approach to use, and getting this wrong means burning time and money on the wrong solution.

There are three fundamentally different things you might want:

1. Change how the model talks. You want it to respond in Sheng, use your company's terminology, follow a specific JSON format, or match a particular tone. This is a behavior problem.
2. Give the model knowledge it does not have. You want it to answer questions about your product docs, your customer data, or yesterday's sales. This is a knowledge problem.
3. Get better results on a specific task. You want it to classify support tickets more accurately, extract information from invoices, or generate better code in your stack. This could be either a behavior or knowledge problem, or both.

Behavior problems are best solved with prompting first, then fine-tuning if prompting is not consistent enough. Knowledge problems are solved with RAG. Most real-world applications involve both, which is why the best production systems layer these approaches together.

The mistake most developers make is jumping straight to the most complex solution. Fine-tuning sounds impressive, but if a well-written system prompt would have done the job, you just spent weeks and thousands of shillings solving a problem that did not exist.

Prompt engineering is the practice of crafting instructions, examples, and context to get the behavior you want from an LLM without modifying the model itself. It is the cheapest approach by far, and it should always be your starting point.

What prompting can do well:

• Set the tone and personality of responses ("You are a friendly customer support agent for a Nairobi-based fintech.")
• Enforce output format ("Respond with valid JSON matching this schema...")
• Add context for a specific conversation ("The customer's name is James, their account was created on March 5, and they have an outstanding payment of KES 4,500.")
• Implement few-shot learning by including examples of desired input/output pairs in the prompt
• Chain reasoning steps ("First analyze the customer's issue, then suggest a resolution, then draft a response.")

The cost math: Prompting costs you only the tokens you send and receive. A typical system prompt is 500-2,000 tokens. With GPT-4o, that is roughly $0.005-0.02 per conversation. With Claude Sonnet, similar range. With smaller models like GPT-4o-mini, you are looking at fractions of a cent. There is no upfront investment, no training time, and no infrastructure to maintain.

Where prompting breaks down:

• Very long context. If your system prompt is 10,000 tokens of instructions and examples, you are paying for those tokens on every single request. At scale, this adds up fast.
• Inconsistent behavior. Models can drift from instructions, especially on edge cases. You tell it to always respond in Swahili, and 95% of the time it does, but 5% of the time it slips into English.
• Knowledge it does not have. You cannot prompt a model into knowing your internal documentation. You can paste chunks of it into the prompt, but that is basically manual RAG with worse retrieval.

For developers in Kenya building their first AI features, prompting alone will handle most use cases. A well-crafted system prompt with 3-5 examples of desired behavior is surprisingly powerful. Do not move to more complex approaches until you have genuinely exhausted what good prompting can do.

RAG (Retrieval-Augmented Generation) adds a retrieval step before the LLM generates a response. Instead of relying on the model's training data alone, you search a knowledge base for relevant documents and inject them into the prompt. The model then generates answers grounded in your actual data.

When RAG is the right choice:

• Your application needs to answer questions about specific documents (product manuals, company policies, legal texts)
• Your data changes regularly and you cannot afford to retrain a model every time
• You need citations and traceability (pointing users to the exact source document)
• You are building a customer support bot that needs to reference your actual help documentation

The cost breakdown for RAG:

Setting up a RAG pipeline involves several cost components. First, you need to embed your documents. Using OpenAI's text-embedding-3-small, embedding 1,000 pages of text costs about $0.02. That is a one-time cost (plus re-embedding when documents change). Second, you need vector storage. Supabase with pgvector handles this on its free tier for most early-stage applications. Pinecone's free tier works too. Third, each query incurs costs: embedding the query ($0.0001), retrieving chunks (essentially free from your database), and generating the response ($0.01-0.05 depending on the model). All in, a small RAG system serving hundreds of queries per day runs under $50/month.

Where RAG falls short:

• It does not change model behavior. RAG gives the model better information, but it does not change how the model responds. If you need a specific tone, format, or reasoning style, RAG alone will not get you there. Pair it with good prompting.
• Retrieval quality is the bottleneck. If your chunking strategy is poor or your documents are badly structured, the model will receive irrelevant context and generate bad answers. Garbage in, garbage out.
• Latency. The retrieval step adds 100-500ms to each request. For real-time applications, this matters.

For Kenyan developers, RAG is the sweet spot for most production AI features. Building a WhatsApp bot that answers questions about your product catalog? RAG. Internal tool that searches your company wiki? RAG. Customer support assistant that references your help docs? RAG. The setup cost is modest and the ongoing costs scale linearly with usage.

Fine-tuning takes a pre-trained LLM and trains it further on your own data. The model updates its internal weights to learn the patterns in your training examples. After fine-tuning, the model produces outputs that reflect those patterns without needing them spelled out in the prompt every time.

What fine-tuning actually costs:

Let us talk real numbers. Fine-tuning GPT-4o-mini on OpenAI costs about $3 per million training tokens. If you have 500 high-quality training examples averaging 1,000 tokens each (input + output), that is 500,000 tokens, costing roughly $1.50 per training run. Sounds cheap, right? But you will run 5-15 training iterations to get the quality right, and you need someone with ML experience to curate the training data, design the evaluation criteria, and debug quality issues. The real cost is the human time, not the compute.

Fine-tuning larger models is significantly more expensive. Fine-tuning GPT-4o costs roughly 10x more per token than GPT-4o-mini. Fine-tuning open-source models (Llama 3, Mistral) can be cheaper on compute but requires your own GPU infrastructure or renting cloud GPUs [TODO: verify current pricing tiers for major providers].

When fine-tuning is genuinely worth it:

• Consistent format compliance. If your application requires the model to always output valid JSON in a specific schema, and prompting alone achieves 95% compliance but you need 99.5%, fine-tuning can close that gap.
• Domain-specific language. If the model needs to use specialized terminology (medical, legal, financial) naturally and correctly, fine-tuning on domain text helps.
• Reducing prompt size. If your system prompt has grown to 5,000 tokens of instructions and examples, fine-tuning can internalize those patterns, letting you use shorter prompts and saving per-query costs at scale.
• Specialized tasks at volume. If you are classifying thousands of support tickets per day into 25 categories, a fine-tuned smaller model can outperform a larger model with prompting alone, at a fraction of the per-query cost.

When fine-tuning is a waste of money:

• You have fewer than 100 high-quality training examples. The model will not learn enough.
• You want to add knowledge (product details, documentation). Use RAG instead.
• You have not tried good prompting first. Most people skip this step and regret it.
• Your data changes frequently. Every update requires retraining.

For most startups in Kenya and across Africa, fine-tuning is not where you should be spending your limited budget. Get your prompting right. Build RAG if you need knowledge. Come back to fine-tuning only when you have a validated product at enough scale that the optimization makes financial sense.

Here is a straightforward way to decide which approach to use. Start at the top and work your way down.

Step 1: Try prompting.

Write a clear system prompt. Include 3-5 examples of desired behavior (few-shot prompting). Test it against 20 real-world inputs. If the results are good enough for your use case, stop here. You are done. Most developers never need to go further.

Step 2: Does the model need knowledge it does not have?

If yes, build a RAG pipeline. Embed your documents, store them in a vector database (pgvector on Supabase is the simplest path), and retrieve relevant chunks at query time. Keep your good system prompt from Step 1. RAG adds knowledge; the prompt shapes behavior.

Step 3: Is the model inconsistent despite good prompting and RAG?

If the model keeps drifting from your desired format, tone, or behavior despite clear instructions, and this inconsistency is actually hurting your product (not just mildly annoying), consider fine-tuning. But only if you have at least 200-500 high-quality training examples that demonstrate the behavior you want.

A real example from the Kenyan market:

Imagine you are building a WhatsApp bot for a M-Pesa lending service. The bot needs to answer customer questions about loan terms, check eligibility, and explain repayment schedules.

• Prompting handles: the bot's personality ("friendly, concise, professional"), response format ("short messages under 160 characters for SMS compatibility"), and basic logic ("if the user asks about eligibility, ask for their M-Pesa phone number first").
• RAG handles: the actual loan terms, interest rates, eligibility criteria, and FAQ answers. When these change (and they will), you update the knowledge base without touching the model.
• Fine-tuning would handle: if you needed the bot to write in a very specific Swahili dialect consistently, or to follow a complex multi-step conversation flow that prompting cannot reliably enforce. For most lending bots, you will never reach this point.

The progression is always: prompting first, RAG second, fine-tuning last. Each step adds cost and complexity. Only move to the next step when the current one genuinely is not sufficient.

The best production AI systems do not choose one approach. They layer all three. Understanding how they fit together is what separates prototypes from real products.

The layered architecture:

1. Fine-tuning sets the baseline behavior. The model is trained to follow your output format, use appropriate language, and handle your domain's terminology naturally. This layer is trained once (or occasionally) and reduces the need for verbose system prompts.
2. RAG provides dynamic knowledge. At query time, relevant documents are retrieved and injected into the context. This gives the fine-tuned model access to current, specific information without retraining.
3. Prompting handles the per-request context. The system prompt contains the conversation history, user-specific information, and any task-specific instructions that vary between requests.

This architecture is how companies like Notion, Intercom, and Shopify build their AI features. But here is the thing: most of the value comes from layers 2 and 3 (RAG and prompting). Layer 1 (fine-tuning) is an optimization, not a requirement.

A practical path for African startups:

Month 1: Ship with prompting only. Get real user feedback. Understand what your users actually ask and where the model fails.

Month 2-3: Add RAG for the knowledge gaps. Index your documentation, FAQs, and product data. This usually solves 80% of the remaining quality issues.

Month 6+: If you have thousands of daily queries and consistent quality issues that prompting cannot fix, evaluate fine-tuning. By this point you will have enough real conversation logs to build a quality training dataset.

This path lets you ship fast, learn from real usage, and invest in complexity only when the data justifies it. Do not build a three-layer architecture on day one. You will spend months engineering solutions for problems your users may never have.

Let us put actual numbers on a hypothetical scenario. You are building a customer support bot for a Nairobi-based e-commerce company. The bot handles 500 conversations per day and needs to answer questions about products, shipping, and returns.

Option A: Prompting Only

• System prompt: 1,500 tokens with instructions and examples
• Average conversation: 3 turns, 500 tokens per turn (input + output)
• Using GPT-4o-mini: roughly $0.003 per conversation
• Daily cost: ~$1.50 (about KES 195)
• Monthly cost: ~$45 (about KES 5,850)
• Setup time: 1-2 days
• Problem: the bot cannot answer product-specific questions accurately because it does not know your catalog

Option B: Prompting + RAG

• One-time embedding cost for 500 product pages: ~$0.01
• Vector storage on Supabase free tier: $0
• Per-query embedding: negligible ($0.0001)
• Per-conversation cost (with retrieval): roughly $0.004
• Daily cost: ~$2.00 (about KES 260)
• Monthly cost: ~$60 (about KES 7,800)
• Setup time: 1-2 weeks
• Result: the bot accurately answers product questions, shipping policies, and return procedures

Option C: Prompting + RAG + Fine-tuning

• All costs from Option B, plus:
• Training data curation: 2-4 weeks of work (the biggest real cost)
• Fine-tuning compute: $5-50 depending on model and iterations
• Per-conversation cost: roughly $0.003 (shorter prompts thanks to fine-tuned behavior)
• Monthly cost: ~$45 (about KES 5,850) plus the upfront time investment
• Result: more consistent tone, shorter prompts, slightly lower per-query cost

Notice that the monthly cost difference between Option B and Option C is only about KES 1,950. The fine-tuning itself is cheap. The expensive part is the weeks of human time spent curating training data and iterating on model quality. For a startup doing 500 conversations per day, that time is almost certainly better spent improving the product, fixing bugs, or talking to customers.

Fine-tuning becomes cost-effective at much higher volumes (think 10,000+ conversations per day) where the per-query savings compound into meaningful numbers, and where the consistency improvements directly impact customer satisfaction metrics.