How to fine-tune a GPT-3 model using Python with your own data for improved performance

Cover image

I'm constantly looking for ways to automate the work with support requests. An idea has been to fine-tune a GPT-3 model to answer common support-related questions.

Here's how you can use Python to fine-tune a GPT-3 model with your own data for improved performance.

Prefer video over text?

Check out my free video course on fine-tuning a GPT-3 model

If you prefer learning through videos, I've got you covered!

I've created a free video course that teaches you how to fine-tune a GPT-3 model with your own data. The course covers everything in this blog post and more, with step-by-step instructions.

You can access the course for free:

Access Course Now

Happy learning!

Here's what we'll use:

1. OpenAI API 🤖

2. Python 🐍

Here are the steps:

1. Get OpenAI API key

2. Create training data

3. Check the training data

4. Upload training data

5. Fine-tune model

6. Check fine-tuning progress

7. Save fine-tuned model

8. Test the new model on a new prompt


This guide walks you through fine-tuning a GPT-3 model in Python, shown in a Jupyter notebook. If you're looking for the steps of fine-tuning right in a terminal, OpenAI has a great guide for fine-tuning in your terminal.

1. Get OpenAI API key

Before we go ahead and fine-tune a model, let's get the OpenAI credentials needed for the API calls.

Go to, log in and click on your avatar and View API keys:

Open AI API keys

Then create a new secret key and save it for the request:

Create Open AI API key

Now we have all the credentials needed to make an API request.

2. Create training data

The next step is to create training data to teach GPT-3 what you'd like to say. The data need to be a JSONL document with a new prompt and the ideal generated text:

{"prompt": "<question>", "completion": "<ideal answer>"}
{"prompt": "<question>", "completion": "<ideal answer>"}
{"prompt": "<question>", "completion": "<ideal answer>"}

Let's start by importing the libraries needed:

import json
import openai

Then add your API key from the previous step:

openai.api_key = api_key

Now create a regular dict with the training data. For this guide, I'll add some support questions I've recently received:

training_data = [{
    "prompt": "Where is the billing ->",
    "completion": " You find the billing in the left-hand side menu.\n"
    "prompt":"How do I upgrade my account ->",
    "completion": " Visit you user settings in the left-hand side menu, then click 'upgrade account' button at the top.\n"

Make sure to end each prompt with a suffix. According to the OpenAI API reference, you can use ->.

Also, make sure to end each completion with a suffix as well; I'm using .\n.

The next step is to convert the dict to a proper JSONL file. JSONL file is a newline-delimited JSON file, so we'll add a \n at the end of each object:

file_name = "training_data.jsonl"

with open(file_name, "w") as output_file:
 for entry in training_data:
  json.dump(entry, output_file)

Now you have the training data as a JSONL file, let's check the training data before starting the fine-tuning.

3. Check the training data

We can check the training data using a CLI data preparation tool provided by OpenAI. It gives you suggestions about how you can reformat the training data.

Let's try it out with our training data. Run this line in Jupyter notebook:

!openai tools fine_tunes.prepare_data -f training_data.jsonl

You'll get suggestions similar to this:

Training data suggestion

I'm only using a couple of prompts for this guide, but the suggestions mention that the recommendations are to have at least a few hundred examples.

You'll also see the approximate time depending on how many jobs are ahead of you in the queue:

Time per job in queue

We're ready to upload the training data in the next section.

4. Upload training data

Now that you checked the improvement suggestions, let's upload the training data to OpenAI:

upload_response = openai.File.create(
  file=open(file_name, "rb"),
file_id =

If you check the response, you'll see the file id which we'll need in the next step when we're training the model:

Uploaded file data

Use this file id in the next step, where we'll fine-tune a model.

5. Fine-tune model

Alrighty, we have the prepared training data, uploaded it, and now we're finally ready to fine-tune the model.

Start the fine-tuning by running this command:

fine_tune_response = openai.FineTune.create(training_file=file_id)

The default model is Curie. But if you'd like to use DaVinci instead, then add it as a base model to fine-tune like this:

openai.FineTune.create(training_file=file_id, model="davinci")

The first response will look something like this:

Fine-tuning status

6. Check fine-tuning progress

You can use two openai functions to check the progress of your fine-tuning.

Option 1: List events

You can use openai.FineTune.list_events() and pass in the fine_tune_response id to list all the current events:

fine_tune_events = openai.FineTune.list_events(

This will give you a list of all the events related to the fine-tuning job you're running:

List fine-tuning events

Option 2: Retrieve fine-tuning job

You can also use openai.FineTune.retrieve( to retrieve a specific fine-tune:

retrieve_response = openai.FineTune.retrieve(

This will give you a JSON object with information about a specific fine-tune job:

Retrieve fine-tune job

When your fine-tuning job is first in line, your fine-tuning event starts:

Fine-tuning started

7. Save fine-tuned model

Troubleshooting fine_tuned_model as null

During the fine-tuning process, the fine_tuned_model key may not be immediately available in the fine_tune_response object returned by openai.FineTune.create().

To check the status of your fine-tuning process, you can call the openai.FineTune.retrieve() function and pass in the This function will return a JSON object with information about the training status, such as the current epoch, the current batch, the training loss, and the validation loss.

After the fine-tuning process is complete, you can check the status of all your fine-tuned models by calling openai.FineTune.list(). This will list all of your fine-tunes and their current status.

Once the fine-tuning process is complete, you can retrieve the fine_tuned_model key by calling the openai.FineTune.retrieve() function again and passing in the This will return a JSON object with the key fine_tuned_model and the ID of the fine-tuned model that you can use for further completions.

Once the fine-tuning is finished, go ahead and save the name of the fine-tuned model:

# Option 1 | if response.fine_tuned_model != null
fine_tuned_model = fine_tune_response.fine_tuned_model

# Option 2 | if response.fine_tuned_model == null
retrieve_response = openai.FineTune.retrieve(
fine_tuned_model = retrieve_response.fine_tuned_model

It will look something like this:

Fine-tuned model

We now have a fine-tuned model. Let's try it out on a new prompt in the next section.

8. Test the new model on a new prompt

We're finally at the last step, where we'll try our fine-tuned model on a new prompt.

I only ran my fine-tuning on 2 prompts, so I'm not expecting a super-accurate completion.

Start by creating a new prompt.

Remember to end the prompt with the same suffix as we used in the training data; ->:

new_prompt = "How do I find my billing? ->"

Next, run the completion with your fine-tuned model:

answer = openai.Completion.create(

Here's what I got:

Fine-tuned completion

Great! Now you have a fine-tuned GPT-3 model ready for future prompts.

Here's a summary of what we did

1. Got our OpenAI API key

2. Created training data

3. Checked the training data

4. Uploaded training data

5. Fine-tuned a GPT-3 model

6. Check the progress of the fine-tuning job

7. Saved the fine-tuned model

8. Tested the new model on a new prompt

9. Improvements

Expand the training data
The training data analysis by OpenAI in step 2 of this guide suggests expanding the training data amount. I only used 2 prompts in this guide. The suggestion says at least a few hundred examples/prompts.

FAQ might not be the best use case
FAQ-related questions might not be the best use case for fine-tuning. If you'd like to automate your support questions, a question-answering approach might be better suited than fine-tuning a GPT-3 model.

Next steps

1. Repo with source code
Here is the repo with a Jupyter notebook with all the source code if you'd like to implement this on your own ⬇️

2. Prefer video over text?
If you prefer learning through videos, I've got you covered!
Check out my free video course on fine-tuning a GPT-3 model:

3. Do you need help with fine-tuning your own GPT-3 model? Or do you have other questions?
I'm happy to help, don't hesitate to reach out ➡️

Or shoot me a DM on Twitter @norahsakal

Need tailored AI solutions?

I provide one-on-one collaboration and custom AI services for businesses.

Let's find the perfect solution for your challenges: Consulting Services

Get notified about new posts
Connect with me
Need a custom solution for your business or idea?
Get a tailored blog post on ChatGPT, GPT-3 and GPT-4 to tackle your specific problem.