In today’s post I am going to show you how you can use Python and the Keyword Planner tool from Google Ads API to generate keyword ideas and enrich your keyword researches. It works in a very similar way to the normal interface, basically we can input keywords and/or page URLs and it will return keyword suggestions with their search volumes, competence levels and their number of searches in the past 12 months.

From my point of view, this API has some advantages that I would like to highlight:

  1. First, it enables you to retrieve up to 1.000 daily keyword reports for free. You can find some information about its pricing over here.
  2. Second, there are already some Python libraries and scripts that make the script development much easier. Specifically, I made use of the library called google-ads-python and this script to connect with this API.
  3. Third, you can find keywords and their metrics for precise geolocations and specific browser languages.
  4. Fourth, this is just my personal opinion, but I believe that the data must be very accurate and reliable as it comes directly from Google.

The only struggle that I encountered with this API was with the credentials obtention and getting the desired format to authenticate as I think that the available information was a bit confusing. For this reason this is something that I am going to explain step by step so that you are able to go through this smoothly.

In a nutshell, what we are going to do in this post is:

  1. Setting up the credentials to authenticate in Google Ads API.
  2. Getting to understand how the previously mentioned Python script works and how the returned output looks like.
  3. Downloading the keywords and their metrics as an Excel file.
  4. Creating graphs to analyze the trend of the number of searches for each keyword.

Does this sound interesting? So let’s get started then!

1.- Obtaining the credentials and authenticating

1.1.- Installing the library

This is the very first step and the easiest one, we need to run in our terminal the command:

pip install google-ads

1.2.- Setting up a project in Google Cloud Platform

You will need to go to the Google Cloud Platform and create a new Project: https://console.cloud.google.com/project.

When you are creating a new project, you will need to name it and click on Create.

After that you will need to go to the credentials section which is under APIs and Services section and create a new OAuth client ID:

When you are generating your OAuth client ID, select in the application type Desktop App and click on Create.

After creating the OAuth client ID, download the JSON file. This JSON file and some of their variables will be used to create the YAML file that we will use to authenticate in Google Ads API.

Finally, you will have to go to the OAuth consent screen section and add your email address as a test user so that you will be able to use your email address to give consent in the OAuth screen:

1.3.- Getting a MCC account

In order to use the Google Ads API we need to have a Manager account. If you do not have a manager account you can create a new one over here.

Once you have created the MCC account, you will need to go to the API Center section and generate a developer token.

In some cases, if you do not have an active Google Ads account, you will also need to create a test account to make use of the API.

1.4.- Getting the refresh token

The last thing that we need to obtain to create the YAML file is the refresh token. To get this token first I downloaded this Python script and then I run the following command in my terminal:

python /path/to/authenticate_in_desktop_application.py --client_secrets_path=/path/to/secrets.json

Note: the client secrets is the JSON file that we have downloaded previously when we set up the project on Google Cloud Platform.

This command will generate an URL that you will need to open in your browser with a code. Copy and paste the code in your terminal and you will finally get the last ingredient for the YAML file recipe!

1.5.- Writing the YAML file

You can write this file with any code editor as long as you follow this structure:

developer_token: insert here the token from Google Ads API center
client_id: can be obtained in the secrets JSON file
client_secret: can be obtained in the secrets JSON file
refresh_token: this is the token obtained previously

Once you have inserted all the variables, save this file with the .yaml final suffix.

2.- Working with Python

After reading everything related to the credentials, we can start working with Python!

First we will have to use the YAML file which was created previously to authenticate:

from google.ads.googleads.client import GoogleAdsClient
client = GoogleAdsClient.load_from_storage("<path to the yaml file>")

If the authentication is made successfully, then we can run the Python script that defines the function that we will use to extract the keyword ideas. This script is basically the one that can be found here, although I made some subtle changes so that it returns a list with the response instead of printing the keywords with their competition levels, volume of searches and annotations.

UPDATE: this article has been updated on the 14th of August thanks to Alex Papageorgiou’s suggestion. The initial piece of code didn’t return the concept groups that Google Ads offers as annotations such as Branded vs. Non-branded. From now on, you will also be able to retrieve this data by running this piece of code and you will be able to export it as an Excel file as the rest of the article has also been updated to support this new feature.

It is possible if you are using Python 3.6 that you might encounter some issues as the Python library googleads seems to support only the deprecated version of Google Ads v6 when using Python 3.6. In my case, I needed to upgrade the Python version of Jupyter Notebook with the command: conda install ipython jupyter. Once I upgraded the Python version, then I was able to access the annotations.

#!/usr/bin/env python
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""This example generates keyword ideas from a list of seed keywords."""


import argparse
import sys
from google.ads.googleads.client import GoogleAdsClient
from google.ads.googleads.errors import GoogleAdsException

# Location IDs are listed here:
# https://developers.google.com/google-ads/api/reference/data/geotargets
# and they can also be retrieved using the GeoTargetConstantService as shown
# here: https://developers.google.com/google-ads/api/docs/targeting/location-targeting
_DEFAULT_LOCATION_IDS = ["1023191"]  # location ID for New York, NY
# A language criterion ID. For example, specify 1000 for English. For more
# information on determining this value, see the below link:
# https://developers.google.com/google-ads/api/reference/data/codes-formats#expandable-7
_DEFAULT_LANGUAGE_ID = "1000"  # language ID for English


# [START generate_keyword_ideas]
def main(
    client, customer_id, location_ids, language_id, keyword_texts, page_url
):
    keyword_plan_idea_service = client.get_service("KeywordPlanIdeaService")
    keyword_competition_level_enum = client.get_type(
        "KeywordPlanCompetitionLevelEnum"
    ).KeywordPlanCompetitionLevel
    keyword_plan_network = client.get_type(
        "KeywordPlanNetworkEnum"
    ).KeywordPlanNetwork.GOOGLE_SEARCH_AND_PARTNERS
    location_rns = _map_locations_ids_to_resource_names(client, location_ids)
    language_rn = client.get_service(
        "LanguageConstantService"
    ).language_constant_path(language_id)
    
    keyword_annotation = client.enums.KeywordPlanKeywordAnnotationEnum
    
    # Either keywords or a page_url are required to generate keyword ideas
    # so this raises an error if neither are provided.
    if not (keyword_texts or page_url):
        raise ValueError(
            "At least one of keywords or page URL is required, "
            "but neither was specified."
        )
    
    
    
    # Only one of the fields "url_seed", "keyword_seed", or
    # "keyword_and_url_seed" can be set on the request, depending on whether
    # keywords, a page_url or both were passed to this function.
    request = client.get_type("GenerateKeywordIdeasRequest")
    request.customer_id = customer_id
    request.language = language_rn
    request.geo_target_constants = location_rns
    request.include_adult_keywords = False
    request.keyword_plan_network = keyword_plan_network
    request.keyword_annotation = keyword_annotation
    
    
    
    # To generate keyword ideas with only a page_url and no keywords we need
    # to initialize a UrlSeed object with the page_url as the "url" field.
    if not keyword_texts and page_url:
        request.url_seed.url = url_seed

    # To generate keyword ideas with only a list of keywords and no page_url
    # we need to initialize a KeywordSeed object and set the "keywords" field
    # to be a list of StringValue objects.
    if keyword_texts and not page_url:
        request.keyword_seed.keywords.extend(keyword_texts)

    # To generate keyword ideas using both a list of keywords and a page_url we
    # need to initialize a KeywordAndUrlSeed object, setting both the "url" and
    # "keywords" fields.
    if keyword_texts and page_url:
        request.keyword_and_url_seed.url = page_url
        request.keyword_and_url_seed.keywords.extend(keyword_texts)

    keyword_ideas = keyword_plan_idea_service.generate_keyword_ideas(
        request=request
    )
    
    list_keywords = []
    for idea in keyword_ideas:
        competition_value = idea.keyword_idea_metrics.competition.name
        list_keywords.append(idea)
    
    return list_keywords

def map_keywords_to_string_values(client, keyword_texts):
    keyword_protos = []
    for keyword in keyword_texts:
        string_val = client.get_type("StringValue")
        string_val.value = keyword
        keyword_protos.append(string_val)
    return keyword_protos


def _map_locations_ids_to_resource_names(client, location_ids):
    """Converts a list of location IDs to resource names.
    Args:
        client: an initialized GoogleAdsClient instance.
        location_ids: a list of location ID strings.
    Returns:
        a list of resource name strings using the given location IDs.
    """
    build_resource_name = client.get_service(
        "GeoTargetConstantService"
    ).geo_target_constant_path
    return [build_resource_name(location_id) for location_id in location_ids]


if __name__ == "__main__":
    # GoogleAdsClient will read the google-ads.yaml configuration file in the
    # home directory if none is specified.
    googleads_client = GoogleAdsClient.load_from_storage("google-ads.yaml")

    parser = argparse.ArgumentParser(
        description="Generates keyword ideas from a list of seed keywords."
    )

    # The following argument(s) should be provided to run the example.
    parser.add_argument(
        "-c",
        "--customer_id",
        type=str,
        required=True,
        help="The Google Ads customer ID.",
    )
    parser.add_argument(
        "-k",
        "--keyword_texts",
        nargs="+",
        type=str,
        required=False,
        default=[],
        help="Space-delimited list of starter keywords",
    )
    # To determine the appropriate location IDs, see:
    # https://developers.google.com/google-ads/api/reference/data/geotargets
    parser.add_argument(
        "-l",
        "--location_ids",
        nargs="+",
        type=str,
        required=False,
        default=_DEFAULT_LOCATION_IDS,
        help="Space-delimited list of location criteria IDs",
    )
    # To determine the appropriate language ID, see:
    # https://developers.google.com/google-ads/api/reference/data/codes-formats#expandable-7
    parser.add_argument(
        "-i",
        "--language_id",
        type=str,
        required=False,
        default=_DEFAULT_LANGUAGE_ID,
        help="The language criterion ID.",
    )
    # Optional: Specify a URL string related to your business to generate ideas.
    parser.add_argument(
        "-p",
        "--page_url",
        type=str,
        required=False,
        help="A URL string related to your business",
    )

    args = parser.parse_args()

    try:
        main(
            googleads_client,
            args.customer_id,
            args.location_ids,
            args.language_id,
            args.keyword_texts,
            args.page_url,
        )
    except GoogleAdsException as ex:
        print(
            f'Request with ID "{ex.request_id}" failed with status '
            f'"{ex.error.code().name}" and includes the following errors:'
        )
        print(f'\tError with message "{error.message}".')
        if error.location:
            for field_path_element in error.location.field_path_elements:
                print(f"\t\tOn field: {field_path_element.field_name}")
        sys.exit(1)

After defining the function, we can run our first keyword suggestion call. However, first I wanted to clarify how the arguments that are needed to run this function work:

  1. Client: we will use the authenticated client that was generated previously (Variable called “client”).
  2. Customer ID: this can be found on the Google Ads account with this format: XXX-XXX-XXXX. However, once you insert it the Customer ID as an argument, you need to remove the hyphens.
  3. Location ID: you can find the location IDS in the CSV file that is attached to this page. You need to insert this argument with a list.
  4. Language ID: you can find the language IDs on this page. English has the language ID: 1000
  5. Keyword Texts: you can insert a seed list with keywords that is going to be used to return the suggested keywords and their metrics.
  6. Page URL: a page URL can also be used as a seed to suggest the new keyword ideas.

After the previous clarification, let’s for instance get the suggested keywords for the seed keyword “mortgage” in the United States and the Spanish language:

list_keywords = main(client, "<your customer id>", ["2840"], "1000", ["mortgage"], None)

The keywords and their metrics will be stored in the list called list_keywords.

3.- Exporting as an Excel file

In order to export the list as an Excel file first we will need to format it properly. We will use a for loop to iterate over the elements from the first list and append them to a new list with the desired format:

list_to_excel = []
for x in range(len(list_keywords)):
    list_months = []
    list_searches = []
    list_annotations = []
    for y in list_keywords[x].keyword_idea_metrics.monthly_search_volumes:
        list_months.append(str(y.month)[12::] + " - " + str(y.year))
        list_searches.append(y.monthly_searches)
        
    for y in list_keywords[x].keyword_annotations.concepts:
        list_annotations.append(y.concept_group.name)
        
        
    list_to_excel.append([list_keywords[x].text, list_keywords[x].keyword_idea_metrics.avg_monthly_searches, str(list_keywords[x].keyword_idea_metrics.competition)[28::], list_keywords[x].keyword_idea_metrics.competition_index, list_searches, list_months, list_annotations ])

After creating such a formatted list, we will only need to export it as an Excel file by using Pandas:

import pandas as pd 

pd.DataFrame(list_to_excel, columns = ["Keyword", "Average Searches", "Competition Level", "Competition Index", "Searches Past Months", "Past Months", "List Annotations"]).to_excel('output.xlsx', header=True, index=False)

And this is how the Excel file would look like:

4.- Analyzing keyword trends

Something that I really like about this API is that we get not only the average volume of searches but the number of searches in the past 12 months. I believe that this can be super useful to detect seasonal behaviors and plan accordingly. With Matplotlib we can draw some graphs that can help us to spot these seasonal behaviors:

import matplotlib.pyplot as plt
import numpy as np
 
for x in range(len(list_to_excel)):
    values=list_to_excel[x][4]
    
    # use the plot function
    plt.plot(values)
    plt.title(list_to_excel[x][0])
    plt.xticks([0,1,2,3,4,5,6,7,8,9,10,11],list_to_excel[0][5], rotation="vertical") 


    # show the graph
    plt.show()

This will create a timeline graph for each keyword:

It is interesting to see how much the volume of searches dropped for “mortgage rates” after the covid crisis got started in April 2020, isn’t it!?

We can also create a heatmap that will enable us to compare the keywords based on their volume of searches in the past 12 months. However, we will have one constraint as we cannot generate an image that exceeds a determined number of pixels. In my case what I will do is separating the keywords in two different chunks and create two separate heatmaps:

import numpy as np
import matplotlib.pyplot as mpl
import matplotlib.cm as cmap

list_trends = []
list_kws = []
for x in range(len(list_to_excel)//2):
    list_integers = list_to_excel[x][4]
    list_integers = [float(i) for i in list_integers]
    
    list_trends.append(list_integers)
    list_kws.append(list_to_excel[x][0])


heatmap_values = np.array(list_trends)


a = heatmap_values
plt.figure(figsize = (20,len(list_kws)))
plt.yticks([i for i in range (len(list_kws))],list_kws)
plt.xticks([0,1,2,3,4,5,6,7,8,9,10,11],list_to_excel[0][5], rotation="vertical") 
plt.imshow(a, cmap="jet", interpolation='nearest', aspect='auto')


for i in range(len(list_kws)):
    for j in range(len(list_to_excel[0][5])):
        plt.text(j, i, int(heatmap_values[i, j]),
                 horizontalalignment='center',
                 verticalalignment='center', size = "large", color = "w", weight = "bold")


plt.show()

This has created a graph which comprises half of the total number of keywords which looks like:

So this is all folks! I hope that you found this article interesting!

Which libraries do you need?

You will need GoogleAdsClient, Pandas, Matplotlib and Numpy.

What will you learn in this post?

You will learn how to find new keyword ideas and represent them with Google Keyword Planner API.

How long will it take?

It will depend on the number of seed keywords that you would insert.