In the past week Google launched a new tool called Destination Insights with Google which is meant to help the travel industry to reach as many costumers as possible. With this tool, you can monitor travel trends and see which are the hottest cities in most of the countries, which are the nationalities that are the most interested in that destination and the comparison between the current volume of searches for that country and the volume of searches from the previous year.

From my point of view, this tool is super useful as now with covid the travel industry is going through a very bad time and segmentation becomes more important than ever. In short, there are many fewer people willing or able to travel so you need to find the ones who are still up to it and with this tool you can detect these opportunities as you would get to know:

  1. Which are the destinations that are worth paying attention based on the current volume of searches.
  2. From where these searches are made to improve the segmentation.

The concept in reality is very simple, for instance, if I know that in France, most of the searches are dedicated to Paris, Nice, Toulouse, Lyon and Marseille and the demand is mainly domestic but people from the UK, the USA and Spain are also quite interested in France, I will allocate most of my efforts in creating campaigns for these cities and for this targeted countries. At the same time, I can get to know that Lorient, Rouen and Rennes are experiencing an important demand growth, so I will keep a close eye on them and I would even try to create and test some campaigns for these cities.

Basically, following this logic I would be able to create a segmented demand driven strategy which will enable me to allocate my time and resources correctly. Does it sound good?

To me it does sound good, but I thought that it would even sound better if we could scrape this page and retrieve the data about all the countries to perform more extensive analysis or create alerts to notify us of new market opportunities or the weekly status for each of the most important markets for your business!

And after this thought, this is when the fun got started for me. On this post what we are going to do is:

  1. Scraping this page with Python and Selenium and retrieving the top demand by destination city, top growth by destination city and top demand by destination city data for each country.
  2. We will export this data as an Excel file where the data for each of the countries will be placed in a different tab.

You can download the Excel file on the button below if you are interested in analyzing it in-depth.

In the Excel file you will fin the Top demand by source country, top growth by destination city and top demand by destination city data for each of the countries in a different tab. It looks like:

1.- Scraping the data with Selenium

First, what we are going to do is scraping the data with Selenium. For that, we will need to set up the proper driver and install some modules that we will use later on.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Chrome(ChromeDriverManager().install())

In order to iterate through the countries, we will need to have a list with all those countries. I extracted such a list from the JavaScript file where they are all compiled: https://destinationinsights.withgoogle.com/static/compiled/index.min.js?cachebust=cedcdda and I created a list which contains dictionaries with the country names and their IDs.

 listcountries = [{"name":"Worldwide","id":"WW"},{"name":"Afghanistan","id":"AF"},{"name":"Albania","id":"AL"},{"name":"Algeria","id":"DZ"},{"name":"\u00c5land Islands","id":"AX"},{"name":"American Samoa","id":"AS"},{"name":"Andorra","id":"AD"},{"name":"Angola","id":"AO"},{"name":"Anguilla","id":"AI"},{"name":"Antarctica","id":"AQ"},{"name":"Antigua and Barbuda","id":"AG"},{"name":"Argentina","id":"AR"},{"name":"Armenia","id":"AM"},{"name":"Aruba","id":"AW"},{"name":"Australia","id":"AU"}, {"name":"Austria","id":"AT"},{"name":"Azerbaijan","id":"AZ"},{"name":"Bahrain","id":"BH"},{"name":"Bangladesh","id":"BD"},{"name":"Barbados","id":"BB"},{"name":"Belarus","id":"BY"},{"name":"Belgium","id":"BE"},{"name":"Belize","id":"BZ"},{"name":"Benin","id":"BJ"},{"name":"Bermuda","id":"BM"},{"name":"Bhutan","id":"BT"},{"name":"Bolivia","id":"BO"},{"name":"Bosnia and Herzegovina","id":"BA"},{"name":"Botswana","id":"BW"},{"name":"Bouvet Island","id":"BV"},{"name":"Brazil","id":"BR"},{"name":"British Indian Ocean Territory","id":"IO"},{"name":"British Virgin Islands", "id":"VG"},{"name":"Brunei","id":"BN"},{"name":"Bulgaria","id":"BG"},{"name":"Burkina Faso","id":"BF"},{"name":"Burundi","id":"BI"},{"name":"Cambodia","id":"KH"},{"name":"Cameroon","id":"CM"},{"name":"Canada","id":"CA"},{"name":"Cape Verde","id":"CV"},{"name":"Caribbean Netherlands","id":"BQ"},{"name":"Cayman Islands","id":"KY"},{"name":"Central African Republic","id":"CF"},{"name":"Chad","id":"TD"},{"name":"Chile","id":"CL"},{"name":"China","id":"CN"},{"name":"Christmas Island","id":"CX"},{"name":"Cocos (Keeling) Islands","id":"CC"},{"name":"Colombia","id":"CO"},{"name":"Comoros", "id":"KM"},{"name":"Cook Islands","id":"CK"},{"name":"Costa Rica","id":"CR"},{"name":"C\u00f4te d'Ivoire","id":"CI"},{"name":"Croatia","id":"HR"},{"name":"Cuba","id":"CU"},{"name":"Cura\u00e7ao","id":"CW"},{"name":"Cyprus","id":"CY"},{"name":"Czechia","id":"CZ"},{"name":"Democratic Republic of the Congo","id":"CD"},{"name":"Denmark","id":"DK"},{"name":"Djibouti","id":"DJ"},{"name":"Dominica","id":"DM"},{"name":"Dominican Republic","id":"DO"},{"name":"Ecuador","id":"EC"},{"name":"Egypt","id":"EG"},{"name":"El Salvador","id":"SV"},{"name":"Equatorial Guinea","id":"GQ"}, {"name":"Eritrea","id":"ER"},{"name":"Estonia","id":"EE"},{"name":"Eswatini","id":"SZ"},{"name":"Ethiopia","id":"ET"},{"name":"Falkland Islands (Islas Malvinas)","id":"FK"},{"name":"Faroe Islands","id":"FO"},{"name":"Federated States of Micronesia","id":"FM"},{"name":"Fiji","id":"FJ"},{"name":"Finland","id":"FI"},{"name":"France","id":"FR"},{"name":"French Guiana","id":"GF"},{"name":"French Polynesia","id":"PF"},{"name":"French Southern Territories","id":"TF"},{"name":"Gabon","id":"GA"},{"name":"Georgia","id":"GE"},{"name":"Germany","id":"DE"}, {"name":"Ghana","id":"GH"},{"name":"Gibraltar","id":"GI"},{"name":"Greece","id":"GR"},{"name":"Greenland","id":"GL"},{"name":"Grenada","id":"GD"},{"name":"Guadeloupe","id":"GP"},{"name":"Guam","id":"GU"},{"name":"Guatemala","id":"GT"},{"name":"Guernsey","id":"GG"},{"name":"Guinea-Bissau","id":"GW"},{"name":"Guinea","id":"GN"},{"name":"Guyana","id":"GY"},{"name":"Haiti","id":"HT"},{"name":"Honduras","id":"HN"},{"name":"Hong Kong","id":"HK"},{"name":"Hungary","id":"HU"},{"name":"Iceland","id":"IS"},{"name":"India","id":"IN"},{"name":"Indonesia","id":"ID"},{"name":"Iran","id":"IR"}, {"name":"Iraq","id":"IQ"},{"name":"Ireland","id":"IE"},{"name":"Isle of Man","id":"IM"},{"name":"Israel","id":"IL"},{"name":"Italy","id":"IT"},{"name":"Jamaica","id":"JM"},{"name":"Japan","id":"JP"},{"name":"Jersey","id":"JE"},{"name":"Jordan","id":"JO"},{"name":"Kazakhstan","id":"KZ"},{"name":"Kenya","id":"KE"},{"name":"Kiribati","id":"KI"},{"name":"Kuwait","id":"KW"},{"name":"Kyrgyzstan","id":"KG"},{"name":"Laos","id":"LA"},{"name":"Latvia","id":"LV"},{"name":"Lebanon","id":"LB"},{"name":"Lesotho","id":"LS"},{"name":"Liberia","id":"LR"},{"name":"Libya","id":"LY"},{"name":"Liechtenstein", "id":"LI"},{"name":"Lithuania","id":"LT"},{"name":"Luxembourg","id":"LU"},{"name":"Macao","id":"MO"},{"name":"Madagascar","id":"MG"},{"name":"Malawi","id":"MW"},{"name":"Malaysia","id":"MY"},{"name":"Maldives","id":"MV"},{"name":"Mali","id":"ML"},{"name":"Malta","id":"MT"},{"name":"Marshall Islands","id":"MH"},{"name":"Martinique","id":"MQ"},{"name":"Mauritania","id":"MR"},{"name":"Mauritius","id":"MU"},{"name":"Mayotte","id":"YT"},{"name":"Mexico","id":"MX"},{"name":"Moldova","id":"MD"},{"name":"Monaco","id":"MC"},{"name":"Mongolia", "id":"MN"},{"name":"Montenegro","id":"ME"},{"name":"Montserrat","id":"MS"},{"name":"Morocco","id":"MA"},{"name":"Mozambique","id":"MZ"},{"name":"Myanmar (Burma)","id":"MM"},{"name":"Namibia","id":"NA"},{"name":"Nauru","id":"NR"},{"name":"Nepal","id":"NP"},{"name":"Netherlands","id":"NL"},{"name":"New Caledonia","id":"NC"},{"name":"New Zealand","id":"NZ"},{"name":"Nicaragua","id":"NI"},{"name":"Niger","id":"NE"},{"name":"Nigeria","id":"NG"},{"name":"Niue","id":"NU"},{"name":"Norfolk Island","id":"NF"},{"name":"North Korea","id":"KP"},{"name":"North Macedonia", "id":"MK"},{"name":"Northern Mariana Islands","id":"MP"},{"name":"Norway","id":"NO"},{"name":"Oman","id":"OM"},{"name":"Pakistan","id":"PK"},{"name":"Palau","id":"PW"},{"name":"Palestinian Territory","id":"PS"},{"name":"Panama","id":"PA"},{"name":"Papua New Guinea","id":"PG"},{"name":"Paraguay","id":"PY"},{"name":"Peru","id":"PE"},{"name":"Philippines","id":"PH"},{"name":"Pitcairn","id":"PN"},{"name":"Poland","id":"PL"},{"name":"Portugal","id":"PT"},{"name":"Puerto Rico","id":"PR"},{"name":"Qatar","id":"QA"},{"name":"Republic of the Congo","id":"CG"},{"name":"R\u00e9union", "id":"RE"},{"name":"Romania","id":"RO"},{"name":"Russia","id":"RU"},{"name":"Rwanda","id":"RW"},{"name":"Saint Barthelemy","id":"BL"},{"name":"Saint Helena, Ascension and Tristan da Cunha","id":"SH"},{"name":"Saint Kitts and Nevis","id":"KN"},{"name":"Saint Lucia","id":"LC"},{"name":"Saint Martin","id":"MF"},{"name":"Saint Pierre and Miquelon","id":"PM"},{"name":"Saint Vincent and the Grenadines","id":"VC"},{"name":"Samoa","id":"WS"},{"name":"San Marino","id":"SM"},{"name":"S\u00e3o Tom\u00e9 and Pr\u00edncipe", "id":"ST"},{"name":"Saudi Arabia","id":"SA"},{"name":"Senegal","id":"SN"},{"name":"Serbia","id":"RS"},{"name":"Seychelles","id":"SC"},{"name":"Sierra Leone","id":"SL"},{"name":"Singapore","id":"SG"},{"name":"Sint Maarten","id":"SX"},{"name":"Slovakia","id":"SK"},{"name":"Slovenia","id":"SI"},{"name":"Solomon Islands","id":"SB"},{"name":"Somalia","id":"SO"},{"name":"South Africa","id":"ZA"},{"name":"South Korea","id":"KR"},{"name":"South Sudan","id":"SS"},{"name":"Spain","id":"ES"},{"name":"Sri Lanka","id":"LK"},{"name":"Sudan","id":"SD"},{"name":"Suriname","id":"SR"},{"name":"Svalbard and Jan Mayen", "id":"SJ"},{"name":"Sweden","id":"SE"},{"name":"Switzerland","id":"CH"},{"name":"Syria","id":"SY"},{"name":"Taiwan","id":"TW"},{"name":"Tajikistan","id":"TJ"},{"name":"Tanzania","id":"TZ"},{"name":"Thailand","id":"TH"},{"name":"The Bahamas","id":"BS"},{"name":"The Gambia","id":"GM"},{"name":"Timor-Leste","id":"TL"},{"name":"Togo","id":"TG"},{"name":"Tokelau","id":"TK"},{"name":"Tonga","id":"TO"},{"name":"Trinidad and Tobago","id":"TT"},{"name":"Tunisia","id":"TN"},{"name":"Turkey","id":"TR"},{"name":"Turkmenistan","id":"TM"},{"name":"Turks and Caicos Islands","id":"TC"}, {"name":"Tuvalu","id":"TV"},{"name":"U.S. Virgin Islands","id":"VI"},{"name":"Uganda","id":"UG"},{"name":"Ukraine","id":"UA"},{"name":"United Arab Emirates","id":"AE"},{"name":"United Kingdom","id":"GB"},{"name":"United States","id":"US"},{"name":"Uruguay","id":"UY"},{"name":"Uzbekistan","id":"UZ"},{"name":"Vanuatu","id":"VU"},{"name":"Vatican","id":"VA"},{"name":"Venezuela","id":"VE"},{"name":"Vietnam","id":"VN"},{"name":"Wallis and Futuna","id":"WF"},{"name":"Yemen","id":"YE"},{"name":"Zambia","id":"ZM"},{"name":"Zimbabwe","id":"ZW"}] 

Now that we have readied the list of countries, installed the required modules and set up the driver, we will start the scraping process. We will access https://destinationinsights.withgoogle.com/ and navigate through it. The first interaction with the page will be done to accept the cookies pop up. We will use Xpath to select many elements.

 driver.get('https://destinationinsights.withgoogle.com/') cookies = driver.find_element_by_xpath('/html/body/div[1]/div/span[2]/a[2]') cookies.click() 

Next step will be ticking the trip type box as Domestic and International trip to get the demand for both types.

 time.sleep(10) input = driver.find_element_by_xpath('/html/body/div[1]/div[4]/div/div/md-content[2]/md-select') input.click() time.sleep(10) choice = driver.find_element_by_xpath('/html/body/div[5]/md-select-menu/md-content/md-option[2]') choice.click() choice.send_keys(Keys.TAB); 

In the next step we will iterate through the countries list and input each of the countries in the drop down menu for the destination country. After that, we will find the elements of our interest and make them an object that we can parse with Beautiful Soup. In this case, to find the right elements we will not use the Xpath but the CSS class.

 counter = 1 listtotal = [] for iteration in listcountries: print("Scraping results for " + iteration["name"]) time.sleep(10) input = driver.find_element_by_xpath('/html/body/div[1]/div[4]/div/div/div[1]/md-content[1]/md-select') input.click() time.sleep(10) choice = driver.find_element_by_xpath('/html/body/div[6]/md-select-menu/md-content/md-option[' + str(counter) + ']') choice.click() time.sleep(15) table1 = driver.find_element_by_xpath("/html/body/div[1]/div[10]/div[2]/div[4]") table2 = driver.find_element_by_xpath("/html/body/div[1]/div[13]/div[2]/div[4]") table3 = driver.find_element_by_xpath("/html/body/div[1]/div[17]/div[2]") table1clean = table1.get_attribute("outerHTML") table2clean = table2.get_attribute("outerHTML") table3clean = table3.get_attribute("outerHTML") table1soup = BeautifulSoup(table1clean, "lxml") table2soup = BeautifulSoup(table2clean, "lxml") table3soup = BeautifulSoup(table3clean, "lxml") mydivsregions = table1soup.find_all("span", {"class": "geographic-demand__name ng-binding"}) mydivsregionsnumber = table1soup.find_all("span", {"class": "geographic-demand__total glue-small-text ng-binding"}) growthregions = table2soup.find_all("span", {"class": "geographic-demand__name ng-binding"}) growthregionsnumber = table2soup.find_all("span", {"class": "growth-scale__label glue-small-text ng-binding"}) mydivsregionssources = table3soup.find_all("span", {"class": "sources__name ng-binding"}) mydivsregionssourcesnumbers = table3soup.find_all("span", {"class": "sources__total glue-small-text ng-binding"}) 

Last final step before exporting to Excel, we will put together all the data under a list which will contain dictionaries with the country names as keys and the data lists as values.

 regionsources = [] for x in range(len(mydivsregionssources)): regionsources.append([mydivsregionssources[x].text,int(mydivsregionssourcesnumbers[x].text)]) growth = [] for x in range(len(growthregions)): growth.append([growthregions[x].text,growthregionsnumber[x].text]) regions = [] for x in range(len(mydivsregions)): regions.append([mydivsregions[x].text,int(mydivsregionsnumber[x].text)]) dictionarytotal = {iteration["name"] : [regionsources, growth, regions]} listtotal.append(dictionarytotal) counter = counter + 1 

This script will be running for around one hour to scrape the data for each of the countries and everything will be stored in the list called “listtotal”.

2.- Importing as an Excel file

Now, we will use openpyxl to create an Excel file, a tab for each of the countries and put down all the data that we have scraped. We will use the name of the countries for the tab names although in some cases we will need to cut down the number of characters as Excel only allows up to 31 characters in the tab names.

 import openpyxl filepath = "<yournamefile.xlsx>" wb = openpyxl.Workbook() wb.save(filepath) for x in listtotal: for k,v in x.items(): key_dict = k value_dict = v ws1 = wb.create_sheet("Sheet_A") if len(str(k)) < 31: ws1.title = str(k) ws = wb[str(k)] else: ws1.title = str(k)[0:30] ws = wb[str(k)[0:30]] counter = 1 for y in value_dict: counter2 = 2 ws.cell(row=1, column=1).value = "TOP DEMAND BY SOURCE COUNTRY" ws.cell(row=1, column=2).value = "Interest" ws.cell(row=1, column=4).value = "TOP GROWTH BY DESTINATION CITY" ws.cell(row=1, column=5).value = "Interest" ws.cell(row=1, column=7).value = "TOP DEMAND BY DESTINATION CITY" ws.cell(row=1, column=8).value = "Interest" for z in y: ws.cell(row=counter2, column=counter).value = z[0] ws.cell(row=counter2, column=counter+1).value = z[1] counter2 = counter2 + 1 counter = counter + 3 del wb["Sheet"] wb.save(filepath) wb.close() 

That is all folks! After replicating these steps you should be able to get an Excel file like the one that you can download above! If you would like to create an alerting system for this data with Python, you can get some inspiration on my previous article where I explain how you can create an alerting system for the newest Google Search Console crawl report.