How to scrape SERP snippets with Python coding

Obtaining visibility in search search snippets is an excellent way to boost CTR and increase organic site traffic. As a quick refresher, below is a visual example of a SERP snippet for the query “paid search vs. seo”:

Snippets take up serious SERP real estate, often driving more traffic than a #1 ranking. There are several great hacks on how to optimize for Google snippets, but not many hacks on how to find snippet opportunities. The process is very manual. It usually requires the following:

• Search for keywords individually on Google
• Record the SERP snippets per keyword
• Identify which existing content/pages can break into those snippets
• Edit the content

Step 3 is the ultimate takeaway, but about 90% of the work goes into steps 1 and 2. However, what if you could automate finding the snippets and preserve your brainpower for winning the snippets? That’s where Python comes in. As a novice Python coder (If I can even call myself that), I’ve quickly realized that Python can significantly reduce the time for SERP research projects. Snippet research is one of those instances.

Rather than individually searching/recording Google snippets for hundreds of keywords, Python can do the grunt work so you can be more efficient with the time spent winning that valuable organic real estate.

First, we must list all the keywords we want to search in a Text file. We choose Text files because they’re minimal and easy for Python to handle. Save the file somewhere easy to access, as Python will need to access through your computer.

*WARNING: IF YOU ARE SAVING AS A RICH TEXT DOCUMENT ON MACBOOK (RFT), MAKE SURE TO EDIT THE FILE SO IT IS A .TXT. WE WANT THE SIMPLEST FILE POSSIBLE SO PYTHON DOESN’T WASTE ITS TIME READING HEADER ELEMENTS WITHIN A COMPLICATED WORD PROCESSOR.

Line 1: with open(“/Users/Desktop/Text_Doc_Python1.txt”, “r”) as f:

We instruct Python to open the list of queries from the file’s absolute path on your computer. In this example, the file was on my desktop and was housed under the path “/Users/Desktop/”. The file with my keywords was titled “Text_Doc_Python1.txt”. Hence, our open instruction is to open  “/Users/Desktop/Text_Doc_Python1.txt” In that same line, we’re giving Python read permissions (‘r’). So now Python can both open and read the file. Lastly, I will define this whole operation of opening and reading the file as “f”. This way,  I can refer back to it via a single letter rather than typing out that long file path whenever I want to use it.

Line 2: queries = [line.strip() for line in f]

Our keyword list might not be perfect, and there might be some stray spaces following our term. To account for this, we will use line.strip() to remove any stray spaces from before or after the KW; this ensures that the term you think is getting Googled is getting Googled. We will define these cleaned-up line items from our handy ‘f’ document as “queries.” This will represent each query we’re going to get a snippet from.

Line 3: print(queries)
For good measure, we’ll also print the queries in our Console to preview what exactly Python will be running through Google search.

For the selenium webdriver module to work, Python needs an application to run the searches through. To get this, you will need to install ChromeDriver from https://chromedriver.chromium.org.

Once you have chromium installed, you will need to find where it’s located on your computer.

Much like we did with our open (“/Users/Desktop/Text_Doc_Python1.txt” command, where we needed to tell Python where the file was, what we’re accessing, and what to do with it, we’re doing the same with Chromedriver. We’re telling Python where the browser is and where these searches will be performed.

In this example, the file was found in /Users/Downloads/ and the file was called “chromedriver 3”. Ergo, we are defining our chromedriver variable as “/Users/Downloads/chromedriver 3”

While we’ve defined which application we’ll use to search things, we haven’t yet set a command to search things. driver = webdriver.Chrome(chromedriver) commands Python to automate a search process via webdriver using the chromedriver browser that we just assigned above. We’ve taken this process of automating a search and defined it as driver.

So to recap, webdriver is our automation and chromedriver is our Google Chrome application where searches will be automated.

Line 1: for item in queries:

In our first line, we’re now establishing an important Python function called a “ for loop,” which essentially repeats an operation for us. Here, we’re simply instructing our program that we’re about to perform an operation for each query (or item ) from our queries variable (our full list of queries we defined earlier).

Line 2: updated_query = item.replace(” “, “+”)

For each query in our doc, we’re now mutating each one into a Google search URL. We know that when you search for something in Google, the URL we get back follows a formula:
Query: “test query”
Google URL: https://www.google.com/search?q=test+query

Easy enough! First, we must replace each space with a “+” sign. As you see in the Google URL above, “test query” is transformed into “test+query” in the Google search URL. We then apply that formula to each query (or item ) and redefine it as updated_query. To recap:
Item = “test query”
updated_query = “test+query”

Line 3: google_url = “https://www.google.com/search?q=” + updated_query

Here, we’re now making another variable called google_url, simply the Google URL search prefix of https://www.google.com/search?q= followed by our URL-ready updated_query.

In these few steps, we’ve now changed “test query” into https://www.google.com/search?q=test+query for each one of our keywords.

Line 4: driver.get(google_url)

In Step 3, we defined the action of performing a search on our Google driver as driver. So now that we have both our Google search operation set up and the specific URL we need to be searched, we’re just instructing driver to perform its operation with our google_url.

Line 5: time.sleep(delay_seconds)
You may recall that in Part 1, we imported our packages but also made the variable delay_seconds generate a decimal number between 1 and 5: delay_seconds = random.randint(100, 500)/100 For each time this script runs, a different number will generate and is assigned as the time.sleep value. time.sleep is the amount of seconds that the program will wait until performing another search. So after each search, the program will wait somewhere between 1.00 and 5.00 seconds before performing the next search.

Why do we do that? If our program waits for the same exact amount of time in between dozens of searches, Google will figure out pretty quickly that this is a bot performing the search, which increases odds of getting blocked by Google. We use the random value for time.sleep to anonymize ourselves to prevent this from happening.

writer.writerow(data)
file.close()

The scraping is done! Now we need to get it into a document we can analyze. You may recall the following code from Part 4.

So since writer has already been defined as the action of writing onto our original ‘innovators.csv’ doc, and since we already have ‘Query’ and ‘Snippet’ written onto the doc as column headers (via writer.writerow(fields) ),  we then invoke the writer.writerow command again to write our data result ( item , which is essentially the query, and ns) which is the snippet result) directly below the appropriate headers in the correct columns.

Once all the loops have run and are written into our document, we then use file.close() to close the file.

Our first query “How do you get health insurance in Vermont” returned no snippet at the time of the search. Meanwhile, “How do you get health insurance in West Virginia” did, and we can see the result along with the URL at the very end.

Now you know how to scrape featured snippets from Google! You can likely make small tweaks to scrape for other features such as People Also Ask fields, but this is a good starting point for your snippet scraping needs.