Web scrapping, a highly effective growth hacking technique
Stepward is an agency dedicated to developing your business through effective outreach and growth hacking techniques.
Are you a startup? A contractor? An SME or any other entity in need of a sales boost? You’ve come to the right place. In this article, we’d like to introduce you to web scrapping for growth hacking.
What is web scrapping?
Before getting to the heart of the matter, let’s start with a quick reminder of what Growth Hacking is all about. Growth Hacking is a set of marketing strategies that will enable a company to increase its number of prospects, its traffic, its visibility and, by the same token, its sales.
We all know that the web is a real goldmine for those who know how to use it. It’s packed with data that can be used to achieve your marketing and business objectives. However, data recovery is not always an easy task. And once you’ve got the data, the most important thing is to know what to do with it and how to use it to your advantage.
Web scrapping is one of the most widely used growth hacking methods. It consists in extracting data from a website and using the information received to prospect for new customers, enrich your customer base or retrieve strategic information. So it’s a great way to gain an edge over your competitors, who use the classic CTRL-C and CTRL-V.
In other words, when you practice web scrapping, you will first identify the most relevant data sources such as a directory, a mapping, a referencing… etc. The aim is to find a site that brings together the data you’re interested in in the most structured way possible, i.e. with a recurring structure. Next, you’ll need to collect all the information you deem useful about the site you’ve identified via a turnkey tool or custom development. Next, you’ll use the information you’ve gathered for your own benefit: prospecting, database, referencing… For example, this will enable you to create your own database of prospects.
Webscrapping is a formidable Growth Hacking tool. Used wisely, scrapping creates a qualitative audience base that can be combined with a marketing automation strategy.
This association is the miracle recipe for developing your business in record time. We’ll go into more detail below. If you don’t have the time, contact us and we’ll be happy to help you with your web scrapping strategy.
Need specific scrapping?
Why web scrapping?
Web scrapping has many uses, and is generally used to give a company a boost. Here, we’re going to show you a few examples of web scrapping applications, just to give you an idea of what can be achieved with this method.
a) Build a prospecting file
This is one of the uses we use the most at Stepward, enabling you to identify a large number of your targets on the fly and then get in touch with them. Let’s take an example: you offer a tool for real estate agents, so you can scrape a directory of all the agents in an agency and then get in touch with them. If you have their contact details directly on the site, that’s fine, otherwise you can combine this information with an enrichment tool as we discuss in this
LINK ENRICHMENT ARTICLE
1. Duplicate content and mashup
The aim is to automatically copy and paste content from one site and duplicate it on another. This will enable scrappers to generate thousands of automatic pages and gain traffic through SEO blackhat. This practice has been sanctioned by the various search engines for a few years now when it is detected, but thanks to AI like GPT-3 you can rewrite the text to avoid being spotted by the search engines and have almost original content!
2. Automated monitoring of your competitors’ prices
This is a much less well-known use of web scrapping, but its application can enable you to monitor your competitors’ prices in an online business. In the event of a variation, you will be alerted directly. But you can also use the data collected to obtain a summary table of your competitors’ prices. This will enable you to compare them directly with your own, an effective way of defining an appropriate pricing policy. This method is mainly used by e-commerce sites in highly competitive environments. What’s more, specialized turnkey tools are available for this purpose.
3. Competitive intelligence
If you have a non-commercial site, web scrapping is an effective way of monitoring your competitors’ sites. You’ll be notified immediately of any changes or new content.
4. Product database extraction
Let’s say you’re a company offering X number of products. Scraping will enable you to extract your competitors’ product catalogs and compare them with your own. You can then easily differentiate your products or create missing products.
Is web scrapping legal?
The term web scrapping is often associated with an important question: is it legal or not? To help you understand, here’s a case study.
A company or an Internet user produces and publishes a set of articles on the web. Publications are then scrapped by a user and published unchanged. In this case, we can immediately speak of copyright infringement, a right that is in force in France and in most other countries around the world. In this case, scrapping violates the intellectual property code.
However, it is important to stress that the data displayed on most websites is intended for public consumption. In other words, it’s perfectly legal to copy them and save them to a file on your computer. It all depends on how you intend to use them. This is where you have to be careful. If you decide to spam people who have no direct interest in your product, or if it runs counter to the interests of the site or the person contacted, this may be illegal.
On the other hand, if the data you download is used for personal purposes (such as an analysis database, for example), then the practice is entirely ethical.
In addition, each website has its own terms of use. Copyright details can easily be found on the site’s home page. So, when launching web scrapping processes, you need to respect the terms of use and copyright declarations of the target websites. It is important to emphasize that conditions of use mainly concern the use of data and access to a site.
By respecting the rules, scrapping becomes a perfectly legal practice that you can use to boost your business. To help you, here are a few best practices for ethical web scrapping:
- Opt for APIs
Today, the vast majority of sites have their own APIs. They are specially designed to allow you to collect data without having to scrape it. In this case, you proceed according to their rules. We’re used to working with APIs, so don’t hesitate to contact us on the subject.
- Respecting Robots.txt files
Also known as the Robots Exclusion Standard, the robots.txt file is what tells browser software which parties are and are not allowed on a given website. Often used to exclude pages from search engines, it can also be used for scrapping.
- Read the general conditions
The general terms and conditions are the part of the site that explains the rules that apply to the site. It is therefore advisable to read the conditions carefully to avoid any unpleasant surprises.
A respectful and ethical practice
Generally speaking, the legality of scrapping is a matter of common sense, and will therefore depend on the above rules, and of course on how you intend to use the data. It is therefore important to avoid any practices that are immoral or that could harm a third party.
Need specific scrapping?
How do I do web scrapping? Tools, challenges and difficulties
To properly scrape a site, the tool of your choice must do more than simply retrieve information on the site’s page(s). It must also be able to crawl any page on the site. Want to do some web scrapping? You can do it with turnkey software, with a script, or with the help of a professional like us.
If you want to start scrapping, the first thing to do is go to the site you’re targeting. Next, you’ll define your objective. Perhaps it’s the extraction of product prices from an e-commerce site? Or is it the extraction of contacts from a web directory?
In any case, here are the essential steps for scrapping a website:
- Define the purpose of scrapping
- Identify the site(s) and/or application(s) to be scrapped
- Creating the data structure
- Choosing the right tool
- Testing the tool on a small scale
- Start scrapping
- Save the result in the format of your choice
To succeed in your task, you can use software/extensions. This will allow you to scrape a site without having to code. Here’s a non-exhaustive list of scrapping software, with varying degrees of user-friendliness. Each tool is specific, so you need to choose them according to your objectives.
- IO: the ideal tool for quickly scrapping a large number of pages without coding.
- SCRAPY: based on a collaborative open source framework, it enables you to extract data quickly and easily.
- WEBSCRAPER: the Google Chrome extension that lets you quickly extract data from a website.
- INSTANT DATA SCRAPPER: Equipped with artificial intelligence, it lets you scrape structured data from a site in 3 clicks.
- PHANTOMBUSTER: an efficient way to scrape without code if the automation block is already available, e.g. for Linkedin, yellow pages… etc.
- APIFY: the tool that converts any website into an API.
- SCRAPPING BOT: the simplest and easiest tool to use, but with a few limitations.
- 80LEGS: flexible and easy to configure, this is the tool used by web giants such as MailChimp, PayPal and others.
- OCTOPARSE: highly interactive, allowing anyone who knows how to navigate to scrape.
Web scrapping can be considered a legal piracy tool. The only problem lies in the use of the extracted data. From a company’s point of view, web scrapping can pose a threat if the data collected is used to drive competitors out of business. However, it’s a practice that’s becoming more widespread by the day, and it enables you to develop your business rapidly and stand out from the competition.
Our customized webscrapping offer: one-off or recurring
Stepward is aware that scrapping is not an easy task for everyone. Successful extraction requires basic knowledge of coding and computing. At the very least, you need to know how a website works and the basic languages such as HTML, CSS, JS and so on.
To make things easier for you, we put our know-how and expertise at your disposal. To meet your needs, we can provide you with a quotation. This enables us to provide you with a tailor-made service that perfectly meets your objectives. We can scrape all the information you want. All we ask is two things:
- a link to the website(s) of your choice
- information of interest to you
We’ll do the work for you, and deliver the result in just a few days.
Want to grow your business with Growth Hacking, but don’t know exactly what you need? We can think along with you and suggest relevant sources of information to help you develop your business. Our experts will guide and advise you.
If you want an up-to-date database, we can automatically and recurrently scrape the site of your choice. You’ll then be up to date with all the relevant information that could be useful in your customer prospecting strategy.
If you have any questions, don’t hesitate to contact us and find out what we can do for you. We have a team of experts ready to answer all your questions as quickly as possible. We’ve set up a system that can meet all your needs and perfectly match your marketing objectives.
- A concrete example of webscrapping Are you still wondering what web scrapping can be used for? So let’s get down to business.
You’re a company that’s doing well, but will soon be running out of customers or unable to accelerate its growth. So you want to grow your business with digital marketing, and you want to do it fast with Growth Hacking. You’ve defined your persona, and now you can use Web Scrapping to build your prospecting file. To do this, you decide to do some scrapping on LinkedIn.
Why LinkedIn? Simply because LinkedIn lets you take advantage of an up-to-date, structured user base of over 500 million members. And best of all, it’s completely legal!
To scrape LinkedIn, you can use Phantombuster, for example, which offers a turnkey scrapping tool for Linkedin. Simply build your search on Linkedin Sales Navigator.
Then you copy and paste your search link into Phantombuster, add your Linkedin cookie and fine-tune the final configuration details according to the information you want to retrieve.
Please note that each Linkedin cookie has limits (250 visits per day per cookie, etc.). If you have a large number of prospects to scrape, I advise you to hire the Mirror Profiles service.
Now all you have to do is wait for the scraper to retrieve all the information, grab a coffee and enjoy your qualified prospect base!
Thanks to web scrapping, you’ve now created a qualified prospecting database that will enable you to contact people potentially interested in your products or services.
