More Needs to be Done to Educate Organizations on Web Scraping, According to New Research by Oxylabs

Published on December 2, 2022

It is estimated that fewer than half of organizations have implemented web scraping due to legal concerns and complexity.

According to Oxylabs’ new research with Censuswide, while many organizations use web scraping to drive business decisions and competitor pricing, legal concerns have taken the top spot, showing more still needs to be done to educate businesses on the practice.

This new research, based on a survey of more than 1,000 UK and US-based senior decision makers, including CTOs, VPs of Engineering, Chief Data Officers, Heads of Business Intelligence, and Heads of Data & Analytics, explains how businesses are using newer, more sophisticated technology to gain an edge over their competition.

Web scraping benefits businesses

The main concern of businesses that have not implemented web scraping has been potential legal complications. More than half (53.06%) of all organizations that haven’t implemented web scraping cite legal risks as the main reason.

Lack of technical expertise is the second most cited reason (39.59%), followed closely by budget restrictions (30.20%). Web scraping has become more accessible than ever, as only 1 in 5 businesses cites complicated technology as an excuse not to implement it (20.82%).

According to Oxylabs’ Head of Legal, Denas Grybauskas, web scraping is still misunderstood and retailers should be aware of the huge benefits it offers – especially during the golden quarter.

“It has been very tricky to overcome the perception that behind every data point lies a legal risk, particularly when it comes to web scraping. Even though organizations should always consult with legal professionals, there are best practices within the industry that, when followed, minimize most risks associated with data collection. These results indicate a need to increase our efforts and educate the industry about the legal aspects of web scraping.

“To help educate business leaders, organizations can turn to initiatives such as Ethical Web Data Collection Initiative (EWDCI), which aims to reach out to businesses and educate them on the importance of ethical data collection, as well as educate users and legislators worldwide in the interest of informed, empowered action. This will encourage industry-led stewardship of critical and widely used technology in order to address current challenges and position the industry for future success.

“Retailers can use web scraping to improve customer experience, which is extremely important during the golden quarter. Companies can maximize profit while staying competitive and without losing the customer. This can be achieved by scraping product and market information to fine-tune the strategies to increase customer satisfaction through areas such as recommendation engines. Since this data also enables you to keep a thorough check on the not-so-static market at large, it helps you create a dynamic pricing strategy to keep up with the changes and help maximize the profit.” 

Lack of technical expertise is a greater barrier to US-based companies (46.77% vs 37.16%), while budget constraints affect the UK more (32.24% vs 24.19%). Risks of legal complications is the greatest factor for both regions with nearly no differences (>50% for both).


“The web scraping industry still suffers from several misconceptions, and organizations need to be educated better on this issue. Web scraping is a powerful tool for analyzing websites, and its benefits far outweigh the drawbacks. Retailers can gain insights into how to increase traffic and sales, manifest ideal conversation rates, and, ultimately, save their businesses and give them a new and refined appearance by analyzing public web data”, – Grybauskas concluded.

To download your copy of the Oxylabs’ Alternative Data Defines Competition in the UK & US eCommerce research findings, please visit HERE.

Julius Cerniauskas is a Grit Daily contributor and the CEO of Oxylabs, one of the biggest companies in the public web data collection industry, employing over 300 specialists. Since joining the company in 2015, Mr. Cerniauskas has implemented a brand new leadership company structure, taking product and service technology to the next level, as well as securing long-term partnerships with dozens of Fortune 500 companies. Mr Cerniauskas is Lithuania’s technology industry leader who speaks on the topics of web scraping, big data, machine learning, technology trends, and business leadership. Today, he continues to lead Oxylabs as a top global provider of premium proxies and data scraping solutions, helping companies and entrepreneurs to realize their full potential by harnessing the power of data.

Read more

More GD News