Half of all financial services organisations prefer managing web scraping practices internally, compared to just 11% that outsource the process completely, according to key findings from the new Oxylabs white paper, Alternative Data Unlocks Key Decisions in the UK & US Finance Industries.
The survey found that completely outsourcing data collection is the least popular method while a hybrid option of combining internal web scraping practices and outsourcing was more popular (38%). The findings demonstrate that web scraping still remains a complicated and intricate practice among organizations, which needs close oversight, particularly in heavily regulated industries such as the financial services sector.
Aleksandras Šulženko, Product Owner at Oxylabs said: “Such a low preference for outsourcing is a little surprising but it’s clear that better trust in web scraping has to be developed. We know that web scraping has become an essential part of financial services, yet so few companies trust third-party services.
“The findings indicate a hidden issue with how web scraping is understood and what goals it can achieve, bringing those third-party companies back to the drawing board. Outsourcers need to better communicate how their services work and how their methods are secure and compliant, which is one of a number of challenges the industry is facing at the moment.”
Among the hurdles experienced by organizations when implementing web scraping activities, ensuring high-quality data tops the list (cited by 42% of respondents), followed closely by managing and processing large datasets at 41%. Other challenges, such as finding the most efficient tools, ensuring a consistent data flow, getting real-time data, or finding reliable partners to outsource to, are equally distributed and similarly important for decision-makers in financial services.
Šulženko continued: “In the finance industry, the quality of data directly defines the quality of business decisions as even small deviations might lead to incorrect conclusions. As a result, organizations put more trust into their internal teams as they can put the necessary checkpoints in place at every stage of the data management process.
“If organizations are to begin putting their trust in third parties, these parties need to communicate how they can tailor collection and analysis more efficiently than an internal team as well as how they can tackle the issues many face when implementing web scraping. For example, outsourcing to a third-party web scraping solutions provider is an easy option that requires minimal start-up costs and the data acquired can be integrated almost immediately.”
Understanding that data must be treated sensitively, the finance industry takes careful measures to stay in line with regulations. Almost all surveyed companies (99%) have a data compliance function – be it internal, external or both. This could be explained by the specifics of the industry, where any possible risk could lead to major financial consequences, coupled with the fact that financial data is inherently confidential.
“Web scraping produces a significant amount of value not only to financial organizations but all companies, especially if that information can be utilized effectively. While a third-party web scraping solutions provider is the easier and cheaper option for organizations, it seems the majority prefer to look internally and will continue to do so until the challenges around web scraping are addressed,” concluded Šulženko.
To download your copy of the Oxylabs Alternative Data Unlocks Key Decisions in the UK & US Finance Industries white paper, please visit HERE.
