Web Scraping — Definition and Business Use Cases
Web scraping is the automated retrieval and processing of data from public websites. It is a broad tool with many legitimate business applications:
- Competitor price monitoring — tracking product prices on e-commerce platforms and price comparison sites.
- Lead generation — collecting publicly available contact data of companies (not individuals) from industry directories.
- Media and reputation monitoring — tracking mentions of brands, products, and executives in media and industry portals.
- Market analysis — collecting data on products, categories, and trends from industry websites.
- Registry data verification — automated verification of company information from public registries.
- Financial data aggregation — collecting public financial reports, market quotes, and macroeconomic data.
Legal Framework in the EU — Four Perspectives
Web scraping in the EU is evaluated from at least four legal perspectives:
1. GDPR — Personal Data on the Web
The key question: does the scraped data contain personal data? A person's name, email address, phone number, photo, or IP address are personal data subject to GDPR, even if they are publicly available.
The mere fact that an individual has made data public (e.g., on LinkedIn) does not provide a legal basis for freely collecting and processing it. You must have one of the six legal bases from Article 6 of GDPR. For business scraping, legitimate interest of the controller is most commonly used — but it requires a balancing test (whether your interest outweighs the individual's right to privacy).
2. Copyright — Database Protection
The Database Directive (96/9/EC) and implementing national legislation protect databases against extraction of substantial elements. Mass downloading of data from websites that constitute protected databases (online stores, real estate portals, employee databases) may infringe the database producer's rights.
The legal test: does collecting this data replace the need for a user to visit the website? If so — there is a risk of infringing the sui generis database right.
3. Terms of Service
Most websites prohibit automated data retrieval in their terms of service. Violating ToS may form the basis for claims of unfair competition or unauthorized access to a computer system.
4. Access to Public Data — Data Act and Open Data
The EU Data Act (coming into force in stages from 2025) and the Open Data Directive create new opportunities for legal access to data — including public data held by government bodies. This is the preferred path for companies that need public data.
Best Practices for Legal Web Scraping
If web scraping is justified from a business standpoint and falls within the legal framework, follow these practices:
- Check robots.txt — the robots.txt file specifies which parts of a site the owner permits for indexing. It is a guideline (not a legal requirement), but respecting it is good practice and reduces legal risk.
- Use official APIs — if a service offers an API for its data, use it instead of scraping. An API is a legal, documented means of access.
- Throttling and rate limiting — do not overload the target server. Aggressive scraping can be classified as a DDoS attack.
- Anonymize personal data — immediately after retrieval, remove or anonymize personal data if it is not essential for the purpose.
- Document the legal basis — before launching a scraping project, prepare legal documentation: purpose, scope of data, GDPR basis (if applicable), balancing of interests analysis.
- Retention and minimization — store data only as long as necessary. Automated deletion policies are mandatory.
Corporate Data vs. Personal Data Scraping — A Key Distinction
The most important practical distinction: company data (name, tax ID, registered address, registration number, industry, revenue from public reports) is generally safe to retrieve from public registries. Personal data — even if publicly available — requires particular caution and usually a strong legal basis.
For enterprises, ESKOM.AI offers integrated access to legitimate registry data sources — with full GDPR compliance, automated caching, and fallback handling between sources. This eliminates the need for scraping where legal APIs are available.
Web Scraping and AI Models — Special Risks
Companies training AI models on scraped data face an additional set of legal risks. Regulations concerning copyright in AI training datasets are evolving rapidly — both at the EU level (AI Act) and in case law (rulings in the US and EU concerning generative AI models).
The general principle: data scraped legally for business analysis purposes may not necessarily be legally used to train commercial AI models. This is a separate legal question requiring an independent assessment.