[eluser]Jaketoolson[/eluser]
A lot of my job requires me to scrape public data made available on public websites. Every so often I come across a site that is 'nearly' impossible to scrape. This is usually when the data I want to scrape is 'appended' or 'inserted' into the DOM using Javascript <em>after</em> the page has loaded and is therefore not directly accessible.
A lot of sites use forms to display the data requested but when a hash or token is required for the form to be changed, this usually requires an additional step or two on the scrapers part.
It is pretty difficult though to prevent. Not only do I run daily scrapes, I also manage a website which we dont want scraped! Some times I simply output charts and raw data to the page in an image form using a php image class.
