+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: External Resources (https://forum.codeigniter.com/forumdisplay.php?fid=7)
+--- Forum: Spotlight (https://forum.codeigniter.com/forumdisplay.php?fid=16)
+--- Thread: Job Board Scraper/Sample Application (/showthread.php?tid=63668)
Job Board Scraper/Sample Application - RobertSF - 11-25-2015
I've developed a small application to assist jobseekers. One of the problems with job boards is that their search results are presented as web pages, which are not in a data-friendly format. I thought it would be nice to search for jobs and get the resulting job listings as a spreadsheet that you can then further manipulate. That's what this application does. The application offers an interface to manage job search URLs, letting you save, edit, delete, and execute them. Executing a job search runs the search and automatically downloads a CSV file with the contents of the search to your local computer.
Since websites are coded differently, the application must "know" the website's code. For this, there's a class file for each website. For now, there are only class files for Indeed and Simply Hired. Adding more job boards will require coding class files for the additional boards, and if Indeed or Simply Hired change their pages' HTML, their class files will have to be modified too.
You can see the application running here. http://www.clicketyhome.com/jobs
You can download the application here: https://github.com/RobertHallsey/cijobs If you have not used GitHub before, look for the button to the right that says Download Zip. Click it to download the application.
The application is complex enough that it could provide a good step up from the tutorials for those just starting with CodeIgniter, but is not so complex that it should overwhelm anyone basically familiar with PHP and MySQL.
I hope someone finds it useful, and if you see any bad practices in it, please let me know!
RE: Job Board Scraper/Sample Application - PaulD - 11-25-2015
Just wanted to say this line is really good:
PHP Code:
$url = ((preg_match($reg_next_url, $page, $matches) == 1) ? self::SITE . $matches[1] : '');
Good work!!
Best wishes
Paul.
RE: Job Board Scraper/Sample Application - PaulD - 11-25-2015
Although I would mention that for 'indeed' in their terms and conditions cite you will not use:
Quote:...any portion of the web pages that are part of the Website without Indeed’s explicit permission
Just to be a pain.
Paul
RE: Job Board Scraper/Sample Application - RobertSF - 11-27-2015
(11-25-2015, 06:18 PM)PaulD Wrote: Just wanted to say this line is really good:
Wow, thank you!
Quote:Although I would mention that for 'indeed' in their terms and conditions cite you will not use:
Quote: Wrote:...any portion of the web pages that are part of the Website without Indeed’s explicit permission
Just to be a pain.
Oh, no worries. We should respect sites' terms and conditions, but I don't think this application does anything a normal visitor wouldn't do, and surely they don't mean people need explicit permission to run a search.
RE: Job Board Scraper/Sample Application - RobertSF - 12-02-2015
I've updated the application to use PHP's DOMDocument instead of regex to scrape web pages.