Being a Professional Developer on Website Scraping, Crawling, Data Mining & Extraction, Parsing and Reporting Services

As you know I have been concentrating on the website scraping stuffs for few months now, I wanted to share some of my experiences in this journey. What I must say before anything is –  its been really an enjoyable field of continuing web development life.

It was late 2005 I had started web scraping & parsing stuffs. I had initially developed some contact grabbers to grab contacts from email clients like Gmail, Hotmail and AOL. Later I had some opportunities to develop scraping tools for some (3 or 4) real estate companies – they used to scrape other competitors’ properties by my automated scraping tool to compare prices, etc. In late 2006, I started working for Beendo Corporation to scrape popular email clients and integrate all emails into one place. It was a big project that wasn’t just scraping emails. We had to scrape emails, attachments, contacts as well as develop processes for sending/replying/forwarding emails using respective clients. Also, we added facilities to grab emails & contacts from popular social networking sites like facebook, myspace, friendster, ringo, hi5, etc. and finally we integrated emails from POP3/IMAP servers. After this development, we developed a facebook version for this whole project and released in May 2008 (see details).

In 2009, I have done quite a few web scraping / data mining projects beside my mainstream development – real estate scraping, media scraping, ecommerce scraping for price comparision, fashion products scraping like t-shirt, shoes data, etc. – these all brought a significant expertise to my development. Then I stopped working on scraping projects and concentrated on other more important fields to receive skills. But scraping projects were enjoyable for me always and I was receiving offers from valuable clients. Thus it has been a part of my regular development.

I have developed huge scraping tools to scrape product data from popular stores (clients use those data for price comparison) like google products,,, adorama, newegg, and many more. Few months back, I have developed a job listing site which lists jobs from popular online job sources – back in the site some scrapers always working to keep it updated.

These all are my past scraping / data mining experiences. Now I’m again concentrating on web scraping / data mining beside other development areas.
Getting good number of scraping offers from past few weeks.

Beside being the professional scraper myself, I am trying to build a team on web scraping to handle more projects. Already started with someone who might need sometime to be expert but going good. Like the E-commerce team, our scraping team will work under its own banner at and will be known as “Professional Scraper“. I am hoping a very nice journey – so far so good.

Please feel free to communicate and discuss about your scraping & data mining projects. Contact us via email: dev <at> proscraper [dot] com



2 thoughts on “Being a Professional Developer on Website Scraping, Crawling, Data Mining & Extraction, Parsing and Reporting Services

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.