Web scraping script

I need a web scraping bot script made for my directory site to scrape other directory sites combine & populate profile listings, reviews and ratings for my directory.

Stealth capability for no tracking.

Will need several methods used to change our outgoing IP.

Spoof the User agent by making a list of user agents and pick a random one for each request.

Detection of honeypots for no follow

Incorporate some random clicks on the page, mouse movements and random actions that will make the bot look like a human.

Scraping will be done from a separate site domain for black list protection

What I am doing is building a HVAC Hub for everything HVAC using edirectory platform

[login to view URL]

My Categories Are:

HVAC Contractors

Heating, Venting , Air Conditioning Professionals

HVAC Tools

HVAC Software

HVAC Consultants

HVAC Training

Indoor Air Quality

Heater and Air Conditioner Distributors

HVAC Supply

HVAC Systems

Boiler Contractors

Air Conditioning Contractors

Air Conditioning Repair Contractors

Furnace Repair Contractors

Heating Contractors

HVAC Installation Contractors

Boiler Repair Contractors

All the scrapping is limited to 5-6 existing web directory sites that specialize in the HVAC industry or have a section for HVAC and have the information.

I have included a sample of existing sites and url locations for most.

The program needs to collect all profile, reviews and ratings from each of these existing sites categories profiles to make 1 combined profile with more details on the profile than any one site individually.

To be clear I need to have reviews and ratings from the same directory's sites on my site and update capability manually or automatically so I have valuable fresh review content going to each of my listed profiles section with multiple tabs with one for each review site.

(a separate project will be code script for search capabilities for all profiles, reviews and ratings)

I have attach the word doc for each site with urls and profiles for each site for your review. We would want flexibility to scrape the internet or any site I find valuable content.

This would need to be used in the beginning once then updated with reviews afterwards. Any new contractors coming on to the other sites we would get flagged and a new profile built.

ALL categories above for contractor, business and professionals locations include :

• USA

• Canada.

Yes I am doing Canada because I have such good seo value their already and I have not started marketing the site.

Look up "HVAC Contractors" on [login to view URL]

I am already on the first page of Google with my small old WP site. [login to view URL]

Dashboard Admin Management

Attached is my dashboard and where I want to put your new section in for management of scraping.

My directory company can install the script on my site. I will include the fields wanted for each category profile plus many more to be filled in when they join.

This will need to be a section for my site built on edirectory platform.

The end product must be very easy to use because I do not come from IT background.

Typical functionality of a harvest or scraper for no duplication, add to list, update, delete, search and more.

See attached for scrape fields

PHP Software Architecture Web Scraping

Kitambulisho cha Mradi: #10239641

Kuhusu mradi