Production of scraping tools and databases

Imefungwa Ilichapishwa Miaka 5 iliyopita Kulipwa wakati wa kujifungua
Imefungwa Kulipwa wakati wa kujifungua

. Contents of request

· Creating a database that automatically updates with the original scraping tool and API and building a cloud environment

· The above test design (unit test - connection test - comprehensive test - acceptance test)

· Immediate use environment construction of the above tools (server selection, program operation verification etc.

· System monitoring / maintenance / troubleshooting support for the above tools

* In particular, please also propose about the whole redundancy.

3. How to propose

"Original scraping tool (IP rotation environment)" + "Database"

4. About the original scraping tool

Reference tool "octopus"

Http://[login to view URL]

Required function

· It must be Api compliant.

· Accessing by accessing IP.

· It is possible to access multiple sites in parallel.

· It is possible to set both of the following two types, or it is a service corresponding to ②

① Send to fixed URL Specify page count = 1 setting × 100 Service up to setting (crawler)

② Simple to load the URL list (csv) = 1 setting x 10 services (crawler) setting

5. About IP rotation environment

· Addition of IP addition is easy.

- Rotation of unrelated 10 to 100 or more IP addresses.

· More than 50 concurrent tasks.

· Setting the access interval (10 seconds to 1 minute)

· Operation on the cloud for 24 hours.

· Efficient rotation of IP.

For example, suppose that there are five access destinations of ABCDE and 10 IPs rotate.

If an IP that rotates at B gets an access refused, it will return B as an IP

Disconnect from access destination and perform rotation only between ACDE.

6. API linkage with database

A database that automatically captures and updates the data collected by the scraping tool

· Automatically update the database by placing it on the cloud. Delivery hope in the state where it can operate immediately

· Since the number of registered products in the database is over 3 million, it can be handled without delay. In case

· The database should always update the latest backup and switch quickly to emergency.

7. Assumed processing flow

1. Automatically extract URL to be scraped from (1) database and fixed URL list

2. Automatically reflect on existing scraping service (api compatible)

3. Automatic extraction of update information from scraping tool

4. Automatically reflect in the database

5. Automatically extract optimized information from database.

8. Supplied materials

· Database template (created with * simple function)

In case

9. point

① The running cost is inexpensive.

* Of course there are no problems with suggestions with existing tools, but evaluation is also high if you propose the same original tool. In case

We will prioritize the adoption of equivalent original tools with only necessary functions.

② It has experience and knowledge on "API", "IP rotation", "access denial" and "large scale database creation", and we can make confident proposals.

③ The interaction with the scraping tool and the database is almost automatic.

④ We will pay the milestone payment, not after the inspection of the finished product.

* If you are absolutely a milestone please appreciate that you do not have any problem by actually seeing the equivalent tool.

Also, since it is not an expert, although explanation about the proposal will of course be obtained, please tell us on our axis whether we can propose our desired function surely.

Please do not hesitate to ask questions about specifications. Thank you.

Kuprogramu Database MySQL Server Web API Web Scraping

Kitambulisho cha Mradi: #18266207

Kuhusu mradi

11 mapendekezo Mradi wa mbali Ipo mtandaoni %project.latestActivity_relativeTime|badilisha%

11 wafanyakazi huru wanazabuni wastani wa $458 kwa kazi hii

zekovicm

Hi there,I am Miljan,Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immediately and finish it within Zaidi

$500 USD kwa siku 7
(Maoni 105)
7.3
pofilippov

Hi there. I need to ask you some questions. I have 8 year experience in very complex and high load projects. I can make your project easily. I will be glad to discuss the project through chat.

$459 USD kwa siku 9
(Maoni 29)
6.7
JinDongZhe

Hello , sir. How are you? I just saw your project description carefully. I am very interested in your project. I have rich experience in Web development and Scraping . I am a full time developer and can work for y Zaidi

$277 USD kwa siku 10
(Maoni 23)
6.3
brianconey

Hello How are you My name is Xu I am a scrping expert and I am sure I can save data into on mysql datatbase . thanks i have full time and I can start to work immediately Please contact me and do let us discus Zaidi

$555 USD kwa siku 10
(Maoni 79)
6.4
smsaurabhv

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON Zaidi

$250 USD kwa siku 7
(Maoni 78)
5.5
Sanjupanchal

Hello there!, I am highly interested in your project I read your project descriptions carefully before bidding. your 100% satisfaction is assured if you allow me to serve. please send me message where we can talk abou Zaidi

$500 USD kwa siku 10
(Maoni 20)
4.1
oswaldoodavidal

Hello, I have experience in web scraping with Python. I can use Selenium, Scrapy, BeautifulSoup and Requests to make the best web scrapers! I can also work with SQL databases! I hope to work with you!

$444 USD kwa siku 15
(Maoni 3)
4.1
satish5006

I am working as a web developer from past four years and providing quality work with long term relationship with clients . I can help in following: PHP, Jquery, Javascript, Ajax, HTaML,CSS. CSS3,HTML5, Joomla, Cakephp, Zaidi

$555 USD kwa siku 10
(Maoni 1)
2.7