Tuma Shindano

Research off the shelf solutions for a back end batch system (scraping application)

Hali: Closed
Zawadi: $170
Wasilisho Zilizopokelewa: 2
Mshindi: TheGuyver2040

Winner

Maelezo ya Shindano

The task is to research and suggest possible off the shelf solutions for a back end batch system (scraping application) that fulfill the following requirements:

Business Objectives:
Get competitors product information for branded shoes on a daily basis. Information like Product name, Product colorname, other product attributes example materials, list of competitors, Competitor prices (Black and discounted prices)).

Proposed Work Flow:
The system fetches a product list from our system with EAN codes (for identification). The system then uses the EAN in google shopping to fetch product information and prices per competitor.

Required Features:
Scraping engine to handle over 15.000 requests per day. The normal request per day will be 50-500. The information should be structured on model level and SKU / EAN level (ie we send over/fetch one size/EAN to scrape).

Inputs:
There will be a product feed into the system from our product system. Alternativly we want to be able to fetch the data through a feed / API.

Dependencies:
The system should have an easy way to export / import information preferably using some API / feed.

Integrations:
There should be an API available. To fetch the scraped products. And possibly what products of interest to scrape.

Server / Hosting:
AWS on a Kubernetes server, if hosted on our servers. Otherwise software as a service is fine.

User Count:
Performance is hard to estimate but roughly 50-500 daily requests but up to 15.000 when going through our total range of products.

Deliverable:
A summary of the system you have found with a short explanation of its functionalities and how it can be used to fulfil our requirements