Build automated data load from websites (automation) & APIs to Postgresql/MS-SQL
₹37500-75000 INR
Imefungwa
Imechapishwa over 5 years ago
₹37500-75000 INR
Kulipwa wakati wa kufikishwa
We are a startup focussing on blockchain products.
One of our project details are as below for which we are looking for a team of freelancers with various skill set who can together build the product.
Phase 1 (Duration: 7 days):
IT solution to collect, load and process data from various websites.
Data will be available :
1. Standard APIs of those websites (json format)
2. Websites crawls (by Automation to collect tabular data)
Need to build batch automated processes to collect these data and load in Postgresql/MS-SQL on defined interval of time. This would require setting of PostgresqlMS-SQL server on Ubuntu latest version and further processing to consolidate collected data in desired tables' format. Which requires processing via stored procedures.
Parameters, like, frequency of batch run and hyperlink of the API/Website to crawl should be read from a database table. Solution should be generalised, such that, similar pattern websites are read similarly.
Ex:
1. Standard APIs (samples):
[login to view URL]
[login to view URL]
[login to view URL]
2. Websites Data Crawl (samples):
[login to view URL]
[login to view URL]
[login to view URL]
Once data is collected directly from these websites, it needs to be processed and collated in target summary tables based upon few business logics.
Phase 2 (duration: 4 weeks) & 3 (duration: 3 weeks):
To be discussed once approaching finish of phase 1.
Skills needed for phase 1:
JSON processing, API processing, Automation to crawl websites and collect data, Postgresql/MS-SQL scripting, database knowledge etc.
Dear ukrblockchains,
I have experience with these skills. I do have similar experience doing something similar to yours "Build automated data load from websites (automation) & APIs to Postgresql/MS-SQL" therefore I am looking forward to discuss and move ahead.
Relevant Skills and Experience
I have 10+ years of experience in API, JSON, Microsoft SQL Server, Selenium Webdriver, Website Design
I can do this within your budget too ₹37500 - ₹75000 INR
please check more from here https://www.freelancer.com/u/mmadi.html?page=portfolio
I am ready to work with you,kindly waiting for your response.
Thanks & Regards,
Mmadi
Looking for Mobile App Solutions, we've got you covered. Contact us today, our developers are eager to solve your problems..
We specialize in Hybrid and Native Mobile Apps. Our prices are affordable. Contact us today.
Hi, I have work experience related to Web Crawling, Data Extraction. Currently Working as Data Science. Have knowledge of Postgress, Distributed System.