Real-Time Analytics with Amazon Kinesis and Amazon Elasticsearch

PROJECT DELIVERABLES: Three documents are required to be submitted with this project- Source Code Document, Screenshots Document, and a Writeup document. If you are offering to complete the project please state the cost and completion date. DESCRIPTION: Real-Time Analytics on Streaming Data with Amazon Kinesis and Amazon Elasticsearch Service to assist Facebook to do a continuous monitoring system to detect sentiment changes in a social media feed to respond to the sentiment in near real time. Problem Statement: Data streaming is used everywhere; from social networks like Facebook to mobile and web applications, IoT devices, instrumentation in data centers, and other sources. As the speed and volume of data increases, the need to perform data analysis in real time with machine learning algorithms and extract a deeper understanding from the data becomes even more important. Domain: Sentiment Analysis of Social Media Platform Analysis to be done: Collect and store streaming data, use Amazon Kinesis Analytics to process and analyze the streaming data, apply machine learning algorithm to detect anomalies in the system. Steps to perform: 1. Create Kinesis delivery stream 2. Simulate streaming application to detect anomalies 3. Open Elasticsearch service and create a new domain 4. Configure Kinesis Firehose to export the results to Amazon ES 5. Update the buffer size and existing IAM role for the process 6. Open the Amazon Kinesis Analytics console and create a new application 7. Connect to the source for further analysis 8. Launch SQL_Editor and start the application 9. Load the processed data into Kinesis Firehose delivery stream 10. Visualize the data using Kibana