Hi, I am a professional software engineer based in Singapore. First, I would like to clarify for a few questions:
1. How fast is the current execution process for 100k data?
2. What execution time frame is acceptable?
3. What is the data format? json?
I can optimize by splitting the load on multiple database and processes and use index to optimize the execution process. Let me know if you are interested thanks =)