SE Radio 631: Abhay Paroha on Cloud Migration for Oil and Gas Operations

Design Insights

In the realm of microservice design, Abhay Paroha shares his approach to defining the scope and evolution of services. Initially, his team focused on data ingestion, evolving from a proof of concept to more complex domains like mapping and API creation. This progression allowed clients to build dashboards and workflows, enhancing the functionality of oil well forecasting and advisory services 1. However, the journey wasn't without challenges. Abhay recalls a significant learning experience when their on-prem software failed, highlighting the importance of proactive monitoring and cloud readiness 2.

Delivering features is not only the important part, the other part is operating as well.

--- Abhay Paroha

These insights underscore the need for a robust cloud strategy and continuous learning in microservice development.

   

State Management

State management in microservices is crucial, particularly when using Kubernetes. Abhay Paroha explains the role of stateful sets in maintaining persistent storage and unique identification for pods, which is essential for applications like databases and message queues 3. The choice of programming language also plays a role in state management. Abhay's team transitioned from Java to Scala, leveraging Akka for its actor-based framework, which supports scalable and fault-tolerant applications 4.

Akka provides a streaming and one important point in Akka streaming it basically handle the asynchronous data ingestion.

--- Abhay Paroha

This framework's integration with Kubernetes enhances the system's ability to manage state effectively.

   

Efficient Scaling

Scaling microservices efficiently is a key focus, with Abhay Paroha detailing the use of Kubernetes and Keda for auto-scaling. Initially, manual scaling was employed, but the team shifted to auto-scaling using custom metrics, allowing them to handle large data volumes from oil wells 5. This approach minimized latency and optimized resource usage, crucial for both ingestion and consumption processes.

We tried to achieve the auto scalability first we tried the manual scalability, then auto scalability.

--- Abhay Paroha

Keda, as a Kubernetes event-driven auto-scaler, played a pivotal role in adapting to fluctuating data streams, ensuring efficient scaling and cost management.