How Utopus Insights Reduced Data Processing Time from Weeks to Hours

Renewable energy analytics provider Utopus Insights (Utopus) wanted to generate more computing power so that it could improve the performance of Scipher, its industrial Internet of Things (IoT) analytics solution platform that assesses the performance of wind and solar energy assets using data in near real time. When the company initially launched the platform, it ran the solution on premises, which required ongoing investments in hardware to scale its infrastructure to meet increasing demand.

To meet its data processing needs, Utopus explored going serverless and decided to migrate to Amazon Web Services (AWS) in 2018. A year later, Utopus became fully native to the cloud, which has helped it scale quickly, accelerate its time to market, and achieve compute cost optimization. The company has maintained high availability on Scipher and reduced the time that it takes to process data from weeks down to hours. By working on AWS and expanding its data processing capabilities, the company can focus on innovating its digital solutions and facilitating the global transition to clean energy.

Exceeding Data Processing Capacity on Premises

Founded in 2017, Utopus aims to accelerate the integration of renewable energy into the modern grid by optimizing energy production using data analytics and insights. Utopus serves over 235 customers globally and holds over 90 granted and pending patents across its suite of products, ranging from asset management coverage to power forecasting insights. This suite is underpinned by the company’s platform, Scipher, which collects data from wind and solar assets to drive reporting and actionable insights using machine learning (ML). Using this solution, businesses can proactively monitor their assets worldwide and predict maintenance needs or part failures, helping to increase the longevity and performance of their renewable energy assets. 

When launching Scipher in 2017, Utopus used an on-premises data center to power its digital solutions and process petabytes of energy data at any given time. As more customers adopted its solution, the company faced challenges in quickly scaling its infrastructure. “The amount of data that we had exceeded our processing capacity,” says Ziad Rida, technical director of architecture at Utopus. “Our main goal was to ingest the data and make it available for data science.” In 2018, Utopus engaged AWS to accelerate its data processing capabilities and optimize its compute resources. To test the viability of working in the cloud, the company developed proofs of concept working alongside the AWS team. Utopus received resources and technical training with subject matter experts, and it kicked off its cloud migration that same year. “AWS helps us scale more rapidly than we could have on our own,” says Michael Wilkinson, chief product officer at Utopus. “At a time when both climate change and energy security are urgent global concerns, making data-driven innovations on AWS is the key to unlocking reliable, smart solutions both superfast and at scale.”

Taking a Serverless-First Approach to Scale to Demand

When migrating to AWS, Utopus opted for fully managed, serverless solutions when possible. “Using serverless solutions on AWS means that we don’t have to worry about scalability, manageability, installation, or configuration,” says Rida. For instance, the company adopted Amazon Kinesis Data Streams, which companies use to easily stream data at virtually any scale. On average, the company streams 7 TB of data through Scipher every day. 

After streaming its data, the company processes it using AWS Lambda, a serverless, event-driven compute service that companies use to run code for virtually any type of application or backend service without provisioning or managing servers. “AWS Lambda is highly scalable and very powerful,” says Rida. “We can manage latency, make our processing faster, and decide how much we want to spend.” Using AWS Lambda, the company processes over 200 billion signals from wind and solar sources every day, and in under 2 hours, it can process the same amount of data that would have taken 2 weeks to process on premises. By gaining deeper and timelier insights, its customers can improve the performance of their renewable energy assets and minimize their carbon footprints. 

Decarbonizing the Electricity System on AWS

Utopus completed its migration to AWS in 2019 and continues to try new services to improve the performance of Scipher. To further reduce its time to market and introduce new capabilities, Utopus is trialing AWS IoT Greengrass, which helps companies build IoT devices faster. The company aims to accelerate the adoption of renewable energy across all industries. “Frankly, we couldn’t be doing the work of trying to decarbonize the electricity system without using AWS,” says Wilkinson. 

Ream more here.