Big data and advanced analytics for energy management

Back to articles

By George Nizharadze, Data and Commercial Optimisation Lead
1 February 2018

Read the full article in Power Engineering International.

The energy world as we know it is changing rapidly. The development of new business models are being driven by decarbonisation of the economy and by technology advancements, including big data and advanced analytics.

Large, flexible, carbon-intensive power plants connected to the transmission system are shutting down and replaced with smaller scale, inflexible renewable generation. Much of this energy is embedded at the edge of the grid, outside the transmission system.

This presents National Grid with a challenge in understanding how much power is going to be available at a point in time in order to balance the system. National Grid and others need to be able to call on distributed energy resources (DERs) to provide flexible generation or demand.

The move to distributed energy

DERs are smaller power sources that can be aggregated to provide the power necessary to meet regular demand, as well as deliver services, such as frequency response. As the electricity system continues to modernize, the share of DERs, such as storage and advanced renewable technologies, keeps increasing.

In addition, evolving information and communication technologies make it possible to unlock extra capacity from previously latent and passive resources, such as power load flexibility (i.e. demand side response), small-scale fossil fuel and biomass/biogas generators, and combined heat-and-power plants.

The opportunities for DERs to create technical and commercial value are becoming widespread and increasingly sophisticated as the speed, accuracy and reliability of a DER’s ability to respond to the grid’s frequency variations approaches that of large-scale conventional power plants.

Cheaper, greener, more efficient energy for all

For the electricity system and its participants, technology developments enable the digitization of the electricity system. The use of big data and advanced analytics means that systems will become more connected and operators will have a better understanding of both supply and demand in real-time. This improved understanding will present opportunities to increase efficiencies and reduce investment requirements.

Less investment in infrastructure should mean a lower cost of energy supply for all, and will remove key barriers for development of new renewable generation in the electricity system.

Big data, technology and analytics

A boom in the technology industry is being driven by cheaper and higher performance data storage and processing. This is enabling the energy system to become increasingly digitized and encouraging a proliferation of data.

The roll out of smart metering at the grid connection, as well as intelligent resource level metering at the asset level (making dumb assets smart), combined with the growth of new generation assets significantly increases the amount of information that is captured, stored and processed.

This provides an increasingly valuable pool of data available to decision makers. Such data informs decisions about how to change the supply and demand of electricity at a given point in time to keep the system balanced. This data enables more efficient use of the assets connected to the system, and avoids the need to build out new, expensive generation to meet peaks in demand.

Aggregation, optimization and dispatch of hundreds, thousands and eventually, tens of thousands of DERs poses significant opportunities, as well as an array of challenges, in terms of data and analytics.  This can generally be thought of in three categories: Big data management; Forecasting, using machine learning; and DER fleet operations optimization.

Big data

Big data management is all about scalability and enabling better decision making based on this data. At Origami Energy we set the objective for our data management platform of being able to “scale out” to practically an unlimited number of DERs and to provide the security required by blue chip organisations who are fundamental to the UK’s critical infrastructure.

We connect our customers’ distributed energy assets (generation, demand and battery storage) to our technology platform using our Energy Router. Our Energy Router is a bespoke design, specified to deliver current and future requirements (we expect it to be functional for the next 20 years of software upgrades). The Energy Router carries out a number of functions, including collecting, storing and transmitting data and providing control of the DER. It is capable of continuously streaming sub-second metrology data to our centralized Technology Platform, which processes and stores it in our Big Data system. This data is then used off-line or in near-real time for solving various operational optimization problems and deriving new insights for medium and long-term strategy.

Forecasting using machine learning and advanced analytics

At Origami Energy, we use advanced analytics and machine learning to develop forecasts for asset availability and behaviour, design of bidding parameters in various ancillary services markets and forecasts of balancing mechanism market prices. This means we have been very successful in bidding into the high value Frequency Response markets offered by National Grid.

For machine learning, we combine our internal data with external sources, such as historical power prices, weather forecasts and power system maintenance schedules published by the system operator. Our machine learning models are constantly developing and getting richer and deeper with time and increasing data.

Our approaches to big data, advanced analytics and machine learning, alongside extreme uptime performance, enables us to provide energy suppliers, energy traders and other electricity system participants with the opportunity to offer new products and services to their customers delivering improved returns for owners of distributed energy resources. We provide our partners with better decision-making capability and the opportunity to automate decisions about when and how to use an asset, leading to more accurate, more reliable and quicker responses to value opportunities.

In addition, we have built the UK’s only dedicated micro-grid emulation and testing laboratory enabling rigorous testing of new applications, algorithms and asset classes such as batteries and electric vehicles, prior to real-world deployment with customers across the country.


As far as commercial and operational optimization is concerned, we rely on the vast body of knowledge and practice from the field of operations research and are also developing our own proprietary frameworks and algorithms for deterministic and stochastic optimization.

We aggregate and optimize the distributed assets of our clients strategically and operationally. At the strategic level, we optimize decisions as to which assets to bid into which submarkets and at which parameters. These submarkets can vary from imbalance price chasing and day-ahead trading to monthly frequency response tenders to annual capacity markets. We also model and simulate expected prices in the different submarkets and quantify and measure associated risks and opportunities.

The operational optimization takes place once assets have already been committed to specific submarkets and we need to realize the objective of meeting the relevant commitments at the least cost. These operational optimization problems are usually deterministic and amenable to mixed integer-linear programming solutions. We articulate the optimization frameworks and formulate efficient mathematical solutions in-house. We then use commercially available powerful solver engines to run our optimization models on. Finally, we integrate all the different components into a single platform.

In a world where big data is being managed securely and effectively, and where the forecasts are operating well (and learning), owners and managers of DERs will be able to make better decisions about where value can be derived. Ultimately it may well be computers that are proposing value strategies and executing them, with minimal intervention by human managers.


If you want to know more about our products and services please get in touch.



Give us a call on 0330 726 0050 to speak to one of our experts. Alternatively you can send us a message and we’ll get back to you.