Key challenges for TSOs and DSOs in the digitalisation of the energy system
Visibility and management: Lack of visibility and effective management of a large number of new, distributed devices on the grid (solar, electric vehicles, batteries).
- Data interoperability: To achieve standardization and easy exchange of data between TSOs, DSOs and other actors at European level.
- Coordinated grid and RES planning: Aligning plans for the expansion of the transmission and distribution network with plans for integration of renewable energy sources.
- Advanced monitoring: The number of sensors in the network and their capabilities are increasing, which leads to different challenges and opportunities.
- Accurate prediction: The increased amount of data allows for the application of advanced algorithms (such as artificial intelligence) to improve system performance.
Comping solution - Data Lake
It provides the TSO (Transmission System Operator) with the flexibility, scalability, and analytical power needed to manage an increasingly complex network that generates more data.
It is used as a single repository for all structured, semi-structured, and unstructured data - allowing engineers and analysts to access everything from one place.
Why Comping Data Lake?
A data, analytics, and AI platform offering both on-premise implementation and easy migration to the cloud.
It is based on open-source technologies with a flexible architecture that allows combining data storage components such as HDFS, HBase, Kudu, and Ozone.
Optimizing power system management with the Comping Data Lake.
The Comping Data Lake has been implemented at a European Transmission System Operator (TSO) where, among other functions, it collects over 6 billion real-time measurements per day, which amounts to more than 50 TB of data annually.
Integration of data from heterogeneous sources.
The Data Lake platform enables data collection from heterogeneous sources without data losses using the "source-aligned data product" methodology. This ensures a high-quality representation of the raw data, which is often normalized or semi-structured.
This approach allows for subsequent data modelling and transformation once the necessary business processes are identified.
Business use cases
Forecasting solar power plant production
For the optimization and stability of the power system, precise forecasting is necessary. It allows for efficient planning of system operations and reduces market risks in buying and selling energy. The growing share of renewable sources, due to their variable nature, represents a key challenge for the accuracy of these forecasts.
The Data Lake platform enables the development of machine learning models for forecasting solar power plant production by combining data from different sources, for instance: meteorological data, ERA5 atmospheric reanalysis data of the global climate, historical solar plant production data, etc.
Regulatory reporting
The EU Agency for the Cooperation of Energy Regulators (ACER) requires the creation of a single, probabilistic grid security assessment for all of Europe. To achieve this, it is necessary to collect and standardize grid incident data from all TSOs.
The Data Lake platform allows for the complete automation of the regulatory reporting process by consolidating data on outages, planned works, and meteorological conditions, and by identifying the causes of disturbances in the power grid.
Transformer insulation condition assessment
Switching and atmospheric transient overvoltages, characterized by high frequencies and amplitudes, pose a significant risk by damaging transformer insulation and other high-voltage equipment. Analyzing these events requires data from multiple systems (e.g., SCADA, lightning location systems), but a critical problem arises when these systems are not time-synchronized.
The Data Lake platform resolves this challenge by unifying all relevant data in a single location. This not only enables faster and automated fault analysis but also facilitates proactive assessment of insulation condition and the calculation of a transformer "health index."
Power grid visualization
The Data Lake platform transforms vast amounts of disconnected data into a clear, real-time visual representation of the power grid's status.
Instead of analyzing data from multiple systems, the platform empowers operators to instantly spot anomalies, correlate cause and effect, and make faster, more accurate decisions — all on a single map.
The direct result is enhanced reliability and security for the entire system.
Words from our satisfied customers
Data Lake is a platform for power network secondary services that enables data retrieval from all the heterogenous sources with no loss, providing high data quality. The need for such a platform has been shown on the example of transient analysis in the vicinity of a power transformer. It is important to monitor such events to have a better understanding of what is happening to the transformer during its operation and interaction with the power network. Continuous measurements of overvoltages at transformer terminals show that real overvoltages have significantly different waveshapes from the standard impulse voltages used for testing transformers.
With a Data Lake platform such analyses will become automatized and eventually it will be possible to use data science techniques to evaluate the stress on the transformer itself. Such fundings aims to improve reliability and safety of the power system in general. Application of Data Lake platform will enable not only to automatically perform post mortem analysis and fault analysis, but also different analysis of stresses on HV equipment which is important and can be used as the basis for the assessment of the transformer insulation condition and estimation of health index. It will also enable TSOs to automatically create reliable and accurate statistics of operating events in the transmission power network and to use machine learning techniques for solving different problems in the power system related to protection, operation, control, asset management, forecasting, etc.
Source: Franc, Bojan ; Filipović-Grčić, Božidar ; Jurišić, Bruno ; Župan, Tomislav ; Jaković, Tihomir ; Ivišić, Antonija ; Šturlić, Ivan ; Župan, Alan; Concept of Data Lake design for analyses of fast transients near power transformers; ICTRAM 2023-CIGRE SC A2 & 6th International Colloquium Transformer Research and Asset Management Joint Colloquium. Split: ICTRAM, 2023;
