Advanced Metering Data Quality & Completion

Data completion approach that ensured full regulation compliance across the globe

data quality, data completion

“The solution provided by BigHub helped our large customer to fully comply with local regulation and to completely mitigate the numerous business issues connected with huge data gaps.”

Karel Novák

Director of Services, EMEA, GoodData s.r.o.

Project background

Client product provides analytics to its customers, specifically using energy usage data from energy meters aka AMI data. AMI data can be messy and incomplete preventing trust and adoption of the analytics product.

‍

Our approach

Identify data issues and replace with estimated data records to ‘complete’ the dataset. With increased trust in the data, customer can launch a product to the remaining clients.

For the purpose of data completion, our solution leveraged the power of similarity modeling based on big data processing in combination with state-of-art recurrent neural network.

Benefits of our approach

More reliable source of data for business activities

First thorough and exact data quality assessment that allows customer to apply additional corrective measures

Full compliance with local regulation reduces risk of disputes and penalties

Identification of Gaps

Thorough data exploration

Detailed data profiling, found continuous gaps in very large data sets
Find the symptoms in the dataset
Several heatmaps were produced to indicate the data issues types and distribution

Completion via LSTM

Precise approach for wider gaps

LSTM with Attention Mechanism proved itself to perform a very precise forecasting capabilities
Suitable for wider gaps that require more complex models
Requires to be trained per segment on GPU enabled HW

Next Solution

info@bighub.cz +420 774 028 175 Vaclavske namesti 802/56, 110 00 Nove Mesto

ID 05388066

VATCZ05388066