| Product(s): | WaterSight |
| Version(s): | 10.00. |
| Area: | Documentation |
WaterSight automatically calculates patterns and forecasts for each sensor and zone based on the collected raw sensor data. The methodology used follows the below main steps:
- Raw data is collected (at any given frequency)
- Raw data gets cleaned up (outliers are automatically removed) using machine learning (AI) and resampled in the following intervals: 15 min,1hour, daily
- Statistical analysis is applied to the processed resampled time series to generate the patterns and confidence bands, using percentiles methodology approach. Anomalous events generated by the solution can be automatically excluded from the pattern calculation (more detail below)
- Specifically when calculating zones patterns, besides the percentiles methodology approach, an other methodology is used to calculate patterns using AI. Variables such as historical data, events generated, special periods (holidays or festive seasons), temperature and precipitation (weather) are used to calculate patterns that can respond better whenever there is a clear relation between zone demand and temperature or precipitation.
2. Raw Data Cleaning
Below it is shown real raw data at green for a specific flow sensor.
Figure 2 - Raw data at green.
And for the same sensor and period it is shown below the real resampled data, every 15 minutes (black line) after the automatic data cleaning process. More than 90% of the data outliers were automatically removed.
Figure 3 - Real 15 minutes time series (processed) at black.
3. Pattern Calculation
Patterns and confidence bands are represented in the graphs by a grey band. Below the patterns calculation for the same period and sensor.
Figure 3 - Pattern confidence bands represented at grey (expected data) .
Patterns and confidence bands (at grey) are compared in real time with real time data (represented by a back line filled with black, yellow and red dots) in order to easily track potential issues in the system. Whenever real data goes outside the pattern confidence bands (above percentile 95 or below percentile 5 of the pattern), an outlier is identified and the real value is color coded with a red circle.
Figure 4 - Real time series (back line with black, yellow and red dots) and pattern confidence bands (grey band).
Pattern and forecasts are continuously and automatically being updated in real time (whenever new data arrives into WaterSight) using always last months of historical data. The historical period used to calculate the patterns is defined by the user, by changing the pattern dropdown window located above each graph. By using always the last month(s) of data, this means that the pattern for today is slightly different than the pattern of the previous day and will be slightly different from tomorrow pattern (as the historical period used also changes). This is particularly useful when there are significant changes in zone demands such as winter and summer, or when zones boundaries are changed. In these cases, patterns automatically adapt to the new sensor/zone condition.
Patterns can be calculated for each individual sensor or for a specific zone. To calculate the zone patterns, the final balance between all zone inflows and outflows (plus storage if exists) is considered. Zones tend to have clear and well defined patterns (the same may not apply to each individual sensor) as they represent a large group of customers with usually well defined demand behaviors that tend to repeat each day, being the main difference usually between weekday (workday) and weekend (not workday), as exemplified in figure below.
Figure 2 - Typical work day flow pattern for a residential zone (obtained from WaterSight)
As mentioned before the patterns are different depending on the day of the week. In WaterSight the default is to have a distinction between Weekday, Saturday and Sunday, but the user can customize differently by having for example a pattern that is different for each day of the week, or other use case. To know more about pattern customization, please click here.
Figure 3 - Pattern forecasts for Saturday, Sunday and week day
4. Pattern Calculation using AI
The AI pattern generated approach uses historical data together with temperature, precipitation, special periods and days to calculate a pattern that in several cases can better fit the real time data and with this minimizing the generation of false alerts. It was seen in several cases a relation between demands and weather conditions, and this can be considered.
In the zone time series detail graph there is an option to switch between the pattern type Percentile based (traditional approach) and the AI generated approach. Please note that the AI generated approach may work better for zones where there is a clear relation between demands and temperatures, however for zones where this relation is not so obvious, the traditional approach might do a better job.
Figure 4 - Switching from the "Percentile based" approach to the "AI generated" approach for the zone pattern calculation.
Comparison
To assess the accuracy of the pattern AI future predictions and relation with temperature and precipitation, the user can click in the comparison shortcut button - icon located on top of the graph - and directly compare the expected zone demands (pattern) with the forecasted (and historical) temperature and precipitation.
5. Forecasts calculation
Using as reference the most recent patterns calculated, WaterSight extrapolates and calculates the future patterns for any sensor and zone, up to one week.
Excluding Anomalous Events from the pattern calculation
Moreover anomalous events that may occur such as pipe breaks and meter failures can be automatically excluded from the patterns calculation so that patterns and forecasts do not get "polluted" by those anomalies. For that to happen, the user needs to make sure he correctly categorizes all the events that are generated. In case the event was not automatically detected by the software but it is "polluting" the patterns, the user can create a manual event - by clicking on the (!) icon located on top of the graph - and give it the right categorization so that those periods can be omitted from the pattern calculation.
For more information please take a look here.
Figure 4 - Possibility to exclude from patterns anomalous events
Patterns and forecasts are mainly used for the real time anomaly detection (to generate alerts based on deviations to patterns) and also as boundary conditions for the model runs.
See Also
Automatically Adjusting model demands
OpenFlows WaterSight TechNotes and FAQ's
WaterSight Learning Resources Guide