improve/fix comments - TODO: debug examples, CSV and after

[epclust.git] / contrat / 2016_IRSDIproject_v3.tex

\documentclass[12pt, a4paper]{article}  

\usepackage[margin=2.5cm]{geometry}
\usepackage[utf8]{inputenc}       % in encoding 
\usepackage[T1]{fontenc}          % out-encoding f
\usepackage{eurosym}
\usepackage{lmodern, microtype}   % goes OK with T1 fontenc
%\usepackage[authoryear, round]{natbib}
\usepackage{natbib}
\usepackage{color, tikz, graphicx, subfig}
\usepackage{amssymb, amsmath, amsthm}
\usepackage{setspace, lineno, url, xcolor}
\usepackage{savetrees}

\newcommand{\todo}[1]{\textcolor{blue}{TODO: #1}} % macro for todo entries

% Style options
\renewcommand\familydefault{\sfdefault} % Use with sans serif font
\setlength{\bibsep}{0.0pt}              % Compact bibliography (natbib)

\title{Disaggregated Electricity Forecasting using Clustering of Individual Consumers \\
      {\normalsize \color{gray}  IRSDI - RESEARCH INITIATIVE IN INDUSTRIAL DATA SCIENCE}}

\author{Benjamin Auder    \and
        Jairo Cugliari    \and
        Yannig Goude      \and
        Jean-Michel Poggi 
}
\date{\normalsize\today
\vspace{-1.2\baselineskip}}



\begin{document}
\maketitle

%\begin{abstract}

%\end{abstract}

% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                                        S E C T I O N
%
\section{Context}

\subsection{Industrial}

Electricity load forecasting is crucial for utilities for production
planning as well as marketing offers. Recently, the increasing deployment of 
smart grids infrastructure requires the development of more flexible data
driven forecasting methods adapting quite automatically to new data sets. 
Electricity load forecasting is crucial for utilities for production planning as 
well as marketing offers. New metering infrastructures as smart meters 
provide new and potentially massive informations about individual (household,
small and medium enterprise) consumption. As an example, in France, 
ERDF (Electricite Reseau Distribution de France the French manager of 
the public electricity distribution network) deployed 250000 smart meters, 
covering a rural and an urban territory and providing half-hourly household 
energy used each day. ERDF plans to install 35 millions of them over the 
French territory by the end of 2020 and exploiting such an amount of data 
is an exciting but challenging task (see \url{http://www.erdf.fr/Linky}).
We propose to build clustering tools useful for forecasting the load 
consumption. The idea is to disaggregate the global signal in such a way that  
the sum of disaggregated forecasts significantly improves the prediction of the 
whole global signal. The strategy is in three steps: first we cluster curves 
defining super-consumers, then we build a hierarchy of partitions within which 
the best one is finally selected with respect to a disaggregated forecast 
criterion. The proposed strategy is applied to a dataset of individual 
consumers from the French electricity provider EDF. A substantial gain 
of $16$ \% in forecast accuracy comparing to the 1-cluster approach is provided 
by disaggregation while preserving meaningful classes of consumers.

\subsection{Academic}

In the context of economic seasonal univariate continuous time series, it is often 
natural to segment it in time, into consecutive curves, for example days, which 
are then treated as a discrete time series of functions. In particular, in the 
electrical context, the shape of the curves exhibits rich information about the 
calendar day type, the meteorological conditions or the existence of special 
electricity tariffs. Using the information contained in the shape of the load 
curves leads to very elegant formulation of functional forecasting.


%Electricity load experts naturally look at daily demand data as time functions
%called load curves. In a recent paper, \cite{shang2013} uses a functional time
%series approach for forecasting short-term electricity demand. This paper is
%illustrated by the half-hourly electricity demand from Monday to Sunday in South
%Australia. The strategy is also to consider a seasonal univariate time series as
%a time series of curves, then to reduce the dimensionality of curves by applying
%a functional principal component analysis and finally, following
%\cite{shang2011}, the principal component scores are forecasted using a
%univariate ARIMA models. In addition, since data points in the daily electricity
%demand are sequentially observed, a forecast updating method based on
%nonparametric bootstrap approach is proposed to improve the accuracy of point
%forecasts. With respect to this strategy, the scheme we propose handles the
%forecasting problem in a functional way avoiding the hour by hour processing and
%considers a more flexible way to construct the distribution leading to the
%prediction interval.

The shape of the curves exhibits rich information about the calendar day type,
the meteorological conditions or the existence of special electricity tariffs. 
Using the information contained in the shape of the load curves, \cite{antoniadis2012prevision} proposed a flexible nonparametric function-valued
forecast model called KWF (\textit{Kernel + Wavelet + Functional}) well suited
to handle nonstationary series. The predictor can be seen as a weighted average
of futures of past situations, where the weights increase with the similarity
between the past situations and the actual one. In addition, this strategy
provides with a simultaneous multiple horizon prediction for a global forecast. 

However, there is a need for local electricity load forecasting at different levels of the grid. 
Bottom-up approaches, based on a two stage process combining clustering and forecasting 
methods, are a promising perspective. First, it 
consists in building classes in a population such that each class could be 
sufficiently well forecast but corresponds to different load shapes or reacts 
differently to exogenous variables like temperature or prices (see e.g. 
\cite{labeeuw} in the context of demand response). The second stage consists in 
aggregating forecasts to forecast the total or any subtotal of the population 
consumption. For example, identify and forecast the consumption of a 
sub-population reactive to an incentive is an important need to optimize a 
demand response program. 

\section{Past work}

Few papers consider the problem of clustering individual consumption for 
forecasting (e.g. \cite{iwafune2014short, Alzate, carevic2010applications, MisitiElec}). Recently, \cite{energycon} proposed to build clustering tools useful for the two tasks simultaneously: clustering individual customers and forecasting the load consumption. The idea is to disaggregate the global signal in such a way that the sum of disaggregated forecasts significantly improves the prediction of the whole global signal. The general strategy is in three steps: first we cluster individual curves defining super-consumers, then we built a hierarchy of partitions within which a best one is finally selected with respect to a disaggregated forecast criterion. The predictions are made with the KWF model which allows one to use it as a off-the-shelve tool.

While this work has ended with an the specification of an algorithm, a current need is a real upscaling proof. A first step on this direction was done in 
\cite{auder2014}.


% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                                        S E C T I O N
%
\section{Aims}

The method proposed in \cite{energycon} has been successfully tested on a small data set of EDF clients. With the current development of smart meters in France the available volume of individual data is increasing day after day. Then, there is a genuine need of measuring the upscale skills of the existent methods. 

This projet's aim is twofold. First, we will evaluate the upscaling capacity of the strategy developed in \cite{energycon} to cope with the upgrowing volume of data. Second, we will study how to adapt the KWF prediction method to take into account an exogeneous variable. In our particular problem the exogeneous variables can be any meteorological measurement that affects the load demand and is available at the moment of the prediction.


% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                                        S E C T I O N
%

\section{Means considered}

\subsection{Methods}
\paragraph{Clustering analysis.} In general, clustering methods look for groups of individuals on data in such a way that those belonging to the same group are more similar than those from other groups. Many methods exists to cluster data: 
hierarchical, center-based, probabilistic, etc. Almost all of them depends heavily 
on the choice of a similarity measure between individuals. For this challenge we plan
to compare individuals in terms of their wavelet spectrum signature. Thanks to this strategy, non
stationary signals may be fairly compared. Moreover, the signals need not to be 
measured on the same temporal grid. However, in order to detect relevant results 
the wavelet signatures should be corrected by exogenous information (e.g. the one
provided as client characteristics).

\paragraph{Wavelet analysis.} Since the objects to analyze (load curves) can be viewed
as functions of time, functional data analysis techniques are one possible choice to
represent these objects. From a stochastic point of view the functions are realizations
of a non stationary random process. Wavelet transform can be used to extract 
relevant information about the functions both on time and frequency. With an 
appropriate representation of the objects, it is then possible to construct
a meaningful distance between load curves.

\paragraph{Forecasting with KWF}
The basic idea of nonparametric forecasting is that similar cases in the past 
have similar future consequences. For example  the electricity consumption is 
divided into blocks of one day size. Then, using a dissimilarity measure, the 
blocks similar to the last observed block are searched in the past and a vector 
of weights is built. Finally, the forecast of the next  day is obtained by a 
weighted average of the most similar future days using previous vector of 
weights. From the statistical point of view, the model is an estimate of the 
regression function using the kernel method, of the last block against all the 
blocks in the past. In \cite{antoniadis2006functional}  this basic model is 
extended to the case of stationary functional random variables. But in the 
context of electrical power demand, the hypothesis of stationarity generally 
fails: an evolving mean level and the existence of groups that may be seen as 
classes of stationarity are to be considered. Corrections to take into 
account these two main nonstationary features are considered in
\cite{antoniadis2012prevision} defining a flexible nonparametric function-valued 
forecast model called KWF (\textit{Kernel + Wavelet + Functional}) well suited 
to handle nonstationary series. The predictor can be seen as a weighted average 
of futures of past situations, where the weights increase with the similarity 
between the past situations and the actual one. Again the similarity is defined 
thanks to the wavelet decompositions of the two segments.


\subsection{Technology}  % to be employed (hardware y software)}


The volume of data to deal for this projet can be handled with standard
but recent tools for data analysis.
The specific software tools will be statistical programming language like \texttt{R} with some popular 
libraries (\texttt{data.table}, \texttt{dplyr}) and specific packages to cope with wavelet analysis. All these elements are open source.

When the computational burden will grow, we have direct access to larger computation capacities. 

All the tools developed on the project will be made available as open source software licences.

\subsection{Research team}  

The proposed team for developing this projet is composed by theree 
academic members :
\begin{itemize}
\item Benjamin Auder, LMO, Univ Paris Saclay
\item Jairo Cugliari, ERIC, Univ Lyon
\item Jean-Michel Poggi, LMO, Univ Paris Saclay, Univ Paris Descartes
\end{itemize}

% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                                        S E C T I O N
%
\section{Data description} 
\begin{itemize}
\item a first dataset already used in \cite{energycon} could be used, at least in a first step, to calibrate the method. 
\item simulated data could be obtained at EDF following \cite{bondu15} or any simulation method preserving confidentiality 
of individual consumers. Obviously, any amount of such data could be produced to benchmark the scalability of our approach.
\item Irish data provided by the Irish commission for energy regulation consisting in 2000 individual consumption (small and 
medium enterprise and residential) at an half-hourly resolution as well as pre and post experiment survey (see \cite{Cer_a, Cer_b}).
\end{itemize}



% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                                        S E C T I O N
%
\section{Budget}
The expected global budget for the projet is of 15000 \euro, which comprises a 1 day workshop.

\paragraph{Internal budget} The members of the research team are based on the Paris area and Lyon. 
The way we work includes video and audio conferences in a regular basis as well as several in-person meetings.

We plan to present the work on international conferences both on data science and energy oriented meetings.

Last, a stress test for the upscale skill of the proposed method will need to hire computing time on a specialized platform. We have access to 
the Centre de Calcul de l'Institut National de Physique Nucléaire et de Physique des Particules (\url{http://cc.in2p3.fr/}) through the laboratory ERIC, Lyon 2.

\paragraph{Worshop organization on Individual Electricity Consumers} 
A 1-day workshop dedicated to Individual Electricity Consumers including
sessions on data, packages and methods, could be organized in September
2017, and could be proposed to The French Statistical Society (SFdS) as a
satellite meeting of the Journées de Statistique 2018 which will be held in
the campus of EDF Lab in May 2018.


\begin{center}
\begin{tabular}{lr} \hline
\textbf{Internal budget}      & \textbf{10 000 \euro}\\
\; Travels                    &  3 000 \euro\\
\; Conference fees            & 3 000 \euro\\
\; Internal meetings          & 2 000 \euro\\
\; Hiring of high performance computing time & 2 000 \euro\\ 
\textbf{Worshop organization} & \textbf{5 000 \euro} \\ 
\; Invitations of researchers & 3 000 \euro\\
\; Organization workshop      & 2 000 \euro\\ \hline
\textbf{Global budget}        & \textbf{15 000 \euro} \\ \hline
\end{tabular}
\end{center}


% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                                        S E C T I O N
%
\section{Vitas}

\paragraph{Benjamin Auder} is CNRS Research Engineer at LMO, University Paris-Sud Orsay in France. 
He obtained his PhD in statistics in 2011 at the university Université Pierre et Marie Curie, Paris. 
His main research areas are Clustering, dimensionality reduction, manifold learning, machine learning
in addition to software development and implementation issues of algorithmic solutions.

(\url{http://auder.net/page-upsud/})

\paragraph{Jairo Cugliari} is Assistant Professor of Statistics at University of Lyon in France. He obtained his PhD in statistics  
in 2011 at the university Paris-Sud 11 Orsay. His main research areas are functional data analysis methods
for classification and prediction for applied statistical problems.

(\url{http://eric.univ-lyon2.fr/~jcugliari/})



\paragraph{Jean-Michel Poggi} is Professor of Statistics at University of Paris Descartes
and at University Paris-Sud Orsay in France. His main research areas are
tree-based methods for classification and regression, nonparametric time
series forecasting, wavelet methods and applied statistical modeling in energy
and environment fields. His publications combine theoretical and practical
contributions together with industrial applications and software development.

\noindent
He is an elected member of the ISI, he was President of the French Statistical
Society (SFdS) and he is Vice-President of the FENStatS, Vice-President of ENBIS and President of ECAS.

(\url{http://www.math.u-psud.fr/~poggi/})

% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%                                        S E C T I O N
%
\section{Associated industrial company} % And members


\paragraph{Yannig Goude} is a research-engineer/project manager at EDF R\&D and associate 
professor at University Paris-Sud Orsay, France. He obtained his PhD in statistics and probability 
in 2008 at the university Paris-Sud 11 Orsay. His research interests are electricity load forecasting, 
more generally time series analysis and forecasting, non-parametric models and expert aggregation.

(\url{https://fr.linkedin.com/in/yannig-goude-768b3980})

\bibliographystyle{plain}
\bibliography{biblio_irsdi} %,predintervals,rapportfinal}

\end{document}



\bibitem{Alzate} 
C.~Alzate and M.~Sinn, 
  Improved electricity load forecasting via kernel spectral clustering of 
  smartmeter, 
  \emph{International Conference on Data Mining}, vol. 948, pp. 943 -- 948, 
  2013
    
\bibitem{antoniadis2006functional}
A.~Antoniadis, E.~Paparoditis and T.~Sapatinas, 
  A functional wavelet-kernel approach for time series prediction,
  \emph{Journal of the Royal Statistical Society, Series B},
  vol. 68(5), pp. 837 -- 857, 2006

\bibitem{antoniadis2013clustering}
A.~Antoniadis, X.~Brossat, J.~Cugliari, and J.-M.~Poggi,
  Clustering functional data using wavelets,
  \emph{International Journal of Wavelets, Multiresolution and Information 
        Processing}, 
  vol. 11(1), 2013

\bibitem{antoniadis2012prevision}
A. Antoniadis, X. Brossat, J. Cugliari, J.-M. Poggi,
  Pr\'{e}vision d'un processus \`{a} valeurs fonctionnelles en pr\'{e}sence de 
  non stationnarit\'{e}s. Application \`{a} la consommation 
  d'\'{e}lectricit\'{e}
  Journal de la Soci\'{e}t\'{e} Fran\c{c}aise de Statistique, 
  Vol. 153, No. 2, 52--78, 2012

\bibitem{brabec2015statistical} 
Brabec, M. and Kon{\'a}r, O. and Mal{\`y}, M. and Kasanick{\`y}, I and Pelik{\'a}n, E., 
  Statistical models for disaggregation and reaggregation of natural gas 
  consumption data, 
  \emph{Journal of Applied Statistics}, vol. 42(5), pp. 921--937, 2015
 
\bibitem{carevic2010applications}
Carevi{\'c}, S. and Capuder, T. and Delimar, M.
  Applications of clustering algorithms in long-term load forecasting
  \emph{Proceedings Energy Conference and Exhibition (EnergyCon), 
  2010 IEEE International} 688--693, 2010
 
\bibitem{Chicco}
G. Chicco 
  Overview and performance assessment of the clustering methods for electrical 
  load pattern grouping, Energy , 42, 68 -- 80, 2012.
  
\bibitem{Figueiredo}	
Figueiredo, V., Rodrigues, F., Vale, Z., Gouveia, J. B. 
  An electric energy consumer characterization framework based on data mining 
  techniques. 
  Power Systems, IEEE Transactions on, 20(2), 596--602, 2005
 
\bibitem{iwafune2014short}	
Iwafune, Y., Yagita, Y., Ikegami, T., Ogimoto K.
  Short-term forecasting of residential building load for distributed energy 
  management 
  \emph{Proceedings Energy Conference (ENERGYCON), 2014 IEEE International}
  1197--1204, 2014
 
\bibitem{kaufmanpj}	
Kaufman, L. and Rousseeuw, P
  Finding groups in data: An introduction to cluster analysis,
  Hoboken NJ John Wiley \& Sons Inc, 1990
 
\bibitem{Kwac}
J. Kwac,  Flora, J., Rajagopal, R. 
  Household Energy Consumption Segmentation Using Hourly Data 
  Smart Grid, IEEE Transactions on, 5, 420--430, 2014
 
\bibitem{labeeuw}
Labeeuw, W., Stragier, J., and Deconinck, G. 
  Potential of active demand reduction with residential wet appliances: 
  A case study for Belgium. 
  Smart Grid, IEEE Transactions on, 6(1), 315--323, 2015

\bibitem{Liao}
Warren Liao, T. 
  Clustering of time series data--a survey
  Pattern recognition, 38(11), 1857--1874, 2005
 
\bibitem{MisitiElec}
M.~Misiti, Y.~Misiti, G.~Oppenheim, and J.-M.~Poggi,
  Optimized Clusters for Disaggregated Electricity Load Forecasting,
  \emph{REVSTAT -- Statistical Journal}, vol. 8(2), pp. 105 -- 124, 2010
   
\bibitem{Mutanen}
  Mutanen, A., Ruska, M., Repo, S., Jarventausta, P. 
  Customer classification and load profiling method for distribution systems. 
  Power Delivery, IEEE Transactions on, 26(3), 1755--1763, 2011

%\bibitem{Piao}
%Piao, M., Lee, H. G., Park, J. H., Ryu, K. H. 
%  Application of Classification Methods for Forecasting Mid-Term 
%  Power Load Patterns.
%  In Advanced Intelligent Computing Theories and Applications. Springer, 2008
 
\bibitem{Rasanen}
T., R\"{a}s\"{a}nen, D., Voukantsis,  H., Niska, K., Karatzas, M., Kolehmainen
  Data-based method for creating electricity use load profiles using large 
  amount of customer-specific hourly measured electricity use data
  Applied Energy, 87(11), 3538--3545, 2010
 
\bibitem{Rhodes}	
J.D. Rhodes, W.J. Cole, C.R. Upshaw, T.F. Edgar, M.E. Webber
  Clustering analysis of residential electricity demand profiles
  Preprint submitted to Applied Energy, March 18, 2014

\bibitem{steinley2008new}
D. Steinley and M. Brusco, 
A new variable weighting and selection procedure for k-means cluster analysis. 
\emph{Multivariate Behavioral Research}, 43:32, 2008.

\bibitem{wijaya2015forecasting} 
Wijaya, T. K., Sinn, M., and Chen, B.,
  Forecasting Uncertainty in Electricity Demand, 
  \emph{AAAI-15 Workshop on Computational Sustainability, EPFL-CONF-203769}, 
        2015

\bibitem{Zhou}
K. Zhou, S. Yang, C. Shen
  A review of electric load classification in smart grid environment, 
  Renewable and Sustainable Energy Reviews, 24, 103 -- 110, 2013.