attempt to fix F_Neighbors2

[talweg.git] / reports / report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<h2>Introduction</h2>\n",
    "\n",
    "J'ai fait quelques essais dans différentes configurations pour la méthode \"Neighbors\"\n",
    "(la seule dont on a parlé) et sa variante récente appelée pour l'instant \"Neighbors2\",\n",
    "avec simtype=\"mix\" : deux types de similarités prises en compte, puis multiplication des poids.\n",
    "Pour Neighbors on prédit le saut (par la moyenne pondérée des sauts passés), et pour Neighbors2\n",
    "on n'effectue aucun raccordement (prévision directe).\n",
    "\n",
    "J'ai systématiquement comparé à une approche naïve : la moyenne des lendemains des jours\n",
    "\"similaires\" dans tout le passé, ainsi qu'à la persistence -- reproduisant le jour courant ou\n",
    "allant chercher le futur similaire une semaine avant.\n",
    "\n",
    "Ensuite j'affiche les erreurs, quelques courbes prévues/mesurées, quelques filaments puis les\n",
    "histogrammes de quelques poids. Concernant les graphes de filaments, la moitié gauche du graphe\n",
    "correspond aux jours similaires au jour courant, tandis que la moitié droite affiche les\n",
    "lendemains : ce sont donc les voisinages tels qu'utilisés dans l'algorithme.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(talweg)\n",
    "\n",
    "P = 7 #instant de prévision\n",
    "H = 17 #horizon (en heures)\n",
    "\n",
    "ts_data = read.csv(system.file(\"extdata\",\"pm10_mesures_H_loc_report.csv\",package=\"talweg\"))\n",
    "exo_data = read.csv(system.file(\"extdata\",\"meteo_extra_noNAs.csv\",package=\"talweg\"))\n",
    "# NOTE: 'GMT' because DST gaps are filled and multiple values merged in above dataset.\n",
    "# Prediction from P+1 to P+H included.\n",
    "data = getData(ts_data, exo_data, input_tz = \"GMT\", working_tz=\"GMT\", predict_at=P)\n",
    "\n",
    "indices_ch = seq(as.Date(\"2015-01-18\"),as.Date(\"2015-01-24\"),\"days\")\n",
    "indices_ep = seq(as.Date(\"2015-03-15\"),as.Date(\"2015-03-21\"),\"days\")\n",
    "indices_np = seq(as.Date(\"2015-04-26\"),as.Date(\"2015-05-02\"),\"days\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<h2 style=\"color:blue;font-size:2em\">Pollution par chauffage</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_nn = computeForecast(data, indices_ch, \"Neighbors\", \"Neighbors\", horizon=H)\n",
    "p_nn2 = computeForecast(data, indices_ch, \"Neighbors2\", \"Zero\", horizon=H)\n",
    "p_az = computeForecast(data, indices_ch, \"Average\", \"Zero\", horizon=H)\n",
    "p_pz = computeForecast(data, indices_ch, \"Persistence\", \"Zero\", horizon=H, same_day=TRUE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e_nn = computeError(data, p_nn, H)\n",
    "e_nn2 = computeError(data, p_nn2, H)\n",
    "e_az = computeError(data, p_az, H)\n",
    "e_pz = computeError(data, p_pz, H)\n",
    "options(repr.plot.width=9, repr.plot.height=7)\n",
    "plotError(list(e_nn, e_pz, e_az, e_nn2), cols=c(1,2,colors()[258], 4))\n",
    "\n",
    "# Noir: Neighbors, bleu: Neighbors2, vert: moyenne, rouge: persistence\n",
    "\n",
    "i_np = which.min(e_nn$abs$indices)\n",
    "i_p = which.max(e_nn$abs$indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "options(repr.plot.width=9, repr.plot.height=4)\n",
    "par(mfrow=c(1,2))\n",
    "\n",
    "plotPredReal(data, p_nn, i_np); title(paste(\"PredReal nn day\",i_np))\n",
    "plotPredReal(data, p_nn2, i_p); title(paste(\"PredReal nn day\",i_p))\n",
    "\n",
    "plotPredReal(data, p_nn2, i_np); title(paste(\"PredReal nn2 day\",i_np))\n",
    "plotPredReal(data, p_nn2, i_p); title(paste(\"PredReal nn2 day\",i_p))\n",
    "\n",
    "plotPredReal(data, p_az, i_np); title(paste(\"PredReal az day\",i_np))\n",
    "plotPredReal(data, p_az, i_p); title(paste(\"PredReal az day\",i_p))\n",
    "\n",
    "# Bleu: prévue, noir: réalisée"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "f_np = computeFilaments(data, p_nn, i_np, plot=TRUE); title(paste(\"Filaments nn day\",i_np))\n",
    "f_p = computeFilaments(data, p_nn, i_p, plot=TRUE); title(paste(\"Filaments nn day\",i_p))\n",
    "\n",
    "f_np2 = computeFilaments(data, p_nn2, i_np, plot=TRUE); title(paste(\"Filaments nn2 day\",i_np))\n",
    "f_p2 = computeFilaments(data, p_nn2, i_p, plot=TRUE); title(paste(\"Filaments nn2 day\",i_p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotFilamentsBox(data, f_np); title(paste(\"FilBox nn day\",i_np))\n",
    "plotFilamentsBox(data, f_p); title(paste(\"FilBox nn day\",i_p))\n",
    "\n",
    "# Generally too few neighbors:\n",
    "#plotFilamentsBox(data, f_np2); title(paste(\"FilBox nn2 day\",i_np))\n",
    "#plotFilamentsBox(data, f_p2); title(paste(\"FilBox nn2 day\",i_p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotRelVar(data, f_np); title(paste(\"StdDev nn day\",i_np))\n",
    "plotRelVar(data, f_p); title(paste(\"StdDev nn day\",i_p))\n",
    "\n",
    "plotRelVar(data, f_np2); title(paste(\"StdDev nn2 day\",i_np))\n",
    "plotRelVar(data, f_p2); title(paste(\"StdDev nn2 day\",i_p))\n",
    "\n",
    "# Variabilité globale en rouge ; sur les 60 voisins (+ lendemains) en noir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotSimils(p_nn, i_np); title(paste(\"Weights nn day\",i_np))\n",
    "plotSimils(p_nn, i_p); title(paste(\"Weights nn day\",i_p))\n",
    "\n",
    "plotSimils(p_nn2, i_np); title(paste(\"Weights nn2 day\",i_np))\n",
    "plotSimils(p_nn2, i_p); title(paste(\"Weights nn2 day\",i_p))\n",
    "\n",
    "# - pollué à gauche, + pollué à droite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fenêtres sélectionnées dans ]0,7] / nn à gauche, nn2 à droite\n",
    "p_nn$getParams(i_np)$window\n",
    "p_nn$getParams(i_p)$window\n",
    "\n",
    "p_nn2$getParams(i_np)$window\n",
    "p_nn2$getParams(i_p)$window"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<h2 style=\"color:blue;font-size:2em\">Pollution par épandage</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_nn = computeForecast(data, indices_ep, \"Neighbors\", \"Neighbors\", horizon=H)\n",
    "p_nn2 = computeForecast(data, indices_ep, \"Neighbors2\", \"Zero\", horizon=H)\n",
    "p_az = computeForecast(data, indices_ep, \"Average\", \"Zero\", horizon=H)\n",
    "p_pz = computeForecast(data, indices_ep, \"Persistence\", \"Zero\", horizon=H, same_day=TRUE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e_nn = computeError(data, p_nn, H)\n",
    "e_nn2 = computeError(data, p_nn2, H)\n",
    "e_az = computeError(data, p_az, H)\n",
    "e_pz = computeError(data, p_pz, H)\n",
    "options(repr.plot.width=9, repr.plot.height=7)\n",
    "plotError(list(e_nn, e_pz, e_az, e_nn2), cols=c(1,2,colors()[258], 4))\n",
    "\n",
    "# Noir: Neighbors, bleu: Neighbors2, vert: moyenne, rouge: persistence\n",
    "\n",
    "i_np = which.min(e_nn$abs$indices)\n",
    "i_p = which.max(e_nn$abs$indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "options(repr.plot.width=9, repr.plot.height=4)\n",
    "par(mfrow=c(1,2))\n",
    "\n",
    "plotPredReal(data, p_nn, i_np); title(paste(\"PredReal nn day\",i_np))\n",
    "plotPredReal(data, p_nn2, i_p); title(paste(\"PredReal nn day\",i_p))\n",
    "\n",
    "plotPredReal(data, p_nn2, i_np); title(paste(\"PredReal nn2 day\",i_np))\n",
    "plotPredReal(data, p_nn2, i_p); title(paste(\"PredReal nn2 day\",i_p))\n",
    "\n",
    "plotPredReal(data, p_az, i_np); title(paste(\"PredReal az day\",i_np))\n",
    "plotPredReal(data, p_az, i_p); title(paste(\"PredReal az day\",i_p))\n",
    "\n",
    "# Bleu: prévue, noir: réalisée"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "f_np = computeFilaments(data, p_nn, i_np, plot=TRUE); title(paste(\"Filaments nn day\",i_np))\n",
    "f_p = computeFilaments(data, p_nn, i_p, plot=TRUE); title(paste(\"Filaments nn day\",i_p))\n",
    "\n",
    "f_np2 = computeFilaments(data, p_nn2, i_np, plot=TRUE); title(paste(\"Filaments nn2 day\",i_np))\n",
    "f_p2 = computeFilaments(data, p_nn2, i_p, plot=TRUE); title(paste(\"Filaments nn2 day\",i_p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotFilamentsBox(data, f_np); title(paste(\"FilBox nn day\",i_np))\n",
    "plotFilamentsBox(data, f_p); title(paste(\"FilBox nn day\",i_p))\n",
    "\n",
    "# Generally too few neighbors:\n",
    "#plotFilamentsBox(data, f_np2); title(paste(\"FilBox nn2 day\",i_np))\n",
    "#plotFilamentsBox(data, f_p2); title(paste(\"FilBox nn2 day\",i_p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotRelVar(data, f_np); title(paste(\"StdDev nn day\",i_np))\n",
    "plotRelVar(data, f_p); title(paste(\"StdDev nn day\",i_p))\n",
    "\n",
    "plotRelVar(data, f_np2); title(paste(\"StdDev nn2 day\",i_np))\n",
    "plotRelVar(data, f_p2); title(paste(\"StdDev nn2 day\",i_p))\n",
    "\n",
    "# Variabilité globale en rouge ; sur les 60 voisins (+ lendemains) en noir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotSimils(p_nn, i_np); title(paste(\"Weights nn day\",i_np))\n",
    "plotSimils(p_nn, i_p); title(paste(\"Weights nn day\",i_p))\n",
    "\n",
    "plotSimils(p_nn2, i_np); title(paste(\"Weights nn2 day\",i_np))\n",
    "plotSimils(p_nn2, i_p); title(paste(\"Weights nn2 day\",i_p))\n",
    "\n",
    "# - pollué à gauche, + pollué à droite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fenêtres sélectionnées dans ]0,7] / nn à gauche, nn2 à droite\n",
    "p_nn$getParams(i_np)$window\n",
    "p_nn$getParams(i_p)$window\n",
    "\n",
    "p_nn2$getParams(i_np)$window\n",
    "p_nn2$getParams(i_p)$window"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "<h2 style=\"color:blue;font-size:2em\">Semaine non polluée</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_nn = computeForecast(data, indices_np, \"Neighbors\", \"Neighbors\", horizon=H)\n",
    "p_nn2 = computeForecast(data, indices_np, \"Neighbors2\", \"Zero\", horizon=H)\n",
    "p_az = computeForecast(data, indices_np, \"Average\", \"Zero\", horizon=H)\n",
    "p_pz = computeForecast(data, indices_np, \"Persistence\", \"Zero\", horizon=H, same_day=FALSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e_nn = computeError(data, p_nn, H)\n",
    "e_nn2 = computeError(data, p_nn2, H)\n",
    "e_az = computeError(data, p_az, H)\n",
    "e_pz = computeError(data, p_pz, H)\n",
    "options(repr.plot.width=9, repr.plot.height=7)\n",
    "plotError(list(e_nn, e_pz, e_az, e_nn2), cols=c(1,2,colors()[258], 4))\n",
    "\n",
    "# Noir: Neighbors, bleu: Neighbors2, vert: moyenne, rouge: persistence\n",
    "\n",
    "i_np = which.min(e_nn$abs$indices)\n",
    "i_p = which.max(e_nn$abs$indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "options(repr.plot.width=9, repr.plot.height=4)\n",
    "par(mfrow=c(1,2))\n",
    "\n",
    "plotPredReal(data, p_nn, i_np); title(paste(\"PredReal nn day\",i_np))\n",
    "plotPredReal(data, p_nn2, i_p); title(paste(\"PredReal nn day\",i_p))\n",
    "\n",
    "plotPredReal(data, p_nn2, i_np); title(paste(\"PredReal nn2 day\",i_np))\n",
    "plotPredReal(data, p_nn2, i_p); title(paste(\"PredReal nn2 day\",i_p))\n",
    "\n",
    "plotPredReal(data, p_az, i_np); title(paste(\"PredReal az day\",i_np))\n",
    "plotPredReal(data, p_az, i_p); title(paste(\"PredReal az day\",i_p))\n",
    "\n",
    "# Bleu: prévue, noir: réalisée"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "f_np = computeFilaments(data, p_nn, i_np, plot=TRUE); title(paste(\"Filaments nn day\",i_np))\n",
    "f_p = computeFilaments(data, p_nn, i_p, plot=TRUE); title(paste(\"Filaments nn day\",i_p))\n",
    "\n",
    "f_np2 = computeFilaments(data, p_nn2, i_np, plot=TRUE); title(paste(\"Filaments nn2 day\",i_np))\n",
    "f_p2 = computeFilaments(data, p_nn2, i_p, plot=TRUE); title(paste(\"Filaments nn2 day\",i_p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotFilamentsBox(data, f_np); title(paste(\"FilBox nn day\",i_np))\n",
    "plotFilamentsBox(data, f_p); title(paste(\"FilBox nn day\",i_p))\n",
    "\n",
    "# Generally too few neighbors:\n",
    "#plotFilamentsBox(data, f_np2); title(paste(\"FilBox nn2 day\",i_np))\n",
    "#plotFilamentsBox(data, f_p2); title(paste(\"FilBox nn2 day\",i_p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotRelVar(data, f_np); title(paste(\"StdDev nn day\",i_np))\n",
    "plotRelVar(data, f_p); title(paste(\"StdDev nn day\",i_p))\n",
    "\n",
    "plotRelVar(data, f_np2); title(paste(\"StdDev nn2 day\",i_np))\n",
    "plotRelVar(data, f_p2); title(paste(\"StdDev nn2 day\",i_p))\n",
    "\n",
    "# Variabilité globale en rouge ; sur les 60 voisins (+ lendemains) en noir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "par(mfrow=c(1,2))\n",
    "plotSimils(p_nn, i_np); title(paste(\"Weights nn day\",i_np))\n",
    "plotSimils(p_nn, i_p); title(paste(\"Weights nn day\",i_p))\n",
    "\n",
    "plotSimils(p_nn2, i_np); title(paste(\"Weights nn2 day\",i_np))\n",
    "plotSimils(p_nn2, i_p); title(paste(\"Weights nn2 day\",i_p))\n",
    "\n",
    "# - pollué à gauche, + pollué à droite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fenêtres sélectionnées dans ]0,7] / nn à gauche, nn2 à droite\n",
    "p_nn$getParams(i_np)$window\n",
    "p_nn$getParams(i_p)$window\n",
    "\n",
    "p_nn2$getParams(i_np)$window\n",
    "p_nn2$getParams(i_p)$window"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}