543 lines
47 KiB
Plaintext
543 lines
47 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "945c9b80",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Table of contents\n",
|
|
"1. [Introduction](#introduction)\n",
|
|
"2. [Aggregate Model Evaluation](#modelevaluation)\n",
|
|
" 1. [Loading the dataset](#modeload)\n",
|
|
" 2. [Perform detections](#modeldetect)\n",
|
|
" 3. [Evaluate detections](#modeldetectionseval)\n",
|
|
" 4. [Calculate results and plot them](#modelshowresults)\n",
|
|
" 5. [View dataset in fiftyone](#modelfiftyonesession)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "01339680",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Introduction <a name=\"introduction\"></a>\n",
|
|
"\n",
|
|
"This notebook loads the test dataset in YOLOv5 format from disk and evaluates the model's performance."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "ff25695e",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/home/zenon/.local/share/miniconda3/lib/python3.7/site-packages/requests/__init__.py:104: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (5.1.0)/charset_normalizer (2.0.4) doesn't match a supported version!\n",
|
|
" RequestsDependencyWarning)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import fiftyone as fo\n",
|
|
"from PIL import Image\n",
|
|
"from detection import detect\n",
|
|
"from detection import detect_yolo_only"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86a5e832",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Aggregate Model Evaluation <a name=\"modelevaluation\"></a>\n",
|
|
"\n",
|
|
"First, load the dataset from the directory containing the images and the labels in YOLOv5 format.\n",
|
|
"\n",
|
|
"### Loading the dataset <a name=\"modeload\"></a>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "bea1038e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"name = \"dataset-new\"\n",
|
|
"dataset_dir = \"dataset\"\n",
|
|
"\n",
|
|
"# The splits to load\n",
|
|
"splits = [\"val\"]\n",
|
|
"\n",
|
|
"# Load the dataset, using tags to mark the samples in each split\n",
|
|
"dataset = fo.Dataset(name)\n",
|
|
"for split in splits:\n",
|
|
" dataset.add_dir(\n",
|
|
" dataset_dir=dataset_dir,\n",
|
|
" dataset_type=fo.types.YOLOv5Dataset,\n",
|
|
" split=split,\n",
|
|
" tags=split,\n",
|
|
" )\n",
|
|
"\n",
|
|
"dataset.persistent = True\n",
|
|
"classes = dataset.default_classes"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "361eeecd",
|
|
"metadata": {},
|
|
"source": [
|
|
"If the dataset already exists because it had been saved under the same name before, load the dataset from fiftyone's folder."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "2d479be8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"dataset = fo.load_dataset('dataset')\n",
|
|
"classes = dataset.default_classes"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4485dce3",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Perform detections <a name=\"modeldetect\"></a>\n",
|
|
"\n",
|
|
"Now we can call the aggregate model to do detections on the images contained in the dataset. The actual detection happens at line 6 where `detect()` is called. This function currently does inference using the GPU via `onnxruntime-gpu`. All detections are saved to the `predictions` keyword of each sample. A sample is one image with potentially multiple detections.\n",
|
|
"\n",
|
|
"> **_NOTE:_** If the dataset already existed beforehand (you used `load_dataset()`), the detections are likely already saved in the dataset and you can skip the next step."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "63f675ab",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" 100% |█████████████████| 640/640 [8.7m elapsed, 0s remaining, 1.4 samples/s] \n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Do detections with model and save bounding boxes\n",
|
|
"with fo.ProgressBar() as pb:\n",
|
|
" for sample in pb(dataset.view()):\n",
|
|
" image = Image.open(sample.filepath)\n",
|
|
" w, h = image.size\n",
|
|
" pred = detect(sample.filepath, '../weights/yolo-final.onnx', '../weights/resnet-fold-7.onnx')\n",
|
|
"\n",
|
|
" detections = []\n",
|
|
" for _, row in pred.iterrows():\n",
|
|
" xmin, xmax = int(row['xmin']), int(row['xmax'])\n",
|
|
" ymin, ymax = int(row['ymin']), int(row['ymax'])\n",
|
|
" rel_box = [\n",
|
|
" xmin / w, ymin / h, (xmax - xmin) / w, (ymax - ymin) / h\n",
|
|
" ]\n",
|
|
" detections.append(\n",
|
|
" fo.Detection(label=classes[int(row['cls'])],\n",
|
|
" bounding_box=rel_box,\n",
|
|
" confidence=int(row['cls_conf'])))\n",
|
|
"\n",
|
|
" sample[\"predictions_yolo_resnet_final\"] = fo.Detections(detections=detections)\n",
|
|
" sample.save()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "10d94167",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Evaluate detections against ground truth <a name=\"modeldetectionseval\"></a>\n",
|
|
"\n",
|
|
"Having saved the predictions, we can now evaluate them by cross-checking with the ground truth labels. If we specify an `eval_key`, true positives, false positives and false negatives will be saved under that key."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "68cfdad2",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Evaluating detections...\n",
|
|
" 100% |█████████████████| 640/640 [2.2s elapsed, 0s remaining, 278.4 samples/s] \n",
|
|
"Performing IoU sweep...\n",
|
|
" 100% |█████████████████| 640/640 [2.4s elapsed, 0s remaining, 270.2 samples/s] \n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results = dataset.view().evaluate_detections(\n",
|
|
" \"predictions_yolo_resnet_final\",\n",
|
|
" gt_field=\"ground_truth\",\n",
|
|
" eval_key=\"eval_yolo_resnet_final\",\n",
|
|
" compute_mAP=True,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94b9751f",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Calculate results and plot them <a name=\"modelshowresults\"></a>\n",
|
|
"\n",
|
|
"Now we have the performance of the model saved in the `results` variable and can extract various metrics from that. Here we print a simple report of all classes and their precision and recall values as well as the mAP with the metric employed by [COCO](https://cocodataset.org/#detection-eval). Next, a confusion matrix is plotted for each class (in our case only one). Finally, we can show the precision vs. recall curve for a specified threshold value."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "86b90e80",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from helpers import set_size\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import seaborn as sns\n",
|
|
"import pandas as pd"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "e34a18f4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Style the plots\n",
|
|
"width = 418\n",
|
|
"sns.set_theme(style='whitegrid',\n",
|
|
" rc={'text.usetex': True, 'font.family': 'serif', 'axes.labelsize': 10,\n",
|
|
" 'font.size': 10, 'legend.fontsize': 8,\n",
|
|
" 'xtick.labelsize': 8, 'ytick.labelsize': 8})"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8ee61fce",
|
|
"metadata": {},
|
|
"source": [
|
|
"The code for the LaTeX table of the classification report can be printed by first converting the results to a pandas DataFrame and then calling the `to_latex()` method of the DataFrame. This code can then be inserted into the LaTeX document."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "b14d2b25",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\\begin{tabular}{lrrrr}\n",
|
|
"\\toprule\n",
|
|
"{} & precision & recall & f1-score & support \\\\\n",
|
|
"\\midrule\n",
|
|
"Healthy & 0.841 & 0.759 & 0.798 & 663.0 \\\\\n",
|
|
"Stressed & 0.726 & 0.810 & 0.766 & 484.0 \\\\\n",
|
|
"micro avg & 0.786 & 0.780 & 0.783 & 1147.0 \\\\\n",
|
|
"macro avg & 0.784 & 0.784 & 0.782 & 1147.0 \\\\\n",
|
|
"weighted avg & 0.793 & 0.780 & 0.784 & 1147.0 \\\\\n",
|
|
"\\bottomrule\n",
|
|
"\\end{tabular}\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
|
|
"\n",
|
|
"# Export DataFrame to LaTeX tabular environment\n",
|
|
"print(results_df.to_latex())\n",
|
|
"# YOLO original with Resnet original and new dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "900e9014",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\\begin{tabular}{lrrrr}\n",
|
|
"\\toprule\n",
|
|
"{} & precision & recall & f1-score & support \\\\\n",
|
|
"\\midrule\n",
|
|
"Healthy & 0.674 & 0.721 & 0.696 & 662.0 \\\\\n",
|
|
"Stressed & 0.616 & 0.543 & 0.577 & 488.0 \\\\\n",
|
|
"micro avg & 0.652 & 0.645 & 0.649 & 1150.0 \\\\\n",
|
|
"macro avg & 0.645 & 0.632 & 0.637 & 1150.0 \\\\\n",
|
|
"weighted avg & 0.649 & 0.645 & 0.646 & 1150.0 \\\\\n",
|
|
"\\bottomrule\n",
|
|
"\\end{tabular}\n",
|
|
"\n",
|
|
"0.49320073714096757\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
|
|
"\n",
|
|
"# Export DataFrame to LaTeX tabular environment\n",
|
|
"print(results_df.to_latex())\n",
|
|
"print(results.mAP())\n",
|
|
"# YOLO original and Resnet final with old dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 51,
|
|
"id": "24df35b4",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" precision recall f1-score support\n",
|
|
"\n",
|
|
" Healthy 0.82 0.74 0.78 662\n",
|
|
" Stressed 0.71 0.78 0.74 488\n",
|
|
"\n",
|
|
" micro avg 0.77 0.76 0.76 1150\n",
|
|
" macro avg 0.77 0.76 0.76 1150\n",
|
|
"weighted avg 0.77 0.76 0.77 1150\n",
|
|
"\n",
|
|
"0.6225848121901868\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Print a classification report for all classes\n",
|
|
"results.print_report()\n",
|
|
"\n",
|
|
"print(results.mAP())\n",
|
|
"# YOLO original and Resnet original with old dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "a6bb272a",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" precision recall f1-score support\n",
|
|
"\n",
|
|
" Healthy 0.66 0.64 0.65 662\n",
|
|
" Stressed 0.68 0.54 0.60 488\n",
|
|
"\n",
|
|
" micro avg 0.67 0.60 0.63 1150\n",
|
|
" macro avg 0.67 0.59 0.63 1150\n",
|
|
"weighted avg 0.67 0.60 0.63 1150\n",
|
|
"\n",
|
|
"0.44258882390400406\n",
|
|
"\\begin{tabular}{lrrrr}\n",
|
|
"\\toprule\n",
|
|
"{} & precision & recall & f1-score & support \\\\\n",
|
|
"\\midrule\n",
|
|
"Healthy & 0.664 & 0.640 & 0.652 & 662.0 \\\\\n",
|
|
"Stressed & 0.680 & 0.539 & 0.601 & 488.0 \\\\\n",
|
|
"micro avg & 0.670 & 0.597 & 0.631 & 1150.0 \\\\\n",
|
|
"macro avg & 0.672 & 0.590 & 0.626 & 1150.0 \\\\\n",
|
|
"weighted avg & 0.670 & 0.597 & 0.630 & 1150.0 \\\\\n",
|
|
"\\bottomrule\n",
|
|
"\\end{tabular}\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Print a classification report for all classes\n",
|
|
"results.print_report()\n",
|
|
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
|
|
"\n",
|
|
"print(results.mAP())\n",
|
|
"print(results_df.to_latex())\n",
|
|
"# YOLO final and Resnet final with old dataset"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "da05e2ba",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Ignoring unsupported argument `thresholds` for the 'matplotlib' backend\n",
|
|
"Ignoring unsupported argument `thresholds` for the 'matplotlib' backend\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAACoCAYAAADtjJScAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAxZUlEQVR4nO3de3BT55kG8McQA75JghiIi48DDobYshMwoV3LuylsaFd4trA4jcVs6A6kMe7SGbwl0G3Z4iQmk83ETilM2lmiJEy3zay12TiQTLHCpSEztczN3GzZ4bZ2JHEztEgy2OayaP9wz6mOLrasu+XnN8OApE/Sa8nn5T3f+S5JLpfLBSIiIqIEMi7WARARERGFGwscIiIiSjgscIiIiCjhsMAhIiKihMMCh4iIiBIOCxwiIiJKOCxwiIiIKOE8FOsAIunkyZNwuVxITk6OdShEY8K9e/eQlJSE+fPnxzqUqGGeIYqekeSYhO7BcblcCGQdQ5fLhbt37wbUNhYYX2jiPT4g/mMMNL5Aj7lEwjwTHYwvNIkS30hyTEL34IhnVEVFRUO26+vrQ2dnJ2bPno3U1NRohDYijC808R4fEP8xBhpfW1tbFKOKD8wz0cH4QpMo8Y0kx0S9wHE6nTAYDACAyspKn22MRiMAwOFwQBAEaDSaqMVHRKMf8wwRRf0Slclkgt1u9/u41WqFyWSCVquFTqeDXq+PXnBElBCYZ4go6j04Wq0WDocDTqfT5+MmkwkZGRnS7YyMDJhMpqDPrlwuF/r6+oZs09/fj7a2Nhw7dgwTJkyQ7lcoFMjKysLdu3fR1dXl9by5c+cCACwWC/r7+2WPZWVlQaFQwG6349q1a9L9qampKCgowOzZswP+GcTX9nyPeMH4QhfvMQYan8vlQlJSUjRCGtJYyTN5eXlYsGBBUDH7is/973jD+EKTKPGNJMfE3Rgci8UClUol3VapVH6TVCDu3buHzs7OIdvY7Xa88MILURt89eSTT+K9994b8fO6u7vDH0wYMb7QxXuMgcTn/p93vEqkPLNr1y50dnbiW9/6FiZPnhzy6yXC72AsMb7QhDPHxF2B44vD4Qj6ucnJycP2lvT39+P999/HwMBARHtwvvrqK9TW1uJf//VfMWPGDKSlpWH8+PHD/gz9/f3o7u7GzJkzkZKSMmz7aGN8oYv3GAON78KFC1GMKrxGW54R88mDBw/w5ptvYtmyZcjPzw/6Z0iU38FYYXyhiUSOibsCJycnR3YmZbfbIQhC0K+XlJQU0IjxoqIi5Ofn+227ePFiv88tLS0NKJYTJ05ICSkrKwutra0oLi4O6LkAkJKSEpej30WML3TxHuNw8cXD5alAJEKeEfPJxIkTAYTvd2e0/w7GGuMLTThzTNysgyMmG41GI5sGZrPZOLuBiMKCeYZo7Ih6D47JZEJzczN6e3shCAK0Wi0AoLy8HI2NjRAEAWVlZTAajXA4HFi7dm20QySiUY55hoiiXuBoNBqfZ0oHDhyQ/i0mIyKiYIylPJOWloZvf/vbsllhRBSHY3DGAnEg4XCzLoDB6aszZsyIdEhENErl5OTgs88+i3UYRHGHBU4UiWdYW7ZsAQCsWrUqoOf993//N+7evYuBgQFMmzYNeXl5EYuRiEaX//u//4PT6Qx4VibRWMECJ4ry8vJw7tw59Pb2BtTeZrNh+fLlqKiokN2/Z88eZGdny+7LyMhg4UM0Bp07dw5f//rXRzwrkyjRscCJsry8PFy8eBE/+tGPUFVVhaysLADAjBkzMH36dNy8eRNdXV3IyMjAsmXLcO7cOfT09KC7uxsTJkxARUUFli9f7vO13QsfFjxERDSWscCJgQcPHuDTTz/Fp59+Kt331ltvYcOGDThw4IDUY3Pu3Dnk5eVhxowZmDRpEvLz8332AIk9PZ6Fz549e5Cfn89Ch4iIxhwWODHg61KVOJB4yZIl+O1vf4tVq1b5vJTlq1gpLi6WvZ5nwSMWSkRERGMFC5wY8VdwTJ482Wu59VOnTqGrqwsDAwPSEtb5+fmy5azdX08seI4ePeq3UCIiIkpkLHDikDh+Rlw6fvHixbh7966sTXt7O9Rqtd/XyMvLkwob9+noHJtDlFjy8vLQ09Mj2zyUiFjgxKW8vDz8/ve/x9SpUwEAn3/+Obq6umSbkCkUCpw4cUJ6jq/CRZyW7jkdnZesiBLHQw89JOUKIvoLFjhxyn0a+Lx58zBx4kRpk74//elPWLduHQwGg9Rm3Lhx+PLLL/HII4/g/PnzUsHjPjans7MTq1atwtGjR/1etmIPD9HoYrVa8corr2Dbtm147LHHYh0OUdxggTMKTZkyBVu3bsWPf/xjAIODivfv3w+VSoXW1lZpR2LPnhp/PTqeON2caPS4desWPv30U7zyyiuxDoUorrDAGaU8BxUvWrQICoUCCxYs8DsLa7iFBjndnIiIEgULnAShUCgADPa4eM7Camtrw6RJk5CXlzdkkTLcdHP27BAR0WjBAicBZWRkQKlUSrOwli5dikuXLgU0uNjXdPPOzs4he3a4GSgREcWbcbEOgMIvLy8Pp06dkmZWbNu2DQBkg4udTmfAryVuGdHa2orW1lbs2bMHALB8+XLMmTMHv/vd72CxWCLwkxDRcKZNm4a33nqLJxpEHljgJKiZM2dK/543bx6AwcHFra2tuH79Ov7t3/4N58+fD/j18vLyUFxcjOLiYqngEQudiooKlJeX48KFC+H8EYgoAA8//DA2bNiA6dOnxzoUorjCAmcMEAcXt7a2YsGCBbBarfjVr36Fo0eP4sSJE/jqq68AAP39/QEXPe49O++99x4A4Pjx4yMqmogodE6nEx9++CFu3rwZ61CI4kpYx+DYbDbZ+i0UPzyniz948ECaLv79738f7777Lo4cOYLFixePaNZUXl4e+vv7pdcBOBiZIoc5xtulS5ek3tnJkyfHOhyiuBFSgdPZ2Qm73S7dNhgM+MUvfhFiSBRpntPFH374YQCD6+sA8Nqks7OzUypifBUss2fPRmNjI+7fv4+Kigqfg5FZ8FAwmGOIKFhBFzjV1dXo7e2VFo8D5HseUXzzVWQ88cQTsllT4t42zz//PE6ePCm1O3LkCL7+9a/LnpuTk4P8/Pwhp5mLWPBQIJhjiCgUQRc4paWlqKiokN332WefhRwQxZa4Vs758+elWVgffPAB+vv7pYLl+PHjePTRR3Hp0iUAg2N3uru7MXHiRBQWFkqvNdy6OiIuJEi+MMcE5+TJk3C5XAB4AkFjW9AFjrjGirucnJyQgqH4MXv2bOnf4sKBYsGSmZmJXbt24aWXXpI9p6ysDLt370ZbW5uUWH2tqxPIQoIiJuixizkmMFevXoUgCLh58yZOnDiBb3zjG7h//770+Mcff+z3c+PxRYks6ALHarXCYDCgqKgIAOByudDU1ISPPvoobMFR/BGT4fPPP49FixYB+EsPzpNPPgm73Y4FCxYA8N0zM5KFBEXc/XxsYo4ZmnjpbuPGjQCAJUuW+Gy3YsWKIV+HC3ZSogq6wGloaIBGo5G6QgHI/k2Jbfr06dK6G319fZg0aRJyc3ORnJyMPXv2yAqWc+fOAYDXHliCICAvLw8PP/wwPv74Y9y+fVt6LCUlBf39/Vi1ahUMBgNmzZoFAEhLS0NOTg4KCgowadIkdHV1eU2PzcrKQlZWVsR+dooO5pihDbe33MDAAC5duoQZM2bgzTfflNatEj377LP46KOPpOP09OnTEY+ZKJqCLnA2bdqEkpIS2X0ajSbkgGh0S05OltbHsdvtmDJlCmbOnImysjLs27dP1vbtt9/GD3/4Q+zduxff+973ZI/91V/9Ff7zP/8TALBlyxav99m/fz+WLFmCLVu24IMPPpA99vLLL2PDhg3SwoP9/f3o6enx2qOL4htzzPAC7dncvn07ampqZPelpaXhH/7hHzAwMIDKykocP34ckydP5nFCCSPoAqekpAS3bt1CU1MTgMH9jgoKCsIWGI1unon37bff9tmDAwyO3WltbZU9lpaWJp2hHj16FADQ09ODDRs2AAC+9a1v4dy5c9i6dat0nygrKwstLS3QarWy+0+fPo0nnngCZ86ckcYocAxC/GKOCZ9HH30Ujz76qOy+EydO4Hvf+x7q6+sB/GUdK/E4IRrtQhqDU11dLf0npdfrsX37dlb/5NNQRcSUKVOkNXh8Pc/9uX//93+Po0ePYtWqVejt7UVxcbF0+cpdWlqaVDRdvHgR//zP/wylUgkAeOaZZ3Djxg2p7YkTJzB//nzYbDb09/ez4IkTzDGR5TmG5/XXX8fmzZtx/PhxpKSk8DigUS/oAmffvn1obGyU3ffWW28x+VBE5eXl+R1z4E6hUKC4uBgA8Pjjj+PXv/61NO394MGDuH//vjSLy2q1Yv78+fj5z3+Obdu2cVBznGCOiSyxh7SzsxP/9E//JF0OdF+RnMs30GgWdIHja7l09zVQiOLJI488Iv1b7H4XZ3GJA5LLysqwbdu2gAooijzmmMgTe0hPnTqFnp4eTJkyBT/96U+xadOmIZdvEPESL8WzkC5RebLZbCEFQxRt7snZ32Uyig3mmOiZOXMmLl26hD/96U+YMmUKPv74Y9y5cwcrV670u3yDiD2eFK+CLnA0Gg1eeOEFqNVqAIDJZJKu5RIRhYo5JrrEMTniJarPP/98yGnonZ2dWLVqFY4ePSq1YY8OxZOgC5yCggK8+uqrMBgMAICtW7dyhgONapmZmVi3bh0yMzNjHQqBOSbaxE1zp02bhpSUFKSlpeH999/HkiVLkJOT41W4iAXRqlWrZPdz7A7Fi5B2ExcEQXZGZbPZ/F6rJYp3OTk5+OUvfxnrMMgNc0x0iZvmpqam4vz583jjjTfwxhtvAPC+FOW50KCvrVdY6FAsBVzg7Nu3DxqNBunp6QCADz/8UPa40+mEyWTCe++9F94IiaKkr68PX375JR5//HGkpqbGOpwxhzkmvrivQyUuy+Crjcjf1issdChWAi5w/uM//gMZGRnSVML/+q//QllZmayN55L5RKPJl19+iQULFqC1tVWaYk7RwxwTf8SiRKfTYerUqTh06BCSk5ORkpICAEhKSsL8+fNl7d2nn7PQoVgKuMDxXI/itdde87oezmXUiShYzDHxKS8vD7/61a/wxz/+EYsXL5Y9NmHCBNy5cwfXr1+H1WqVBhmz0KF4EPQYnL1798JsNmPp0qWorq5GRkYGysrKOAiQiMKCOSZ+iKuNe86qSkpKgs1mw9/+7d/i/PnzAORjdYYqdDi9nCJtXLBPLCoqwnPPPYeGhgbk5+fjF7/4Bex2exhDI6KxjDkm/uTl5aG4uFj6M3/+fGRnZ+N3v/sdfvvb3wIALBaLz+eJm/CK7Y4ePSoVRUSREHQPjkKhAAA0NTXhtddeAwBprx+i0WjcuHHIyMjAuHFB1/0URswxo4d7T8zZs2dRWFiIS5cuydoolUpZO3F6OS9ZUaSEvJKx1WpFfn4+rFYrnE5n2AIjirZ58+bxdziOMMeMLuKlqMzMTOzatQsvvfSS7PHvfOc7+OSTT4Ycm+O5BAAXDqRQBF3gLF26FAaDAR999BF6e3thMBgwefLkgJ5rNBoBAA6HA4Ig+Bw4uH79elRVVQEYvBa/adOmYEMlolEolBwDMM/EgliMPP/881i0aJHssbS0NHR0dKC3txfJycnIzs7Gxx9/DABYsWKF3y0hOFaHghV0gZORkYEXX3xRur1x48aA9omxWq0wmUyora0FAKxZs8Zn4rHZbFi9ejUKCwuxffv2YMMkClhHRweee+45fPjhhxzIGgeCzTEA80ysTZ8+HdOnT5fdd/78eTz++ONebR0OBw4fPowLFy7I7nc6nVi3bp20FUR/fz+6u7sxMDCAadOmseihYUV9oT+TySQt8Q0MJjGTyeSVfNauXQutVhtoeH65XC709fUN2aa/v1/2d7xhfP7fb7jv1rP9UG7evImOjg7cvHkzoNcNp0T5jl0uF5KSkoJ6j3Au9Mc8M3KRjm/GjBk4ffq014KB169fx7/8y7/g8OHDsvt/+tOfAvDeCkJ0+vRpAINbTMSDsf79hioSOSbqC/1ZLBaoVCrptkql8nldva2tDcBgdQ8MLjQVjHv37qGzszOgtt3d3UG9R7QwPvn7dHd3Y9KkSSN+XrhfN5wS4TueMGFCUK8dzoX+mGeCF+n4PI8tu92OH//4x14F4vTp09HX1+fVs5adnQ2bzYbm5masW7cOb731FmbNmoWcnJyIxh2osf79hiqcOSYuFvoTk4s792vhS5YswdKlS6VZFSORnJw8bIUvdn3OnDlTWqEznjA+uYGBAQDAzJkzkZ+fP2z7QOMb6euGU6J8x56XGUYi0gv9Mc8MLZbx+TveHn/8cTz33HMAgDt37uDKlSu4f/8+Vq9ejSlTpgCANJj56NGjUKvV6O7ulgrhjIyMqPXw8PsNTSRyTNBjcARBwHvvvQedTof09HS0tLSgqKho2Ofl5OTIzqTsdjsEQZC1MRqNaGtrk5KPQqGA1WqFWq0ecZxJSUkB7yuUkpIS13sQMb6/vE8w7zdc+2BfN5xG+3cc7OUpX4LNMQDzTCjiKb7U1FRpZlVfXx86OzulExHP2Vj3799HamoqXn/9dXzwwQfSa0R7kHI8fX6+jPb4RpJjgl7wo6mpSdZdXFJSApPJNOzzNBqN1C0MDA7yE8/KxIQkCAJKS0ulNk6nM6ikQzQSubm52LNnD3Jzc2MdCiH4HAMwz4wV7gsIintibd26Fa2trdKCgr42CaWxIegeHJVKhYqKihE/TxAElJWVwWg0wuFwYO3atdJj5eXlaGxshFqthtFolM6wdu3aFWyYRAFTqVRYtmxZrMOgPws2xwDMM2ONew/NrFmzMGvWLGRlZeHll19GVlZWDCOjWAq6wDlz5oxsxgMwOGDv29/+9rDP9Tdr4cCBA15twjHDgSgQV69exa5du7BmzRo88sgjsQ5nzAslxwDMM2NdVlYWXnnllViHQTEUdIGj0+mwYsUK5OTkICMjAx0dHXj11VfDGRtRVF2+fBmbN2/G3/3d37HAiQPMMRQKp9OJlpYWlJSUBDVwnEa/kAYZNzY2oqmpCU6nEy+99JLXID4iomAxx1AoLly4AK1Wi9bWVhQXF8c6HIqBkHYVNBgMMJlMePHFF2Gz2XDr1q1wxUVExBxDREELusCpr6+HQqGQZiaMZIYDEdFwmGOIKBRBFzhFRUWoqKhglzElDJVKhe9+97uyFXApdphjiCgUQRc4vja9c193gmi0yc3NxYcffsh1cOIEcwyFYuLEiXjssccwceLEWIdCMRL0IOOCggKUl5dj8uTJMJlMMJlM2LhxYzhjI4qqu3fvoqenB9OmTQt6PyUKH+YYCoVarQ5p6xAa/YLuwSkpKcH27duRn58Pl8uFrVu3SpvkEY1G7e3tEAQB7e3tsQ6FwBxDRKEJugfn2WefRVVVFc+oiCgimGMoFGfOnMEzzzyDgwcP4oknnoh1OBQDQffg6HQ6rxVFW1paQg6IiAhgjqHQ3L9/Hzdu3MD9+/djHQrFSNA9OElJSXj55ZeRk5MDQRDgcDhgNBrZhUxEYcEcQ0ShCLrAeeedd1BSUoKbN29KO/7a7fZwxUVEYxxzDBGFIugCp7a21utMit3H4bF+/Xrs2LFDdl9XVxfef/99XLlyBTqdDgBgsVhQWloqLYQWDkajEQDgcDggCILP116/fj2qqqoAAHv37sUPf/hDAMD+/fsxMDAAs9kMrVYbVFxmsxnV1dVobGyM+v4x8+bNw8DAAJKTk6VYdu7cCZvNho0bN0Kj0cBoNGLLli2oqKhAVVXViGIc7mfr6upCVVUVfv/733PvHDDHRFq85xmj0Sgb9P+d73wHANDX14fJkyfjk08+CXoLhljmGV+xRDPPdHZ24gc/+AEOHjwY85890kZU4HR2dmLv3r3IycnBc8895/U4u45DZzQa0dLSAqvVKlvgbNasWUhOTkZra6uUeABg7ty5OHDgQFgWQ7NarTCZTKitrQUArFmzxmfisdlsWL16NQoLC7F9+3YAg4kxOTkZOp0OTqcTzzzzDI4dOzbiGNRqdcwWdhs3bpxszQy1Wo3S0lKYzWbpc9Bqtaivr8fKlStHnBw8fzaDwSD7LmfNmoWCgoIQf4rRjTkmOuI9zzidTrzzzjtobGyEIAhYuHChVOB0dHRg/Pjxw77PnDlzYDKZMGfOHK/HYplnPEU7z+Tn56OwsDA8wce5gAcZt7S0YMWKFTAajairq8OPfvSjSMY1ZjkcDlRUVKChoSGg9gqFAk6nMyzvbTKZkJGRId3OyMjwuTT+2rVrcezYMezatUs6+G7duoUjR45IMSmVSpjN5rDEFS3nzp3DokWLcO7cuai8n8FgiMr7jBbMMdET73lGoVCgsbERwGBB5F4APfXUU7h3796w75Oeno6SkhKkp6eHJe7RaiznmYB7cAwGA44dOyb9YtbX18NmsyE7OztiwY01TqdT6q4tLy/Hpk2bhmxvMBhQUlICtVodlve3WCyybQpUKpXPpCauJutwOAAMdh2Ly+qLHA5HQHEZjUbpdZRKJbRarezxuro6lJaWorm5GStXroQgCOjq6kJ6ejra2tqwe/du1NbWwmw2w2q1QqFQwGg0SmeHI3Hr1i188cUXXhs6Wq1WqUvd/ed2j7GoqEhKxGq12mfc7kwmE5xOJwwGAwRBwLx582SPAYOX/3bs2CF1V2/fvh0ajQbr16+HIAjD/n6MNswx0TFa8oz43s3NzVJP8UjYbDb8/Oc/x4YNG9De3j7iPOMrp4Qjz/gT6TwzdepUKJVK2WNAYueZgAuc7OxsWdVdVVWFlpaWhEk+NpsNAwMDSElJCevrqlSqgJf+N5lM0oEnCAJMJpNX1217e7t0EGg0GlnXo8jpdGLnzp1+38fXgeCP50EGQPYLv2TJEixatEj2eE1NDbZu3Trsa5vNZungslqtqKur80o8KpVK+gz0ej1qa2vR0tKC8ePHIzc3F0899RQASJc1NBqN7CAW3b59G9u3b8dDD/n+lV+5cqXfOAVBkMVVX18v/dtgMEClUkmPr1mzBrt27fIZtzuNRgOFQiF9f319fdJj4gaTYsLTarWyM9zS0lKf3/tol+g5BmCe8cdXngEGlwoQBAH19fX4yU9+EtBriXp6erBt2zaUlpbiiy++GHGe8ZVTwpFn/H0mkc4zfX196OzslB4bC3km4AInJydHdjsjIwMul0t2X2dnJ/Lz88MTWRTduHED5eXlePDgQdhfe/z48bh69SoyMzOHbeu+z05hYSEaGhq8Ek92drbXwelJoVAEVXXn5OTIzqTsdrvXwWg0GtHW1ia9vkKhkO0ZZDQaodFoho0RGEwWpaWlAAYPbs8BjyKDwQCn0ynNoFm+fDl+85vfYPPmzZg/fz527NiBqqoq1NfXQ6/Xo6CgwOu10tLSUF1djdTUVL/xXL9+fdiYPZnNZlkXu/uidJ5xB8pXAtTpdNi5c6ffxJoIEjnHAMwzokDyDDBYQIn/CVdXV2PRokVB/e63tLQElWd85ZRw5JlgMM8EJ+ACx2q1enXd22w26T673Y6Ghga8+uqr4Y0wCjIzM9HY2IjMzMyInFkFknTMZrOsutdqtVi4cGFQ7xnsmZVGo0FdXZ1022azSYlPTDaCIMgGvTmdTuTn56OzsxOHDx+WEpLZbJbai8/1pFKpYLFYZK/l3s5gMMBut6OyshJmsxltbW0wm804fPgwenp60NTUhN27d3sNWqypqfEaPBlKD85Q1Go1LBaL9DmZzWa/cfvq4nc6nTCZTHj66ael+3wlFrVaDZvNBoPBgKVLlwYVa7xL5BwDMM+IAskzBoMBFotFKqCUSuWw/+H6yzPp6elB5Zn29navnBKOPBPM4OZw5JnPP/9c1hs6FvJMwAWOXq/Hu+++K7vP5XJJ3WgulwtJSUmjNvlkZ2cjPz8/7JV3IEwmE+rr66HT6aQuQXGAbk1NDVatWoVr165h3759uHLlis8uZXfBnlkJgoCysjJpXMzatWulx8rLy9HY2Ai1Wg2j0Sj15OzatQsAcO3aNdTU1CApKQnA4AF19uxZ6bm1tbVeMVdWVqKurg56vR6CIEgHvtVqhcFgkAol925Tq9WKa9euIT09HUeOHJGeZ7FYpC5199cSBXJmlZKSAr1eL/UkWK1WNDc3w2azSZ+5+Nk0NDSgqqoKOp0Oer0eBoMBSqUSgiCgsLDQZ9zuP1tlZSV0Oh0MBoOUkLq6unDp0iVp1oM4GFPsZl66dCnMZvOo7zb2J9FzDMA8AwSWZ5YuXSptsNrc3AydTiedSJnNZqSmpqKtrU26tCI+11eeWbZsGQ4ePDjiPOMrp4Qjz3iKVp557LHHAAz2goqPJ3yecQVoy5YtLqfT6fePw+Fw1dXVBfpyUXHmzBnXmTNnhm13+/Zt1/Hjx123b9+OQlQjlwjxNTc3h+39WltbXQBcra2tAbWP98/P5Qosxvb29rB+jiMR6GcY6DHny2jMMS4X80y0iPH94Q9/8Hv8ux8fX331lWvdunWur776KqrxxfvnN1x8scozkcgxAU8TX7lyJTIyMvz+USgUKCsri1wlRqOW2WyOmzUnhnLjxg28++67uHHjRqxDkRHPGNvb28O62Fq8YY6hUHjmmZycHPzyl7/0GttFviVingm4wAlkAbKxvkgZ+eZ5nTpeWSwWVFZWyq7XxwNxlkOiL87FHEOh8MwzfX19OHHihGyGIvmXiHkm6K0aiAIVyIwq8q+ysjLWIRDFPc888+WXX2LBggVobW0NekuHsSQR80zAPThEREREowULHCIiIko4LHCI/iw9PR3f/OY3x/zeNUREiYBjcIj+bM6cOTh06FCswyCiMBg3bhwyMjIwbhzP48cqFjhxaP369V5LgHd1deH999/HlStXpAWYLBYLSktLwzqlT5wq6HA4pA35hmqrUChkG0WKi2kBwQ0uNpvNqK6uRmNjo89VSSPpwYMHuHfvHpKTk6Wk6L4Tr1KphMPhiIsFsKxWK6qrq7Fx48aEmdJJ0RXveUZc7M5sNkOr1Up55tChQ0hPT8cnn3yCiRMn+t0EdN68eX438YxlnvGFeSYyWODEGaPRiJaWFq8pj7NmzUJycjJaW1tlv/hz587FgQMHwjIN23Mp8jVr1vj9pXY6nXjnnXdkq5D+4Ac/wNtvvw2FQoHy8vKgChy1Wh2zKeWnTp2Szbowm81wOp3S7AKr1YqGhgapvbgSaCwIgoCSkpKYvDeNfvGeZ8QVlnU6HZxOJ5555hl88cUX6OrqQkdHB27duoVly5ahvr5eWk19JGKZZzwxz0QO++7ijMPhQEVFhewXfCgKhcLvWcpIiUt2i9w3d/PU1NQk26ukq6tLeq7ZbEZjY2NYYoolh8Mh25hQEARp0z5AftZFNJrEe55xOBzSfQqFAkqlEp2dnWhvb8fUqVNlbcViyFNHRwfUajU6OjrCEnekMM9EDntw/sxms2FgYCAim+Dl5uYG1NbpdErdteXl5cPu82IwGFBSUuK3i3akLBYLVCqVdFulUvlMamazWdovRfS///u/uHTpkrQXSk1NjXSGNhSxGxoY7Jr17PWpq6tDaWkpmpubpY3qurq6kJ6ejra2NuzevRu1tbUwm82wWq1QKBQwGo0BvfdwNBoN9Ho9Fi5ciKVLl0Kn00lnmiaTCU6nEwaDQfrOTCaT9HPr9Xps374dCoUCdXV1KCoqgtVqlZ7vGWtnZyeOHDkCh8OBQ4cOoba21ut5arUaer1eSvgdHR2yREjxj3kmsDyj0WhkvToOhwP5+fk4cuQIzp8/L91vtVphtVp9xjYwMICOjg4MDAwElWd85ZTRnGd+8pOfoKurCzabDVOnTpXiT+Q8wwIHg0v0l5eX48GDB2F/7fHjx+Pq1asB7fRrMpmkA08QBJ+b3bW3t0uFhUaj8dl1Gewuv76IScGd1Wr1ShB9fX1QKBRSomlvb/e7s63IbDZj79692LFjB6xWK+rq6rxeV6VSSZ+BXq9HbW0tWlpaMH78eOTm5uKpp54CAOzduxc5OTnQaDQ+d8kNdjfxXbt2SRvaideitVqttDGd++cv3qdUKqWkYzAYoFKppJ9rzZo1KCgo8Ip13759GD9+PNRqNaZPn+7zeTqdDlarVUqq/nrXKD4xz/jnK8+IampqsHXrVgDAN77xDTQ3N2PcuHFSr8dwY2i6urrwxRdfjDjP+Mop4cgzvj6TaOWZw4cP48knn8TixYuhVCoTPs+wwAGQmZmJxsZGZGZmRuTMKpCkA0DWTVlYWIiGhgavxJOdnT3s2JZgd/nNycmRnUnZ7Xavg1EcRCzuJm61WjF16lRMmzYNd+/eldoplUq/Z1aivXv3SmcGgiB4DXgUGQwGOJ1O2O12AMDy5cvxm9/8Bps3b8b8+fOxY8cOVFVVob6+Hnq9HgUFBV6vFcguv9evX5fdFgs08U9lZSXWrFkz7Ofv/jObzWZZF/zGjRshCIJXrC+88AJeeeUV/OM//iPUajVUKpXX89x3Hgcg6+an+Mc8MyiQPCMyGo3QaDTQarXSlgvr1q3DRx99hNzcXCgUimG3FmhpaQkqz/jKKeHIM56ilWfeeOMNLFu2DEajEb/+9a9RUFCQ8HmGBc6fZWdnIz8/f0S/mOFkNptl1b1Wq8XChQuDeq1gz6w0Gg3q6uqk2zabTUp8TqcTCoVCtpx3W1sbioqKkJ+fj9u3b+Pw4cPSY+7dpOJzPalUKtm+T57tDAYD7HY7KisrYTab0dbWBrPZjMOHD6OnpwdNTU3YvXu316DFmpoar8GTgZxZFRYWwmq1Ytq0adLP4HA4ZMnf82B3Op2yM2JParUaFotFeg2z2YympiavWA8dOoQXX3wR+fn5eOONN/DII4/g9u3bsudpNBrZf069vb0+35PiF/NMYHkGGOw5UCgU0Gg0MJvNSE5Oxu3bt7Fz507cuXMHt27dQmFhodTeX55JT08PKs+0t7d7HafhyDOen0m08ozNZsPhw4fxs5/9DKmpqaipqUn4PMMCJw6YTCbU19dDp9NJXZHiwLmamhqsWrUK165dw759+3DlyhWfXcrugj2zEgQBZWVl0vVq9xlS5eXlsimVJpNJmoWRm5uLtLQ0PPvss9JZ0MaNG6W25eXlqK2t9Yq5srISdXV1Uq+QeOBbrVYYDAYpsbl3kVqtVly7dg3p6ek4cuSI9DyLxSJ1qbu/lijQM6vs7GzZbXEDOofDAYvFgtdee016TKfTyc52xOvz7rMedDod9Ho9DAYDlEql31htNht6e3ths9kgCAIqKyu9nqfVaqV4xNh8nX0T+TKa8ozD4UB1dbV0v9PpxMmTJ9Hb24uCggLp+HcfA+OZZ3Jzc7Fnzx48/fTT0Ov1I84zvo7TcOUZT9HIM9nZ2ejp6cH+/fsxceLEsZFnXAnszJkzrjNnzgzb7vbt267jx4+7bt++HYWoRi4R4mtubg7b+7W2troAuFpbWwNqH+jnd/HiRdd3v/td18WLF8MR5ogkwnfscgV+zCUS5pnoEOP7wx/+4Pf4D2eeGanR8vmN9vhGkmM4TZwizmw2x82aE0Ox2+34n//5H+kaPBGNHp555urVq/j3f/93XL16NYZRUSyxwKGI87xOTUQUbp555vLly9i8eTMuX74cw6golljgUMQFs6IxEdFIMM+Qp5gMMg50H5Lh2hAR+cM8QzS2Rb0HR5xqp9VqpZHfwbQhCrevfe1reP311/G1r30t1qFQiJhniCjqPTj+9iFxP3MKpE2gXC6XtECUP/39/bK/4w3j8/1+p06dCug979y5gytXrsDhcGDixIlDti0uLsbevXtl902cOBEzZ84EAJw/f95rJdpHH30UkyZNwrVr17wGKE+ZMgVTp05Ff3+/bC0OYHD12dmzZwMAzp49i8uXL+PYsWOYMGECgMEp62lpabhx4wb++Mc/yp6rUCiQlZWFu3fvoqury+vnmDt3LoDBZfE9P6OsrCwoFArY7XZcu3ZN9lhaWhqys7Px4MED2ZL4d+/exaRJk6TPwR+Xy4WkpKQh20QD88zIjZb47ty5A2D44//y5ctYvHgxzp49i1OnTnk9PmfOHCQlJcFqtXp9d9OnT4dKpfJ5jKSmpkIQBLhcLpw7d066/+7du7hx4wZu3LiB9PR0XL582WsNmczMTDz88MO4desWLl26JHss0nnmzp07OHr0qCzHAPGTZyKRY6Je4ASyD0mgeyIF4t69e+js7AyobXd3d1DvES2Mb1BPTw8A4Pvf/35U3o8GJSUlYf/+/bJj0xf35BkrzDPBi/f4xG0dAj3+P//880iGQ2EU7hwTFwv9DbUPyUja+JKcnCydJfvT39+P7u5uzJw5M+xLqIcD45PLz8/H6dOnA15hU+zBycrKGrYHJxZnVsBfenAyMzPjugdn3rx5Q37HFy5c8PtYrDHPDG20xPf000+P6Pi/f/8+Ll686HV/pHpwFi5cGNc9OCqVKq57cMKZY6Je4ASyD8lI9ioZTlJSUsCrSqakpMRsCfVAML6/eOKJJwJu29fXh87OzrAskR+pXXXnz58fdIyLFy/2+1go8f7N3/yN9G/xMxzuO46Hy1MA80woRkN8Izn+AeCb3/xmRGL567/+a+nf8Z5n+vr6oFQqg44v0nkmEjkm6oOMPfe58NyHZLg2RETDYZ4hoiSXy+WK9pu6T81UKpXS+gVLliyR9jvy12YkTpw4AZfLNez1OpfLhXv37iE5OTluzkDdMb7QxHt8QPzHGGh8d+/eRVJSEoqLi6MYnW/MMyPD+ELD+EITiRwTkwInWk6ePAmXy4Xk5ORYh0I0Jty7dw9JSUmYP39+rEOJGuYZougZSY5J6AKHiIiIxiZu1UBEREQJhwUOERERJRwWOERERJRwWOAQERFRwmGBQ0RERAmHBQ4RERElHBY4RERElHBY4BAREVHCYYFDRERECYcFDhERESUcFjhERESUcFjgEBERUcJ5KNYBRJvRaAQAOBwOCIIAjUYTVJtYx+dwOGA2m6HVauMuPve2CoUiLuPT6/UQBAEAoNVq4yo+sY0oWvE5nU4YDAYAQGVlpc82sTw2RgvmmOjE6N6WeWbk8Y2JPOMaQywWi2vLli3S7dWrVwfVJlICee/29nZXU1OTy+VyuRwOh+upp56Kq/hEDofDtWLFCinWaAg0vtWrV7scDofL5XK5VqxYEZXYXK7A4nM4HK533nlHuu3ePtKamppcb775puz93cXy2BgtmGNCxzwTGuaZvxhTl6hMJhMyMjKk2xkZGTCZTCNuE8v4HA6HdJ9CoYBSqYTZbI6b+ERNTU1YunRpVOISBRKf2WyW2pjNZjQ2NsZVfAqFAgaDQfpO3dtHmlarRU5Ojt/HY3lsjBbMMdGJUcQ8E1x8YyXPjKkCx2KxQKVSSbdVKhWcTueI28QyPo1Gg9raWum2w+GAWq2Om/iAwQM6FpcuAomvvb0dNpsNVqsVAFBTUxNX8QHAxo0bUV5ejvLyclRVVUUtvuHE8tgYLZhjQsc8E/n4gLGRZ8ZUgeOLw+EIS5tIGeq9a2pqsHXr1ihG481XfFarVbruHGue8TmdTiiVSqjVaqjVarS3t0f17NSTr8+vra0NjY2NUCqVWL16dfSDGoFYHhujBXNM6JhnQjNW88yYKnA8u8XsdrvXARJIm0gZyXsbjUZoNJqoDlwLJD69Xi/F19bWBpPJFLUDO5D4BEGQ3adUKqWzrHiIz2g0orS0FGq1Grt27UJhYWHcXAaK5bExWjDHhI55JvLxjZU8M6YKHI1Gg7a2Num2zWaTujjFLrCh2sRDfMDgNUqFQgGtVguz2Ry1AyeQ+CorK6HVaqHVaqXR79Hq3g70+3X/vKxWa1x9vw6HA0qlUvYc99uxEA/HxmjBHBOdGJlnQotvrOSZJJfL5QpLdKOE+/QzpVIpnZ0sWbIEjY2NUCgUftvEQ3wOhwPl5eVSe6fTibNnz8ZNfAqFAsBggqyvr0d2djY2bdoUtTPUQL9fh8MBp9MJQRDi6vtVKBTQ6/XS5xjN3z+TyYSGhgb09vZCp9PF3bExWjDHRD5G5pnQ4xsLeWbMFThERESU+MbUJSoiIiIaG1jgEBERUcJhgUNEREQJhwUOERERJRwWOERERJRwWOBQWJnNZtTU1GDu3Lmoq6uDXq+HXq9HTU1NxNbSMJlMKC8vl3ao9bxNRImFeYYCwWniFHZOpxMLFy7EsWPHZOtVVFdX4+DBg9J94SSu6aDT6XzeJqLEwjxDw2EPDkWFRqOB0+mMm+XAiSjxMM+QOxY4FBXiPjHR3JWYiMYW5hly91CsA6DEJe5nYzabYbfbceDAAdlS6uIGeYIgoK2tDZs2bQIwuG9LQ0MDioqK4HA4sHTpUmn5boVCAavVCovFIrUnorGLeYb8YQ8ORYxGo5H+tLS0yDZzs1qtqK+vlzbNy8nJgV6vh9PpxJo1a1BVVQWtVguLxSIN4quuroYgCNDpdOjt7ZX2KyGisYt5hvxhDw5FnFqtRmFhIerr61FbWwsAaGhogFKplF0rb2trg0KhgCAI0gDBqqoq6XFxMKHVaoXdbo/qDsdEFN+YZ8gTCxyKioyMDHz22Wey+woKCqDRaKTbOp0Oer0eGRkZ0n3uMyF27twJlUoFrVYbtV2DiWj0YJ4hd7xERVGRk5MjnQmZzWaUlZWhpaVF1sZkMkGr1aKjo8PrfpPJhI6ODlRWVkIQBPT29kqPiZxOp+x5nreJKLExz5A7roNDYWU2m7F3715YrVYUFRVBo9FIMxrWr1+PoqIiad0Ik8mE5uZmFBUVARi8lq5QKHzeDwA/+9nPsHLlSum9GhoaUFZWBkEQsGXLFgDA1q1bAUB2mzMqiBIL8wwFggUOERERJRxeoiIiIqKEwwKHiIiIEg4LHCIiIko4LHCIiIgo4bDAISIiooTDAoeIiIgSDgscIiIiSjgscIiIiCjhsMAhIiKihMMCh4iIiBIOCxwiIiJKOCxwiIiIKOH8P/sn2nxl1xapAAAAAElFTkSuQmCC\n",
|
|
"text/plain": [
|
|
"<Figure size 578.387x178.731 with 2 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"fig_save_dir = '../../thesis/graphics/'\n",
|
|
"\n",
|
|
"fig, ax = plt.subplots(1, 2, figsize=set_size(width, subplots=(1,2)))\n",
|
|
"results.plot_pr_curves(classes=classes, iou_thresh=0.5, backend='matplotlib', ax=ax[0], color='black', linewidth=1)\n",
|
|
"results.plot_pr_curves(classes=classes, iou_thresh=0.95, backend='matplotlib', ax=ax[1], color='black', linewidth=1)\n",
|
|
"# Set the labels for the legends manually\n",
|
|
"ax[0].get_lines()[0].set_linestyle('dashed')\n",
|
|
"ax[1].get_lines()[0].set_linestyle('dashed')\n",
|
|
"ax[0].legend(['AP: 0.52, Healthy', 'AP: 0.46, Stressed'], frameon=False)\n",
|
|
"ax[1].legend(['AP: 0.31, Healthy', 'AP: 0.29, Stressed'], frameon=False)\n",
|
|
"fig.tight_layout()\n",
|
|
"fig.savefig(fig_save_dir + 'APmodel-final.pdf', format='pdf', bbox_inches='tight')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "dc5941e4",
|
|
"metadata": {},
|
|
"source": [
|
|
"The confusion matrix for the aggregate model seems to not show the cases where the object detection was successful but the class was wrong. For example, in the matrix below all classifications were correct or the detection failed. Under column _Stressed_ and row _Healthy_ not a single item is recorded. It seems that this evaluation metric does not have as much relevance when compared to the AP curves above or the mAP values."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "f1586bd5",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhcAAAFbCAYAAABmq6zCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAoyklEQVR4nO3df3xU9Z3v8fdgH/xyfilWYmBSoSrmV/VW6cqgWxVXBpTasjWhrW7lItDd7gO8ive2t118iLrevcGuWL0rjVTd+rglYWu1KkyygN1VTryilhomEbQqmSEGlDpzMjXCY93cP2KOGTKByeQkJznzevYxD3O+Z07mE5M6n/l8P+f79XR3d3cLAADAJuOcDgAAALgLyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALDV55wOoNefF13idAgYQ57as9HpEDCGtD35lNMhYAy58G/uGJHXGcr73r93vGRjJPYbNckFAACFZJzH43QIw4ZpEQAAYCsqFwAAOMAj91YuSC4AAHCAm6dFSC4AAHCAm5MLei4AAICtSC4AAICtmBYBAMABNHQCAABbubnnguQCAAAHuDm5oOcCAADYiuQCAADYimkRAAAcQEMnAACwlZt7LkguAABwgJuTC3ouAACArahcAADgAHouAACArZgWAQAAyBGVCwAAHEDlAgAAIEdULgAAcAANnQAAwFZunhYhuQAAwAFuTi7ouQAAALYiuQAAALZiWgQAAAd4XDwtQnIBAIAD3Dx1QHIBAIAD3Fy5cHPiBAAAHJBXcnHffffZHQcAAHCJvKZF2tratGXLFgUCAYXDYXm9XrvjAgDA1dy8zkVeycWGDRusr7ds2aK9e/cqEolozpw5tgUGAICbjWP570xNTU2SpM2bNysej2vhwoUKBAKqr69XZWWlSktLbQ0SAAC3cXNDZ17JxerVq7VgwQKtXLlSZWVl1nhZWZkaGxtJLgAAKGB5JRd33XWX5s+fbx2n02l5vV41NDQokUjYFhwAAG41zr2Fi/ySi/nz56uxsVGS1N3drW3btun+++/PSDgAAMDAmBY5zurVq1VZWWkdm6ZpW0AAAGBsyyu5WLhwYUaVorq62raAAAAoBG6+WyTnRbTS6bT18Hg8ampqUiKRUDqdVn19/XDGCACA64zzePJ+jHY5Vy6uvPJKVVZWqru7u9+5RCKhZcuW2RoYAAAYm3JOLh577LGM2077amlpsS0gAAAKwRgoQOQt5+RioMTiZOcAAEB/Y2F6I195r9DZu9R3Q0OD/H4/S38P0fd+/H098cDjSptpSdJZJcW67q++oYPvJjTt7OkZ5y66bLYkyRvwqvTCMu18erv2N+9zLHY462D7IT3/25dUXDxV7e2HdN2iq+Tznep0WHDQpDOKFLp8kfb/S22/c97pMzTBf5qOmh9KktKJdyRJ4/1BTSm7WMfMP2q8/3Qdfu0FfXLs6IjGDffIa1fUvgtlhcNhpkWG6LzKWfr2396YMfaT+gf0xAOP65knntbOp7frez/+vnVuXe096kya+rdnn9fBdxO6s/aekQ4Zo8iP7viJbvj2dbry8kt0xeWX6N6ajU6HBAcFZvaskDz5zOJ+57zTZyj4xXIdaXlNx8wPFfrqtda5Ly66UYdfe0FHWl5T8q29OmvOX4xYzIXKM4T/jXaDSi7q6+u1ePFi/exnP9Nf/uVfavHixbrpppvk9/uHK76CcNYXinXw3c8Stt7KRG+lYn/zPn3tr75hnV+7/EcZlYp0Kj1CkWK0Odh+KON4WvFUvfJqs0PRYDRIvd2qrg86sp4LffVavdf0r5KkY2ZSf3jmF5J6kg5JVqWi64MOnVF+0QhEC7ca1LRIVVWVqqqqMqZFMDRfvfYK/duzz2dUJryB7FvYn1c5S/ub9+nVF3ZbY5cvmqff/OLXwx4nRqfdrzTL78v8e/H7vdq3/23NOm+mQ1FhNBrvD+qUCZP0ybGjmnRGkY6ZH+qYmZQknTJ+YtZrJp1RNGCigqFz8/Lfea1zUVlZmXG8adOm4YzRtbx+b9aqw/7X92na2dOt4/MqZ0mSfEF/xtj3fvx9vfLvL+uZJ54e/mAxKqXTf8o6njKpZiHTpDPO0idHuxSYWapj5oeaUnaRNYXS9cF7mhA4vc9ziyRJp0yc5EishcLj8eT9GO1yrlx84xvfUElJCetc2OiKr83Lmhi819auf7rrQS264To9/5sdOusLPXOnncnPllnf37xP7QcO6ns//r5V/QB6pdMfOR0CRpnPTZykCYHTlU68rU+OHdWRlldVefMPtOf/3KljZlLtxr9qStmXlXwrpvH+0yRJn3zc5XDU7sbdIpI2bNjAOhc2uuiy2Xr+NzsGPP/Lh57QWSXFKv7CNL367z3TIO0HDmY8J22m9fwzO/WPW36qheddZfVooHB4vafK7Mz8vZtmWl7vZIciwmh11PxQ//Fxl9VX0fvP3qmPw3sMjfcHNd5/mtKJtyVJxz69owQYrJynRQZKLFpbW9lmPU9XfG2eFt1wnRbdcJ2mnT1dN6z6rjUFclZJsd5ra9f+5n0q/sI07fv9G0qbaV102Ww990aj9T16E47iL0xz5GeAs2ZfXJl1vHTWF0c4Eox2J0sUxvuDOmYm1fVBh8b7T9NHh9u5FRV5y2udi3g8rtraWpmmqe7ubnk8Hl199dV2x+Zqr76wW6++8Nnx7et/qKf/+dd6r61dkvRI42Oq/spipc20vnbj1/Xw3Q9J6pkaeaVPQ+d5X5qlzqTJOhcFalrx1Izjg+2HdP6smaxzAUnSKeMnWAnCMTOprvfbrbHx/qCOpv5oNWye980Van1igz45dlRTyi9W+0vbnQy9ILh4ViS/5KKurk4LFixQLBZTOBxWKpWyO66C4fV7tejGr0uSvvO3N+o3v3hK+5v36eG7H9JFfz5b/qBfzz+z07pDZH/zPu18ersW3XCdJOnir35FN199k0PRYzS4585b9dDDT6j0/HPU+sZbuufOW50OCQ7yTp8h3/SeytWZX75MHx0+qNTbrZKkdxu26Kw5f6Gu99s16fPF1q2okvTeS9vlnT5Tn5s4Sck/xKzFtTB8nNwVNRqNyu/3KxwOW8eSlEqlFAqFTjp+Mp7ubB2aJ9F7K+qWLVt0/fXX23Jr6p8XXTKk61FYntrDQlHIXduTTzkdAsaQC//mjhF5nf/+X67P+9r//bsteV9rmqZuuukmrVixQpFIxJqNWLdunSRp6dKlevTRRwccz0VelYtdu3appaVF3d3duu+++2SaJuteAAAwBmzbtk0LFiywjg3DkM/ns459Pp8Mw1A8Hs86nkv1Iq/kYs2aNdbX9fX1qqioyOfbAABQsIZyK+q8efNOeH7Hjux3I/a2M/ROd0hSW1ubgsGgdRwMBmWa5oDjuchrb5GWlhYtW7ZMra2tqqioGBMLegAAUOji8bhCodBJnzdQL2WuPZZ574q6adMmq9eisbFRpaWl+XwrAAAK0lA+mA9UmTiR2tpahUIhRaNRNTc3W4lGSUlJRkUimUxaCchA4yeTV3Jx/DfnbhEAAAZnpPcWWb58ufV1c3OzKisrVV5eLr/fr5qaGutcIpFQOBxWPB7POp6LvJKL119/XfF4XMlkUoZhZMzJAACA0cswDDU1NSkej6u8vFyhUEgLFy5UNBpVKpXSihUrJGnA8Vzk3dBZX1+vAwcOqKKiQlVVVfl8GwAACpZT/YrhcFhPPvlkxlgkEsn63IHGTybn5CKdTsvr/Wxr597t17OdAwAAJ5bXHRVjRM4/28aNG5VIJLI+Nm5kQSMAAAaDLdfVs+R3Q0ODuru7ZZqm/H6/dc40Td12223DEiAAABhbck4u7rrrLs2fP1+S1NDQYH3dewwAACANIrnom0wcX5Lpew4AAJzcSN+KOpJy7rloamrK6xwAAOiPngtJNTU1qqyslNSzkMauXbusc3v37tWvfvUr+6MDAMCl3Fy5yDm5CIVCKi8vlyTrn72SyaStQQEAgLEr5+Ri5cqVKisry3qO7dYBAECvnJOLgRILqf9eIwAA4MTGQu9EvvJa/hsAAAyNi3MLkgsAAJwwzsXZhZuXNgcAAA6gcgEAgAPcW7egcgEAAGxG5QIAAAe4ueeC5AIAAAe4OLdgWgQAANiLygUAAA5gES0AAGArNi4DAAC2cnPlgp4LAABgK5ILAABgK6ZFAABwgItnRUguAABwAotoAQAAW7k4t6DnAgAA2IvkAgAA2IppEQAAHODmaRGSCwAAHODmRbRILgAAcICbl/+m5wIAANiKygUAAA5gWgQAANjKxbkF0yIAAMBeVC4AAHAAlQsAAIAcUbkAAMABbFwGAABs5eLcYvQkF0/t2eh0CBhDHvz6PzgdAsaQb/3dZU6HAPTj5ltR6bkAAAC2IrkAAAC2GjXTIgAAFBIXz4qQXAAA4ASSCwAAYCsaOgEAAHJE5QIAAAeMc2/hgsoFAACwF5ULAAAc4OaeC5ILAAAc4OLcgmkRAABgLyoXAAA4wM2VC5ILAAAcQM8FAAAY86LRqEKhkPbu3StJqq6utsYlKZVKKRQKKRwOn3D8ZOi5AADAAR5P/o98mKapn/3sZyovL9eCBQu0du1aSVI8HpdhGIpEIqqurlZtbe0Jx3NB5QIAgDFm3rx5Jzy/Y8eOfmN+v19PPvmkpJ7EobcKYRiGfD6f9TyfzyfDMBSPx7OO51K9ILkAAMABTrVc1NXVadeuXdqwYYMkqa2tTcFg0DofDAZlmuaA47kguQAAwAGeIaz/na0ykavq6mqFQiGtX79e69aty/qcVCo1qPHj0XMBAIADRrrnQpJVeQiHw9q2bZsMw1BJSUnGc5LJpEKh0IDjuSC5AACgANTV1Wnjxo3WcSAQUCAQUDgcVnNzszWeSCQUDocHHM8F0yIAABSABQsWyDAMGYahXbt2qbq6WuXl5ZKkhQsXKhqNKpVKacWKFZKkUCiUdTwXJBcAADhgpBfR8vv9ikQiktSvAtE7fryBxk+G5AIAAAe4eIFOkgsAAJzg5uSChk4AAGArKhcAADjAzRuXUbkAAAC2onIBAIADXFy4ILkAAMAJJBefampqGvBcXV2d7r///qHGAwAAxrhBJRc1NTWaO3euuru71dLSounTp8vv98s0TXV2dg5XjAAAuM5QNi4b7QaVXGzYsMHatKShoUHz58+3zjU0NNgbGQAALubmaZFB3S3Sdze04/d0d/MtNQAAIHd5N3SmUindcccdqqioUFtbm4LBoI1hAQDgbm7+UJ73Ohc333yztR1rZWWlli1bZmdcAABgjMo7uWhpaVFdXZ2+9a1vafr06WptbbUzLgAAXM3jyf8x2uWdXDQ1NennP/+5ksmkysrKFI/H7YwLAABXc3NykXfPRd/mTqmnBwMAAOTGzT0XeScXr7/+uuLxuJLJpAzDoKETAIDBcG9ukf+0yJo1a+Tz+ZRKpVRSUkJDJwAAkDTEvUWqqqpUVVWlRCJhVzwAAGCMyzu5uOOOO7RkyRJt3LhRgUCA6gUAAIPA8t9ZRCIRTZ8+XS0tLWpsbGT5bwAABsHF/Zz5JxemaSoajeqSSy6R5O6uVyccbD+k53/7koqLp6q9/ZCuW3SVfL5TnQ4LDpoxu0KSNNE7WcVlMxXb/pI69r0rSQoWn6mZsyvUZaZ12rQz1bLzZSXbD0uSimadrZmfXltc+kU9c2+tjqY/cuRnwMh4K3FYP63frg23fjtjvONISi/+/k0VTQmo40hKkTkV8k6aaJ3fs79NHUdSKpoSkCRdeF7JiMZdaNz8vpl3Q2coFNKBAwd0++23q76+Xs3NzXbGVfB+dMdPdMO3r9OVl1+iKy6/RPfWbHQ6JDjsm/esVpeZVuvzL+vDxGF9857V1rmyK7+i157eqdbnX5bxxLOae+Mi69zM2RUynnhWxhPPqr31D7rxwR85ET5GyIu/f1OS9IfE4X7n7n38OX3zyot16QXn6tILztVP63dY5/bsb9OLe/YrMqdSRVMCenDLjn7XA7nKO7koKyuz7hgJh8O67bbb7IyroB1sP5RxPK14ql55leSt0P3LjzZYlQpJ+rhP9aHsqjlZrymadbbm/tV11nHLzpdVfP4MBYvPHLY44axLLzhX50zv//vtOJK5FlHRlID27G+zjh/cskM3LbrUOnf39xYPb6BwtbyTizvuuEOtra265ZZbVFtbq02bNtkZV0Hb/Uqz/D5vxpjf79W+/W87FBFGg3d277W+Lpv3Z3rt1599svzYTOvmx+5RsPhMzZhdobdf7klGO/a9q2fvrbWeN9E3WZLUZaZHKGqMFnv2t8k3eWLGmG/yRL2VOKyOIyl1fvSxvJN6jtNdH1tTIxg+bl6hM+/kom9D55133qnp06fbGVdBS6f/lHU8xRtCwSuadbbmfX+J3n65Wa89vdMa3/I/N0iSVj15v8qu/DO1Pv+yda7v1+VXXaK3X26m56IApbuOZh//6GO9lTgs3+SJn/Zj+BVt2mtNr2D4kFxkQUPnyEvzhlDwOva9qxcf/41OmzZVpVd8xRqfObtCOx76pZ699xFd9I15uuZ/9L8tfIJ3skqv+DMrEQGknqQj/dHH6jiS0oXnheSdNFGRORX6X48/53Ro7jfOk/9jlBtSQ2dbWxsNncPA6z1VZmdmlcI00/J6JzsUEUaTo+mP1LLz/+n6e2/RBO9kBYvPVHHZTL2ze69ee3qnHlh8i8rmXdKvr+Kq739LT6y6l6pFgfJOmqDOjz7OGOuZCpmgoikBnTppgnXnSO8/38rSFArkIu/kQpJisZgSiYQqKiq0YMECu2IqeLMvrsw6XjrriyMcCUaLGbMrdHvjZ70THx7s+Y/+adPO1FmzzlZ7y2f9OMn2w9r1z09b/RWSFL7hWu36xTNKth/WBO9kTSBRLTgD3VZ6TuhM+iscwrRIFsdvuc4S4PaZVjw14/hg+yGdP2sm61wUsC4zndHQedass9Vl/kkd+97Ve/veVXHZzIznTwp4rTtLSq/4it7b966VWJTPu4TqRYFId31WqTg+geg4ktK5oanyTpqooikBnRuaaj2/d62LbHedALlgy/VR6p47b9VDDz+h0vPPUesbb+meO291OiQ4qGPfu4ptb9KXr7tSkjTzK5WqvalnvYpk+2G9/fJehW+4Vh939iQNr/66p9kzWHymrr/3lozv1WX+KaMZFO6yZ3+bfrfvgCRpy/bdOrekSJdecK4k6YffvUaPPvOCzi0p0pttHfrBdxda1/3guwv12DMv6pzQVL0VP8StqCPAzb2Knu7u7u58Lly/fr1OO+00JZNJSVIwGBzS3iJ/7Ph93tei8Dz49X9wOgSMId/6u8ucDgFjyLnX/PWIvM5vb/tveV97+X3/aGMk9st7WmTlypXWluuhUIhNywAAgKQhTIvccsst2rBhg6qqquyMBwCAwjAGbinNV96Vi5tvvlnxeNw63rJliy0BAQBQCNx8t0jelYu1a9fK7/dLkrq7u3Xw4EFdf/31tgUGAICbubmhM+/kYt26dZoz57PNklpaWmwJCAAAjG15T4v0TSyknl1SAQAAhrSIVq+GhoaMYwAAcGJu7rnIO7nouyJnOBxmWgQAgMFwcXYx6J6L+vp6bd68Wel0Wps3b1Z3d7c8Ho+WLFkyHPEBAOBKniHt7jW6DTq5qKqqUlVVlZqamvr1XQAAANjW0AkAACANsnJRX1+veDyucDisOXPmqLGxUZs3b5bf79fdd98tr9c7XHECAOAqrHPxqUAgoDlz5igUCqmlpUXr169XY2OjJOm+++7TbbfdNixBAgDgOu7NLQaXXJimaW21XldXl7GvyPFbsAMAgIG5uXKRd8/Ftm3bNHfuXOs4GAzaEQ8AABjjBlW56O7u1qZNm/T666+roqJCpaWlknoW1EqlUsMSIAAAbsStqJ+qqqpSS0uLysrKrLtFWlpaZJqmtYkZAADIgYunRQa9zsXxe4iUlZWxrwgAALDkvSsqAADIn4sLF/k3dAIAAGRD5QIAAAd4xo186SIajSqVSikWiykSiSgcDlvjkpRKpRQKhU46fjIkFwAAOGGEc4tYLCZJqq6ulmmamjdvnnbv3q14PC7DMLRu3TpJ0tKlSxUOhwcczwXTIgAAOMDj8eT9yEcqlZJhGJIkv9+vQCCgWCwmwzDk8/ms5/l8PhmGMeB4LqhcAAAwxsybN++E53fs2NFvLBwOZ1QeUqmUysvLtXXr1oyFMIPBoEzTVFtbW9bxXFC5AACgwKxdu1Z33XXXgOcHWhgz1wUzqVwAAOCEIdyLmq0ykatoNKpwOKxIJCJJKikpyahIJJNJa7+wgcZPhsoFAAAO8IzL/5EvwzDk9/sViUQUi8UUj8cVDofV3NxsPSeRSFhTKNnGc0HlAgAAB4z0rqjxeFyrV6+2jk3T1L59+yRJCxcutG5TXbFihaSe3c6zjeeC5AIAgAIQCoW0e/furOd6p0hyHT8ZkgsAAJzg4vW/6bkAAAC2onIBAIADhtKYOdqRXAAA4ASmRQAAAHJD5QIAAAe4uHBBcgEAgCNcnF0wLQIAAGxF5QIAAAd4xlG5AAAAyAmVCwAAnODinguSCwAAHODi3ILkAgAAR9BzAQAAkBuSCwAAYCumRQAAcIDHxU0XJBcAADjBvbnF6EkujrzW5HQIGEO+vqzM6RAwhky5KOx0CEA/LKIFAACQo1FTuQAAoKC4uOeCygUAALAVlQsAABzA3SIAAMBeLk4umBYBAAC2onIBAIADPC7+eE9yAQCAE5gWAQAAyA2VCwAAHODmu0WoXAAAAFtRuQAAwAku3luE5AIAAAe4eVqE5AIAACe4OLmg5wIAANiK5AIAANiKaREAAJxAQycAALATDZ0AAMBe7s0t6LkAAAD2onIBAIADPC7uuaByAQAAbEXlAgAAJ9DQCQAA7OTmu0WYFgEAALaicgEAgBNcXLkguQAAwAkuvluE5AIAAAfQcwEAAJAjkgsAAGArpkUAAHCCi6dFSC4AAHCAm5f/JrkAAMAJHvd2Jrj3JwMAAI7Iq3KRSCTU0tIiSQqFQgqFQvJ6vbYGBgAAxqZBJRcNDQ3atWuXPB6PysvLFQgE1NzcrOeee06JREIrV65UaWnpcMUKAIB7jHBDp2maqqurkyQtX77cGo9Go5KkVCqlUCikcDh8wvFc5JxcPPLII5o7d67mz58/4HOamprU2Nioq6++OucAAAAoRCPd0GkYhpLJpILBoDUWj8dlGIbWrVsnSVq6dKnC4fCA47nKObmorq6Wz+c74XPmzJmjzs7OnF8cAICCNYTKxbx58054fseOHf3GIpGIUqmUTNO0xgzDyHhv9/l8MgxD8Xg863iuCUbODZ19X6SlpUXLli1Ta2urWlpa1NramvV5AABgAB5P/g+btLW1ZVQygsGgTNMccDxXeTV0NjU1adOmTWpqatKcOXPU2NhIrwUAACMkW2XCLqlUalDj2eSVXIRCobxfEAAAjA4lJSUZFYlkMmm9xw80nou8kovXX39d8XhcyWRShmFklE4AAMDJjYZdUcPhsGpqaqzjRCJhNXRmG89VXsnFmjVrVF9frwMHDqiyslILFizI59sAAFC4HLhbZNeuXers7FQoFFIkElEoFNLChQsVjUaVSqW0YsUKSRpwPFee7u7u7nyCbGhokMfjUXd3t7Zt26b7778/n29jeXPrw0O6fix6K35IP63frg23fSdjvOODpF78/ZsqmhJQx5GUIuFKeSdNtK6RpHNCU9XxQVLprqM6JzR1xGN3WtfBD5wOYcRNmlKkaZddo7ee2tRvXJK6jnRovC+oU8ZPVNeRDknSeF9Q3mkz9MnRLo33nabUO6061pkc6dAdN33RIqdDGDUOth/S8799ScXFU9XefkjXLbpKPt+pToc1qpxedMGIvM7bO3+e97Uzr/yvNkZiv7wqF6tXr1ZlZaV1PJgOUvR4cc9+FU0J6A+Jw/3O3fv4c1bC0fFBUj+t264f3nStJCna1KxoU7Mk6cLzSvSDm64ZuaDhmMCM83XMTGry58/qd+700i9rSumXJUmdibd1YMev+lxXqvdfb7KOp126UAdf3Dr8AWPU+tEdP9Fjtf8gqSfRuLdmo/5+3a0ORwW3ySu5WLhwYcZiWtXV1bYFVCguvfC8rOMdHyQzjovOCGrP/jbr+JzpZ2rz3/+1JFnVDLhf6p03BjzX9cF72vt4z9zofx47mnEuMLMsI7lAYTvYfijjeFrxVL3yarND0YCNy47j9/vV2tqqRCKhdDqt+vp6u+MqWHv2t8k3OTNp8E2eaE2HSD1JBYkF+vrPY0f7JRaS9MnRLp3z9WU90yPFM5Q++LYD0WG02P1Ks/y+zH2g/H6v9u3n78IR4zz5P0a5IU2L9LZrJBIJLVu2zNbAClW6q/8bhCSlP/rYOv/inv2SpDfjh7RgTqWKzgiOVHgYhU4ZP1GBGedLkiZ9vlh/bH3N6qs4sONXmrnwBp2/5G91pPU1pkQKXDr9p6zjKTM9wpHA7fJKLh577DGVlZVZx707pGL49CYdfZs7i6YE9OOHn9QjPx7djT0YXkfeeM2qWhwzk5qx8DvaV/eQJMk3bYY6Xt6p8f6gpl/W059DgoHjpdMfOR1CQRoNt6IOl7ymRcrKytTa2mqt0tk30cDQeCdNUOenVYpenR99LO+kCZKkjg8+W7Cs6Iyeu0mO79NAYZngO836+mjnh5rgP03jfUGN9wU16fPFSre/oz++8Tu9sflBBWaWarwv6FywcJTXe6rMzswqhWmm5fVOdigiuFVeyUVDQ4N++ctfWrehbtmyxe64CtaF55VkHT+nZKreih/Sj/7pV/3OeU+l/6JQTZpSpBnXfKff+H8c7dKkM4rU9X67NXasM6n39xg6ZTx/L4Vq9sWVWcdLZ31xhCOBpFGxt8hwyWtaRJK1DaskGjqHKN318WdTHcf1T3R8kNS5oanyTpqoojOkpddeap3bs69Ncy84l+bOAjNu/ARrGuRo54fqeHmndc43bYaSb7foP48dVdcHHTq99MsZd5qcMnGStQYGCs+04sw1cQ62H9L5s2ayzoVDPC6+WySv5OL4eSKW/x68PfsO6Hef3mK6ZftunRuaat2e+sPvXqNHn3lB54am6s34IWstC++kiTonNFX/smO3vJMm6L0jKWv9C7ibt3iGvNNnSJLOvHCuut5vV+qdN3qSiPff0+e/NEefHPtY4/2nqW3Hk5J6KhXpxDvWOUn6Y+trjv0MGB3uufNWPfTwEyo9/xy1vvGW7rmTNS4cMwbu+shXXit0PvLII/J4PAqFQorH45I05LtFCnGFTuSvEFfoRP5YoRODMVIrdL5r/N+8rz07/G0bI7FfXjWZm2++WT6fTy+++KJ8Ph+3oQIAAEvePRdVVVWqqqqSJKXTaXm93pNcAQAALGOgMTNfeScXjY2N1tdbt24d8sZlAAAUEjevc8HGZQAAOIG7RTKxcRkAABgIG5cBAOAAzzhP3o/Rjo3LAACArdi4DAAAJ9DQmen4jcrYuAwAgEFycUNnzj9Z31tPT4RNzAAAKGw5Vy5KS0u1fv16felLX9LVV1+dcS6RSMgwDMXjca1cudL2IAEAcBvWuZAUCoW0Zs0aNTQ0aO3atUqlUurs7JTH49H06dO1ZMkSa8VOAABwEi6eFhl0z8X8+fMz1rgAAADoK+/lvwEAQP7GwnoV+XJvTQYAADiCygUAAE6goRMAANjJQ0MnAACwlYsrF+5NmwAAgCNILgAAgK2YFgEAwAn0XAAAADu5eZ0LkgsAAJzg4sqFe38yAADgCCoXAAA4wM27olK5AAAAtqJyAQCAE1zcc0FyAQCAA5gWAQAAyBGVCwAAnDDOvZ/vSS4AAHCAm3dFde9PBgAAHEHlAgAAJ9DQCQAAkBsqFwAAOMDNPRckFwAAOMHF0yIkFwAAOMDNlQv3/mQAAMARJBcAAMBWTIsAAOAEF0+LkFwAAOAAN29cRnIBAIAT2FsEAACMddFoVJKUSqUUCoUUDoeH5XXcmzYBADCKeTzj8n7kIx6PyzAMRSIRVVdXq7a21uaf6DMkFwAAFADDMOTz+axjn88nwzCG5bWYFgEAwAFDaeicN2/eCc/v2LGj31hbW5uCwaB1HAwGZZpm3jGcyKhJLs5d+D2nQwAAYMScXnSB0yEolUoNy/cdNckFAADITbbKxMmUlJRkVCqSyaRCoZCdYVnouQAAoACEw2E1Nzdbx4lEYtjuFvF0d3d3D8t3BgAAo0rfW1EDgYAikciwvA7JBQAAsBXTIgAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFzaLxWJatWqVFi9ebO02F41GNXv2bNXU1Ax6k5hYLKarrrpqwOtisZhmz549bJvPAAAwWCQXNisvL9fcuXNVUVFhLasaiUQUCAS0ZMkS+f3+QX+/vmu/19XV9TtfUVEx9MDhiLq6OusRjUb7/X6dEo/HMxJkjE7xeNxacXG4maap2traEXktjH0kF2PMaHnzwdDFYjGZpqnq6mpVV1ervLxcbW1t1nknf9ehUEhz5sxx7PWRm82bNw/b8s3H8/v9CofDI5bMYGwjuRgmvZ8oeh/Hb2tbU1OjaDSq2tpaxWIxa8wwDNXU1Cgej/f7noZhyDRN1dXV9ftEaRiGotGoVq1aJemzqZje561atUo1NTXD8aMiT6lUKmMToVAopLlz51rHJJI4kWg0qsrKyhF9zfLycqpZyAnJxTAJhUKKRCLWIxAIWOfq6uoUDAYViUS0fPlyrV+/XpIUDAYVDoc1d+7crOXHcDgsv9+v6urqfjvZ+f1+RSIRVVZWKhqNKhKJaMGCBdb5uXPn6vbbbx+mnxb5CIfD6uzs1OzZs7V27VrFYjHr95otkTQMQ1dddZUMw9DSpUtlmma/JDUWiykajcowDK1du1aSso5J2RPc2tpaa4qmpaVlhP+NYDAMw8ioWvT2Xx3/QaNXbW1txgeek12T7e+jV7YPP0Bfn3M6gEIUi8Xk8/msN401a9ZY5+rq6mSappLJ5KC+Z9++jF7V1dXauHGjwuFwRnKD0ePRRx9VLBaTYRhavXq11qxZo0gkkpFI9uodCwQC2rBhg7Zt22YlqZK0dOlSlZWVqaSkJON3vnXr1n5jfRPc3murq6sVj8e1bt06SeIT6ih3/H8jevuvAoGAwuGwVT2NRCJWFaz397127VqFQqEBr0mlUv3+Ph599FHrdWKxWNb/5gC9qFw4oLy8XFLPm0XvJ9W6ujolk8mMqsTxnxZ6mabZb94zW/JQXl6uRCKhurq6fpUOOK/391teXq7ly5dr+/btOU2FlJeXy+/3KxaLKZlMyjAMGYahNWvWaOXKlYrFYlq8eLE2btwoSVnHsl1rGIb1tylJPp9vGH5q2KWzszPreLam8eOTgVAolJE8Hn9Ntr+PXnxQQS5ILmwWj8e1a9cu7d27N+NW1FQqpc2bN1sNfMFg0Co/S1JFRYU6OztlGIbVnxGPxxWLxRSPx603nerqatXV1Vn/Meh73jRNGYahrVu3WremLliwQLFYbNB3qWD4xePxftWB49/QsyWSvbIlqdu2bdO6deu0fft2BYNBxePxrGPZrg2HwxkNpQO9eWF0G+iDRt+pjL5/A9muyfb30SuVSlG1wEl5uru7u50OAsMnFosplUpRuRiFepPOQCCgVCqltrY2rVy50koEexPG8vJyhcNhxWIx3XTTTVqzZo01XVJbW2tNlYRCIW3dutVq8ovH41q+fLlqamr6jWW7try8XLW1tdYbR11dnXw+nx544IGR/leDHKxdu1bLly+3fl+xWEyrV6+27j5avXq1fD6f7r77bvn9fut32/vhpbq6+oTX9H6I6fv3IfX0YvT9OwWyIblwqb5zrX3n7QG4QywW0969e0f8/9+rVq0i4cRJMS3iUr2NWSywBbjT8euijATDMLRy5coRfU2MTVQuAGCMMk1T27ZtG5HqRW9P10gt2oWxjeQCAADYimkRAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgq/8PY0SVokXiwVIAAAAASUVORK5CYII=\n",
|
|
"text/plain": [
|
|
"<Figure size 578.387x357.463 with 2 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"fig, ax = plt.subplots(1, 1, figsize=set_size(width, subplots=(1,1)))\n",
|
|
"# Manually set confusion matrix values obtained from results.plot_confusion_matrix()\n",
|
|
"matrix = np.array([[493, 0, 169], [0, 382, 106], [105, 158, 0]])\n",
|
|
"labels = ['Healthy', 'Stressed', '(none)']\n",
|
|
"sns.heatmap(matrix, annot=True, xticklabels=labels, yticklabels=labels, fmt=\".0f\", cmap=sns.cubehelix_palette(as_cmap=True, start=.3, hue=1, light=.9))\n",
|
|
"fig.tight_layout()\n",
|
|
"fig.savefig(fig_save_dir + 'CMmodel-final.pdf', format='pdf', bbox_inches='tight')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3871c398",
|
|
"metadata": {},
|
|
"source": [
|
|
"### View dataset in fiftyone <a name=\"modelfiftyonesession\"></a>\n",
|
|
"\n",
|
|
"We can launch a fiftyone session in a new tab to explore the dataset and the results."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "bfb39b5d",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Session launched. Run `session.show()` to open the App in a cell output.\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/javascript": [
|
|
"window.open('http://localhost:5151/');"
|
|
],
|
|
"text/plain": [
|
|
"<IPython.core.display.Javascript object>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"session = fo.launch_app(dataset, auto=False)\n",
|
|
"session.view = dataset.view()\n",
|
|
"session.open_tab()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 104,
|
|
"id": "e1d00573",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"session.close()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "53a67321",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.15"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|