{
"cells": [
{
"cell_type": "markdown",
"id": "945c9b80",
"metadata": {},
"source": [
"# Table of contents\n",
"1. [Introduction](#introduction)\n",
"2. [Aggregate Model Evaluation](#modelevaluation)\n",
" 1. [Loading the dataset](#modeload)\n",
" 2. [Perform detections](#modeldetect)\n",
" 3. [Evaluate detections](#modeldetectionseval)\n",
" 4. [Calculate results and plot them](#modelshowresults)\n",
" 5. [View dataset in fiftyone](#modelfiftyonesession)"
]
},
{
"cell_type": "markdown",
"id": "01339680",
"metadata": {},
"source": [
"## Introduction \n",
"\n",
"This notebook loads the test dataset in YOLOv5 format from disk and evaluates the model's performance."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ff25695e",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/zenon/.local/share/miniconda3/lib/python3.7/site-packages/requests/__init__.py:104: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (5.1.0)/charset_normalizer (2.0.4) doesn't match a supported version!\n",
" RequestsDependencyWarning)\n"
]
}
],
"source": [
"import fiftyone as fo\n",
"from PIL import Image\n",
"from detection import detect\n",
"from detection import detect_yolo_only"
]
},
{
"cell_type": "markdown",
"id": "86a5e832",
"metadata": {},
"source": [
"## Aggregate Model Evaluation \n",
"\n",
"First, load the dataset from the directory containing the images and the labels in YOLOv5 format.\n",
"\n",
"### Loading the dataset "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bea1038e",
"metadata": {},
"outputs": [],
"source": [
"name = \"dataset-new\"\n",
"dataset_dir = \"dataset\"\n",
"\n",
"# The splits to load\n",
"splits = [\"val\"]\n",
"\n",
"# Load the dataset, using tags to mark the samples in each split\n",
"dataset = fo.Dataset(name)\n",
"for split in splits:\n",
" dataset.add_dir(\n",
" dataset_dir=dataset_dir,\n",
" dataset_type=fo.types.YOLOv5Dataset,\n",
" split=split,\n",
" tags=split,\n",
" )\n",
"\n",
"dataset.persistent = True\n",
"classes = dataset.default_classes"
]
},
{
"cell_type": "markdown",
"id": "361eeecd",
"metadata": {},
"source": [
"If the dataset already exists because it had been saved under the same name before, load the dataset from fiftyone's folder."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "2d479be8",
"metadata": {},
"outputs": [],
"source": [
"dataset = fo.load_dataset('dataset')\n",
"classes = dataset.default_classes"
]
},
{
"cell_type": "markdown",
"id": "4485dce3",
"metadata": {},
"source": [
"### Perform detections \n",
"\n",
"Now we can call the aggregate model to do detections on the images contained in the dataset. The actual detection happens at line 6 where `detect()` is called. This function currently does inference using the GPU via `onnxruntime-gpu`. All detections are saved to the `predictions` keyword of each sample. A sample is one image with potentially multiple detections.\n",
"\n",
"> **_NOTE:_** If the dataset already existed beforehand (you used `load_dataset()`), the detections are likely already saved in the dataset and you can skip the next step."
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "63f675ab",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 100% |█████████████████| 640/640 [8.9m elapsed, 0s remaining, 1.4 samples/s] \n"
]
}
],
"source": [
"# Do detections with model and save bounding boxes\n",
"with fo.ProgressBar() as pb:\n",
" for sample in pb(dataset.view()):\n",
" image = Image.open(sample.filepath)\n",
" w, h = image.size\n",
" pred = detect(sample.filepath, '../weights/yolo-final.onnx', '../weights/resnet-fold-7.onnx')\n",
"\n",
" detections = []\n",
" for _, row in pred.iterrows():\n",
" xmin, xmax = int(row['xmin']), int(row['xmax'])\n",
" ymin, ymax = int(row['ymin']), int(row['ymax'])\n",
" rel_box = [\n",
" xmin / w, ymin / h, (xmax - xmin) / w, (ymax - ymin) / h\n",
" ]\n",
" detections.append(\n",
" fo.Detection(label=classes[int(row['cls'])],\n",
" bounding_box=rel_box,\n",
" confidence=int(row['cls_conf'])))\n",
"\n",
" sample[\"predictions_model_optimized_relabeled\"] = fo.Detections(detections=detections)\n",
" sample.save()"
]
},
{
"cell_type": "markdown",
"id": "10d94167",
"metadata": {},
"source": [
"### Evaluate detections against ground truth \n",
"\n",
"Having saved the predictions, we can now evaluate them by cross-checking with the ground truth labels. If we specify an `eval_key`, true positives, false positives and false negatives will be saved under that key."
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "68cfdad2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Evaluating detections...\n",
" 100% |█████████████████| 640/640 [2.9s elapsed, 0s remaining, 242.0 samples/s] \n",
"Performing IoU sweep...\n",
" 100% |█████████████████| 640/640 [2.8s elapsed, 0s remaining, 235.9 samples/s] \n"
]
}
],
"source": [
"results = dataset.view().evaluate_detections(\n",
" \"predictions_model_optimized_relabeled\",\n",
" gt_field=\"ground_truth\",\n",
" eval_key=\"eval_model_optimized_relabeled\",\n",
" compute_mAP=True,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "94b9751f",
"metadata": {},
"source": [
"### Calculate results and plot them \n",
"\n",
"Now we have the performance of the model saved in the `results` variable and can extract various metrics from that. Here we print a simple report of all classes and their precision and recall values as well as the mAP with the metric employed by [COCO](https://cocodataset.org/#detection-eval). Next, a confusion matrix is plotted for each class (in our case only one). Finally, we can show the precision vs. recall curve for a specified threshold value."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "86b90e80",
"metadata": {},
"outputs": [],
"source": [
"from helpers import set_size\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "e34a18f4",
"metadata": {},
"outputs": [],
"source": [
"# Style the plots\n",
"width = 418\n",
"sns.set_theme(style='whitegrid',\n",
" rc={'text.usetex': True, 'font.family': 'serif', 'axes.labelsize': 10,\n",
" 'font.size': 10, 'legend.fontsize': 8,\n",
" 'xtick.labelsize': 8, 'ytick.labelsize': 8})"
]
},
{
"cell_type": "markdown",
"id": "8ee61fce",
"metadata": {},
"source": [
"The code for the LaTeX table of the classification report can be printed by first converting the results to a pandas DataFrame and then calling the `to_latex()` method of the DataFrame. This code can then be inserted into the LaTeX document."
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "f7ad63b0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\\begin{tabular}{lrrrr}\n",
"\\toprule\n",
"{} & precision & recall & f1-score & support \\\\\n",
"\\midrule\n",
"Healthy & 0.679 & 0.525 & 0.592 & 766.0 \\\\\n",
"Stressed & 0.646 & 0.447 & 0.529 & 494.0 \\\\\n",
"micro avg & 0.667 & 0.494 & 0.568 & 1260.0 \\\\\n",
"macro avg & 0.663 & 0.486 & 0.560 & 1260.0 \\\\\n",
"weighted avg & 0.666 & 0.494 & 0.567 & 1260.0 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\n",
"0.3374377395168513\n"
]
}
],
"source": [
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
"\n",
"# Export DataFrame to LaTeX tabular environment\n",
"print(results_df.to_latex())\n",
"# YOLO second hyp with Resnet optimized and relabeled dataset\n",
"print(results.mAP())"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "d73cca50",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\\begin{tabular}{lrrrr}\n",
"\\toprule\n",
"{} & precision & recall & f1-score & support \\\\\n",
"\\midrule\n",
"Healthy & 0.653 & 0.604 & 0.628 & 766.0 \\\\\n",
"Stressed & 0.566 & 0.492 & 0.527 & 494.0 \\\\\n",
"micro avg & 0.620 & 0.560 & 0.589 & 1260.0 \\\\\n",
"macro avg & 0.610 & 0.548 & 0.577 & 1260.0 \\\\\n",
"weighted avg & 0.619 & 0.560 & 0.588 & 1260.0 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\n",
"0.36171308664990176\n"
]
}
],
"source": [
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
"\n",
"# Export DataFrame to LaTeX tabular environment\n",
"print(results_df.to_latex())\n",
"# YOLO original with Resnet optimized and relabeled dataset\n",
"print(results.mAP())"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "7ba5cd14",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\\begin{tabular}{lrrrr}\n",
"\\toprule\n",
"{} & precision & recall & f1-score & support \\\\\n",
"\\midrule\n",
"Healthy & 0.665 & 0.554 & 0.604 & 766.0 \\\\\n",
"Stressed & 0.639 & 0.502 & 0.562 & 494.0 \\\\\n",
"micro avg & 0.655 & 0.533 & 0.588 & 1260.0 \\\\\n",
"macro avg & 0.652 & 0.528 & 0.583 & 1260.0 \\\\\n",
"weighted avg & 0.655 & 0.533 & 0.588 & 1260.0 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\n",
"0.35812991936475147\n"
]
}
],
"source": [
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
"\n",
"# Export DataFrame to LaTeX tabular environment\n",
"print(results_df.to_latex())\n",
"# YOLO optimized with Resnet optimized and relabeled dataset\n",
"print(results.mAP())"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "f2b178e8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\\begin{tabular}{lrrrr}\n",
"\\toprule\n",
"{} & precision & recall & f1-score & support \\\\\n",
"\\midrule\n",
"Healthy & 0.711 & 0.555 & 0.623 & 766.0 \\\\\n",
"Stressed & 0.570 & 0.623 & 0.596 & 494.0 \\\\\n",
"micro avg & 0.644 & 0.582 & 0.611 & 1260.0 \\\\\n",
"macro avg & 0.641 & 0.589 & 0.609 & 1260.0 \\\\\n",
"weighted avg & 0.656 & 0.582 & 0.612 & 1260.0 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\n",
"0.38379973332791195\n"
]
}
],
"source": [
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
"\n",
"# Export DataFrame to LaTeX tabular environment\n",
"print(results_df.to_latex())\n",
"# YOLO original with Resnet original and relabeled dataset\n",
"print(results.mAP())"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "b14d2b25",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\\begin{tabular}{lrrrr}\n",
"\\toprule\n",
"{} & precision & recall & f1-score & support \\\\\n",
"\\midrule\n",
"Healthy & 0.841 & 0.759 & 0.798 & 663.0 \\\\\n",
"Stressed & 0.726 & 0.810 & 0.766 & 484.0 \\\\\n",
"micro avg & 0.786 & 0.780 & 0.783 & 1147.0 \\\\\n",
"macro avg & 0.784 & 0.784 & 0.782 & 1147.0 \\\\\n",
"weighted avg & 0.793 & 0.780 & 0.784 & 1147.0 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\n"
]
}
],
"source": [
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
"\n",
"# Export DataFrame to LaTeX tabular environment\n",
"print(results_df.to_latex())\n",
"# YOLO original with Resnet original and new dataset"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "900e9014",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\\begin{tabular}{lrrrr}\n",
"\\toprule\n",
"{} & precision & recall & f1-score & support \\\\\n",
"\\midrule\n",
"Healthy & 0.674 & 0.721 & 0.696 & 662.0 \\\\\n",
"Stressed & 0.616 & 0.543 & 0.577 & 488.0 \\\\\n",
"micro avg & 0.652 & 0.645 & 0.649 & 1150.0 \\\\\n",
"macro avg & 0.645 & 0.632 & 0.637 & 1150.0 \\\\\n",
"weighted avg & 0.649 & 0.645 & 0.646 & 1150.0 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\n",
"0.49320073714096757\n"
]
}
],
"source": [
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
"\n",
"# Export DataFrame to LaTeX tabular environment\n",
"print(results_df.to_latex())\n",
"print(results.mAP())\n",
"# YOLO original and Resnet final with old dataset"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "24df35b4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" Healthy 0.82 0.74 0.78 662\n",
" Stressed 0.71 0.78 0.74 488\n",
"\n",
" micro avg 0.77 0.76 0.76 1150\n",
" macro avg 0.77 0.76 0.76 1150\n",
"weighted avg 0.77 0.76 0.77 1150\n",
"\n",
"0.6225848121901868\n"
]
}
],
"source": [
"# Print a classification report for all classes\n",
"results.print_report()\n",
"\n",
"print(results.mAP())\n",
"# YOLO original and Resnet original with old dataset"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "a6bb272a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" precision recall f1-score support\n",
"\n",
" Healthy 0.66 0.64 0.65 662\n",
" Stressed 0.68 0.54 0.60 488\n",
"\n",
" micro avg 0.67 0.60 0.63 1150\n",
" macro avg 0.67 0.59 0.63 1150\n",
"weighted avg 0.67 0.60 0.63 1150\n",
"\n",
"0.44258882390400406\n",
"\\begin{tabular}{lrrrr}\n",
"\\toprule\n",
"{} & precision & recall & f1-score & support \\\\\n",
"\\midrule\n",
"Healthy & 0.664 & 0.640 & 0.652 & 662.0 \\\\\n",
"Stressed & 0.680 & 0.539 & 0.601 & 488.0 \\\\\n",
"micro avg & 0.670 & 0.597 & 0.631 & 1150.0 \\\\\n",
"macro avg & 0.672 & 0.590 & 0.626 & 1150.0 \\\\\n",
"weighted avg & 0.670 & 0.597 & 0.630 & 1150.0 \\\\\n",
"\\bottomrule\n",
"\\end{tabular}\n",
"\n"
]
}
],
"source": [
"# Print a classification report for all classes\n",
"results.print_report()\n",
"results_df = pd.DataFrame(results.report()).transpose().round(3)\n",
"\n",
"print(results.mAP())\n",
"print(results_df.to_latex())\n",
"# YOLO final and Resnet final with old dataset"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "da05e2ba",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Ignoring unsupported argument `thresholds` for the 'matplotlib' backend\n",
"Ignoring unsupported argument `thresholds` for the 'matplotlib' backend\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAACoCAYAAADtjJScAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAwg0lEQVR4nO3dfXRUZX4H8G/ACAHmBUyMQG7KW8DJJC2EcrqZtO6yRDvQXSNhJR6L5yRijNoWlMLW7llSDbZqE1+gui7OrmCP22bWY3bZnkNGXtS2h4nABhaSIZCgYecOIgFhZkJeytv0j/Tezk0myWReb2a+n3M4Mvdl5sck9+tzn3vv86T4fD4fiIiIiBLIhHgXQERERBRpbOAQERFRwmEDh4iIiBIOGzhERESUcNjAISIiooTDBg4RERElHDZwiIiIKOHcEe8Coun48ePw+XxITU2NdylESeHGjRtISUnBkiVL4l1KzDBniGJnLBmT0D04Pp8PwYxj6PP5cP369aC2jQfWFx611weov8Zg6wv2mEskzJnYYH3hSZT6xpIxCd2DI51R5efnj7hdb28v2trasGDBAkyZMiUWpY0J6wuP2usD1F9jsPW1tLTEsCp1YM7EBusLT6LUN5aMiXkDx+v1wmq1AgAqKysDbmOz2QAAHo8HgiDAZDLFrD4iGv+YM0QU80tUdrsdbrd72PWiKMJut8NsNqOsrAwWiyV2xRFRQmDOEFHMe3DMZjM8Hg+8Xm/A9Xa7HRqNRn6t0Whgt9tDPrvy+Xzo7e0dcZu+vj60tLTg6NGjuPPOO+XlWq0WM2fOxPXr19HZ2Tlkv0WLFgEAnE4n+vr6FOtmzpwJrVYLt9uNixcvYsqUKRAEARqNBgsWLBjTv0F678GfoRasL3xqrzHY+nw+H1JSUmJR0ogSLWc0Gg1u376Nnp4eud6x5shoEuV3MF5YX3iikTGquwfH6XRCr9fLr/V6/bAhFYwbN26gra1txG3cbjcef/zxmN181dDQgOzs7DHvd+7cucgXE0GsL3xqrzGY+vz/561W4zFnFi5ciPb2dvl1qDkymkT4HYwn1heeSGaM6ho4gXg8npD3TU1NHfVMp6+vD++99x76+/uj0oNz5MgR1NTU4NVXX8W+ffswf/78MZ199fX14dy5c5gzZw7S0tKC3i9WWF/41F5jsPWdPXs2hlVFllpzBgDWr1+PF154AdnZ2Thz5gzWr1+Pu+++GwaDIeSaA9WXCL+D8cL6whONjFFdAyc7O1txJuV2uyEIQsjvl5KSEtQd4/n5+TAYDMNuu3z58mH3LSoqGvG9Z86ciZqaGhQXF+OHP/zhqLUMJy0tTZV3v0tYX/jUXuNo9anh8lQwxlPOHDt2DMDACVVBQQEMBgPuvPPOEd8nHOP9dzDeWF94IpkxqhkHRwobk8mkeAzM5XIlzNMNt27dgtfrxa1bt+JdClFSSoScmTFjBtatW4cZM2bEuxQiVYvLU1SHDh2C3W6XH9MEgNLSUni9XgiCgFWrVsFms8FqteLJJ5+MdYlR097eDp1OhxMnTsS7FKKElsg5c+nSJbz99tu4dOlSvEshUrWYX6IymUwBz5QOHDgg/91sNseyJCJKMImcM6Io4q//+q9RWFiIjIyMeJdDpFqquURFREREFCmqu8k4kUlPSEjX/kVRhM/nC/iop0ajQU5OTkzrIyIiShRs4MSANKDY1q1bAQDl5eVB7dfe3s5GDhERUQjYwImBnJwctLe3o7u7G06nE7NmzcIdd9yB//7v/8azzz6LDz74QDGeRVtbG9atW4cjR46gu7sbfX196OrqiuiYF0Q0Pmk0GjzwwAOKkZiJaCg2cGJE6okpKCgYss5gMCiWS8G1bt06xXa//OUvsXjxYvbqECWxnJwcfPzxx/Eug0j12MBRIf8eHwD44osvsHbtWqxduxYAsGfPHhgMBjZ0iJLQrVu30NPTg6lTp2LixInxLodItdjAiaPZs2fjtddew+zZs4es82+83HvvvWhoaMDNmzexdu1alJSUAOA9OkTJ6MSJE1i6dCmam5sD9ggT0QA2cOIoMzMTFRUV6OzsxPnz5+XlaWlp8v02Z8+exaxZs5CdnQ2DwYD29nYcOXJEvkcHABs5REREg7CBE2cHDhyQLz1JlixZgmPHjuHSpUvIyclRjHzs35iR7tFhTw4REZESGzhxVlxcjObmZsUyaSZVURQBAN3d3Zg8ebK8XrpHR+rJke7VISIiogFs4MTZ9OnTMX369DHvl5OTIzds2tra5OUcIJCIiIgNnHGhv78fkydPhtPpRG9vL4CBhsxwj5PzkhVR4srPz0dXVxf0en28SyFSNTZwVExqwNy4cQMA8Morr+D999+X17e3tyseJ5cGCOQlK6LElZqaykk2iYLAyTZVTLrXZtmyZQCA559/Hs3Nzfjggw8wYcIEdHd345577gEw0BiSnrxqa2vDsWPHcOzYMXR0dMStfiKKvC+++AIPPvggvvjii3iXQqRq7MFRuZycHPmyVHZ2NqZMmQKNRoNnnnkGgiCgubkZy5cvBwC88847AIZestqzZw+ysrIUy3ivDtH45PF48B//8R944YUX4l0KkaqxgTMO5eTk4B//8R+h1WqxdOlS7NmzByUlJXj66aeRmZmJvXv34urVqzh58iQ2bdokDww4GBs+RESUqNjAGae0Wi2AgQbJgw8+KN+Lk5qaCo1Gg+9+97vweDyKfXbu3In58+ePqeHDBg8REY1HEW3guFyuIT0CFBuDGyFHjx4dcrPxPffcg5dffhk/+clPFMtLSkrwwx/+EIcPHw7Y8GGDh9SCGUNEwQqrgdPW1ga32y2/tlqtePPNN8MsiSJhuEbIhg0bUFFRoVh211134dq1a9i0aZNi+TvvvIOnn356SIOHj6FTrDBjhhppDjsi+n8hN3A2btyI7u5u+VFmQDngHKnTcA2Tvr4+eURll8uFkpISrFmzBitWrMD3v/99nDlzBs899xzeeOMNPoZOMcGMCSwzM3PIyQgRDRVyA6eoqGjIHEoff/xx2AVRfKSlpckzExcUFKCjowMZGRnIyMjAW2+9hfvvvx9vvPEGAODTTz/lpSqKOmZMYFevXsWBAwdQXFwc0ijoRMki5HFwBEEYsiw7OzusYkg9FixYIP+9uLgY7e3t2L17NwBg8+bNWLhwId544w38+te/xvHjx+FyueJUKSUqZkxgnZ2dWLt2LTo7O+NdCpGqhdyDI4oirFYr8vPzAQA+nw+NjY346KOPIlYcqUdOTg5ycnJgMpnw8ssvY9euXYpu8iVLlkCv1+PSpUvo6enBrFmzcP/998exYhrvmDFEFI6QGzj19fUwmUzw+XzyMv+/U2LKycnByy+/jAcffBCdnZ1yI+f48eP4zne+o9j2Rz/6ER566CFMnDgR33zzDe666y7FekEQkJGRgStXruDcuXOKdbwERswYIgpHyA2cLVu2oLCwULHMZDKFXRCpX2ZmJh566CEAwPe+9z10dXXh3LlzmDNnDi5duoSjR4/in/7pn+Q/w/m7v/s7rF+/HocPH8Zjjz2mWJeXl4ePP/4YX3/9NQA2eJIRM4aIwhFyA6ewsBDXrl1DY2MjAGDlypXIzc2NWGE0PuTk5GD27NmYPHkyDAYDpkyZgoceegjf+c530NXVhYULF+LChQsBBxV89dVX8eqrr+KDDz7Ar371K8X9FVLvjv+jsHv27MEDDzyAyZMno7OzE1evXpXXsQGUeJgxgaWlpWHJkiVIS0uLdylEqhbWPTgbN26UbwS0WCzYvn27POEjJbfB99/4z3oukR5Hl+bOCjS+TnNzs7xdSUkJOjo6sGDBAmzduhW/+MUvhnwGGzmJgxkTmMFgwLFjx3D8+HF0dHTwd55oGCE3cPbt24eGhgbFstdeey3pw4cCCxTCBQUFaG9vx5EjR7Bu3TocOXJkyLYFBQXydt3d3fIottu2bZPv/3G73fiv//ovTJs2LQb/EooVZszIvvWtb+H69ets2BMNI+QGTqDh0vPy8sIqhpKPfzBLPTmBpoYYHOBz587F3Llz5dff/e53Y1AtxRIzZmS7d+/Go48+yoE3iYYR1iWqwTgWCoUiJycH7e3taGtrky9F+RvtDNXr9aKpqQmFhYXyJKQ0/jFjRrZo0aJ4l0CkaiE3cEwmEx5//HEYjUYAgN1ux+bNmyNWGCUXqZfG/16dtrY2rFu3btQz1LNnz8JsNqO5uVkejZnGP2YMEYUj5AZObm4uXnzxRVitVgAD90TwCQcKF+8lIAkzZmQGgwGtra2YN29evEshUqWwZhMXBEFxRuVyuQJeNyciCgUzZnhpaWm4++67FROQcrgEov8XdANn3759MJlM8pMqH374oWK91+uF3W7Hz3/+88hWSERJgRkzNleuXMHf/M3fyD1cEj5VRTQg6Mk2f/rTn6KlpUV+/e///u/weDzyH5/Ppxh4jShWJk2ahPnz52PSpEnxLoXCwIwZmxkzZmDbtm1obm5Gc3MzDhw4gOeffx5TpkyJd2lEqhB0D87g8SheeumlIdfDOYw6xYPRaMTZs2fjXQaFiRkzdoN7alasWBGnSojUJ+genMH27t2LDz/8ENeuXcP69evx7LPP8hFOIooYZszYdHd347PPPuO4OET/J+QGTn5+Ph5++GHU19fDYDDgzTffhNvtjmBpRME5efIkMjIycPLkyXiXQhHEjBmbjo4OLF++HL/5zW/Q0dER73KI4i7kp6ikAdUaGxvx0ksvAQB0Ol1kqiIag5s3b+Ly5cu4efNmvEuhCGLGKPk/LTWYRqOBRqMBgBHndiNKJmGPZCyKIgwGA0RRhNfrjVhhRJTcmDEDBjdchtPe3o729nb87ne/w3PPPYf+/n6cO3cOc+bMiUGVROoTcgNn5cqVsFqt+Oijj9Dd3Q2r1Yrp06cHta/NZgMAeDweCIIQ8MbBDRs2oKqqCsDAtfgtW7aEWioRjUPhZAyQODkzeITvwaQRv48cOQKDwYD58+fj008/hV6vR05ODo4ePcqeHEpKITdwNBoNnnjiCfn15s2bg7oBUBRF2O121NTUAAAqKioCBo/L5UJ5eTny8vKwffv2UMskonEq1IwBEi9nRmqgDNfD86tf/Qoej4c3HVPSivlAf3a7XT4ggYGD0263DwmfJ598EmazOdjyhuXz+dDb2zviNn19fYr/qk2y1uf/viP9DLOysvDJJ58gKysr4HZq//4A9dcYbH0+nw8pKSkhfUYkB/pLppyZPXs2Tpw4ITdkzpw5g/Xr1+PKlSvy541WWzTrixTWF55EqW8sGRN0A+enP/0pNBoNCgsLAQwMwrVq1SrFNsEMwuV0OqHX6+XXer0+4HV1acAvj8cDACgrKwu2VIUbN26MeHOev3PnzoX0GbGSbPVJ73fu3DlMnjx5xG21Wm3A2acDvZ+aqb3GYOq78847Q3rvSGUMkJw5Ix0j0oCXX331FQDA4XCMevz4S4TfwXhifeGJZMaoYqA/KVz8+V8LLy4uxsqVK+WnKsYiNTUVCxYsGHGbvr4++Wa8tLS0MX9GtCVrff39/QCAOXPmwGAwDLvd+fPnsWPHDmzYsAGzZ8+OWX2RpPYag60vnAEXoz3QX7LkjHTcSJe1pkyZgvT0dLhcLmg0mmHrTJTfwXhhfeGJRsaEfA+OIAj4+c9/jrKyMkybNg1NTU3Iz88fdb/s7GzFmZTb7YYgCIptbDYbWlpa5PCRzs6NRuOY60xJSQl66PK0tDRVD3OebPVJv+SjvW93dzfeeustVFRUjLid2r8/QP01jlZfqJenAgk1Y4DkzhnpuJFuTp46dSpefvll/OQnP8Ht27dHfXx8vP8OxhvrC08kMybkgf4aGxsV3cWFhYWw2+2j7mcymRTzzbhcLvmsTAokQRBQVFQkb+P1ekMKHSIav0LNGIA5I8nJycGsWbOwYcMG/Ou//isA4Msvv4xzVUSxEXIPjl6vx9q1a8e8nyAIWLVqFWw2GzweD5588kl5XWlpKRoaGmA0GmGz2eQzrF27doVaJhGNU6FmDMCcAYYODDh16lQAAzdv33fffYr1Go0m4OVdovEs5AbOyZMnFU88AAM37D3wwAOj7jvcUwsHDhwYsk0knnAgovEnnIwBkjdnRhsY8Fvf+hY+/fRT/MVf/IVi+YkTJ6JeG1EshdzAKSsrw+rVq5GdnQ2NRoNTp07hxRdfjGRtREFJT0/HM888g/T09HiXQhHEjAnNcAMDulwulJSUoLKyEgDw+uuv4+6770ZXVxc2bdoEURThdDpx4sQJCIKA+++/Px7lE0VMWDcZNzQ0oLGxEV6vF3/7t3875CY+oljIzs7G22+/He8yKMKYMaELdBNxQUEB2tvb0dbWhpKSEmzatEmx/nvf+57i9TvvvIMVK1ZwFGQat0Ju4ACA1WpFa2sr3nzzTTQ1NWH69OmK7mSiWOjt7cXp06dx7733qvrpABo7Zkxk5eTkDNvD09fXh6amJjidTvzLv/wLnn76aQDAnj17YDAY2NChcSfkp6jq6uqg1WrlJxPG8oQDUSSdPn0aS5cuxenTp+NdCkUQMyZ6cnJyUFBQoPizZMkSLF++XB7s8PXXXwcAlJSUYOHChejo6IhnyURjFnIPTn5+Pv78z/8cTU1NkayHiAgAMyZe5syZg1/+8pcoLi5GV1cXXnnlFQADl6y+/e1vQxAEaDQa9uiQ6oXcgxNo0jv/cSeIiMLBjImP6dOn4+GHH8b06dPx7LPP4p133gEAvPHGG3jooYewdOlSLFy4EA0NDezVIVULuQcnNzcXpaWlmD59Oux2O+x2OzZv3hzJ2ogoiTFj4i8zMxNPPfUU5s+fj66uLkybNg1ffvklNm3ahDVr1gDAqCMjE8VLyD04hYWF2L59OwwGA3w+H7Zt2yZPkkcUSxMmTIBGo8GECSH/OpMKMWPU4/7778df/uVfoqSkZMjTVg6HI05VEY0s5B6cNWvWoKqqimdUFHeLFy8OOFM0jW/MGHWSnsKyWq3YunUrWltbYTQa2YtDqhPyKW9ZWdmQEUV5MyARRQozRr1ycnLwgx/8AACwdetWLFy4EL/+9a95Tw6pSsg9OCkpKfiHf/gHZGdnQxAEeDwe2Gw2diFTzJ06dQoPP/wwPvzwQ+Tm5sa7HIoQZoy63XvvvWhvb8eaNWvQ0tKC1atXA+A9OaQeIffgvPvuu/D5fLh69SpOnjyJ3//+93C73REsjSg4/f39OHXqFPr7++NdCkUQM0b9cnJyUFNTAwDYtm0bAAwZQJAoXkLuwampqRlyJsXu48jYsGEDduzYoVjW2dmJ9957DxcuXJAH4nI6nSgqKpIHQosEm80GAPB4PBAEYcT3ttls0Gq1WLx4MQBgy5YteOaZZwAAe/fuxZYtW8b8+Q6HAxs3bkRDQ8PYi48wh8OBnTt3wuVyYfPmzTCZTLDZbNi6dSvWrl2LqqoqaLXaMb2f9G8LtF9nZyeqqqrwySefjOl9ExUzJroilTPZ2dkABm4Kf+utt4KaTiOYnPF6vbBarQAgz58FAPv378ekSZPk16FMlDrasRhLsc6ZtrY2PPXUUzh48GDc/+3RNqYGTltbG/bu3Yvs7Gw8/PDDQ9az6zh8NpsNTU1NEEVRERRz585Famoqmpub5eABgEWLFuHAgQMRmaNHFEXY7Xb5jKyiomLYUPN6vXj33Xfx5JNPysvOnz+P8vJy5OXlYfv27SHVYDQaVTPfkNFoRFFRERwOh/w9mM1m1NXV4ZFHHhlzOAz+t1mtVsXPcu7cuUl/iY0ZExvRyJnp06fjr/7qr0b97GBzxm63w+12Q6/Xy8t6enpw/vx5+USquro6pAZOMueMwWBAXl5eZIpXuaAvUTU1NWH16tWw2Wyora3Fc889F826kpbH48HatWtRX18f1PZarTZiTxDZ7XZoNBr5tUajGXZo/MbGRqxcuVKxrKKiAkePHsWuXbsS/swgEqSzUxrAjImdaOSMx+PBBx98gCtXroy4XbA5Yzab5d4hydSpU/HRRx/Jj6b7vw8Flsw5E3QPjtVqxdGjR+VfqLq6OrhcLmRlZUWtuGTj9Xrl7trS0tJRL/FYrVYUFhbCaDRG5POdTqfibEmv1wcMNelMQ+pmlpw6dQqTJk2Cx+MBAMVZw3BsNpu8vU6nG3I2lp6ejpaWFhw8eBCPPPIIBEGAw+GAKIrQarWw2WzYtGkT3n77bXR2dqK/vx82m00+O4wEURQV/1apXkltbS3y8/MhiiJMJhOMRiNqa2tRVFSEQ4cOyXX7s9vtche8IAjyZT5pHTBwmW/Hjh1yd/X27dthMpmwYcMGCIIQ0iVANWPGxEa0cuarr77CY489hubmZsyYMWPY7YLNmeFs2LABpaWlMBqN2L17d1D7jJYzgY7XwTlTU1MTcFmkRDtnMjIyoNPpFOuAxM6ZoBs4WVlZitZyVVUVmpqaEiZ8XC4X+vv7kZaWFtH31ev1mDdvXlDb2u12+cATBAF2u31I121ra6t8EJhMpoCNCK/Xi507dw77OYEOhOEMPsiAgQMxULfwxo0b5dm8i4uLsXLlyhF7chwOh3xwiaKI2traIe9769Yt5Ofno7+/HxaLBTU1NfIlDJPJBJ1OB71ej56eHty8eRN5eXmKg1jS09OD7du34447Av/Kj/SdCIKgqKuurk7+u9VqhV6vl9dXVFRg165d0Ov18s9OqtufyWSCVquVf369vb3yOmmCSSnwzGaz4gy3qKgoqMbjeJPoGQMkZs5cvHgR6enpOHDgANLT0/Fv//ZvihOSYATKmeGcOnUKDQ0NqKurQ3l5+aj36wWTM4GO18E5AyDgMn9qzpne3l60tbXJ65IhZ4Ju4AzuKtRoNPD5fIplbW1tMBgMkakshi5fvozS0lLcvn074u89ceJEfP3110hPTx91W/95dvLy8lBfXz8keLKyska95qzVakNqdWdnZyvOpNxu95CD0WKxQBAE2Gw2tLS0QBRFZGRk4PDhw9i3bx/+/u//Xq5BFMURz/r27t2LoqIiAAMH9+AbHiUHDx6EXq+Xn6CpqqpCXV0dLBYLcnNz8aMf/Qg9PT04evSovGzwe02dOlXRAIsUh8Oh6GL3H5TOarXC6/WO+cmfQAFYVlaGnTt3DhusiSCRMwZI3Jw5duwYXnvtNRQXF2P37t149NFHUVBQMOx+weTMcA4fPow/+ZM/gdFoxK5du1BdXR2wgeYv2JwZfLwOzpkdO3YEXOaPOaMuQTdwRFHEtWvXFMtcLpe8zO12o76+Hi+++GJkK4yB9PR0NDQ0ID09PSpnVsGEjsPhULTuzWYzli1bFtJnhtqDYzKZUFtbK792uVxycHi9Xmi1WsXTDC0tLcjPz4fBYEBnZyfuvfdeRQ1S40badzC9Xg+n06nYx3+7gwcPYuLEiVixYgUmTZqElpYWOBwOtLa2ymcq1dXVOH78ON5880188sknqKurQ3V19ZCbJ8M5sxqJ0WiE0+mUvyeHwwGr1Qq3243Kyko4HA657kCNPa/XC7vdjvvuu09eFihYjEYjXC4XrFbrkHufEkUiZwyQuDkz1h6cYHJmONeuXVMcH/7/Iw41Z4Y7XgfnzOCbo8dbznz66aeK3tBkyJmgGzgWiwU/+9nPFMt8Pp/cjebz+ZCSkjJuwycrKwsGgyHiLe9g2O121NXVoaysTO4SlG6iq66uxrp163Dx4kXs27cPFy5cGPWMJdQeHEEQsGrVKvl6tf8TUqWlpYrHDu12u/wUxrx58zB37ly4XC65Z2fXrl2KfWtqaobUXFlZidraWrlXSDrwRVGE1WrFvHnzMGHCBLS0tMgDh4miCKfTKXefC4KAzMxMpKam4vDhw+jq6lK8lySUMytRFHHo0CG4XC75O5e+m/r6elRVVaGsrAwWiwVWqxU6nQ6CICAvLw8Oh0PR3SuKouLfVllZibKyMlitVjmQOjs7cf78efmpB+lmTKmbeeXKlXA4HOO+23g4iZ4xQGLmjNSD8/3vfx+nT59GZWUlFi1aNOx+weaM3W7HoUOH0N3dDUEQcN9992HFihU4fPgwzp49C2Dgf9LS8RNqzphMpoDH6+CcEQQh4DJ/as6Z+fPnAxjoBZXWJ3zO+IK0detWn9frHfaPx+Px1dbWBvt2MXHy5EnfyZMnR92up6fH99vf/tbX09MTg6rGLhHqO3To0Jjft7m52QfA19zcHNZ2av/+fL7gamxtbQ3pe4yEYL/DYI+5QMZjxvh8zJlgj9NwRStnIiVRfr7xyploZEzQj4k/8sgj0Gg0w/7RarVYtWpV1BpiNH45HA7VjDkxHklnjK2trREd1FFtmDEUDuZMeBIxZ4Ju4AQzAFmyD1JGgQ2+Th1per0eP/jBDxSPniYS6SmHRB+cixkzvrW1tSElJQXHjh2Ly+dHO2cSXSLmTMhTNRAFK5SRRsdi3rx5+PDDD6P6GfHkf2M3EQUW7ZxJdImYMyFPtkmkFtevX4fL5cL169fjXQoREakEGzg07rW2tkIQBLS2tsa7FCIiUgk2cIiIiCjh8B4cIiIK27x589DR0ZFQU2vQ+MYGjgpt2LBhyBDgnZ2deO+993DhwgV5ACan04mioqKIPtInPSro8XjkCflG2lar1coTRe7fvx/9/f1wOBwwm80h1eVwOLBx48ZR55eJFf+ZeHU6HTwejyoGwBJFERs3bsTmzZsT5pFOiq1I58ykSZOwYMGCoD47mJyRJokEht4AKw3aB4R2c7F/zow0cnKsMGeigw0clbHZbPIIwf6PPM6dOxepqalobm5W/OIvWrQIBw4ciMjjkYOHIq+oqBj2l9rr9eLdd9+VRyHt7OxEamoqysrK4PV6sWLFChw9enTMNRiNRtU86ulwOOD1euVwFUUR9fX18nppJNB4EAQBhYWFcflsGv+ikTPnz5/H66+/jm3btmHu3LnDbhdsztjtdrjd7iHDPzz11FN46623oNVqUVpaGlIDhzkTnPGeM7wHR2U8Hg/Wrl2r+AUfiVarVUxcFw5pyG6J/+RugzU2NirmKrl27RoOHz4s16TT6eRh4KNt8eLF6O/vl3uSIsXj8SgmJhQEQZ60D1CedRGNJ9HIGa/Xi1/84he4evXqiNsFmzNms3nIBKydnZ3yvg6HQzU9veFgzkQPe3D+j8vlQn9/f1QmwZs3b15Q23q9Xrm7trS0dNT5pKxWKwoLC0ecsXssnE6n4mxJr9cHDDWHwyHPlyLJz8/H2rVr5dcejyeouqQ5V4CBrtnBZ2Pp6eloaWlRTN7ncDggiiK0Wi1sNhtqampw9uzZIcvCZTKZYLFYsGzZMqxcuRJlZWXymabdbpe70KWfmd1uR3V1NWpqamCxWLB9+3ZotVrU1tYiPz8foijK+w+uta2tDYcPH4bH48Fnn32GmpqaIfsZjUZYLBa5AXnq1ClFEJL6MWeCz5lAvvzyS5w/f16ec0k63kYzWs7U1taiqKgIhw4dGjFnAi0LV6xy5vnnn0dnZydcLhcyMjLk+hM5Z9jAAXD58mWUlpbi9u3bEX/viRMn4uuvvw5qpl+73S4feIIgBJzsrrW1VW5YmEymgF2Xoc4mHogUCv5EURyxW7i6uhrbtm0b9b0dDgf27t2LHTt2QBRF1NbWDnnfW7duIT8/H/39/bBYLKipqcHevXuRnZ0tzyTc3t6Oxx57DOXl5SgvLw84S26os/zu2rVLntBOuhYt3V+k1WoV37+0TKfTyaFjtVqh1+vlf1dFRQVyc3MV9QPAvn37MHHiRBiNRmRmZgbcr6ysDKIoyqE6XO8aqVOi5sxYZxMPJFDOBNLb2wutVis3tlpbW4edQVsSTM7o9Xr5OxguZwAEXOZP7Tnz+eef44/+6I+wfPly6HS6hM8ZNnAw0EvQ0NCA9PT0qJxZBRM6ABTdlHl5eaivrx8SPFlZWaNecw51NvHs7GzFmZTb7R5yMEo390mzhouiiIyMDHm9zWaDyWQK6rr43r175TMDQRCG3PAoOXjwIPR6PdxuNwCgqqoKdXV1sFgsyM3NRXl5OU6cOIHTp0+jtLQUubm5Q94rlFl+peCU/lRWVqKiomLUf5t/2DocDkUX/ObNmyEIgqL+HTt24PHHH8cLL7yARx99FEajEXq9fsh+/jOPA1B085P6JWrOSLOJFxcXY/fu3Xj00UdRUFAw7H7B5Mxw7r77bsWAnjqdDqIojtjACTZnrFYrvF7vsDmzY8eOgMv8qTlnXnnlFTz44IOw2Wx4//33kZubm/A5wwbO/8nKyoLBYBjTL2YkORwORevebDZj2bJlIb1XqD04JpMJtbW18muXyyUHn9frhVarVTzN0NLSgvz8fBgMBrS1teHzzz+HVquFyWSCw+GAVquFIAjyvoPp9Xo4nU5F3f7bHTx4EBMnTsSKFSswadIktLS0wOFwoLW1VT67qK6uxsWLF6HRaPDEE0+goKAA1dXVQ26eDOXMShRFeDweRfgPPti9Xq/ijHgwo9EIp9Mpv4fD4UBjY6OiflEU8dlnn+GJJ56AwWDAK6+8gnvuuQc9PT2K/Uwmk+J/Tt3d3QE/k9QrEXNG6sE5dOgQSkpK8Jvf/GbEHpxgcmY4eXl5+Pzzz+XX/pdjQs0Zq9UKt9uNyspKOByOYXNm8M3R4y1nXC4XPv/8c/z4xz/GlClTUF1dnfA5wwaOCtjtdtTV1aGsrEzuipRu0K2ursa6detw8eJF7Nu3DxcuXAjYpewv1B4cQRCwatUq+Xq19IQUAJSWlioeqbTb7fJTGPPmzcPFixdRXV2NlJQUAAMH5JkzZ+R9a2pqhtRcWVmJ2tpauVdIOvBFUYTVasW8efMwYcIEtLS0ICcnR17ndDrl7nNBEJCZmYnU1FQcPnwYXV1diveShHJmJX2e9H04nU689NJL8rqysjLF2Y50fd7/qYeysjJYLBZYrVbodDoIgjCkfkEQ4HK50N3dDZfLBUEQUFlZOWQ/s9ks1yPVFujsmyiQaOWM1IMjnWCMJticsdvtOHToELq7uyEIAu677z5MnToVa9askXtbNm/eLGdSqDkjnZD5X4oJlDPDHbv+1JwzWVlZ6Orqwv79+zFp0qSkyJkUn8/ni3cR0SK1QvPz80fcrre3F21tbXE9sxpJItQ3WlgGcuzYMSxduhTNzc0jBudo26n9+wPUX2Ow9QV7zCWSZM8Z6fj7z//8T/T19aGwsDAqY8tEK2ciJVF/vrESjYzhY+IUdQ6HI6pjTmRnZ8NisQx5pJSIYkd6+ODs2bNx+fxo5wyNP2zgUNQNvk4daenp6XjiiSeCvsmSiBJPtHOGxh82cCjqQhlpdCwuX76Mn/3sZ7h8+XJUP4eI1CvaOUPjT1xuMg5mHpKxzIlEyc3pdKKyshIFBQXsxSEZc4YoucW8B0d61M5sNst3foeyDRHRcJgzsXfnnXdi/vz5mDRpUrxLIQIQhx6c4eYh8T9zCmabYPl8PvT29o64TV9fn+K/apOs9Unv97vf/W7E95YeR9+/fz/6+vrg8/nQ3t4ur79+/TouX76My5cvY9q0afjqq6+GjO2Qnp6Ou+66C9euXcP58+cV6yZNmoQ5c+YAADo6OoaMRPsHf/AHmDx5Mi5evCgPEiaZMWMGMjIy0NfXpxiLAxgYfVaaffnMmTP46quvcPToUdx5550ABsZMmTp1Ki5fvoxvvvlGsa9Wq8XMmTNx/fp1dHZ2DvlOFi1aBGCgd2vwdzdz5kxotVq43W5cvHhRsW7q1KnIysrC7du30dHRofgOJ0+eLH8Pw/H5fPJQAfHEnBm7UOuTtu/u7sb777+P3/72t/K8dJLs7GykpaXh0qVLuHLlimKdXq9HZmYm+vv78fvf/16xbsKECfIQEe3t7Th//rziGJk9ezamTZuGb775Zsglao1Gg1mzZuHmzZv44osvhtS9cOFCpKSkQBTFIT+7zMxMeXDRwcfIlClTIAjCuMuZ//mf/8GRI0cU3x+gnpyJRsbEvIETzDwk4cxVMtiNGzfQ1tYW1Lbnzp0L6TNiJdnq6+rqAgCsX78+qO2ff/75iH4+KaWkpGD//v1DZncezD8844U5E7qx1jfW45RoOJHOGFUM9BfMPCTBzlUyWGpqqnyWPJy+vj6cO3cOc+bMifgQ6pGQrPUZDAacOHEiqJE0XS4XfD7fiGdWy5YtU+WZFfD/PTjp6emq7sFZvHjxiD/jeD0iHAzmzMhCrW/wcXr27FncunVLsU0ke3D8jxE19uCoNWekHhy9Xq/qHpxIZkzMGzjBzEMSzlwlg6WkpAQ9qFFaWpoqB0CSJGN9f/iHfxjSfn/6p38q/z2SA1xFa1bdJUuWhFzj8uXLh10XTr1/9md/Jv9d+g5H+xmr4fIUwJwJRyj1+R+najxGvv3tb0elpvGUM729vdDpdCHXF+2ciUbGxPwm48HzXAyeh2S0bYiIRsOcIaK4TNXg/2imTqeTxy8oLi6W5yEZbpuxOHbsGHw+36jX63w+H27cuIHU1FTVnIH6Y33hUXt9gPprDLa+69evIyUlJag5iaKNOTM2rC88rC880ciYhJ6L6vjx4/D5fEhNTY13KURJ4caNG0hJScGSJUviXUrMMGeIYmcsGZPQDRwiIiJKTpyqgYiIiBIOGzhERESUcNjAISIiooTDBg4RERElHDZwiIiIKOGwgUNEREQJhw0cIiIiSjhs4BAREVHCYQOHiIiIEg4bOERERJRw2MAhIiKihMMGDhERESWcO+JdQKzZbDYAgMfjgSAIMJlMIW0T7/o8Hg8cDgfMZrPq6vPfVqvVqrI+i8UCQRAAAGazWVX1SdtIYlWf1+uF1WoFAFRWVgbcJp7HxnjBjIlNjf7bMmfGXl9S5IwviTidTt/WrVvl1+Xl5SFtEy3BfHZra6uvsbHR5/P5fB6Px/fHf/zHqqpP4vF4fKtXr5ZrjYVg6ysvL/d5PB6fz+fzrV69Oia1+XzB1efxeHzvvvuu/Np/+2hrbGz0/fM//7Pi8/3F89gYL5gx4WPOhIc58/+S6hKV3W6HRqORX2s0Gtjt9jFvE8/6PB6PvEyr1UKn08HhcKimPkljYyNWrlwZk7okwdTncDjkbRwOBxoaGlRVn1arhdVqlX+m/ttHm9lsRnZ29rDr43lsjBfMmNjUKGHOhFZfsuRMUjVwnE4n9Hq9/Fqv18Pr9Y55m3jWZzKZUFNTI7/2eDwwGo2qqQ8YOKDjcekimPpaW1vhcrkgiiIAoLq6WlX1AcDmzZtRWlqK0tJSVFVVxay+0cTz2BgvmDHhY85Evz4gOXImqRo4gXg8nohsEy0jfXZ1dTW2bdsWw2qGClSfKIryded4G1yf1+uFTqeD0WiE0WhEa2trTM9OBwv0/bW0tKChoQE6nQ7l5eWxL2oM4nlsjBfMmPAxZ8KTrDmTVA2cwd1ibrd7yAESzDbRMpbPttlsMJlMMb1xLZj6LBaLXF9LSwvsdnvMDuxg6hMEQbFMp9PJZ1lqqM9ms6GoqAhGoxG7du1CXl6eai4DxfPYGC+YMeFjzkS/vmTJmaRq4JhMJrS0tMivXS6X3MUpdYGNtI0a6gMGrlFqtVqYzWY4HI6YHTjB1FdZWQmz2Qyz2Szf/R6r7u1gf77+35coiqr6+Xo8Huh0OsU+/q/jQQ3HxnjBjIlNjcyZ8OpLlpxJ8fl8vohUN074P36m0+nks5Pi4mI0NDRAq9UOu40a6vN4PCgtLZW393q9OHPmjGrq02q1AAYCsq6uDllZWdiyZUvMzlCD/fl6PB54vV4IgqCqn69Wq4XFYpG/x1j+/tntdtTX16O7uxtlZWWqOzbGC2ZM9GtkzoRfXzLkTNI1cIiIiCjxJdUlKiIiIkoObOAQERFRwmEDh4iIiBIOGzhERESUcNjAISIiooTDBg5FlMPhQHV1NRYtWoTa2lpYLBZYLBZUV1dHbSwNu92O0tJSeYbawa+JKLEwZygYfEycIs7r9WLZsmU4evSoYryKjRs34uDBg/KySJLGdCgrKwv4mogSC3OGRsMeHIoJk8kEr9ermuHAiSjxMGfIHxs4FBPSPDGxnJWYiJILc4b83RHvAihxSfPZOBwOuN1uHDhwQDGUujRBniAIaGlpwZYtWwAMzNtSX1+P/Px8eDwerFy5Uh6+W6vVQhRFOJ1OeXsiSl7MGRoOe3Aoakwmk/ynqalJMZmbKIqoq6uTJ83Lzs6GxWKB1+tFRUUFqqqqYDab4XQ65Zv4Nm7cCEEQUFZWhu7ubnm+EiJKXswZGg57cCjqjEYj8vLyUFdXh5qaGgBAfX09dDqd4lp5S0sLtFotBEGQbxCsqqqS10s3E4qiCLfbHdMZjolI3ZgzNBgbOBQTGo0GH3/8sWJZbm4uTCaT/LqsrAwWiwUajUZe5v8kxM6dO6HX62E2m2M2azARjR/MGfLHS1QUE9nZ2fKZkMPhwKpVq9DU1KTYxm63w2w249SpU0OW2+12nDp1CpWVlRAEAd3d3fI6idfrVew3+DURJTbmDPnjODgUUQ6HA3v37oUoisjPz4fJZJKfaNiwYQPy8/PlcSPsdjsOHTqE/Px8AAPX0rVabcDlAPDjH/8YjzzyiPxZ9fX1WLVqFQRBwNatWwEA27ZtAwDFaz5RQZRYmDMUDDZwiIiIKOHwEhURERElHDZwiIiIKOGwgUNEREQJhw0cIiIiSjhs4BAREVHCYQOHiIiIEg4bOERERJRw2MAhIiKihMMGDhERESUcNnCIiIgo4bCBQ0RERAmHDRwiIiJKOP8Lpn/A0iZNySYAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig_save_dir = '../../thesis/graphics/'\n",
"\n",
"fig, ax = plt.subplots(1, 2, figsize=set_size(width, subplots=(1,2)))\n",
"results.plot_pr_curves(classes=classes, iou_thresh=0.5, backend='matplotlib', ax=ax[0], color='black', linewidth=1)\n",
"results.plot_pr_curves(classes=classes, iou_thresh=0.95, backend='matplotlib', ax=ax[1], color='black', linewidth=1)\n",
"# Set the labels for the legends manually\n",
"ax[0].get_lines()[0].set_linestyle('dashed')\n",
"ax[1].get_lines()[0].set_linestyle('dashed')\n",
"#ax[0].legend(['AP: 0.46, Healthy', 'AP: 0.48, Stressed'], frameon=False)\n",
"#ax[1].legend(['AP: 0.16, Healthy', 'AP: 0.16, Stressed'], frameon=False)\n",
"fig.tight_layout()\n",
"fig.savefig(fig_save_dir + 'APmodel-model-optimized-relabeled.pdf', format='pdf', bbox_inches='tight')"
]
},
{
"cell_type": "markdown",
"id": "dc5941e4",
"metadata": {},
"source": [
"The confusion matrix for the aggregate model seems to not show the cases where the object detection was successful but the class was wrong. For example, in the matrix below all classifications were correct or the detection failed. Under column _Stressed_ and row _Healthy_ not a single item is recorded. It seems that this evaluation metric does not have as much relevance when compared to the AP curves above or the mAP values."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "f1586bd5",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAhcAAAFbCAYAAABmq6zCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAoyklEQVR4nO3df3xU9Z3v8fdgH/xyfilWYmBSoSrmV/VW6cqgWxVXBpTasjWhrW7lItDd7gO8ive2t118iLrevcGuWL0rjVTd+rglYWu1KkyygN1VTryilhomEbQqmSEGlDpzMjXCY93cP2KOGTKByeQkJznzevYxD3O+Z07mE5M6n/l8P+f79XR3d3cLAADAJuOcDgAAALgLyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALAVyQUAALDV55wOoNefF13idAgYQ57as9HpEDCGtD35lNMhYAy58G/uGJHXGcr73r93vGRjJPYbNckFAACFZJzH43QIw4ZpEQAAYCsqFwAAOMAj91YuSC4AAHCAm6dFSC4AAHCAm5MLei4AAICtSC4AAICtmBYBAMABNHQCAABbubnnguQCAAAHuDm5oOcCAADYiuQCAADYimkRAAAcQEMnAACwlZt7LkguAABwgJuTC3ouAACArahcAADgAHouAACArZgWAQAAyBGVCwAAHEDlAgAAIEdULgAAcAANnQAAwFZunhYhuQAAwAFuTi7ouQAAALYiuQAAALZiWgQAAAd4XDwtQnIBAIAD3Dx1QHIBAIAD3Fy5cHPiBAAAHJBXcnHffffZHQcAAHCJvKZF2tratGXLFgUCAYXDYXm9XrvjAgDA1dy8zkVeycWGDRusr7ds2aK9e/cqEolozpw5tgUGAICbjWP570xNTU2SpM2bNysej2vhwoUKBAKqr69XZWWlSktLbQ0SAAC3cXNDZ17JxerVq7VgwQKtXLlSZWVl1nhZWZkaGxtJLgAAKGB5JRd33XWX5s+fbx2n02l5vV41NDQokUjYFhwAAG41zr2Fi/ySi/nz56uxsVGS1N3drW3btun+++/PSDgAAMDAmBY5zurVq1VZWWkdm6ZpW0AAAGBsyyu5WLhwYUaVorq62raAAAAoBG6+WyTnRbTS6bT18Hg8ampqUiKRUDqdVn19/XDGCACA64zzePJ+jHY5Vy6uvPJKVVZWqru7u9+5RCKhZcuW2RoYAAAYm3JOLh577LGM2077amlpsS0gAAAKwRgoQOQt5+RioMTiZOcAAEB/Y2F6I195r9DZu9R3Q0OD/H4/S38P0fd+/H098cDjSptpSdJZJcW67q++oYPvJjTt7OkZ5y66bLYkyRvwqvTCMu18erv2N+9zLHY462D7IT3/25dUXDxV7e2HdN2iq+Tznep0WHDQpDOKFLp8kfb/S22/c97pMzTBf5qOmh9KktKJdyRJ4/1BTSm7WMfMP2q8/3Qdfu0FfXLs6IjGDffIa1fUvgtlhcNhpkWG6LzKWfr2396YMfaT+gf0xAOP65knntbOp7frez/+vnVuXe096kya+rdnn9fBdxO6s/aekQ4Zo8iP7viJbvj2dbry8kt0xeWX6N6ajU6HBAcFZvaskDz5zOJ+57zTZyj4xXIdaXlNx8wPFfrqtda5Ly66UYdfe0FHWl5T8q29OmvOX4xYzIXKM4T/jXaDSi7q6+u1ePFi/exnP9Nf/uVfavHixbrpppvk9/uHK76CcNYXinXw3c8Stt7KRG+lYn/zPn3tr75hnV+7/EcZlYp0Kj1CkWK0Odh+KON4WvFUvfJqs0PRYDRIvd2qrg86sp4LffVavdf0r5KkY2ZSf3jmF5J6kg5JVqWi64MOnVF+0QhEC7ca1LRIVVWVqqqqMqZFMDRfvfYK/duzz2dUJryB7FvYn1c5S/ub9+nVF3ZbY5cvmqff/OLXwx4nRqfdrzTL78v8e/H7vdq3/23NOm+mQ1FhNBrvD+qUCZP0ybGjmnRGkY6ZH+qYmZQknTJ+YtZrJp1RNGCigqFz8/Lfea1zUVlZmXG8adOm4YzRtbx+b9aqw/7X92na2dOt4/MqZ0mSfEF/xtj3fvx9vfLvL+uZJ54e/mAxKqXTf8o6njKpZiHTpDPO0idHuxSYWapj5oeaUnaRNYXS9cF7mhA4vc9ziyRJp0yc5EishcLj8eT9GO1yrlx84xvfUElJCetc2OiKr83Lmhi819auf7rrQS264To9/5sdOusLPXOnncnPllnf37xP7QcO6ns//r5V/QB6pdMfOR0CRpnPTZykCYHTlU68rU+OHdWRlldVefMPtOf/3KljZlLtxr9qStmXlXwrpvH+0yRJn3zc5XDU7sbdIpI2bNjAOhc2uuiy2Xr+NzsGPP/Lh57QWSXFKv7CNL367z3TIO0HDmY8J22m9fwzO/WPW36qheddZfVooHB4vafK7Mz8vZtmWl7vZIciwmh11PxQ//Fxl9VX0fvP3qmPw3sMjfcHNd5/mtKJtyVJxz69owQYrJynRQZKLFpbW9lmPU9XfG2eFt1wnRbdcJ2mnT1dN6z6rjUFclZJsd5ra9f+5n0q/sI07fv9G0qbaV102Ww990aj9T16E47iL0xz5GeAs2ZfXJl1vHTWF0c4Eox2J0sUxvuDOmYm1fVBh8b7T9NHh9u5FRV5y2udi3g8rtraWpmmqe7ubnk8Hl199dV2x+Zqr76wW6++8Nnx7et/qKf/+dd6r61dkvRI42Oq/spipc20vnbj1/Xw3Q9J6pkaeaVPQ+d5X5qlzqTJOhcFalrx1Izjg+2HdP6smaxzAUnSKeMnWAnCMTOprvfbrbHx/qCOpv5oNWye980Van1igz45dlRTyi9W+0vbnQy9ILh4ViS/5KKurk4LFixQLBZTOBxWKpWyO66C4fV7tejGr0uSvvO3N+o3v3hK+5v36eG7H9JFfz5b/qBfzz+z07pDZH/zPu18ersW3XCdJOnir35FN199k0PRYzS4585b9dDDT6j0/HPU+sZbuufOW50OCQ7yTp8h3/SeytWZX75MHx0+qNTbrZKkdxu26Kw5f6Gu99s16fPF1q2okvTeS9vlnT5Tn5s4Sck/xKzFtTB8nNwVNRqNyu/3KxwOW8eSlEqlFAqFTjp+Mp7ubB2aJ9F7K+qWLVt0/fXX23Jr6p8XXTKk61FYntrDQlHIXduTTzkdAsaQC//mjhF5nf/+X67P+9r//bsteV9rmqZuuukmrVixQpFIxJqNWLdunSRp6dKlevTRRwccz0VelYtdu3appaVF3d3duu+++2SaJuteAAAwBmzbtk0LFiywjg3DkM/ns459Pp8Mw1A8Hs86nkv1Iq/kYs2aNdbX9fX1qqioyOfbAABQsIZyK+q8efNOeH7Hjux3I/a2M/ROd0hSW1ubgsGgdRwMBmWa5oDjuchrb5GWlhYtW7ZMra2tqqioGBMLegAAUOji8bhCodBJnzdQL2WuPZZ574q6adMmq9eisbFRpaWl+XwrAAAK0lA+mA9UmTiR2tpahUIhRaNRNTc3W4lGSUlJRkUimUxaCchA4yeTV3Jx/DfnbhEAAAZnpPcWWb58ufV1c3OzKisrVV5eLr/fr5qaGutcIpFQOBxWPB7POp6LvJKL119/XfF4XMlkUoZhZMzJAACA0cswDDU1NSkej6u8vFyhUEgLFy5UNBpVKpXSihUrJGnA8Vzk3dBZX1+vAwcOqKKiQlVVVfl8GwAACpZT/YrhcFhPPvlkxlgkEsn63IHGTybn5CKdTsvr/Wxr597t17OdAwAAJ5bXHRVjRM4/28aNG5VIJLI+Nm5kQSMAAAaDLdfVs+R3Q0ODuru7ZZqm/H6/dc40Td12223DEiAAABhbck4u7rrrLs2fP1+S1NDQYH3dewwAACANIrnom0wcX5Lpew4AAJzcSN+KOpJy7rloamrK6xwAAOiPngtJNTU1qqyslNSzkMauXbusc3v37tWvfvUr+6MDAMCl3Fy5yDm5CIVCKi8vlyTrn72SyaStQQEAgLEr5+Ri5cqVKisry3qO7dYBAECvnJOLgRILqf9eIwAA4MTGQu9EvvJa/hsAAAyNi3MLkgsAAJwwzsXZhZuXNgcAAA6gcgEAgAPcW7egcgEAAGxG5QIAAAe4ueeC5AIAAAe4OLdgWgQAANiLygUAAA5gES0AAGArNi4DAAC2cnPlgp4LAABgK5ILAABgK6ZFAABwgItnRUguAABwAotoAQAAW7k4t6DnAgAA2IvkAgAA2IppEQAAHODmaRGSCwAAHODmRbRILgAAcICbl/+m5wIAANiKygUAAA5gWgQAANjKxbkF0yIAAMBeVC4AAHAAlQsAAIAcUbkAAMABbFwGAABs5eLcYvQkF0/t2eh0CBhDHvz6PzgdAsaQb/3dZU6HAPTj5ltR6bkAAAC2IrkAAAC2GjXTIgAAFBIXz4qQXAAA4ASSCwAAYCsaOgEAAHJE5QIAAAeMc2/hgsoFAACwF5ULAAAc4OaeC5ILAAAc4OLcgmkRAABgLyoXAAA4wM2VC5ILAAAcQM8FAAAY86LRqEKhkPbu3StJqq6utsYlKZVKKRQKKRwOn3D8ZOi5AADAAR5P/o98mKapn/3sZyovL9eCBQu0du1aSVI8HpdhGIpEIqqurlZtbe0Jx3NB5QIAgDFm3rx5Jzy/Y8eOfmN+v19PPvmkpJ7EobcKYRiGfD6f9TyfzyfDMBSPx7OO51K9ILkAAMABTrVc1NXVadeuXdqwYYMkqa2tTcFg0DofDAZlmuaA47kguQAAwAGeIaz/na0ykavq6mqFQiGtX79e69aty/qcVCo1qPHj0XMBAIADRrrnQpJVeQiHw9q2bZsMw1BJSUnGc5LJpEKh0IDjuSC5AACgANTV1Wnjxo3WcSAQUCAQUDgcVnNzszWeSCQUDocHHM8F0yIAABSABQsWyDAMGYahXbt2qbq6WuXl5ZKkhQsXKhqNKpVKacWKFZKkUCiUdTwXJBcAADhgpBfR8vv9ikQiktSvAtE7fryBxk+G5AIAAAe4eIFOkgsAAJzg5uSChk4AAGArKhcAADjAzRuXUbkAAAC2onIBAIADXFy4ILkAAMAJJBefampqGvBcXV2d7r///qHGAwAAxrhBJRc1NTWaO3euuru71dLSounTp8vv98s0TXV2dg5XjAAAuM5QNi4b7QaVXGzYsMHatKShoUHz58+3zjU0NNgbGQAALubmaZFB3S3Sdze04/d0d/MtNQAAIHd5N3SmUindcccdqqioUFtbm4LBoI1hAQDgbm7+UJ73Ohc333yztR1rZWWlli1bZmdcAABgjMo7uWhpaVFdXZ2+9a1vafr06WptbbUzLgAAXM3jyf8x2uWdXDQ1NennP/+5ksmkysrKFI/H7YwLAABXc3NykXfPRd/mTqmnBwMAAOTGzT0XeScXr7/+uuLxuJLJpAzDoKETAIDBcG9ukf+0yJo1a+Tz+ZRKpVRSUkJDJwAAkDTEvUWqqqpUVVWlRCJhVzwAAGCMyzu5uOOOO7RkyRJt3LhRgUCA6gUAAIPA8t9ZRCIRTZ8+XS0tLWpsbGT5bwAABsHF/Zz5JxemaSoajeqSSy6R5O6uVyccbD+k53/7koqLp6q9/ZCuW3SVfL5TnQ4LDpoxu0KSNNE7WcVlMxXb/pI69r0rSQoWn6mZsyvUZaZ12rQz1bLzZSXbD0uSimadrZmfXltc+kU9c2+tjqY/cuRnwMh4K3FYP63frg23fjtjvONISi/+/k0VTQmo40hKkTkV8k6aaJ3fs79NHUdSKpoSkCRdeF7JiMZdaNz8vpl3Q2coFNKBAwd0++23q76+Xs3NzXbGVfB+dMdPdMO3r9OVl1+iKy6/RPfWbHQ6JDjsm/esVpeZVuvzL+vDxGF9857V1rmyK7+i157eqdbnX5bxxLOae+Mi69zM2RUynnhWxhPPqr31D7rxwR85ET5GyIu/f1OS9IfE4X7n7n38OX3zyot16QXn6tILztVP63dY5/bsb9OLe/YrMqdSRVMCenDLjn7XA7nKO7koKyuz7hgJh8O67bbb7IyroB1sP5RxPK14ql55leSt0P3LjzZYlQpJ+rhP9aHsqjlZrymadbbm/tV11nHLzpdVfP4MBYvPHLY44axLLzhX50zv//vtOJK5FlHRlID27G+zjh/cskM3LbrUOnf39xYPb6BwtbyTizvuuEOtra265ZZbVFtbq02bNtkZV0Hb/Uqz/D5vxpjf79W+/W87FBFGg3d277W+Lpv3Z3rt1599svzYTOvmx+5RsPhMzZhdobdf7klGO/a9q2fvrbWeN9E3WZLUZaZHKGqMFnv2t8k3eWLGmG/yRL2VOKyOIyl1fvSxvJN6jtNdH1tTIxg+bl6hM+/kom9D55133qnp06fbGVdBS6f/lHU8xRtCwSuadbbmfX+J3n65Wa89vdMa3/I/N0iSVj15v8qu/DO1Pv+yda7v1+VXXaK3X26m56IApbuOZh//6GO9lTgs3+SJn/Zj+BVt2mtNr2D4kFxkQUPnyEvzhlDwOva9qxcf/41OmzZVpVd8xRqfObtCOx76pZ699xFd9I15uuZ/9L8tfIJ3skqv+DMrEQGknqQj/dHH6jiS0oXnheSdNFGRORX6X48/53Ro7jfOk/9jlBtSQ2dbWxsNncPA6z1VZmdmlcI00/J6JzsUEUaTo+mP1LLz/+n6e2/RBO9kBYvPVHHZTL2ze69ee3qnHlh8i8rmXdKvr+Kq739LT6y6l6pFgfJOmqDOjz7OGOuZCpmgoikBnTppgnXnSO8/38rSFArkIu/kQpJisZgSiYQqKiq0YMECu2IqeLMvrsw6XjrriyMcCUaLGbMrdHvjZ70THx7s+Y/+adPO1FmzzlZ7y2f9OMn2w9r1z09b/RWSFL7hWu36xTNKth/WBO9kTSBRLTgD3VZ6TuhM+iscwrRIFsdvuc4S4PaZVjw14/hg+yGdP2sm61wUsC4zndHQedass9Vl/kkd+97Ve/veVXHZzIznTwp4rTtLSq/4it7b966VWJTPu4TqRYFId31WqTg+geg4ktK5oanyTpqooikBnRuaaj2/d62LbHedALlgy/VR6p47b9VDDz+h0vPPUesbb+meO291OiQ4qGPfu4ptb9KXr7tSkjTzK5WqvalnvYpk+2G9/fJehW+4Vh939iQNr/66p9kzWHymrr/3lozv1WX+KaMZFO6yZ3+bfrfvgCRpy/bdOrekSJdecK4k6YffvUaPPvOCzi0p0pttHfrBdxda1/3guwv12DMv6pzQVL0VP8StqCPAzb2Knu7u7u58Lly/fr1OO+00JZNJSVIwGBzS3iJ/7Ph93tei8Dz49X9wOgSMId/6u8ucDgFjyLnX/PWIvM5vb/tveV97+X3/aGMk9st7WmTlypXWluuhUIhNywAAgKQhTIvccsst2rBhg6qqquyMBwCAwjAGbinNV96Vi5tvvlnxeNw63rJliy0BAQBQCNx8t0jelYu1a9fK7/dLkrq7u3Xw4EFdf/31tgUGAICbubmhM+/kYt26dZoz57PNklpaWmwJCAAAjG15T4v0TSyknl1SAQAAhrSIVq+GhoaMYwAAcGJu7rnIO7nouyJnOBxmWgQAgMFwcXYx6J6L+vp6bd68Wel0Wps3b1Z3d7c8Ho+WLFkyHPEBAOBKniHt7jW6DTq5qKqqUlVVlZqamvr1XQAAANjW0AkAACANsnJRX1+veDyucDisOXPmqLGxUZs3b5bf79fdd98tr9c7XHECAOAqrHPxqUAgoDlz5igUCqmlpUXr169XY2OjJOm+++7TbbfdNixBAgDgOu7NLQaXXJimaW21XldXl7GvyPFbsAMAgIG5uXKRd8/Ftm3bNHfuXOs4GAzaEQ8AABjjBlW56O7u1qZNm/T666+roqJCpaWlknoW1EqlUsMSIAAAbsStqJ+qqqpSS0uLysrKrLtFWlpaZJqmtYkZAADIgYunRQa9zsXxe4iUlZWxrwgAALDkvSsqAADIn4sLF/k3dAIAAGRD5QIAAAd4xo186SIajSqVSikWiykSiSgcDlvjkpRKpRQKhU46fjIkFwAAOGGEc4tYLCZJqq6ulmmamjdvnnbv3q14PC7DMLRu3TpJ0tKlSxUOhwcczwXTIgAAOMDj8eT9yEcqlZJhGJIkv9+vQCCgWCwmwzDk8/ms5/l8PhmGMeB4LqhcAAAwxsybN++E53fs2NFvLBwOZ1QeUqmUysvLtXXr1oyFMIPBoEzTVFtbW9bxXFC5AACgwKxdu1Z33XXXgOcHWhgz1wUzqVwAAOCEIdyLmq0ykatoNKpwOKxIJCJJKikpyahIJJNJa7+wgcZPhsoFAAAO8IzL/5EvwzDk9/sViUQUi8UUj8cVDofV3NxsPSeRSFhTKNnGc0HlAgAAB4z0rqjxeFyrV6+2jk3T1L59+yRJCxcutG5TXbFihaSe3c6zjeeC5AIAgAIQCoW0e/furOd6p0hyHT8ZkgsAAJzg4vW/6bkAAAC2onIBAIADhtKYOdqRXAAA4ASmRQAAAHJD5QIAAAe4uHBBcgEAgCNcnF0wLQIAAGxF5QIAAAd4xlG5AAAAyAmVCwAAnODinguSCwAAHODi3ILkAgAAR9BzAQAAkBuSCwAAYCumRQAAcIDHxU0XJBcAADjBvbnF6EkujrzW5HQIGEO+vqzM6RAwhky5KOx0CEA/LKIFAACQo1FTuQAAoKC4uOeCygUAALAVlQsAABzA3SIAAMBeLk4umBYBAAC2onIBAIADPC7+eE9yAQCAE5gWAQAAyA2VCwAAHODmu0WoXAAAAFtRuQAAwAku3luE5AIAAAe4eVqE5AIAACe4OLmg5wIAANiK5AIAANiKaREAAJxAQycAALATDZ0AAMBe7s0t6LkAAAD2onIBAIADPC7uuaByAQAAbEXlAgAAJ9DQCQAA7OTmu0WYFgEAALaicgEAgBNcXLkguQAAwAkuvluE5AIAAAfQcwEAAJAjkgsAAGArpkUAAHCCi6dFSC4AAHCAm5f/JrkAAMAJHvd2Jrj3JwMAAI7Iq3KRSCTU0tIiSQqFQgqFQvJ6vbYGBgAAxqZBJRcNDQ3atWuXPB6PysvLFQgE1NzcrOeee06JREIrV65UaWnpcMUKAIB7jHBDp2maqqurkyQtX77cGo9Go5KkVCqlUCikcDh8wvFc5JxcPPLII5o7d67mz58/4HOamprU2Nioq6++OucAAAAoRCPd0GkYhpLJpILBoDUWj8dlGIbWrVsnSVq6dKnC4fCA47nKObmorq6Wz+c74XPmzJmjzs7OnF8cAICCNYTKxbx58054fseOHf3GIpGIUqmUTNO0xgzDyHhv9/l8MgxD8Xg863iuCUbODZ19X6SlpUXLli1Ta2urWlpa1NramvV5AABgAB5P/g+btLW1ZVQygsGgTNMccDxXeTV0NjU1adOmTWpqatKcOXPU2NhIrwUAACMkW2XCLqlUalDj2eSVXIRCobxfEAAAjA4lJSUZFYlkMmm9xw80nou8kovXX39d8XhcyWRShmFklE4AAMDJjYZdUcPhsGpqaqzjRCJhNXRmG89VXsnFmjVrVF9frwMHDqiyslILFizI59sAAFC4HLhbZNeuXers7FQoFFIkElEoFNLChQsVjUaVSqW0YsUKSRpwPFee7u7u7nyCbGhokMfjUXd3t7Zt26b7778/n29jeXPrw0O6fix6K35IP63frg23fSdjvOODpF78/ZsqmhJQx5GUIuFKeSdNtK6RpHNCU9XxQVLprqM6JzR1xGN3WtfBD5wOYcRNmlKkaZddo7ee2tRvXJK6jnRovC+oU8ZPVNeRDknSeF9Q3mkz9MnRLo33nabUO6061pkc6dAdN33RIqdDGDUOth/S8799ScXFU9XefkjXLbpKPt+pToc1qpxedMGIvM7bO3+e97Uzr/yvNkZiv7wqF6tXr1ZlZaV1PJgOUvR4cc9+FU0J6A+Jw/3O3fv4c1bC0fFBUj+t264f3nStJCna1KxoU7Mk6cLzSvSDm64ZuaDhmMCM83XMTGry58/qd+700i9rSumXJUmdibd1YMev+lxXqvdfb7KOp126UAdf3Dr8AWPU+tEdP9Fjtf8gqSfRuLdmo/5+3a0ORwW3ySu5WLhwYcZiWtXV1bYFVCguvfC8rOMdHyQzjovOCGrP/jbr+JzpZ2rz3/+1JFnVDLhf6p03BjzX9cF72vt4z9zofx47mnEuMLMsI7lAYTvYfijjeFrxVL3yarND0YCNy47j9/vV2tqqRCKhdDqt+vp6u+MqWHv2t8k3OTNp8E2eaE2HSD1JBYkF+vrPY0f7JRaS9MnRLp3z9WU90yPFM5Q++LYD0WG02P1Ks/y+zH2g/H6v9u3n78IR4zz5P0a5IU2L9LZrJBIJLVu2zNbAClW6q/8bhCSlP/rYOv/inv2SpDfjh7RgTqWKzgiOVHgYhU4ZP1GBGedLkiZ9vlh/bH3N6qs4sONXmrnwBp2/5G91pPU1pkQKXDr9p6zjKTM9wpHA7fJKLh577DGVlZVZx707pGL49CYdfZs7i6YE9OOHn9QjPx7djT0YXkfeeM2qWhwzk5qx8DvaV/eQJMk3bYY6Xt6p8f6gpl/W059DgoHjpdMfOR1CQRoNt6IOl7ymRcrKytTa2mqt0tk30cDQeCdNUOenVYpenR99LO+kCZKkjg8+W7Cs6Iyeu0mO79NAYZngO836+mjnh5rgP03jfUGN9wU16fPFSre/oz++8Tu9sflBBWaWarwv6FywcJTXe6rMzswqhWmm5fVOdigiuFVeyUVDQ4N++ctfWrehbtmyxe64CtaF55VkHT+nZKreih/Sj/7pV/3OeU+l/6JQTZpSpBnXfKff+H8c7dKkM4rU9X67NXasM6n39xg6ZTx/L4Vq9sWVWcdLZ31xhCOBpFGxt8hwyWtaRJK1DaskGjqHKN318WdTHcf1T3R8kNS5oanyTpqoojOkpddeap3bs69Ncy84l+bOAjNu/ARrGuRo54fqeHmndc43bYaSb7foP48dVdcHHTq99MsZd5qcMnGStQYGCs+04sw1cQ62H9L5s2ayzoVDPC6+WySv5OL4eSKW/x68PfsO6Hef3mK6ZftunRuaat2e+sPvXqNHn3lB54am6s34IWstC++kiTonNFX/smO3vJMm6L0jKWv9C7ibt3iGvNNnSJLOvHCuut5vV+qdN3qSiPff0+e/NEefHPtY4/2nqW3Hk5J6KhXpxDvWOUn6Y+trjv0MGB3uufNWPfTwEyo9/xy1vvGW7rmTNS4cMwbu+shXXit0PvLII/J4PAqFQorH45I05LtFCnGFTuSvEFfoRP5YoRODMVIrdL5r/N+8rz07/G0bI7FfXjWZm2++WT6fTy+++KJ8Ph+3oQIAAEvePRdVVVWqqqqSJKXTaXm93pNcAQAALGOgMTNfeScXjY2N1tdbt24d8sZlAAAUEjevc8HGZQAAOIG7RTKxcRkAABgIG5cBAOAAzzhP3o/Rjo3LAACArdi4DAAAJ9DQmen4jcrYuAwAgEFycUNnzj9Z31tPT4RNzAAAKGw5Vy5KS0u1fv16felLX9LVV1+dcS6RSMgwDMXjca1cudL2IAEAcBvWuZAUCoW0Zs0aNTQ0aO3atUqlUurs7JTH49H06dO1ZMkSa8VOAABwEi6eFhl0z8X8+fMz1rgAAADoK+/lvwEAQP7GwnoV+XJvTQYAADiCygUAAE6goRMAANjJQ0MnAACwlYsrF+5NmwAAgCNILgAAgK2YFgEAwAn0XAAAADu5eZ0LkgsAAJzg4sqFe38yAADgCCoXAAA4wM27olK5AAAAtqJyAQCAE1zcc0FyAQCAA5gWAQAAyBGVCwAAnDDOvZ/vSS4AAHCAm3dFde9PBgAAHEHlAgAAJ9DQCQAAkBsqFwAAOMDNPRckFwAAOMHF0yIkFwAAOMDNlQv3/mQAAMARJBcAAMBWTIsAAOAEF0+LkFwAAOAAN29cRnIBAIAT2FsEAACMddFoVJKUSqUUCoUUDoeH5XXcmzYBADCKeTzj8n7kIx6PyzAMRSIRVVdXq7a21uaf6DMkFwAAFADDMOTz+axjn88nwzCG5bWYFgEAwAFDaeicN2/eCc/v2LGj31hbW5uCwaB1HAwGZZpm3jGcyKhJLs5d+D2nQwAAYMScXnSB0yEolUoNy/cdNckFAADITbbKxMmUlJRkVCqSyaRCoZCdYVnouQAAoACEw2E1Nzdbx4lEYtjuFvF0d3d3D8t3BgAAo0rfW1EDgYAikciwvA7JBQAAsBXTIgAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFwAAwFYkFzaLxWJatWqVFi9ebO02F41GNXv2bNXU1Ax6k5hYLKarrrpqwOtisZhmz549bJvPAAAwWCQXNisvL9fcuXNVUVFhLasaiUQUCAS0ZMkS+f3+QX+/vmu/19XV9TtfUVEx9MDhiLq6OusRjUb7/X6dEo/HMxJkjE7xeNxacXG4maap2traEXktjH0kF2PMaHnzwdDFYjGZpqnq6mpVV1ervLxcbW1t1nknf9ehUEhz5sxx7PWRm82bNw/b8s3H8/v9CofDI5bMYGwjuRgmvZ8oeh/Hb2tbU1OjaDSq2tpaxWIxa8wwDNXU1Cgej/f7noZhyDRN1dXV9ftEaRiGotGoVq1aJemzqZje561atUo1NTXD8aMiT6lUKmMToVAopLlz51rHJJI4kWg0qsrKyhF9zfLycqpZyAnJxTAJhUKKRCLWIxAIWOfq6uoUDAYViUS0fPlyrV+/XpIUDAYVDoc1d+7crOXHcDgsv9+v6urqfjvZ+f1+RSIRVVZWKhqNKhKJaMGCBdb5uXPn6vbbbx+mnxb5CIfD6uzs1OzZs7V27VrFYjHr95otkTQMQ1dddZUMw9DSpUtlmma/JDUWiykajcowDK1du1aSso5J2RPc2tpaa4qmpaVlhP+NYDAMw8ioWvT2Xx3/QaNXbW1txgeek12T7e+jV7YPP0Bfn3M6gEIUi8Xk8/msN401a9ZY5+rq6mSappLJ5KC+Z9++jF7V1dXauHGjwuFwRnKD0ePRRx9VLBaTYRhavXq11qxZo0gkkpFI9uodCwQC2rBhg7Zt22YlqZK0dOlSlZWVqaSkJON3vnXr1n5jfRPc3murq6sVj8e1bt06SeIT6ih3/H8jevuvAoGAwuGwVT2NRCJWFaz397127VqFQqEBr0mlUv3+Ph599FHrdWKxWNb/5gC9qFw4oLy8XFLPm0XvJ9W6ujolk8mMqsTxnxZ6mabZb94zW/JQXl6uRCKhurq6fpUOOK/391teXq7ly5dr+/btOU2FlJeXy+/3KxaLKZlMyjAMGYahNWvWaOXKlYrFYlq8eLE2btwoSVnHsl1rGIb1tylJPp9vGH5q2KWzszPreLam8eOTgVAolJE8Hn9Ntr+PXnxQQS5ILmwWj8e1a9cu7d27N+NW1FQqpc2bN1sNfMFg0Co/S1JFRYU6OztlGIbVnxGPxxWLxRSPx603nerqatXV1Vn/Meh73jRNGYahrVu3WremLliwQLFYbNB3qWD4xePxftWB49/QsyWSvbIlqdu2bdO6deu0fft2BYNBxePxrGPZrg2HwxkNpQO9eWF0G+iDRt+pjL5/A9muyfb30SuVSlG1wEl5uru7u50OAsMnFosplUpRuRiFepPOQCCgVCqltrY2rVy50koEexPG8vJyhcNhxWIx3XTTTVqzZo01XVJbW2tNlYRCIW3dutVq8ovH41q+fLlqamr6jWW7try8XLW1tdYbR11dnXw+nx544IGR/leDHKxdu1bLly+3fl+xWEyrV6+27j5avXq1fD6f7r77bvn9fut32/vhpbq6+oTX9H6I6fv3IfX0YvT9OwWyIblwqb5zrX3n7QG4QywW0969e0f8/9+rVq0i4cRJMS3iUr2NWSywBbjT8euijATDMLRy5coRfU2MTVQuAGCMMk1T27ZtG5HqRW9P10gt2oWxjeQCAADYimkRAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgK5ILAABgq/8PY0SVokXiwVIAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import numpy as np\n",
"fig, ax = plt.subplots(1, 1, figsize=set_size(width, subplots=(1,1)))\n",
"# Manually set confusion matrix values obtained from results.plot_confusion_matrix()\n",
"matrix = np.array([[493, 0, 169], [0, 382, 106], [105, 158, 0]])\n",
"labels = ['Healthy', 'Stressed', '(none)']\n",
"sns.heatmap(matrix, annot=True, xticklabels=labels, yticklabels=labels, fmt=\".0f\", cmap=sns.cubehelix_palette(as_cmap=True, start=.3, hue=1, light=.9))\n",
"fig.tight_layout()\n",
"fig.savefig(fig_save_dir + 'CMmodel-relabeled.pdf', format='pdf', bbox_inches='tight')"
]
},
{
"cell_type": "markdown",
"id": "3871c398",
"metadata": {},
"source": [
"### View dataset in fiftyone \n",
"\n",
"We can launch a fiftyone session in a new tab to explore the dataset and the results."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "bfb39b5d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Connected to FiftyOne on port 5151 at localhost.\n",
"If you are not connecting to a remote session, you may need to start a new session and specify a port\n",
"Session launched. Run `session.show()` to open the App in a cell output.\n"
]
},
{
"data": {
"application/javascript": [
"window.open('http://localhost:5151/');"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"session = fo.launch_app(dataset, auto=False)\n",
"session.view = dataset.view()\n",
"session.open_tab()"
]
},
{
"cell_type": "code",
"execution_count": 104,
"id": "e1d00573",
"metadata": {},
"outputs": [],
"source": [
"session.close()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "53a67321",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}