Add extensive documentation
This commit is contained in:
parent
43cff6dff5
commit
30d0e6a360
@ -1,5 +1,39 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b1e57c8a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Table of contents\n",
|
||||
"1. [Introduction](#introduction)\n",
|
||||
"2. [Aggregate Model Evaluation](#modelevaluation)\n",
|
||||
" 1. [Loading the dataset](#modeload)\n",
|
||||
" 2. [Perform detections](#modeldetect)\n",
|
||||
" 3. [Save detections](#modeldetectionssave)\n",
|
||||
" 4. [Evaluate detections](#modeldetectionseval)\n",
|
||||
" 5. [Calculate results and plot them](#modelshowresults)\n",
|
||||
" 6. [View dataset in fiftyone](#modelfiftyonesession)\n",
|
||||
"3. [YOLO Evaluation](#yoloevaluation)\n",
|
||||
" 1. [Load OIDv6](#yololoadoid)\n",
|
||||
" 2. [Merge labels into one](#yolomergelabels)\n",
|
||||
" 3. [Load YOLOv5 dataset](#yololoadv5)\n",
|
||||
" 4. [Perform detections](#yoloperformdetections)\n",
|
||||
" 5. [Evaluate detections](#yolodetectionseval)\n",
|
||||
" 6. [Calculate results and plot them](#yoloshowresults)\n",
|
||||
" 7. [View dataset in fiftyone](#yolofiftyonesession)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "12921db4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Introduction <a name=\"introduction\"></a>\n",
|
||||
"\n",
|
||||
"This notebook loads the test dataset in YOLOv5 format from disk and evaluates the model's performance."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
@ -9,35 +43,47 @@
|
||||
"source": [
|
||||
"import fiftyone as fo\n",
|
||||
"from PIL import Image\n",
|
||||
"from detection import detect"
|
||||
"from detection import detect\n",
|
||||
"from detection import detect_yolo_only"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d46bd91d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Aggregate Model Evaluation <a name=\"modelevaluation\"></a>\n",
|
||||
"\n",
|
||||
"First, load the dataset from the directory containing the images and the labels in YOLOv5 format.\n",
|
||||
"\n",
|
||||
"### Loading the dataset <a name=\"modeload\"></a>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "32f0f8ec",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"name = \"dataset\"\n",
|
||||
"dataset_dir = \"dataset\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "6343aa55",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 100% |█████████████████| 640/640 [716.7ms elapsed, 0s remaining, 894.6 samples/s] \n"
|
||||
"ename": "ValueError",
|
||||
"evalue": "Dataset 'dataset' already exists; use `fiftyone.load_dataset()` to load an existing dataset",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[0;32m/tmp/ipykernel_395832/1016218199.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0;31m# Load the dataset, using tags to mark the samples in each split\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mdataset\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfo\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDataset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 9\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msplit\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msplits\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m dataset.add_dir(\n",
|
||||
"\u001b[0;32m~/.local/share/miniconda3/lib/python3.7/site-packages/fiftyone/core/singletons.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(cls, name, _create, *args, **kwargs)\u001b[0m\n\u001b[1;32m 31\u001b[0m ):\n\u001b[1;32m 32\u001b[0m \u001b[0minstance\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__new__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 33\u001b[0;31m \u001b[0minstance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_create\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_create\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 34\u001b[0m \u001b[0mname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minstance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m \u001b[0;31m# `__init__` may have changed `name`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 35\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_instances\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minstance\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[0;32m~/.local/share/miniconda3/lib/python3.7/site-packages/fiftyone/core/dataset.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, name, persistent, overwrite, _create, _virtual, **kwargs)\u001b[0m\n\u001b[1;32m 240\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0m_create\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 241\u001b[0m doc, sample_doc_cls, frame_doc_cls = _create_dataset(\n\u001b[0;32m--> 242\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpersistent\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpersistent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 243\u001b[0m )\n\u001b[1;32m 244\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[0;32m~/.local/share/miniconda3/lib/python3.7/site-packages/fiftyone/core/dataset.py\u001b[0m in \u001b[0;36m_create_dataset\u001b[0;34m(obj, name, persistent, _patches, _frames, _clips, _src_collection)\u001b[0m\n\u001b[1;32m 5881\u001b[0m \u001b[0;34m\"to load an existing dataset\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5882\u001b[0m )\n\u001b[0;32m-> 5883\u001b[0;31m \u001b[0;34m%\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5884\u001b[0m )\n\u001b[1;32m 5885\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[0;31mValueError\u001b[0m: Dataset 'dataset' already exists; use `fiftyone.load_dataset()` to load an existing dataset"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"name = \"dataset\"\n",
|
||||
"dataset_dir = \"dataset\"\n",
|
||||
"\n",
|
||||
"# The splits to load\n",
|
||||
"splits = [\"val\"]\n",
|
||||
"\n",
|
||||
@ -54,9 +100,38 @@
|
||||
"classes = dataset.default_classes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8bd95ca9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"If the dataset already exists because it had been saved under the same name before, load the dataset from fiftyone's folder."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"execution_count": 3,
|
||||
"id": "d9c393d6",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset = fo.load_dataset('dataset')\n",
|
||||
"classes = dataset.default_classes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5dd071ae",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Perform detections <a name=\"modeldetect\"></a>\n",
|
||||
"\n",
|
||||
"Now we can call the aggregate model to do detections on the images contained in the dataset. The actual detection happens at line 6 where `detect()` is called. This function currently does inference using the GPU via `onnxruntime-gpu`. All detections are saved to the `predictions` keyword of each sample. A sample is one image with potentially multiple detections."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "29827e3f",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
@ -64,7 +139,7 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 100% |█████████████████| 640/640 [8.8m elapsed, 0s remaining, 1.5 samples/s] \n"
|
||||
" 100% |█████████████████| 640/640 [6.2m elapsed, 0s remaining, 2.1 samples/s] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
@ -93,18 +168,39 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "06e1b4c0",
|
||||
"cell_type": "markdown",
|
||||
"id": "a9294bf2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset.persistent = True"
|
||||
"### Save detections <a name=\"modeldetectionssave\"></a>\n",
|
||||
"\n",
|
||||
"We have to make sure that the predictions for each sample are saved within the dataset. That is why we must call `dataset.save()` manually. The `persistent` flag is set again just to make sure."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"execution_count": 16,
|
||||
"id": "06e1b4c0",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"dataset.persistent = True\n",
|
||||
"dataset.save()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f75ef7aa",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Evaluate detections against ground truth <a name=\"modeldetectionseval\"></a>\n",
|
||||
"\n",
|
||||
"Having saved the predictions, we can now evaluate them by cross-checking with the ground truth labels. If we specify an `eval_key`, true positives, false positives and false negatives will be saved under that key."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "8ad67806",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
@ -113,9 +209,9 @@
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Evaluating detections...\n",
|
||||
" 100% |█████████████████| 640/640 [2.0s elapsed, 0s remaining, 319.5 samples/s] \n",
|
||||
" 100% |█████████████████| 640/640 [2.1s elapsed, 0s remaining, 307.0 samples/s] \n",
|
||||
"Performing IoU sweep...\n",
|
||||
" 100% |█████████████████| 640/640 [2.1s elapsed, 0s remaining, 297.3 samples/s] \n"
|
||||
" 100% |█████████████████| 640/640 [2.3s elapsed, 0s remaining, 280.1 samples/s] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
@ -128,9 +224,19 @@
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d1e5e4b9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Calculate results and plot them <a name=\"modelshowresults\"></a>\n",
|
||||
"\n",
|
||||
"Now we have the performance of the model saved in the `results` variable and can extract various metrics from that. Here we print a simple report of all classes and their precision and recall values as well as the mAP with the metric employed by COCO. Next, a confusion matrix is plotted for each class (in our case only one). Finally, we can show the precision vs. recall curve for a specified threshold value."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"execution_count": 5,
|
||||
"id": "b180420b",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
@ -147,7 +253,7 @@
|
||||
" macro avg 0.77 0.76 0.76 1150\n",
|
||||
"weighted avg 0.77 0.76 0.77 1150\n",
|
||||
"\n",
|
||||
"0.6225828327927432\n"
|
||||
"0.6225848121901868\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
@ -167,7 +273,7 @@
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "0e0a5d23d3f148419bd62ecf4a07dce9",
|
||||
"model_id": "2faa9ee3a0ce499986d12b6f9c4e1ef8",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
@ -176,7 +282,7 @@
|
||||
" 'data': [{'mode': 'markers',\n",
|
||||
" 'opacity': 0.1,\n",
|
||||
" 'type': 'scatter',\n",
|
||||
" 'uid': 'c432eebe-8bf5-4b15-834c-abdc3af7c18b',\n",
|
||||
" 'uid': 'a30f32cd-dedd-40c9-b4e8-755f4a701673',\n",
|
||||
" 'x': array([0, 1, 2, 0, 1, 2, 0, 1, 2]),\n",
|
||||
" 'y': array([0, 0, 0, 1, 1, 1, 2, 2, 2])},\n",
|
||||
" {'colorscale': [[0.0, 'rgb(255,245,235)'], [0.125,\n",
|
||||
@ -187,7 +293,7 @@
|
||||
" 'hoverinfo': 'skip',\n",
|
||||
" 'showscale': False,\n",
|
||||
" 'type': 'heatmap',\n",
|
||||
" 'uid': '01498ed3-dbe3-4c9e-baf9-49d814522d20',\n",
|
||||
" 'uid': '352f91ad-547a-412a-ad31-a9cef7ad7c16',\n",
|
||||
" 'z': array([[105, 158, 0],\n",
|
||||
" [ 0, 382, 106],\n",
|
||||
" [493, 0, 169]]),\n",
|
||||
@ -202,7 +308,7 @@
|
||||
" 'hovertemplate': '<b>count: %{z}</b><br>truth: %{y}<br>predicted: %{x}<extra></extra>',\n",
|
||||
" 'opacity': 0.25,\n",
|
||||
" 'type': 'heatmap',\n",
|
||||
" 'uid': '10618ef9-35c5-451a-b915-32b70d7bfb5e',\n",
|
||||
" 'uid': 'd8669a84-9bf1-4f67-a3aa-32ce811302b7',\n",
|
||||
" 'z': array([[105, 158, 0],\n",
|
||||
" [ 0, 382, 106],\n",
|
||||
" [493, 0, 169]]),\n",
|
||||
@ -247,25 +353,24 @@
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "915261c4504943ec9b07813fbe75b8c9",
|
||||
"model_id": "09acc41587fe4bed8572f205b7456502",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"FigureWidget({\n",
|
||||
" 'data': [{'customdata': array([99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 98.8,\n",
|
||||
" 98. , 98. , 97.7, 97. , 97. , 97. , 96.9, 96.4, 96. , 95.7, 94.8, 94.3,\n",
|
||||
" 93.8, 92.8, 92.2, 91.5, 90.9, 90.7, 89.3, 88.3, 87.8, 87.1, 86.8, 86.3,\n",
|
||||
" 85.8, 85.1, 84.6, 84.1, 83.2, 82. , 81.7, 80.8, 79.7, 79.4, 78.6, 77.8,\n",
|
||||
" 77.4, 76.3, 75.5, 74.6, 73.6, 72.3, 71. , 69.9, 68.6, 67.5, 66.4, 65.4,\n",
|
||||
" 64.3, 63.2, 62.2, 61. , 60.2, 59.1, 58.1, 56.9, 50.4, 49. , 47.3, 46.4,\n",
|
||||
" 40.8, 25.3, 10. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ]),\n",
|
||||
" 'data': [{'customdata': array([99., 99., 99., 99., 99., 99., 99., 99., 99., 99., 99., 98., 98., 98.,\n",
|
||||
" 97., 97., 97., 97., 96., 96., 96., 95., 94., 93., 93., 92., 91., 91.,\n",
|
||||
" 90., 89., 88., 87., 87., 86., 86., 85., 85., 84., 82., 82., 81., 80.,\n",
|
||||
" 80., 79., 78., 78., 77., 76., 75., 74., 72., 70., 70., 68., 67., 66.,\n",
|
||||
" 65., 64., 62., 62., 61., 59., 58., 56., 53., 52., 51., 50., 0., 0.,\n",
|
||||
" 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
|
||||
" 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
|
||||
" 0., 0., 0.]),\n",
|
||||
" 'hovertemplate': ('<b>class: %{text}</b><br>recal' ... 'customdata:.3f}<extra></extra>'),\n",
|
||||
" 'line': {'color': '#3366CC'},\n",
|
||||
" 'mode': 'lines',\n",
|
||||
" 'name': 'Healthy (AP = 0.631)',\n",
|
||||
" 'name': 'Healthy (AP = 0.562)',\n",
|
||||
" 'text': array(['Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy',\n",
|
||||
" 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy',\n",
|
||||
" 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy',\n",
|
||||
@ -284,7 +389,7 @@
|
||||
" 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy',\n",
|
||||
" 'Healthy', 'Healthy', 'Healthy', 'Healthy', 'Healthy'], dtype='<U7'),\n",
|
||||
" 'type': 'scatter',\n",
|
||||
" 'uid': '7d365acd-11ac-471f-95c1-0c45b4af1e64',\n",
|
||||
" 'uid': '57f76d88-109a-4e5d-a9f4-543291c9e521',\n",
|
||||
" 'x': array([0. , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11,\n",
|
||||
" 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 , 0.21, 0.22, 0.23,\n",
|
||||
" 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32, 0.33, 0.34, 0.35,\n",
|
||||
@ -295,35 +400,34 @@
|
||||
" 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.9 , 0.91, 0.92, 0.93, 0.94, 0.95,\n",
|
||||
" 0.96, 0.97, 0.98, 0.99, 1. ]),\n",
|
||||
" 'y': array([1. , 1. , 1. , 1. , 1. , 1. ,\n",
|
||||
" 1. , 1. , 1. , 1. , 1. , 0.97197452,\n",
|
||||
" 0.88968238, 0.88968238, 0.88968238, 0.88968238, 0.88968238, 0.88968238,\n",
|
||||
" 0.88968238, 0.88962774, 0.88906313, 0.88406656, 0.87159671, 0.86864529,\n",
|
||||
" 0.86658584, 0.86482334, 0.86414826, 0.86302011, 0.86087578, 0.8598177 ,\n",
|
||||
" 0.85442276, 0.84805184, 0.84805184, 0.84805184, 0.84805184, 0.84805184,\n",
|
||||
" 0.84805184, 0.84805184, 0.84805184, 0.84805184, 0.84805184, 0.84805184,\n",
|
||||
" 0.84805184, 0.84805184, 0.84805184, 0.84805184, 0.84805184, 0.84727236,\n",
|
||||
" 0.84615682, 0.84387252, 0.84024854, 0.83876528, 0.8378955 , 0.83308277,\n",
|
||||
" 0.82724578, 0.82424648, 0.82113648, 0.8193439 , 0.81764747, 0.81764747,\n",
|
||||
" 0.81764747, 0.81716972, 0.81710833, 0.8164769 , 0.81554506, 0.81418573,\n",
|
||||
" 0.81247877, 0.81064826, 0.73478697, 0.73331137, 0.73195383, 0.73122457,\n",
|
||||
" 0.65219773, 0.41090604, 0.16526846, 0. , 0. , 0. ,\n",
|
||||
" 1. , 1. , 1. , 1. , 1. , 0.85350318,\n",
|
||||
" 0.85350318, 0.85350318, 0.85350318, 0.85350318, 0.85350318, 0.85350318,\n",
|
||||
" 0.85350318, 0.85350318, 0.85350318, 0.83529412, 0.82162162, 0.82142857,\n",
|
||||
" 0.82142857, 0.8195122 , 0.81696429, 0.81696429, 0.81304348, 0.80526316,\n",
|
||||
" 0.80526316, 0.80526316, 0.80526316, 0.80526316, 0.80526316, 0.80526316,\n",
|
||||
" 0.80526316, 0.80526316, 0.80526316, 0.80526316, 0.80526316, 0.80526316,\n",
|
||||
" 0.80526316, 0.80526316, 0.80526316, 0.80526316, 0.80526316, 0.79746835,\n",
|
||||
" 0.79301746, 0.79075426, 0.78117647, 0.77181208, 0.77181208, 0.76685934,\n",
|
||||
" 0.76685934, 0.76685934, 0.76685934, 0.76685934, 0.76685934, 0.76685934,\n",
|
||||
" 0.76685934, 0.76208178, 0.76146789, 0.75631769, 0.75167785, 0.75167785,\n",
|
||||
" 0.75167785, 0.75167785, 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ])},\n",
|
||||
" {'customdata': array([99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. , 99. ,\n",
|
||||
" 98. , 98. , 98. , 98. , 98. , 98. , 97.7, 97. , 97. , 96.7, 96.5, 96. ,\n",
|
||||
" 95.7, 95.7, 95.4, 94.7, 94.6, 93.6, 93.3, 92.4, 91.6, 91.1, 90.5, 90. ,\n",
|
||||
" 89.3, 88.6, 88.2, 87.4, 87.1, 86.2, 85.4, 84.9, 84. , 83.4, 82. , 81.5,\n",
|
||||
" 80.5, 79.4, 78.5, 77.5, 76.4, 75.6, 74.6, 73.7, 72.9, 71.8, 70.9, 70.2,\n",
|
||||
" 69.5, 68.5, 67.1, 66.6, 65.5, 63.8, 62.3, 61.1, 60.1, 53.5, 52.6, 51.2,\n",
|
||||
" 45. , 43.9, 38.2, 37.4, 31.3, 25.5, 10. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ]),\n",
|
||||
" {'customdata': array([99., 99., 99., 99., 99., 99., 99., 99., 99., 99., 99., 99., 98., 98.,\n",
|
||||
" 98., 98., 98., 98., 97., 97., 97., 96., 96., 96., 95., 95., 95., 94.,\n",
|
||||
" 93., 92., 92., 91., 90., 89., 89., 88., 88., 87., 86., 86., 85., 84.,\n",
|
||||
" 83., 82., 81., 80., 79., 77., 76., 75., 75., 74., 73., 72., 71., 70.,\n",
|
||||
" 69., 69., 68., 66., 64., 63., 61., 60., 58., 56., 54., 53., 51., 0.,\n",
|
||||
" 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
|
||||
" 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
|
||||
" 0., 0., 0.]),\n",
|
||||
" 'hovertemplate': ('<b>class: %{text}</b><br>recal' ... 'customdata:.3f}<extra></extra>'),\n",
|
||||
" 'line': {'color': '#DC3912'},\n",
|
||||
" 'mode': 'lines',\n",
|
||||
" 'name': 'Stressed (AP = 0.614)',\n",
|
||||
" 'name': 'Stressed (AP = 0.532)',\n",
|
||||
" 'text': array(['Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed',\n",
|
||||
" 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed',\n",
|
||||
" 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed',\n",
|
||||
@ -342,7 +446,7 @@
|
||||
" 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed',\n",
|
||||
" 'Stressed', 'Stressed', 'Stressed', 'Stressed', 'Stressed'], dtype='<U8'),\n",
|
||||
" 'type': 'scatter',\n",
|
||||
" 'uid': 'c00a4fd2-8937-4000-bfbe-6429b1528d2d',\n",
|
||||
" 'uid': '057f171b-c861-47e6-9fab-3f379448b690',\n",
|
||||
" 'x': array([0. , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11,\n",
|
||||
" 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 , 0.21, 0.22, 0.23,\n",
|
||||
" 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32, 0.33, 0.34, 0.35,\n",
|
||||
@ -354,18 +458,18 @@
|
||||
" 0.96, 0.97, 0.98, 0.99, 1. ]),\n",
|
||||
" 'y': array([1. , 1. , 1. , 1. , 1. , 1. ,\n",
|
||||
" 1. , 1. , 1. , 1. , 1. , 1. ,\n",
|
||||
" 0.91667218, 0.91667218, 0.91667218, 0.91667218, 0.91667218, 0.91667218,\n",
|
||||
" 0.89886256, 0.8706589 , 0.8706589 , 0.86176273, 0.8566061 , 0.8426063 ,\n",
|
||||
" 0.8414663 , 0.8414663 , 0.8414663 , 0.83779842, 0.83730636, 0.82590162,\n",
|
||||
" 0.82226179, 0.81571494, 0.80136981, 0.79562115, 0.79030661, 0.78516763,\n",
|
||||
" 0.77779393, 0.77779393, 0.7766316 , 0.77013304, 0.76981741, 0.7687736 ,\n",
|
||||
" 0.76706332, 0.76615004, 0.76345747, 0.76120645, 0.75633043, 0.75442522,\n",
|
||||
" 0.75216493, 0.74937749, 0.74462707, 0.73753607, 0.73544297, 0.73534102,\n",
|
||||
" 0.73534102, 0.73491603, 0.73330167, 0.73209432, 0.73192653, 0.73073596,\n",
|
||||
" 0.7298439 , 0.72819654, 0.72615935, 0.72491458, 0.72298246, 0.71613728,\n",
|
||||
" 0.71363144, 0.71260932, 0.71076601, 0.64219247, 0.63885976, 0.636554 ,\n",
|
||||
" 0.57072089, 0.56905625, 0.49989485, 0.4972456 , 0.42550943, 0.35394558,\n",
|
||||
" 0.14235075, 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0.89361702, 0.89361702, 0.89361702, 0.89361702, 0.89361702, 0.89361702,\n",
|
||||
" 0.83333333, 0.83333333, 0.83333333, 0.8 , 0.8 , 0.8 ,\n",
|
||||
" 0.79503106, 0.79503106, 0.79503106, 0.77714286, 0.77222222, 0.75510204,\n",
|
||||
" 0.75510204, 0.74519231, 0.72850679, 0.72222222, 0.72222222, 0.71641791,\n",
|
||||
" 0.71641791, 0.71641791, 0.71641791, 0.71641791, 0.71326165, 0.70877193,\n",
|
||||
" 0.70748299, 0.70627063, 0.70418006, 0.69811321, 0.68484848, 0.68144044,\n",
|
||||
" 0.68144044, 0.68144044, 0.68144044, 0.67938931, 0.67938931, 0.67938931,\n",
|
||||
" 0.67938931, 0.6778043 , 0.6778043 , 0.6778043 , 0.6778043 , 0.66590389,\n",
|
||||
" 0.66444444, 0.66444444, 0.65450644, 0.64876033, 0.64876033, 0.64788732,\n",
|
||||
" 0.63547758, 0.63249516, 0.625 , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ])}],\n",
|
||||
@ -395,10 +499,20 @@
|
||||
"matrix = results.plot_confusion_matrix(classes=classes)\n",
|
||||
"matrix.show()\n",
|
||||
"\n",
|
||||
"pr_curves = results.plot_pr_curves(classes=classes)\n",
|
||||
"pr_curves = results.plot_pr_curves(classes=classes, iou_thresh=0.95)\n",
|
||||
"pr_curves.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9997cd3f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### View dataset in fiftyone <a name=\"modelfiftyonesession\"></a>\n",
|
||||
"\n",
|
||||
"We can launch a fiftyone session in a new tab to explore the dataset and the results."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
@ -434,19 +548,18 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"execution_count": 6,
|
||||
"id": "6eed9a86",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def export_dataset(dataset, export_dir):\n",
|
||||
"# Write final dataset to disk\n",
|
||||
"def export_dataset(dataset, export_dir, classes=[\"Healthy\", \"Stressed\"]):\n",
|
||||
" label_field = \"ground_truth\"\n",
|
||||
"\n",
|
||||
" # The splits to export\n",
|
||||
" splits = [\"val\"]\n",
|
||||
"\n",
|
||||
" classes = [\"Healthy\", \"Stressed\"]\n",
|
||||
"\n",
|
||||
" # Export the splits\n",
|
||||
" for split in splits:\n",
|
||||
" split_view = dataset.match_tags(split)\n",
|
||||
@ -459,18 +572,542 @@
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "64e89754",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## YOLO Model Evaluation <a name=\"yoloevaluation\"></a>\n",
|
||||
"\n",
|
||||
"In this section we look at the object detection model in detail by evaluating it separately from the classification model. The object detection model was trained on the Open Images Dataset v6 on the two classes _Plant_ and _Houseplant_ which come with the dataset. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5782d392",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load OIDv6 <a name=\"yololoadoid\"></a>\n",
|
||||
"\n",
|
||||
"Since we are only interested in evaluating the model, we only load the _test_ split of the dataset. The only classes of interest to us are _Plant_ and _Houseplant_ and we do not want to load keypoint detections or segmentation masks, which is why we specify the `label_types` parameter. There are 12106 images in the test split."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"execution_count": 7,
|
||||
"id": "19c5b271",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Downloading split 'test' to '/home/zenon/fiftyone/open-images-v6/test' if necessary\n",
|
||||
"Necessary images already downloaded\n",
|
||||
"Existing download of split 'test' is sufficient\n",
|
||||
"Loading 'open-images-v6' split 'test'\n",
|
||||
" 100% |█████████████| 12106/12106 [1.0m elapsed, 0s remaining, 161.8 samples/s] \n",
|
||||
"Dataset 'open-images-v6-test' created\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import fiftyone as fo\n",
|
||||
"import fiftyone.zoo as foz\n",
|
||||
"oid = foz.load_zoo_dataset(\n",
|
||||
" \"open-images-v6\",\n",
|
||||
" split=\"test\",\n",
|
||||
" classes=[\"Plant\", \"Houseplant\"],\n",
|
||||
" label_types=[\"detections\"],\n",
|
||||
" shuffle=True,\n",
|
||||
")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9bfbf8a4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Export dataset for conversion <a name=\"yoloexportoid\"></a>\n",
|
||||
"\n",
|
||||
"Unfortunately, the OID dataset does not adhere to the YOLOv5 label format understood by the object detection model. That is why we export the model as a YOLOv5Dataset using fiftyone's converter. The target directory will contain the proper folder structure as well as a `.yaml` file pointing to the images and labels. Take note that the exported files require around 4.2G of space."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "ebdde519",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
"source": [
|
||||
"# The directory to which to write the exported dataset\n",
|
||||
"import os\n",
|
||||
"\n",
|
||||
"export_dir = \"/home/zenon/testdir\"\n",
|
||||
"\n",
|
||||
"# Only export if export_dir doesn't exist already\n",
|
||||
"if not os.path.isdir(export_dir):\n",
|
||||
" # The name of the sample field containing the label that you wish to export\n",
|
||||
" # Used when exporting labeled datasets (e.g., classification or detection)\n",
|
||||
" label_field = \"detections\" # for example\n",
|
||||
"\n",
|
||||
" # The type of dataset to export\n",
|
||||
" # Any subclass of `fiftyone.types.Dataset` is supported\n",
|
||||
" dataset_type = fo.types.YOLOv5Dataset # for example\n",
|
||||
"\n",
|
||||
" # Export the dataset\n",
|
||||
" oid.export(\n",
|
||||
" export_dir=export_dir,\n",
|
||||
" dataset_type=dataset_type,\n",
|
||||
" label_field=label_field,\n",
|
||||
" classes=['Plant', 'Houseplant']\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "065e0dca",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Merge labels into one <a name=\"yolomergelabels\"></a>\n",
|
||||
"\n",
|
||||
"The label files contain a 0 at the beginning of each line if the ground truth specifies a plant and a 1 if it specifies a houseplant. We do not care about the distinction between the two and only want to detect plants in general. That means we have to change all 1s at the beginning of each line in each label file into 0s. The YOLOv5 format requires that the labels start at 0 and not at 1, which is why 1s are changed to 0s and not vice-versa. To accomplish this task, we use a simple bash script in the labels directory:\n",
|
||||
"```bash\n",
|
||||
"for file in `ls test`\n",
|
||||
"do\n",
|
||||
" sed -i 's/^./0/g' test/$file\n",
|
||||
"done\n",
|
||||
"```\n",
|
||||
"This script calls sed to change the first character in each file to a 0. It performs the conversion in place (`-i` flag). For this script to work, the `val` directories inside `images` and `labels` must be renamed to `test` and the path to the directory changed in the `data.yaml` file. I believe `val` is the wrong name for a test dataset and it should be named accordingly."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "030e4550",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load YOLOv5 dataset <a name=\"yololoadv5\"></a>\n",
|
||||
"\n",
|
||||
"Now that the labels are in the correct format and we only have one class to deal with, we can import the dataset into the variable `yolo`."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"id": "002ae8fa",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 100% |█████████████| 12106/12106 [14.9s elapsed, 0s remaining, 804.0 samples/s] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"yolo_dataset_dir = export_dir\n",
|
||||
"\n",
|
||||
"# The type of the dataset being imported\n",
|
||||
"dataset_type = fo.types.YOLOv5Dataset\n",
|
||||
"\n",
|
||||
"# Import the dataset\n",
|
||||
"yolo = fo.Dataset.from_dir(\n",
|
||||
" dataset_dir=yolo_dataset_dir,\n",
|
||||
" dataset_type=dataset_type,\n",
|
||||
")\n",
|
||||
"yolo.name = 'yolo'\n",
|
||||
"yolo.persistent = True"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a9ea9ba1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In case the yolo dataset already exists because it had been saved earlier, we can simply load the dataset from fiftyone's database."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 28,
|
||||
"id": "42b72a2d",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"['dataset', 'yolo']"
|
||||
]
|
||||
},
|
||||
"execution_count": 28,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"yolo = fo.load_dataset('yolo')\n",
|
||||
"fo.list_datasets()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ebffbda",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Perform detections <a name=\"yoloperformdetections\"></a>\n",
|
||||
"\n",
|
||||
"We can proceed as before by calling the model and saving the detections to the `predictions` field of each sample. Note that line 7 does not call `detect()` but `detect_yolo_only()`. The detections on all 12106 images take around 1.2h on a GTX 750Ti."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "030e9c7c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 100% |█████████████| 12106/12106 [1.2h elapsed, 0s remaining, 2.7 samples/s] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Do detections with model and save bounding boxes\n",
|
||||
"yolo_view = yolo.view()\n",
|
||||
"with fo.ProgressBar() as pb:\n",
|
||||
" for sample in pb(yolo_view):\n",
|
||||
" image = Image.open(sample.filepath)\n",
|
||||
" w, h = image.size\n",
|
||||
" pred = detect_yolo_only(sample.filepath, '../weights/yolo.onnx')\n",
|
||||
"\n",
|
||||
" detections = []\n",
|
||||
" for _, row in pred.iterrows():\n",
|
||||
" xmin, xmax = int(row['xmin']), int(row['xmax'])\n",
|
||||
" ymin, ymax = int(row['ymin']), int(row['ymax'])\n",
|
||||
" rel_box = [\n",
|
||||
" xmin / w, ymin / h, (xmax - xmin) / w, (ymax - ymin) / h\n",
|
||||
" ]\n",
|
||||
" detections.append(\n",
|
||||
" fo.Detection(label='Plant',\n",
|
||||
" bounding_box=rel_box,\n",
|
||||
" confidence=float(row['box_conf'])))\n",
|
||||
"\n",
|
||||
" sample[\"predictions\"] = fo.Detections(detections=detections)\n",
|
||||
" sample.save()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d0789cc2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Evaluate detections against ground truth <a name=\"yolodetectionseval\"></a>\n",
|
||||
"\n",
|
||||
"Having saved the predictions, we can now evaluate them by cross-checking with the ground truth labels. If we specify an `eval_key`, true positives, false positives and false negatives will be saved under that key."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 29,
|
||||
"id": "b6b35ed4",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Evaluating detections...\n",
|
||||
" 100% |█████████████| 12106/12106 [42.8s elapsed, 0s remaining, 294.8 samples/s] \n",
|
||||
"Performing IoU sweep...\n",
|
||||
" 100% |█████████████| 12106/12106 [43.5s elapsed, 0s remaining, 300.3 samples/s] \n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"results = yolo.evaluate_detections(\"predictions\", gt_field=\"ground_truth\", eval_key=\"eval\", compute_mAP=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "124d92a4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Calculate results and plot them <a name=\"yoloshowresults\"></a>\n",
|
||||
"\n",
|
||||
"Now we have the performance of the model saved in the `results` variable and can extract various metrics from that. Here we print a simple report of all classes and their precision and recall values as well as the mAP with the metric employed by COCO. Next, a confusion matrix is plotted for each class (in our case only one). Finally, we can show the precision vs. recall curve for a specified threshold value."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 30,
|
||||
"id": "0c3c446e",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" precision recall f1-score support\n",
|
||||
"\n",
|
||||
" Plant 0.52 0.54 0.53 22535\n",
|
||||
"\n",
|
||||
" micro avg 0.52 0.54 0.53 22535\n",
|
||||
" macro avg 0.52 0.54 0.53 22535\n",
|
||||
"weighted avg 0.52 0.54 0.53 22535\n",
|
||||
"\n",
|
||||
"0.3623395579880134\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": []
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": []
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "06770c63d5d34d6cb9c7c09821916697",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"FigureWidget({\n",
|
||||
" 'data': [{'mode': 'markers',\n",
|
||||
" 'opacity': 0.1,\n",
|
||||
" 'type': 'scatter',\n",
|
||||
" 'uid': '245dadac-2cdf-4379-b02c-25054e792f00',\n",
|
||||
" 'x': array([0, 1, 0, 1]),\n",
|
||||
" 'y': array([0, 0, 1, 1])},\n",
|
||||
" {'colorscale': [[0.0, 'rgb(255,245,235)'], [0.125,\n",
|
||||
" 'rgb(254,230,206)'], [0.25, 'rgb(253,208,162)'],\n",
|
||||
" [0.375, 'rgb(253,174,107)'], [0.5, 'rgb(253,141,60)'],\n",
|
||||
" [0.625, 'rgb(241,105,19)'], [0.75, 'rgb(217,72,1)'],\n",
|
||||
" [0.875, 'rgb(166,54,3)'], [1.0, 'rgb(127,39,4)']],\n",
|
||||
" 'hoverinfo': 'skip',\n",
|
||||
" 'showscale': False,\n",
|
||||
" 'type': 'heatmap',\n",
|
||||
" 'uid': 'feaf31e2-40ce-4a97-953b-36efe673c3d7',\n",
|
||||
" 'z': array([[11496, 0],\n",
|
||||
" [12237, 10298]]),\n",
|
||||
" 'zmax': 12237,\n",
|
||||
" 'zmin': 0},\n",
|
||||
" {'colorbar': {'len': 1, 'lenmode': 'fraction'},\n",
|
||||
" 'colorscale': [[0.0, 'rgb(255,245,235)'], [0.125,\n",
|
||||
" 'rgb(254,230,206)'], [0.25, 'rgb(253,208,162)'],\n",
|
||||
" [0.375, 'rgb(253,174,107)'], [0.5, 'rgb(253,141,60)'],\n",
|
||||
" [0.625, 'rgb(241,105,19)'], [0.75, 'rgb(217,72,1)'],\n",
|
||||
" [0.875, 'rgb(166,54,3)'], [1.0, 'rgb(127,39,4)']],\n",
|
||||
" 'hovertemplate': '<b>count: %{z}</b><br>truth: %{y}<br>predicted: %{x}<extra></extra>',\n",
|
||||
" 'opacity': 0.25,\n",
|
||||
" 'type': 'heatmap',\n",
|
||||
" 'uid': '7aa2a8da-7437-4ccc-a929-903b3de13a4b',\n",
|
||||
" 'z': array([[11496, 0],\n",
|
||||
" [12237, 10298]]),\n",
|
||||
" 'zmax': 12237,\n",
|
||||
" 'zmin': 0}],\n",
|
||||
" 'layout': {'clickmode': 'event',\n",
|
||||
" 'margin': {'b': 0, 'l': 0, 'r': 0, 't': 30},\n",
|
||||
" 'template': '...',\n",
|
||||
" 'title': {},\n",
|
||||
" 'xaxis': {'constrain': 'domain',\n",
|
||||
" 'range': [-0.5, 1.5],\n",
|
||||
" 'tickmode': 'array',\n",
|
||||
" 'ticktext': [Plant, (none)],\n",
|
||||
" 'tickvals': array([0, 1])},\n",
|
||||
" 'yaxis': {'constrain': 'domain',\n",
|
||||
" 'range': [-0.5, 1.5],\n",
|
||||
" 'scaleanchor': 'x',\n",
|
||||
" 'scaleratio': 1,\n",
|
||||
" 'tickmode': 'array',\n",
|
||||
" 'ticktext': array(['(none)', 'Plant'], dtype=object),\n",
|
||||
" 'tickvals': array([0, 1])}}\n",
|
||||
"})"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"<fiftyone.core.plots.plotly.PlotlyNotebookPlot object at 0x7f509b1bcb10>\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": []
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": []
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "972da7a137ab49bab9498d1deba213ef",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
"FigureWidget({\n",
|
||||
" 'data': [{'customdata': array([0.9709574 , 0.94905514, 0.94097441, 0.93560362, 0.93146831, 0.92766279,\n",
|
||||
" 0.92468315, 0.92186761, 0.91872227, 0.91542256, 0.91216522, 0.90852749,\n",
|
||||
" 0.90351772, 0.89783657, 0.8911835 , 0.88456821, 0.87576878, 0.86684024,\n",
|
||||
" 0.85590148, 0.84403592, 0.82939762, 0.81318164, 0.79340947, 0.77035546,\n",
|
||||
" 0.74586266, 0.72073823, 0.69323456, 0.65981126, 0.61941642, 0.57270879,\n",
|
||||
" 0.52519083, 0.46561444, 0.40196586, 0.3248868 , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ]),\n",
|
||||
" 'hovertemplate': ('<b>class: %{text}</b><br>recal' ... 'customdata:.3f}<extra></extra>'),\n",
|
||||
" 'line': {'color': '#FF6D04'},\n",
|
||||
" 'mode': 'lines',\n",
|
||||
" 'name': 'Plant (AP = 0.278)',\n",
|
||||
" 'text': array(['Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant', 'Plant',\n",
|
||||
" 'Plant', 'Plant', 'Plant', 'Plant', 'Plant'], dtype='<U5'),\n",
|
||||
" 'type': 'scatter',\n",
|
||||
" 'uid': 'aebfd54d-341f-4988-a891-7c11134709e7',\n",
|
||||
" 'x': array([0. , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 , 0.11,\n",
|
||||
" 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 , 0.21, 0.22, 0.23,\n",
|
||||
" 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32, 0.33, 0.34, 0.35,\n",
|
||||
" 0.36, 0.37, 0.38, 0.39, 0.4 , 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47,\n",
|
||||
" 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59,\n",
|
||||
" 0.6 , 0.61, 0.62, 0.63, 0.64, 0.65, 0.66, 0.67, 0.68, 0.69, 0.7 , 0.71,\n",
|
||||
" 0.72, 0.73, 0.74, 0.75, 0.76, 0.77, 0.78, 0.79, 0.8 , 0.81, 0.82, 0.83,\n",
|
||||
" 0.84, 0.85, 0.86, 0.87, 0.88, 0.89, 0.9 , 0.91, 0.92, 0.93, 0.94, 0.95,\n",
|
||||
" 0.96, 0.97, 0.98, 0.99, 1. ]),\n",
|
||||
" 'y': array([1. , 0.93883661, 0.93883661, 0.93883661, 0.93883661, 0.93883661,\n",
|
||||
" 0.93883661, 0.93883661, 0.93883661, 0.93883661, 0.93719807, 0.93587326,\n",
|
||||
" 0.93308167, 0.92715232, 0.92366189, 0.92006525, 0.91499619, 0.90937129,\n",
|
||||
" 0.90153027, 0.89138577, 0.87890016, 0.86260933, 0.84492161, 0.82207422,\n",
|
||||
" 0.79579226, 0.76820289, 0.73332499, 0.69479333, 0.64924375, 0.59197536,\n",
|
||||
" 0.53650214, 0.47404492, 0.41206719, 0.33913995, 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. , 0. ,\n",
|
||||
" 0. , 0. , 0. , 0. , 0. ])}],\n",
|
||||
" 'layout': {'margin': {'b': 0, 'l': 0, 'r': 0, 't': 30},\n",
|
||||
" 'shapes': [{'line': {'dash': 'dash'}, 'type': 'line', 'x0': 0, 'x1': 1, 'y0': 1, 'y1': 0}],\n",
|
||||
" 'template': '...',\n",
|
||||
" 'xaxis': {'constrain': 'domain', 'range': [0, 1], 'title': {'text': 'Recall'}},\n",
|
||||
" 'yaxis': {'constrain': 'domain',\n",
|
||||
" 'range': [0, 1],\n",
|
||||
" 'scaleanchor': 'x',\n",
|
||||
" 'scaleratio': 1,\n",
|
||||
" 'title': {'text': 'Precision'}}}\n",
|
||||
"})"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Print a classification report for all classes\n",
|
||||
"results.print_report()\n",
|
||||
"\n",
|
||||
"print(results.mAP())\n",
|
||||
"\n",
|
||||
"# Plot confusion matrix\n",
|
||||
"matrix = results.plot_confusion_matrix(classes=['Plant'])\n",
|
||||
"matrix.show()\n",
|
||||
"\n",
|
||||
"pr_curves = results.plot_pr_curves(classes=['Plant'], iou_thresh=0.9)\n",
|
||||
"print(pr_curves)\n",
|
||||
"pr_curves.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cfff898d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### View dataset in fiftyone <a name=\"yolofiftyonesession\"></a>\n",
|
||||
"\n",
|
||||
"We can launch a fiftyone session in a new tab to explore the dataset and the results."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "75fd090c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Session launched. Run `session.show()` to open the App in a cell output.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"application/javascript": [
|
||||
"window.open('http://localhost:5151/');"
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.Javascript object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"session = fo.launch_app(yolo_view, auto=False)\n",
|
||||
"session.plots.attach(matrix)\n",
|
||||
"session.open_tab()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ebdde519",
|
||||
"id": "751f3d2b",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user