Commit 4f9f2715 authored by Micaela Verucchi's avatar Micaela Verucchi
Browse files

Update READMEs



Signed-off-by: default avatarMicaela Verucchi <micaelaverucchi@gmail.com>
parent 0292e21c
......@@ -19,6 +19,7 @@ If you use tkDNN in your research, please cite the [following paper](https://iee
### What's new (20 July 2021)
- [x] Support to sematic segmentation [README](docs/README_seg.md)
- [x] Support 2D/3D Object Detection and Tracking [README](docs/README_2d3dtracking.md)
- [ ] Support to TensorRT8 (WIP)
## FPS Results
......@@ -70,15 +71,13 @@ Results for COCO val 2017 (5k images), on RTX 2080Ti, with conf threshold=0.001
- [tkDNN](#tkdnn)
- [Index](#index)
- [Dependencies](#dependencies)
- [About OpenCV](#about-opencv)
- [How to compile this repo](#how-to-compile-this-repo)
- [Workflow](#workflow)
- [How to export weights](#how-to-export-weights)
- [Run the demo](#run-the-demo)
- [mAP demo](#map-demo)
- [Exporting weights](#exporting-weights)
- [Run the demos](#run-the-demos)
- [tkDNN on Windows 10 (experimental)](#tkdnn-on-windows-10-experimental)
- [Existing tests and supported networks](#existing-tests-and-supported-networks)
- [References](#references)
- [tkDNN on Windows 10 (experimental)](#tkdnn-on-windows-10-experimental)
## Dependencies
......@@ -97,7 +96,7 @@ sudo apt install libyaml-cpp-dev curl libeigen3-dev
```
## About OpenCV
#### About OpenCV
To compile and install OpenCV4 with contrib us the script ```install_OpenCV4.sh```. It will download and compile OpenCV in Download folder.
```
bash scripts/install_OpenCV4.sh
......@@ -125,15 +124,21 @@ Steps needed to do inference on tkDNN with a custom neural network.
## Exporting weights
For specific details on how to export weights see [here](./docs/exporting_weights.md)
For specific details on how to export weights see [HERE](./docs/exporting_weights.md).
## Run the demo
## Run the demos
For specific details on how to run the demos see [here](./docs/demo.md)
For specific details on how to run:
- 2D object detection demos, details on FP16, INT8 and batching see [HERE](./docs/demo.md).
- segmentation demos see [HERE](./docs/README_seg.md).
- 2D/3D object detection and tracking demos see [HERE](./docs/README_2d3dtracking.md).
- mAP demo to evaluate 2D object detectors see [HERE](./docs/mAP_demo.md).
## mAP demo
![demo](https://user-images.githubusercontent.com/11562617/72547657-540e7800-388d-11ea-83c6-49dfea2a0607.gif)
For specific details on how to run the mAP demo see [here](./docs/mAP_demo.md)
## tkDNN on Windows 10 (experimental)
For specific details on how to run tkDNN on Windows 10 see [HERE](./docs/windows.md).
## Existing tests and supported networks
......@@ -160,14 +165,20 @@ For specific details on how to run the mAP demo see [here](./docs/mAP_demo.md)
| resnet101_cnet | Centernet (Resnet101 backend)<sup>4</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/5BTjHMWBcJk8g3i/download) |
| csresnext50-panet-spp | Cross Stage Partial Network <sup>7</sup> | [COCO 2014](http://cocodataset.org/) | 80 | 416x416 | [weights](https://cloud.hipert.unimore.it/s/Kcs4xBozwY4wFx8/download) |
| yolo4 | Yolov4 <sup>8</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 416x416 | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download) |
| yolo4_320 | Yolov4 <sup>8</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 320x320 | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download) |
| yolo4_512 | Yolov4 <sup>8</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download) |
| yolo4_608 | Yolov4 <sup>8</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 608x608 | [weights](https://cloud.hipert.unimore.it/s/d97CFzYqCPCp5Hg/download) |
| yolo4_berkeley | Yolov4 <sup>8</sup> | [BDD100K ](https://bair.berkeley.edu/blog/2018/05/30/bdd/) | 10 | 540x320 | [weights](https://cloud.hipert.unimore.it/s/nkWFa5fgb4NTdnB/download) |
| yolo4tiny | Yolov4 tiny <sup>9</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 416x416 | [weights](https://cloud.hipert.unimore.it/s/iRnc4pSqmx78gJs/download) |
| yolo4x | Yolov4x-mish <sup>9</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 640x640 | [weights](https://cloud.hipert.unimore.it/s/5MFjtNtgbDGdJEo/download) |
| yolo4x | Yolov4x-mish <sup>9</sup> | [COCO 2017](http://cocodataset.org/) |
| yolo4tiny_512 | Yolov4 tiny <sup>9</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/iRnc4pSqmx78gJs/download) |
80 | 640x640 | [weights](https://cloud.hipert.unimore.it/s/5MFjtNtgbDGdJEo/download) |
| yolo4x-cps | Scaled Yolov4 <sup>10</sup> | [COCO 2017](http://cocodataset.org/) | 80 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/AfzHE4BfTeEm2gH/download) |
| shelfnet | ShelfNet18_realtime<sup>11</sup> | [Cityscapes](https://www.cityscapes-dataset.com/) | 19 | 1024x1024 | [weights](https://cloud.hipert.unimore.it/s/mEDZMRJaGCFWSJF/download) |
| shelfnet_berkeley | ShelfNet18_realtime<sup>11</sup> | [DeepDrive](https://bdd-data.berkeley.edu/) | 20 | 1024x1024 | [weights](https://cloud.hipert.unimore.it/s/m92e7QdD9gYMF7f/download) |
| dla34_cnet3d | Centernet3D (DLA34 backend)<sup>4</sup> | [KITTI 2017](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) | 1 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/2MDyWGzQsTKMjmR/download) |
| dla34_ctrack | CenterTrack (DLA34 backend)<sup>12</sup> | [NuScenes 3D](https://www.nuscenes.org/) | 7 | 512x512 | [weights](https://cloud.hipert.unimore.it/s/rjNfgGL9FtAXLHp/download) |
### tkDNN on Windows 10 (experimental)
For specific details on how to run the demos on Windows 10 see [here](./docs/windows.md)
## References
......@@ -181,3 +192,5 @@ For specific details on how to run the demos on Windows 10 see [here](./docs/win
8. Bochkovskiy, Alexey, Chien-Yao Wang, and Hong-Yuan Mark Liao. "YOLOv4: Optimal Speed and Accuracy of Object Detection." arXiv preprint arXiv:2004.10934 (2020).
9. Bochkovskiy, Alexey, "Yolo v4, v3 and v2 for Windows and Linux" (https://github.com/AlexeyAB/darknet)
10. Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. "Scaled-YOLOv4: Scaling Cross Stage Partial Network." arXiv preprint arXiv:2011.08036 (2020).
11. Zhuang, Juntang, et al. "ShelfNet for fast semantic segmentation." Proceedings of the IEEE International Conference on Computer Vision Workshops. 2019.
12. Zhou, Xingyi, Vladlen Koltun, and Philipp Krähenbühl. "Tracking objects as points." European Conference on Computer Vision. Springer, Cham, 2020.
# 2D/3D Object Detection and Tracking
Currently tkDNN supports only CenterTrack as OD & Tracker network.
Currently tkDNN supports only CenterTrack as 3DOD & 2D/3D Tracker network.
## 3D Object Detection
To run the 3D object detection demo follow these steps (example with CenterNet based on DLA34):
```
rm dla34_cnet3d_fp32.rt # be sure to delete(or move) old tensorRT files
./test_dla34_cnet3d # run the yolo test (is slow)
./demo3D dla34_cnet3d_fp32.rt ../demo/yolo_test.mp4 NULL c
```
The demo3D program takes the same parameters of the demo program:
```
./demo3D <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh>
```
where
* ```<calibration-file>``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
## Object Detection and Tracking
To run the 3D object detection & tracking demo follow these steps (example with CenterTrack based on DLA34):
```
rm dla34_ctrack_fp32.rt # be sure to delete(or move) old tensorRT files
./test_dla34_ctrack # run the yolo test (is slow)
./demoTracker dla34_ctrack_fp32.rt ../demo/yolo_test.mp4 NULL c
```
The demoTracker program takes the same parameters of the demo program:
```
./demoTracker <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh> <2D/3D-flag>
```
where
* ```<calibration-file>``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
* ```<2D/3D-flag>``` if set to 0 the demo will be in the 2D mode, while if set to 1 the demo will be in the 3D mode (Default is 1 - 3D mode).
## FPS Results
......
......@@ -2,19 +2,6 @@
Currently tkDNN supports only ShelfNet as semantic segmentation network.
## Export weights from Shelfnet
To get the weights needed to run Shelfnet tests use [this](https://git.hipert.unimore.it/mverucchi/shelfnet) fork of a Pytorch implementation of Shelfnet network.
```
git clone https://git.hipert.unimore.it/mverucchi/shelfnet
cd shelfnet
cd ShelfNet18_realtime
conda env create --file shelfnet_env.yml
conda activate shelfnet
mkdir layer debug
python export.py
```
## Run the demo
......@@ -46,15 +33,7 @@ NB) The batching is not used to work on more streams, rather to work on more til
For other demo videos refer to [this playlist](https://www.youtube.com/playlist?list=PLv0nEQYDD45y5EdSiywwCGPBmJVUzIWwe).
## Existing tests and supported networks
| Test Name | Network | Dataset | N Classes | Input size | Weights |
| :---------------- | :-------------------------------------------- | :-----------------------------------------------------------: | :-------: | :-----------: | :------------------------------------------------------------------------ |
| shelfnet | ShelfNet18_realtime<sup>1</sup> | [Cityscapes](https://www.cityscapes-dataset.com/) | 19 | 1024x1024 | [weights](https://cloud.hipert.unimore.it/s/mEDZMRJaGCFWSJF/download) |
| shelfnet_berkeley | ShelfNet18_realtime<sup>1</sup> | [DeepDrive](https://bdd-data.berkeley.edu/) | 20 | 1024x1024 | [weights](https://cloud.hipert.unimore.it/s/m92e7QdD9gYMF7f/download) |
1. Zhuang, Juntang, et al. "ShelfNet for fast semantic segmentation." Proceedings of the IEEE International Conference on Computer Vision Workshops. 2019.
NB) The gif and the videos are obtained with Mapillary Vistas weights, that we cannot publicly share due to its license restrictions. However, you can train Shelfnet using Mapillary and [this](https://git.hipert.unimore.it/mverucchi/shelfnet) fork of the original repo.
## FPS Results
......
# tkDNN Demo
# 2D Object Detection with tkDNN
## Supported Network
2D Object Detection:
## Supported Networks
* Yolo4, Yolo4-csp, Yolo4x, Yolo4_berkeley, Yolo4tiny
* Yolo3, Yolo3_berkeley, Yolo3_coco4, Yolo3_flir, Yolo3_512, Yolo3tiny, Yolo3tiny_512
......@@ -11,28 +9,12 @@
* Resnet101_cnet, Dla34_cnet
* Mobilenetv2ssd, Mobilenetv2ssd512, Bdd-mobilenetv2ssd
3D Object Detection:
* Dla34_cnet3d
2D/3D Object Detection and Tracking:
* Dla34_cnet3d_track
## Index
- [Run the demo](#run-the-demo)
- [2D Object Detection](#2d-object-detection)
- [3D Object Detection](#3d-object-detection)
- [Object Detection and Tracking](#object-detection-and-tracking)
- [FP16 inference](#fp16-inference)
- [INT8 inference](#int8-inference)
- [Batching](#batching)
- [Run the demo on Windows](#run-the-demo-on-windows)
## Run the demo
N.b. By default it is used FP32 inference
- [2D Object Detection](#2d-object-detection)
- [FP16 inference](#fp16-inference)
- [INT8 inference](#int8-inference)
- [Batching](#batching)
### 2D Object Detection
This is an example using yolov4.
......@@ -66,45 +48,11 @@ where
* ```<show-flag>``` if set to 0 the demo will not show the visualization but save the video into result.mp4 (if n-batches ==1)
* ```<conf-thresh>``` confidence threshold for the detector. Only bounding boxes with threshold greater than conf-thresh will be displayed.
N.B. By default it is used FP32 inference
![demo](https://user-images.githubusercontent.com/11562617/72547657-540e7800-388d-11ea-83c6-49dfea2a0607.gif)
### 3D Object Detection
To run the 3D object detection demo follow these steps (example with CenterNet based on DLA34):
```
rm dla34_cnet3d_fp32.rt # be sure to delete(or move) old tensorRT files
./test_dla34_cnet3d # run the yolo test (is slow)
./demo3D dla34_cnet3d_fp32.rt ../demo/yolo_test.mp4 NULL c
```
The demo3D program takes the same parameters of the demo program:
```
./demo3D <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh>
```
where
* ```calibration-file``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
### Object Detection and Tracking
To run the 3D object detection & tracking demo follow these steps (example with CenterTrack based on DLA34):
```
rm dla34_ctrack_fp32.rt # be sure to delete(or move) old tensorRT files
./test_dla34_ctrack # run the yolo test (is slow)
./demoTracker dla34_ctrack_fp32.rt ../demo/yolo_test.mp4 NULL c
```
The demoTracker program takes the same parameters of the demo program:
```
./demoTracker <network-rt-file> <path-to-video> <calibration-file> <kind-of-network> <number-of-classes> <n-batches> <show-flag> <conf-thresh> <2D/3D-flag>
```
where
* ```calibration-file``` is the camera calibration file (opencv format). It is important that the file contains entry "camera_matrix" with sub-entry "rows", "cols", "data". If you do not want to pass the calibration file, pass "NULL" instead.
* ```<2D/3D-flag>``` if set to 0 the demo will be in the 2D mode, while if set to 1 the demo will be in the 3D mode (Default is 1 - 3D mode).
### FP16 inference
......@@ -115,7 +63,7 @@ rm yolo3_fp16.rt # be sure to delete(or move) old tensorRT files
./test_yolo3 # run the yolo test (is slow)
./demo yolo3_fp16.rt ../demo/yolo_test.mp4 y
```
N.b. Using FP16 inference will lead to some errors in the results (first or second decimal).
N.B. Using FP16 inference will lead to some errors in the results (first or second decimal).
### INT8 inference
......@@ -169,49 +117,3 @@ rm yolo3_fp32.rt # be sure to delete(or move) old tensorRT fil
./test_yolo3 # build RT file
./test_rtinference yolo3_fp32.rt 4 # test with a batch size of 4
```
### Run the demo on Windows
This example uses yolo4_tiny.\
To run the object detection file create .rt file bu running:
```
.\test_yolo4tiny.exe
```
Once the rt file has been successfully create,run the demo using the following command:
```
.\demo.exe yolo4tiny_fp32.rt ..\demo\yolo_test.mp4 y
```
For general info on more demo paramters,check Run the demo section on top
To run the test_all_tests.sh on windows,use git bash or msys2
### FP16 inference windows
This is an untested feature on windows.To run the object detection demo with FP16 interference follow the below steps(example with yolo4tiny):
```
set TKDNN_MODE=FP16
del /f yolo4tiny_fp16.rt
.\test_yolo4tiny.exe
.\demo.exe yolo4tiny_fp16.rt ..\demo\yolo_test.mp4
```
### INT8 inference windows
To run object detection demo with INT8 (example with yolo4tiny):
```
set TKDNN_MODE=INT8
set TKDNN_CALIB_LABEL_PATH=..\demo\COCO_val2017\all_labels.txt
set TKDNN_CALIB_IMG_PATH=..\demo\COCO_val2017\all_images.txt
del /f yolo4tiny_int8.rt # be sure to delete(or move) old tensorRT files
.\test_yolo4tiny.exe # run the yolo test (is slow)
.\demo.exe yolo4tiny_int8.rt ..\demo\yolo_test.mp4 y
```
### Known issues with tkDNN on Windows
Mobilenet and Centernet demos work properly only when built with msvc 16.7 in Release Mode,when built in debug mode for the mentioned networks one might encounter opencv assert errors
All Darknet models work properly with demo using MSVC version(16.7-16.9)
It is recommended to use Nvidia Driver(465+),Cuda unknown errors have been observed when using older drivers on pascal(SM 61) devices.
......@@ -8,6 +8,7 @@
- [3)Export weights for CenterNet](#3export-weights-for-centernet)
- [4)Export weights for MobileNetSSD](#4export-weights-for-mobilenetssd)
- [5)Export weights for CenterTrack](#5export-weights-for-centertrack)
- [6)Export weights for ShelfNet](#6export-weights-for-shelfnet)
- [Darknet Parser](#darknet-parser)
## How to export weights
......@@ -30,7 +31,7 @@ make
mkdir layers debug
./darknet export <path-to-cfg-file> <path-to-weights> layers
```
N.b. Use compilation with CPU (leave GPU=0 in Makefile) if you also want debug.
N.B. Use compilation with CPU (leave GPU=0 in Makefile) if you also want debug.
### 2)Export weights for DLA34 and ResNet101
To get weights and outputs needed to run the tests dla34 and resnet101 use the Python script and the Anaconda environment included in the repository.
......@@ -72,6 +73,19 @@ git clone https://github.com/sapienzadavide/CenterTrack.git
python demo.py tracking,ddd --load_model ../models/nuScenes_3Dtracking.pth --dataset nuscenes --pre_hm --track_thresh 0.1 --demo /path/to/image/or/folder/or/video/or/webcam --test_focal_length 633 --exp_wo --exp_wo_dim 512 --input_h 512 --input_w 512
```
### 6)Export weights for ShelfNet
To get the weights needed to run Shelfnet tests use [this](https://git.hipert.unimore.it/mverucchi/shelfnet) fork of a Pytorch implementation of Shelfnet network.
```
git clone https://git.hipert.unimore.it/mverucchi/shelfnet
cd shelfnet
cd ShelfNet18_realtime
conda env create --file shelfnet_env.yml
conda activate shelfnet
mkdir layer debug
python export.py
```
## Darknet Parser
tkDNN implement and easy parser for darknet cfg files, a network can be converted with *tk::dnn::darknetParser*:
```
......
# mAP demo
# Run the mAP demo
## Run the mAP demo
To compute mAP, precision, recall and f1score, run the map_demo.
To compute mAP, precision, recall and f1score to evaluate 2D object detectors, run the map_demo.
A validation set is needed.
To download COCO_val2017 (80 classes) run (form the root folder):
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment