YOLOv4 | Haobin Tan

YOLOv4: Run Pretrained YOLOv4 on COCO Dataset

Wed, 04 Nov 2020 00:00:00 +0000

Here we will learn how to get YOLOv4 Object Detection running in the Cloud with Google Colab step by step.

Clone and build DarkNet

Clone darknet from AlexeyAB’s repository,

!git clone https://github.com/AlexeyAB/darknet

Adjust the Makefile to enable OPENCV and GPU for darknet

# change makefile to have GPU and OPENCV enabled
%cd darknet
!sed -i 's/OPENCV=0/OPENCV=1/' Makefile
!sed -i 's/GPU=0/GPU=1/' Makefile
!sed -i 's/CUDNN=0/CUDNN=1/' Makefile
!sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile

Verify CUDA

# verify CUDA
!/usr/local/cuda/bin/nvcc --version

Build darknet

Note: Do not worry about any warnings when running the !make cell!

# make darknet 
# (builds darknet so that you can then use the darknet executable file 
# to run or train object detectors)
!make

Download pretrained YOLO v4 weights

YOLOv4 has been trained already on the coco dataset which has 80 classes that it can predict. We will grab these pretrained weights so that we can run YOLOv4 on these pretrained classes and get detections.

!wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights

Define helper functions

import cv2
import matplotlib.pyplot as plt
%matplotlib inline


def imShow(path):
 """
 Show image
 """
 image = cv2.imread(path)
 height, width = image.shape[:2]
 resized_image = cv2.resize(image, (3*width, 3*height), interpolation = cv2.INTER_CUBIC)

 fig = plt.gcf()
 fig.set_size_inches(18, 10)
 plt.axis("off")
 plt.imshow(cv2.cvtColor(resized_image, cv2.COLOR_BGR2RGB))
 plt.show()


def upload():
 """
 upload files to Google Colab
 """
 from google.colab import files
 uploaded = files.upload()
 for name, data in uploaded.items():
 with open(name, 'wb') as f:
 f.write(data)
 print(f'saved file {name}')


def download(path):
 """
 Download from Google Colab
 """
 from google.colab import files
 files.download(path)

Run detections with Darknet and YOLOv4

The object detector can be run using the following command

!./darknet detector test <path to .data file> <path to config> <path to weights> <path to image>

This will output the image with the detections shown. The most recent detections are always saved to ‘predictions.jpg’

Note: After running detections OpenCV can’t open the image instantly in the cloud so we must run:

imShow('predictions.jpg')

Darknet comes with a few images already installed in the darknet/data/ folder. Let’s test one of the images inside:

# run darknet detection on test images
!./darknet detector test cfg/coco.data cfg/yolov4.cfg yolov4.weights data/person.jpg

imShow('predictions.jpg')

Run detections using uploaded image

We can also mount Google drive into the cloud VM a

from google.colab import drive
drive.mount('/content/gdrive')

# this creates a symbolic link 
# so that now the path /content/gdrive/My\ Drive/ is equal to /mydrive
!ln -s /content/gdrive/My\ Drive/ /mydrive
!ls /mydrive

nd run YOLOv4 with images from Google drive using the following command:

!./darknet detector test cfg/coco.data cfg/yolov4.cfg yolov4.weights /mydrive/<path to image>

For example, I uploaded an image called “pedestrian.jpg” in images/ folder:

and run detection on it:

!./darknet detector test cfg/coco.data cfg/yolov4.cfg yolov4.weights /mydrive/images/pedestrian.jpg
imShow('predictions.jpg')

Reference

YOLOv4 in the CLOUD: Install and Run Object Detector (FREE GPU)
- Google Colab Notebook
- https://github.com/theAIGuysCode/YOLOv4-Cloud-Tutorial
- Video Tutorial

YOLOv4: Train on Custom Dataset

Wed, 04 Nov 2020 00:00:00 +0000

Clone and build Darknet

Clone darknet repo

git clone https://github.com/AlexeyAB/darknet

Change makefile to have GPU and OPENCV enabled

cd darknet
sed -i 's/OPENCV=0/OPENCV=1/' Makefile
sed -i 's/GPU=0/GPU=1/' Makefile
sed -i 's/CUDNN=0/CUDNN=1/' Makefile
sed -i 's/CUDNN_HALF=0/CUDNN_HALF=1/' Makefile

Verify CUDA

/usr/local/cuda/bin/nvcc --version

Compile on Linux using `make`

Make darknet

make

GPU=1 : build with CUDA to accelerate by using GPU
CUDNN=1 : build with cuDNN v5-v7 to accelerate training by using GPU
CUDNN_HALF=1 to build for Tensor Cores (on Titan V / Tesla V100 / DGX-2 and later) speedup Detection 3x, Training 2x
OPENCV=1 to build with OpenCV 4.x/3.x/2.4.x - allows to detect on video files and video streams from network cameras or web-cams
DEBUG=1 to bould debug version of Yolo
OPENMP=1 to build with OpenMP support to accelerate Yolo by using multi-core CPU

Do not worry about any warnings when running make command.

Prepare custom dataset

The custom dataset should be in YOLOv4 or darknet format:

For each .jpg image file, there should be a corresponding .txt file
- In the same directory, with the same name, but with .txt-extension
  
  For example, if there’s an .jpg image named BloodImage_00001.jpg, there should also be a corresponding .txt file named BloodImage_00001.txt
In this .txt file: object number and object coordinates on this image, for each object in new line.

Format:
```
<object-class> <x_center> <y_center> <width> <height>
```
- <object-class> : integer object number from 0 to (classes-1)
- <x_center> <y_center> <width> <height> : float values relative to width and height of image, it can be equal from (0.0 to 1.0]
  - <x_center> <y_center> are center of rectangle (are not top-left corner)

Configure files for training

For training cfg/yolov4-custom.cfg download the pre-trained weights-file yolov4.conv.137

cd darknet
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.conv.137

In folder ./cfg, create custom config file (let’s call it custom-yolov4-detector.cfg) with the same content as in yolov4-custom.cfg and
- change line batch to batch=64
- change line subdivisions to subdivisions=16
- change line max_batches to classes*2000 but
  - NOT less than number of training images
  - NOT less than number of training images
  - NOT less than 6000
  e.g. max_batches=6000 if you train for 3 classes
- change line steps to 80% and 90% of max_batches (e.g. steps=4800, 5400)
- set network size width=416 height=416 or any value multiple of 32
- change line classes=80 to number of objects in each of 3 [yolo]-layers
- change [filters=255] to $ \text{filters}=(\text{classes} + 5) \times 3$ in the 3 [convolutional] before each [yolo] layer, keep in mind that it only has to be the last [convolutional] before each of the [yolo] layers.
  Note: Do not write in the cfg-file: filters=(classes + 5) x 3!!!
  
  It has to be the specific number!
  
  E.g. classes=1 then should be filters=18; classes=2 then should be filters=21
  
  So for example, for 2 objects, your custom config file should differ from yolov4-custom.cfg in such lines in each of 3 [yolo]-layers:
```
[convolutional]
filters=21

[region]
classes=2
```
- when using [Gaussian_yolo] layers, change [filters=57] $ \text{filters}=(\text{classes} + 9) \times 3$ in the 3 [convolutional] before each [Gaussian_yolo] layer
Create file obj.names in the directory data/, with objects names - each in new line
Create fiel obj.data in the directory data/, containing (where classes = number of objects):

For example, if we two objects
```
classes = 2
train = data/train.txt
valid = data/test.txt
names = data/obj.names
backup = backup/
```
Put image files (.jpg) of your objects in the directory data/obj/
Create train.txt in directory data/ with filenames of your images, each filename in new line, with path relative to darknet.

For example containing:
```
data/obj/img1.jpg
data/obj/img2.jpg
data/obj/img3.jpg
```
Download pre-trained weights for the convolutional layers and put to the directory darknet (root directory of the project)
- for yolov4.cfg, yolov4-custom.cfg (162 MB): yolov4.conv.137
- for yolov4-tiny.cfg, yolov4-tiny-3l.cfg, yolov4-tiny-custom.cfg(19 MB): yolov4-tiny.conv.29
- for csresnext50-panet-spp.cfg (133 MB): csresnext50-panet-spp.conv.112
- for yolov3.cfg, yolov3-spp.cfg (154 MB): darknet53.conv.74
- for yolov3-tiny-prn.cfg , yolov3-tiny.cfg (6 MB): yolov3-tiny.conv.11
- for enet-coco.cfg (EfficientNetB0-Yolov3) (14 MB): enetb0-coco.conv.132

Start training

./darknet detector train data/obj.data custom-yolov4-detector.cfg yolov4.conv.137 -dont_show

file yolo-obj_last.weights will be saved to the backup\ for each 100 iterations
-dont_show: disable Loss-Window, if you train on computer without monitor (e.g remote server)

To see the mAP & loss0chart during training on remote server:

use command ./darknet detector train data/obj.data yolo-obj.cfg yolov4.conv.137 -dont_show -mjpeg_port 8090 -map
then open URL http://ip-address:8090 in Chrome/Firefox browser)

After training is complete, you can get weights from backup/

If you want the training to output only main information (e.g loss, mAP, remaining training time) instead of full logging, you can use this command

./darknet detector train data/obj.data custom-yolov4-detector.cfg yolov4.conv.137 -dont_show -map 2>&1 | tee log/train.log | grep -E "hours left|mean_average"

Then the output will look like followings:

 1189: 1.874030, 2.934438 avg loss, 0.002610 rate, 2.930427 seconds, 76096 images, 3.905244 hours left

Notes

If during training you see nan values for avg (loss) field - then training goes wrong! 🤦‍♂️

But if nan is in some other lines - then training goes well.
if error Out of memory occurs then in .cfg-file you should increase subdivisions=16, 32 or 64

Train tiny-YOLO

Do all the same steps as for the full yolo model as described above. With the exception of:

Download file with the first 29-convolutional layers of yolov4-tiny:
```
wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.conv.29
```
(Or get this file from yolov4-tiny.weights file by using command: ./darknet partial cfg/yolov4-tiny-custom.cfg yolov4-tiny.weights yolov4-tiny.conv.29 29)

Make your custom model yolov4-tiny-obj.cfg based on cfg/yolov4-tiny-custom.cfg instead of yolov4.cfg

import re

# num_classes: number of object classes
max_batches = max(num_classes * 2000, num_train_images, 6000)
steps1 = .8 * max_batches
steps2 = .9 * max_batches
num_filters = (num_classes + 5) * 3

# Assuming that we have already defined the following hyperparameters:
# - TINY_CONFIG_FILE: config file we're gonna use for training
# - WIDTH, HEIGHT: width and height of image
with open("cfg/yolov4-tiny-custom.cfg", "r") as reader, open(TINY_CONFIG_FILE, "w") as writer:
 content = reader.read()

 content = re.sub("subdivisions=\d*", f"subdivisions={SUBDIVISION}", content)
 content = re.sub("width=\d*", f"width={WIDTH}", content)
 content = re.sub("height=\d*", f"height={HEIGHT}", content)
 content = re.sub("max_batches = \d*", f"max_batches = {max_batches}", content)
 content = re.sub("steps=\d*,\d*", f"steps={steps1},{steps2}", content)
 content = re.sub("classes=\d*", f"classes={num_classes}", content)
 content = re.sub("pad=1\nfilters=\d*", f"pad=1\nfilters={num_filters}", content)

 writer.write(content)

Start training:

./darknet detector train data/obj.data yolov4-tiny-obj.cfg yolov4-tiny.conv.29

Google Colab Notebook

Colab Notebook

Small hacks to keep colab notebook training

Open up the inspector view on Chrome
Switch to the console window

Paste the following code

function ClickConnect(){
console.log("Working");
document
 .querySelector('#top-toolbar > colab-connect-button')
 .shadowRoot.querySelector('#connect')
 .click()
}
setInterval(ClickConnect,60000)

and hit Enter.

It will click the screen every 10 minutes so that you don’t get kicked off for being idle!

Convert YOLOv4 to TensorRT through ONNX

To convert YOLOv4 to TensorRT engine through ONNX, I used the code from TensorRT_demos following its step-by-step instructions. For more details about the code, check out this blog post.

Note that the Code in this repo was designed to run on Jetson platforms. In my case, conversion from YOLOv4 to TensorRT engine was conducted on Jetson Nano.

Convert YOLOv4 for custom trained models

To apply the conversion for custom trained models, see TensorRT YOLOv3 For Custom Trained Models. You need to stick to the naming convention {yolo_version}-{custom_name}-{image_size}. Otherwise you’ll get errors during conversion.

Reference

Guide from AlexeyAB/darknet repo: How to train (to detect your custom objects)
Tutorials
- 👨‍🏫 How to Train YOLOv4 on a Custom Dataset in Darknet
  - Colab Notebook
  - Blog post: https://blog.roboflow.com/training-yolov4-on-a-custom-dataset/
  - Video tutorial:
  - YOLOv4 - Ten Tactics to Build a Better Model
- Train YOLOv4-tiny on custom dataset: Train YOLOv4-tiny on Custom Data - Lightning Fast Object Detection
- YOLOv4 in the CLOUD: Build and Train Custom Object Detector (FREE GPU)
  - Colab Notebook
  - Video tutorial:
- Custom YOLOv4 Model on Google Colab
  - Colab Notebook
- TensorRT YOLOv4
- YOLOv4 on Jetson Nano

YOLOv4: Training Tips

Sat, 19 Dec 2020 00:00:00 +0000

Model zoo

YOLOv4 model zoo

Pretrained models
Proper configuration based on GPU

We do NOT suggest you train the model with subdivisions equal or larger than 32, it will takes very long training time.

FAQ

Low accuracy ¹

The most common problem - you do NOT follow strictly the manual.

You must use
- default anchors
- learning_rate=0.001
- batch=64
- max_batches = max(6000, number_of_training_images, 2000*classes)
You can only change subdivisions
Do not do anything that is not written in the manual. 🙅‍♂️

Your datasets are wrong.

check the AP50 (average precision) for validation and training dataset by using ./darknet detector map obj.data yolo.cfg yolo.weights
- If you get high mAP for both Training and Validation datasets, but the network detects objects poorly in real life, then your training dataset is not representative –> add more images from real life to it
- If you get high mAP for Training dataset, but low for Validation dataset, then your Training dataset isn’t suitable for Validation dataset.
  
  For example
  - Training dataset contains: cars (rear view) from distance 100m
  - Test dataset contains: cars (side view) from distance 5m
- if you get low mAP for both Training and Validation datasets, then labels in your Training dataset are wrong
  - Run training with flag -show_imgs, i.e. ./darknet detector train ... -show_imgs , do you see correct bounded boxes?
  - Or check your dataset by using Yolo_mark tool

Darknet training/detection crashes with an error ²

If CUDA Out of memory error occurs, then increase subdivisions= 2 times in cfg-file, but not higher than batch= (don’t change batch)!
- If it doesn’t help - set random=0 and width=416 height=416 in cfg-file.
Check content of files bad.list and bad_label.list if they exist near with ./darknet executable file.
Do not move some files from Darknet folder - you may forget the necessary files.
Download libraries CUDA, cuDNN, OpenCV, … only from official sources. Don’t download libs from other sites.
Make sure that you do everything in accordance with the manual, and do not do anything that is not written in the manual.

Train with multiple GPUs ³

Train it first on 1 GPU for like 1000 iterations:

./darknet detector train cfg/coco.data cfg/yolov4.cfg yolov4.conv.137

Then stop and by using partially-trained model /backup/yolov4_1000.weights. Run training with multigpu (up to 4 GPUs): ./darknet detector train cfg/coco.data cfg/yolov4.cfg /backup/yolov4_1000.weights -gpus 0,1,2,3

If you get a Nan, then for some datasets better to decrease learning rate, for 4 GPUs set learning_rate = 0,00065 (i.e. learning_rate = 0.00261 / GPUs). In this case also increase 4x times burn_in = in your cfg-file. I.e. use burn_in = 4000 instead of 1000.

Train custom datasets

Configuration setup see: Train YOLO v4 on Custom Dataset

Start training:

./darknet detector train data/obj.data <custom-cfg> yolov4.conv.137

File <custom-cfg>_last.weights will be saved to backup/ for each 100 iterations
File <custom-cfg>_xxxx.weights will be saved to backup/ for each 1000 iterations
if you train on server without monitor, disable Loss-window by using argument --dont_show. I.e.
```
./darknet detector train data/obj.data <custom-cfg> yolov4.conv.137 -dont_show
```
To see the mAP & Loss-chart during training on remote server without GUI, use
```
./darknet detector train data/obj.data <custom-cfg> yolov4.conv.137 -dont_show -mjpeg_port 8090 -map
```
Then open URL http://ip-address:8090 in browser
For training with mAP calculation for each 4 Epochs, you need to
- set valid=valid.txt or train.txt in obj.data file
- run training with -map argument
```
./darknet detector train data/obj.data <custom-cfg> yolov4.conv.137 -map
```
After training is complete - get result yolo-obj_final.weights from backup/
After each 100 iterations you can stop and later start training from this point. For example, after 2000 iterations you can stop training, and later just start training using:
```
./darknet detector train data/obj.data <custom-cfg> backup/yolo-obj_2000.weights
```
You can get result earlier than all 45000 iterations.

Notes 📝

If during training you see nan values for avg (loss) field, then training goes wrong. 😭

But if nan is in some other lines, then training goes well. 🙏
If you changed width= or height= in your cfg-file, then new width and height must be divisible by 32.
If error Out of memory occurs then in .cfg-file you should increase subdivisions=16, 32 or 64

When should I stop training ⁴

Usually sufficient 2000 iterations for each class(object),
- but NOT less than number of training images and
- NOT less than 6000 iterations in total.
During training, you will see varying indicators of error, and you should stop when no longer decreases 0.XXXXXXX avg
For example

9002: 0.211667, 0.60730 avg, 0.001000 rate, 3.868000 seconds, 576128 images Loaded: 0.000000 seconds
- 9002 - iteration number (number of batch)
- 0.60730 avg - average loss (error) - the lower, the better
he final avgerage loss can be from 0.05 (for a small model and easy dataset) to 3.0 (for a big model and a difficult dataset).
if you train with flag -map then you will see mAP indicator like Last accuracy mAP@0.5 = 18.50% in the console. This indicator is better than Loss, so keep training while mAP increases.

Choose the best weights

Once training is stopped, you should take some of last .weights-files from backup/ and choose the best of them.

For example, you stopped training after 9000 iterations, but the best result can give one of previous weights (7000, 8000, 9000). It can happen due to overfitting.

In order to choose best weight, just train with -map flag

./darknet detector train data/obj.data <custom-cfg> yolov4.conv.137 -dont_show -map

So you will see mAP-chart (red-line) in the Loss-chart Window looks like the following figure. mAP will be calculated for each 4 Epochs using valid=valid.txt file that is specified in obj.data file (1 Epoch = images_in_train_txt / batch iterations)

How to improve object detection⁵

Before training

Set flag random=1 in your .cfg-file - it will increase precision by training Yolo for different resolutions
increase network resolution in your .cfg-file (height=608, width=608 or any value multiple of 32) - it will increase precision
Check that each object that you want to detect is mandatory labeled in your dataset - no one object in your data set should not be without label.
- In the most training issues, there are wrong labels in your dataset. Always check your dataset by using: https://github.com/AlexeyAB/Yolo_mark
My Loss is very high and mAP is very low, is training wrong?

–> Run training with -show_imgs flag at the end of training command, do you see correct bounded boxes of objects? If no, your training dataset is wrong.
For each object which you want to detect - there must be at least 1 similar object in the Training dataset with about the same: shape, side of object, relative size, angle of rotation, tilt, illumination.
- So desirable that your training dataset include images with objects at diffrent: scales, rotations, lightings, from different sides, on different backgrounds
- You should preferably have 2000 different images for each class or more, and you should train 2000*classes iterations or more
Desirable that your training dataset include images with non-labeled objects that you do not want to detect, i.e. negative samples without bounded box (empty .txt files). Use as many images of negative samples as there are images with objects.
More see: https://github.com/AlexeyAB/darknet#how-to-improve-object-detection

After training, for detection:

Increase network-resolution by set in your .cfg-file (height=608 and width=608) or (height=832 and width=832) or (any value multiple of 32). This increases the precision and makes it possible to detect small objects.
It is not necessary to train the network again, just use .weights-file already trained for 416x416 resolution
To get even greater accuracy you should train with higher resolution 608x608 or 832x832.
- Note: if error Out of memory occurs then in .cfg-file you should increase subdivisions=16, 32 or 64

Useful resources

Tips from Roboflow: YOLOv4 - Ten Tactics to Build a Better Model
Articles from Aleksey Bochkovskiy (author of YOLOv4)
- YOLOv4 — the most accurate real-time neural network on MS COCO dataset.
- Scaled YOLO v4 is the best neural network for object detection on MS COCO dataset
DARKNET FAQ

YOLOv4 | Haobin Tan

YOLOv4: Run Pretrained YOLOv4 on COCO Dataset

Clone and build DarkNet

Download pretrained YOLO v4 weights

Define helper functions

Run detections with Darknet and YOLOv4

Run detections using uploaded image

Reference

YOLOv4: Train on Custom Dataset

Clone and build Darknet

Compile on Linux using `make`

Prepare custom dataset

Configure files for training

Start training

Notes

Train tiny-YOLO

Google Colab Notebook

Small hacks to keep colab notebook training

Convert YOLOv4 to TensorRT through ONNX

Convert YOLOv4 for custom trained models

Reference

YOLOv4: Training Tips

Model zoo

FAQ

Low accuracy ¹

The most common problem - you do NOT follow strictly the manual.

Your datasets are wrong.

Darknet training/detection crashes with an error ²

Train with multiple GPUs ³

Train custom datasets

Notes 📝

When should I stop training ⁴

Choose the best weights

How to improve object detection⁵

Other questions

Will darknet automaticly resize the image size?

Does the network have to be perfectly square?

Detection with aspect ratio change

Useful resources

YOLOv4 | Haobin Tan

YOLOv4: Run Pretrained YOLOv4 on COCO Dataset

Clone and build DarkNet

Download pretrained YOLO v4 weights

Define helper functions

Run detections with Darknet and YOLOv4

Run detections using uploaded image

Reference

YOLOv4: Train on Custom Dataset

Clone and build Darknet

Compile on Linux using make

Prepare custom dataset

Configure files for training

Start training

Notes

Train tiny-YOLO

Google Colab Notebook

Small hacks to keep colab notebook training

Convert YOLOv4 to TensorRT through ONNX

Convert YOLOv4 for custom trained models

Reference

YOLOv4: Training Tips

Model zoo

FAQ

Low accuracy 1

The most common problem - you do NOT follow strictly the manual.

Your datasets are wrong.

Darknet training/detection crashes with an error 2

Train with multiple GPUs 3

Train custom datasets

Notes 📝

When should I stop training 4

Choose the best weights

How to improve object detection5

Other questions

Will darknet automaticly resize the image size?

Does the network have to be perfectly square?

Detection with aspect ratio change

Useful resources

Compile on Linux using `make`

Low accuracy ¹

Darknet training/detection crashes with an error ²

Train with multiple GPUs ³

When should I stop training ⁴

How to improve object detection⁵