Automatic Number Plate Recognition using AI
Problem Statement — Our client, a leading textile manufacturer faced various security related issues at their site. Through a detailed assessment of the site, it was noticed that the operating modules lacked standardization and monitoring of the same was done manually. They also had a significantly large amount of man-guarding coupled with underutilization of the said resources with little to no accountability. Through the integration of tech with the mobile application, it was possible to track the daily activities and provide the same information through a maker checker concept for further validation of processes.
Let’s for now focus only on the tracking of vehicles through mobile application. This was done by the security workforce deployed at every checkpoint where they were responsible to capture the picture of every vehicle’s number plate on entry and exit through the mobile app. To detect the license plate, we made use of an AI based Indian license plate detector. In this blog I will explain my approach to solve this problem. Let’s get going!
Tools and Library dependencies
1. Python — used Python 3.8 for building the machine learning models.
2. IDE — PyCharm and Jupyter Notebook
3. NumPy and Pandas — NumPy is an extension package for performing numerical computing with Python that replaced NumArray and Numeric. It supports multidimensional arrays (tables) and matrices.
Pandas simplifies analysis by converting CSV, JSON, and TSV data files or a SQL database into a data frame, a Python object looking like an Excel or an SPSS table with rows and columns.
4. Matplotlib — used for visualization
5. Scikit-learn — It is an open source Python machine learning library built on top of SciPy (Scientific Python), NumPy, and matplotlib provides a number of well-established algorithms for supervised and unsupervised learning.
6. Keras — Keras is a Python deep learning library capable of running on top off Theano and TensorFlow. The library can run on GPU and CPU and support both recurrent and convolutional networks, as well as their combinations.
7. Flask — used flask 1.1.2 for model deployment
Below is an overview of the steps we are going to follow.
- Input Source — As discussed, the number plate image would be taken by the security workforce using the mobile application.
- Analyzing and performing image processing on the number plate — Using OpenCV’s grayscale, threshold, erode, dilate, contour detection and by some parameter tuning, we may easily be able to generate enough information about the plate to decide if the data is useful enough to be passed on to further processes or not (sometime if the image is very distorted or not proper, we may only get suppose 8 out of 10 characters, then there’s no point passing the data down the pipeline but to ignore it and look at the next frame for the plate), also before passing the image to the next process we need to make sure that it is noise-free and processed.
- Segmenting the alphanumeric characters from the license plate — If everything in the above steps works fine, we should be ready to extract the characters from the plate, this can be done by thresholding, eroding, dilating and blurring the image skillfully such that at the end the image we have is almost noise-free and easy for further functions to work on. We now again use contour detection and some parameter tuning to extract the characters.
- Character Prediction — Since we have all the characters, we need to pass the characters one by one into our trained model to predict all the characters of our number plate. We’ll be using Keras for our Convolutional Neural Network model.
- Model Deployment — Model deployment is done using Flask framework.
Step 1: Create a Flask project in PyCharm
- In the New Project dialog, do the following:
- Specify project type Flask.
- Specify project location.
2. Next, click to expand the Python Interpreter node, and select the new environment or previously configured interpreter, by clicking the corresponding radio-button.
New environment using: if this option has been selected, choose the tool to be used to create a virtual environment. To do that, click the list and choose Virtual env, Pipenv , or Conda.
Next, specify the Location and Base interpreter of the new virtual environment. If necessary, click the Inherit global site-packages and Make available to all projects check boxes.
When configuring the base interpreter, you need to specify the path to the Python executable. If PyCharm detects no Python on your machine, it provides two options: to download the latest Python versions from python.org or to specify a path to the Python executable (in case of non-standard installation).
3. Click (More Settings), and specify the following:
From the Template language list, select the language to be used.
In the Templates folder field, specify the directory where the templates will be stored, and where they will be loaded from. You can specify the name of the directory that doesn’t yet exist; in this case, the directory will be created.
4. Click Create.
PyCharm creates an application and produces specific directory structure, which you can explore in the Project tool window. Besides that, PyCharm creates a stub Python script with the name app.py, which provides a simple “Hello, World!” example.
Now you can add files as per your requirement in the project. Before deploying our project, let’s make our model for the number plate recognition.
Step 2: Let’s start with some image processing to make the segmentation of characters easier.
def segment_characters(image) :# Preprocess cropped license plate image
img_lp = cv2.resize(image, (333, 75))
img_gray_lp = cv2.cvtColor(img_lp, cv2.COLOR_BGR2GRAY)
_, img_binary_lp = cv2.threshold(img_gray_lp, 200, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)
img_binary_lp = cv2.erode(img_binary_lp, (3,3))
img_binary_lp = cv2.dilate(img_binary_lp, (3,3))LP_WIDTH = img_binary_lp.shape[0]
LP_HEIGHT = img_binary_lp.shape[1]# Make borders white
img_binary_lp[0:3,:] = 255
img_binary_lp[:,0:3] = 255
img_binary_lp[72:75,:] = 255
img_binary_lp[:,330:333] = 255# Estimations of character contours sizes of cropped license plates
dimensions = [LP_WIDTH/6,
LP_WIDTH/2,
LP_HEIGHT/10,
2*LP_HEIGHT/3]
plt.imshow(img_binary_lp, cmap='gray')
plt.show()
cv2.imwrite('contour.jpg',img_binary_lp)# Get contours within cropped license plate
char_list = find_contours(dimensions, img_binary_lp)return char_list
Functions used for image processing are:
1. cv2.resize() — resizes the image to a dimension such that all characters seem distinct and clear.
2. cv2.cvtColor() — converts images from one color-space to another, like BGR Gray, BGR HSV etc. to convert images from one color-space to another, like BGR Gray, BGR HSV etc. For BGR Gray conversion we use the flags cv2.COLOR_BGR2GRAY. Similarly for BGR HSV, we use the flag cv2.COLOR_BGR2HSV.
3. cv2.erode() and cv2.dilate() — Morphological transformations are some simple operations based on the image shape. It is normally performed on binary images. It needs two inputs, one is our original image, second one is called structuring element or kernel which decides the nature of operation. Two basic morphological operators are Erosion and Dilation.
Erosion — The basic idea of erosion is just like soil erosion only, it erodes away the boundaries of foreground object (Always try to keep foreground in white). So what does it do? The kernel slides through the image (as in 2D convolution). A pixel in the original image (either 1 or 0) will be considered 1 only if all the pixels under the kernel is 1, otherwise it is eroded (made to zero).
Dilation — It is just opposite of erosion. Here, a pixel element is ‘1’ if at least one pixel under the kernel is ‘1’. So it increases the white region in the image or the size of foreground object increases. Normally, in cases like noise removal, erosion is followed by dilation. Because, erosion removes white noises, but it also shrinks our object. So we dilate it. Since noise is gone, they won’t come back, but our object area increases. It is also useful in joining broken parts of an object.
4. The next step now is to make the boundaries of the image white. This is to remove any out of the frame pixel in case it is present.
5. Next, we define a list of dimensions that contains 4 values with which we’ll be comparing the character’s dimensions for filtering out the required characters.
6. Through the above processes, we have reduced our image to a processed binary image and we are ready to pass this image for character extraction.
Step 3: Segmenting the alphanumeric characters from the number plate.
def find_contours(dimensions, img) :# Find all contours in the image
cntrs, _ = cv2.findContours(img.copy(), cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)# Retrieve potential dimensions
lower_width = dimensions[0]
upper_width = dimensions[1]
lower_height = dimensions[2]
upper_height = dimensions[3]
# Check largest 5 or 15 contours for license plate or character respectively
cntrs = sorted(cntrs, key=cv2.contourArea, reverse=True)[:15]
ii = cv2.imread('contour.jpg')
iii = cv2.imread('contour.jpg')
cv2.drawContours(iii, cntrs, -1, (0,255,0), 3)
plt.imshow(iii, cmap='gray')
plt.show()
x_cntr_list = []
target_contours = []
img_res = []
for cntr in cntrs :
#detects contour in binary image and returns the coordinates of rectangle enclosing it
intX, intY, intWidth, intHeight = cv2.boundingRect(cntr)
#checking the dimensions of the contour to filter out the characters by contour's size
if intWidth > lower_width and intWidth < upper_width and intHeight > lower_height and intHeight < upper_height :
x_cntr_list.append(intX) #stores the x coordinate of the character's contour, to used later for indexing the contourschar_copy = np.zeros((44,24))
#extracting each character using the enclosing rectangle's coordinates.
char = img[intY:intY+intHeight, intX:intX+intWidth]
char = cv2.resize(char, (20, 40))
cv2.rectangle(ii, (intX,intY), (intWidth+intX, intY+intHeight), (50,21,200), 2)
plt.imshow(ii, cmap='gray')# Make result formatted for classification: invert colors
char = cv2.subtract(255, char)# Resize the image to 24x44 with black border
char_copy[2:42, 2:22] = char
char_copy[0:2, :] = 0
char_copy[:, 0:2] = 0
char_copy[42:44, :] = 0
char_copy[:, 22:24] = 0img_res.append(char_copy) #List that stores the character's binary image (unsorted)
#Return characters on ascending order with respect to the x-coordinate (most-left character first)
plt.show()
#arbitrary function that stores sorted list of character indeces
indices = sorted(range(len(x_cntr_list)), key=lambda k: x_cntr_list[k])
img_res_copy = []
for idx in indices:
img_res_copy.append(img_res[idx])# stores character images according to their index
img_res = np.array(img_res_copy)return img_res
1. cv.findContours() , cv.drawContours() — Contours can be explained simply as a curve joining all the continuous points (along the boundary), having same color or intensity. The contours are a useful tool for shape analysis and object detection and recognition.
For better accuracy, use binary images. So before finding contours, apply threshold or canny edge detection.
Since OpenCV 3.2, findContours() no longer modifies the source image.
In OpenCV, finding contours is like finding white object from black background. So remember, object to be found should be white and background should be black.
To draw the contours, cv.drawContours function is used. It can also be used to draw any shape provided you have its boundary points.
2. After finding all the contours we consider them one by one and calculate the dimension of their respective bounding rectangle. Now consider bounding rectangle is the smallest rectangle possible that contains the contour.
3. Since we have the dimensions of these bounding rectangles, all we need to do is do some parameter tuning and filter out the required rectangle containing required characters. For this, we will be performing some dimension comparison by accepting only those rectangles that has a width in range of 0, (length of the image) / (number of characters) and length in a range of (width of the pic)/2, 4*(width of the pic)/5. If everything works well, we will have all the characters extracted as binary images.
Step 4: Creating a Machine Learning model and training it for the characters.
- The data is all clean and ready, now it’s time to create a Neural Network that will be intelligent enough to recognize the characters after training.
model = Sequential()
model.add(Conv2D(32, (24,24), input_shape=(28, 28, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.4))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(36, activation='softmax'))model.compile(loss='categorical_crossentropy', optimizer=optimizers.Adam(lr=0.00001), metrics=['accuracy'])
model.summary()
2. To keep the model simple, we’ll start by creating a sequential object.
3. The first layer will be a convolutional layer with 32 output filters, a convolution window of size (5,5), and ‘Relu’ as activation function.
4. Next, we’ll be adding a max-pooling layer with a window size of (2,2).
5. Max pooling is a sample-based discretization process. The objective is to down-sample an input representation (image, hidden-layer output matrix, etc.), reducing its dimensionality and allowing for assumptions to be made about features contained in the sub-regions binned.
6. Now, we will be adding some dropout rate to take care of overfitting.
7. Dropout is a regularization hyperparameter initialized to prevent Neural Networks from Overfitting. Dropout is a technique where randomly selected neurons are ignored during training. They are “dropped-out” randomly. We have chosen a dropout rate of 0.4 meaning 60% of the node will be retained.
8. Now it’s time to flatten the node data so we add a flatten layer for that. The flatten layer takes data from the previous layer and represents it in a single dimension.
9. Finally, we will be adding 2 dense layers, one with the dimensionality of the output space as 128, activation function=’relu’ and other, our final layer with 36 outputs for categorizing the 26 alphabets (A-Z) + 10 digits (0–9) and activation function=’softmax’.
Step 5: Train the model
1. The data we will be using contains images of alphabets (A-Z) and digits (0–9) of size 28x28, also the data is balanced so we won’t have to do any kind of data tuning here.
2. We’ll be using ImageDataGenerator class available in keras to generate some more data using image augmentation techniques like width shift, height shift.
3. It’s time to train our model now!
We will use ‘categorical_crossentropy’ as loss function, ‘Adam’ as optimization function and ‘Accuracy’ as our error matrix.
from tensorflow.keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(rescale=1./255, width_shift_range=0.1, height_shift_range=0.1)
train_generator = train_datagen.flow_from_directory(
'data/data/train',
batch_size=1,
target_size=(28,28),
class_mode='categorical')validation_generator = train_datagen.flow_from_directory(
'data/data/val', # this is the target directory
target_size=(28,28), # all images will be resized to 28x28
batch_size=1,
class_mode='categorical')
Step 6: Save the model
1. In machine learning, while working with scikit learn library, we need to save the trained models in a file and restore them in order to make predictions on the new data or to compare the models with other models.
2. The saving of data is called Serialization, while restoring the data is called Deserialization.
3. There are various methods to achieve this, using pickle, joblib or Keras H5 format. I have used Keras H5 format and is a light-weight alternative to TensorFlow’s SavedModel.
model.save(“anpr_model.h5”)4. Calling save(‘my_model.h5’) creates a h5 file `my_model.h5` which can be used to make prediction on new data.
Step 7: Model Deployment using Flask
- First we will import the Flask class. An instance of this class will be our WSGI application.
- Next we create an instance of this class. The first argument is the name of the application’s module or package. If you are using a single module (as in this example), you should use __name__ because depending on if it’s started as application or imported as module the name will be different (‘__main__’ versus the actual import name). This is needed so that Flask knows where to look for templates, static files, and so on. For more information, have a look at the Flask documentation.
- We then use the route () decorator to tell Flask what URL should trigger our function.
- The function is given a name which is also used to generate URLs for that particular function, and returns the message we want to display in the user’s browser.
from flask import Flask, render_template, send_from_directory, request, abort
from keras.models import load_model
import numpy as np
import cv2
import os
from werkzeug.utils import secure_filename
UPLOAD_FOLDER = 'static/images'
app = Flask(__name__)@app.route("/")
@app.route("/index")
def index():
return render_template('index.html')
5. The next part was to make an API which receives image through GUI and returns the number plate number based on our model. For this I de- serialized the model in the form of python object by adding the below line in my project’s app.py file (save the anpr_model.h5 file in your project).
model = load_model('anpr_model.h5')6. Make a functions that performs the image processing, segmentation and finally prediction on the image selected using the saved model.
@app.route('/segment_and_predict', methods=['GET', 'POST'])
def segment_and_predict():
# Preprocess cropped license plate image
if request.method == 'POST':
uploaded_file = request.files['image']
original_image = uploaded_file.filename
print('original_image', original_image)
if not uploaded_file:
return render_template('index.html', label="No file")
if uploaded_file.filename != '':
filename = secure_filename(uploaded_file.filename)
file_ext = os.path.splitext(filename)[1]
# if file_ext not in app.config['UPLOAD_EXTENSIONS']:
# abort(400)
uploaded_file.save(os.path.join(app.config['UPLOAD_PATH'], filename))
image = cv2.imread("static/images/"+uploaded_file.filename, cv2.IMREAD_COLOR)
img_lp = cv2.resize(image, (333, 75))
img_gray_lp = cv2.cvtColor(img_lp, cv2.COLOR_BGR2GRAY)
_, img_binary_lp = cv2.threshold(img_gray_lp, 200, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)
img_binary_lp = cv2.erode(img_binary_lp, (3,3))
img_binary_lp = cv2.dilate(img_binary_lp, (3,3))
LP_WIDTH = img_binary_lp.shape[0]
LP_HEIGHT = img_binary_lp.shape[1]
# Make borders white
img_binary_lp[0:3,:] = 255
img_binary_lp[:,0:3] = 255
img_binary_lp[72:75,:] = 255
img_binary_lp[:,330:333] = 255
# Estimations of character contours sizes of cropped license plates
dimensions = [LP_WIDTH/6,
LP_WIDTH/2,
LP_HEIGHT/10,
2*LP_HEIGHT/3]
cv2.imwrite('contour.jpg',img_binary_lp)
# Get contours within cropped license plate
char_list = find_contours(dimensions, img_binary_lp)
dic = {}
characters = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
for i, c in enumerate(characters):
dic[i] = c
output = []
for i, ch in enumerate(char_list): # iterating over the characters
img_ = cv2.resize(ch, (28, 28))
img = fix_dimension(img_)
img = img.reshape(1, 28, 28, 3)
y_ = model.predict_classes(img)[0] # predicting the class
character = dic[y_] #
output.append(character)
plate_number = ''.join(output)
return render_template('results.html', prediction=plate_number)7. Run the project to view the results.
Hope the blog was helpful! Kindly let me know your views in the comment section and suggestions if any.
Also, you can find the complete project on GitHub -https://github.com/PranjaliParnerkar/ANPRDemo
