286 - Object detection using Mask RCNN: end-to-end from annotation to prediction

Code generated in the video can be downloaded from here:

All other code:

This video helps you with end-to-end Mask RCNN project, all the way from annotations to training to prediction. Handling of VGG and Coco style JSON annotations is demonstrated in the video. Code is also made available for both approaches.

You can try other annotation tools like:

[mengyingzhang]

I have been watching your entire series on Mask RCNN this afternoon and learned so much from your video, Sreeni! Super grateful for your wonderful work here! I'll be following your code tomorrow step by step :)

[joaosousa]

I'm sure the is more elegant ways of referencing paths than the crude form I employed here, but it fixed my problems. Running the code in full, no problem was detected. It just wasn't generating the weights. Going line by line or groups of lines by group of lines, following the video, I corrected some things that were wrong, printing things to see were the paths were pointing to. My changes are crude, so you will have to manually change the paths, as your path will not be like mine. All this on the coco style

In my file detection using mask RCNN - end to end\286-marbles_maskrcnn_coco_style_labels.py', I had to make the following changes to make it work:

line before change:
dataset_train.load_data('marble_dataset/train/labels/marbles_two_class_coco_json_format.json', 'marble_dataset/train')

line after change:
detection using mask RCNN - end to end/marble_dataset/train/labels/marbles_two_class_coco_json_format.json', detection using mask RCNN - end to end/marble_dataset/train')


similarly for the 'COCO_WEIGHTS_PATH = '... line, the line after the change became:

COCO_WEIGHTS_PATH = detection using mask RCNN - end to

Also added lines to save the weights (I'm not sure how Sreenivas was able to save it without those lines...), below the last line of the training section of the code, 'model.train(dataset_train, dataset_train, learning_rate=config.LEARNING_RATE, epochs=3, layers='heads')':

# Save the trained model weights
##print(os.path.join(DEFAULT_LOGS_DIR,
model_path = os.path.join(DEFAULT_LOGS_DIR,

print(f"Trained weights saved to {model_path}")


-==-=-

Now on the inference (which I put in a separate file), I've made these changes:

line before change:
model = MaskRCNN(mode='inference', model_dir='logs', config=cfg)

line after the change:
model = MaskRCNN(mode='inference', detection using mask RCNN - end to end/logs', config=cfg)

line before change:
model.load_weights('logs/mask_rcnn_trained_weights.h5', by_name=True)

line after change:
detection using mask RCNN - end to end/logs/mask_rcnn_trained_weights.h5', by_name=True)


-==-=-=-

Now on the section 'Show detected objects in color and all others in B&W' I changed 1 line (below '#line changed 1')and added another (below '#line added 2') to specify an output folder, otherwise it was outputting in my C: in my user. The changed code is below:



#Show detected objects in color and all others in B&W
def color_splash(img, mask):
"""Apply color splash effect.
image: RGB image [height, width, 3]
mask: instance segmentation mask [height, width, instance count]
Returns result image.
"""
# Make a grayscale copy of the image. The grayscale copy still
# has 3 RGB channels, though.
gray = * 255
# Copy color pixels from the original color image where mask is set
if mask.shape[-1] > 0:
# We're treating all instances as one, so collapse the mask into one layer
mask = (np.sum(mask, -1, keepdims=True) >= 1)
splash = np.where(mask, img, gray).astype(np.uint8)
else:
splash = gray.astype(np.uint8)
return splash

import skimage
def detect_and_color_splash(model, image_path=None, video_path=None):
assert image_path or video_path
# line added 2
output_dir = detection using mask RCNN - end to end/viz/'

# Image or video?
if image_path:
# Run model detection and generate the color splash effect
#print("Running on {}".format(img))
# Read image
img =
# Detect objects
r = model.detect([img], verbose=1)[0]
# Color splash
splash = color_splash(img, r['masks'])

#line changed 1
file_name = output_dir +

skimage.io.imsave(file_name, splash)
elif video_path:
import cv2
# Video capture
vcapture = cv2.VideoCapture(video_path)
width =
height =
fps =

# Define codec and create video writer
file_name =
vwriter = cv2.VideoWriter(file_name,
cv2.VideoWriter_fourcc(*'MJPG'),
fps, (width, height))

count = 0
success = True
while success:
print("frame: ", count)
# Read next image
success, img = vcapture.read()
if success:
# OpenCV returns images as BGR, convert to RGB
img = img[..., ::-1]
# Detect objects
r = model.detect([img], verbose=0)[0]
# Color splash
splash = color_splash(img, r['masks'])
# RGB -> BGR to save image to video
splash = splash[..., ::-1]
# Add image to video writer
vwriter.write(splash)
count += 1
vwriter.release()
print("Saved to ", file_name)

detect_and_color_splash(model, detection using mask RCNN - end to



=-=-=-=-

[GuideEver]

Your videos are always long enough, informative and beyond what anybody does. Very polite and understanding delivery. Kudos!

[Kutulu]

Your videos on Mask RCNN is great thank you!

[karlkeller]

Very useful presentation. Nice to see an end-to-end example!

[caiyu]

Keep on learning from your great lectures.

[elmRoz]

Thank you for the presentation, makes the whole thing so much clearer.

[datapro]

Fantastic video! I'm going to run through the code in detail next. Many thanks.

[Dheeraj]

highly informative presentation and explanation!!

[noureddineelharyky]

nice tutorial, thank you.what's about the confusion matrix

[saqibqamar]

Hi,
Excellent explanation step by step to understand MaskRCNN. Pls, let us know how we can calculate the accuracy of generated mask instead of the mAP of object detection?

[justinamichael]

Hello sir... Wonderful lecture... I tried the same... Couldnt see where the trained weights are stored... So they are not loaded during the 'inference' and shows Train mAP value as 0.000. Please address the issue...

[alele]

your presentation is nice and pretty good. would please show how to classify image(Remote sensing image such as sentinel 1, 2 or land sat) by CNN in jupyter notebook

[abhishekwadighare]

Hi Sreeni sir, thank you for sharing mask rcnn workflow, I was using it, I am struggling to deploy it using redis as this model has custom objects and class, could you please help

[GuideEver]

What could be wrong if the training stops at the message "Epoch 1/30"? There is no loss information or anything loading like in your case. This occurs after model.train has been run.
The process where it gets stuck is:
get_next_batch() -> get() -> get() -> wait() -> wait() -> wait(), so I believe it doesn't get the required responses just yet or it's looping.

[mayaltaee]

Please make a video for object detection using Faster R-CNN from annotation to end.

[tmacnba-eger]

sir please make a video about nnunet with costum dataset on colab.

[moumitamoitra]

I have learned a lot from your videos. Could you please upload any video on Cell Nuclei segmentation using Mask R-CNN?

[saqibqamar]

Thanks for video. I have trained MaskRNN model with my custom dataset. Could you tell me about how to measure large image where few objects span to another patch. In that case, objects metrics are not accurate.

[sambitpritam]

I am getting the train mAP as 0.000. Could you please let me know what could be the reason behind this?