YOLO V8 Pose Trained on MS-COCO Data

Detect and localize human joints in an image

YOLO (You Only Look Once) Version 8 by Ultralytics is the latest version of the YOLO models. Just like its predecessor, YOLO Version 5, YOLO Version 8 is an anchor-free model that was trained with mosaic augmentation. It features the use of new "C2f" blocks, which employ additional dense connections between bottleneck modules. Although YOLO models are historically object detection models, these models perform both object detection and human joint keypoint regression at the same time.

Training Set Information

Microsoft COCO, a dataset for image recognition, segmentation, captioning, object detection and keypoint estimation, consisting of more than three hundred thousand images.

Model Information

Examples

Download Example Notebook

Open in Wolfram Cloud

Resource retrieval

Get the pre-trained net:

In[1]:=

Pick a non-default net by specifying the parameters:

In[4]:=

Evaluation function

Write an evaluation function to scale the result to the input image size and suppress the least probable detections:

In[27]:=

$netevaluate[net_, img_, detectionThreshold_ : .25, overlapThreshold_ : .5] := Module[ {imgSize, w, h, probableObj, probableBoxes, probableScores, probableKeypoints, max, scale, padx, pady, results, nms, x1, y1, x2, y2}, (*define image dimensions*) imgSize = 640; {w, h} = ImageDimensions[img]; (*get inference*) results = net[img]; (*filter by probability*) (*very small probability are thresholded*) probableObj = UnitStep[ results["Objectness"] - detectionThreshold]; {probableBoxes, probableScores, probableKeypoints} = Map[Pick[#, probableObj, 1] &, {results["Boxes"], results["Objectness"], results["KeyPoints"]}]; If[Or[Length[probableBoxes] == 0, Length[probableKeypoints] == 0], Return[{}]]; max = Max[{w, h}]; scale = max/imgSize; {padx, pady} = imgSize*(1 - {w, h}/max)/2; (*transform keypoints coordinates to fit the input image size*) probableKeypoints = Apply[ { { Clip[Floor[scale*(#1 - padx)], {1, w}], Clip[Floor[scale*(imgSize - #2 - pady)], {1, h}] }, #3 } &, probableKeypoints, {2} ]; (*transform coordinates into rectangular boxes*) probableBoxes = Apply[ ( x1 = Clip[Floor[scale*(#1 - #3/2 - padx)], {1, w}]; y1 = Clip[Floor[scale*(imgSize - #2 - #4/2 - pady)], {1, h}]; x2 = Clip[Floor[scale*(#1 + #3/2 - padx)], {1, w}]; y2 = Clip[Floor[scale*(imgSize - #2 + #4/2 - pady)], {1, h}]; Rectangle[{x1, y1}, {x2, y2}] ) &, probableBoxes, 1 ]; (*gather the boxes of the same class and perform non- max suppression*) nms = ResourceFunction["NonMaximumSuppression"][ probableBoxes -> probableScores, "Index", MaxOverlapFraction -> overlapThreshold]; results = Association[]; results["ObjectDetection"] = Part[Transpose[{probableBoxes, probableScores}], nms]; results["KeypointEstimation"] = Part[probableKeypoints, nms][[All, All, 1]]; results["KeypointConfidence"] = Part[probableKeypoints, nms][[All, All, 2]]; results ];$

Basic usage

Obtain the detected bounding boxes with their corresponding classes and confidences as well as the locations of human joints for a given image:

In[29]:=

The "ObjectDetection" key contains the coordinates of the detected objects as well as their confidences and classes:

In[32]:=

The "KeypointConfidence" key contains the confidences for each person’s keypoints:

In[34]:=

Out[36]=

HighlightImage[testImage, AssociationThread[
Range[Length[Transpose@keypoints]] -> Transpose@keypoints], ImageLabels -> None, ImageLegends -> labels]

Out[38]=

$getSkeleton[personKeypoints_] := Line[DeleteMissing[ Map[personKeypoints[[#]] &, {{1, 2}, {1, 3}, {2, 4}, {3, 5}, {1, 6}, {1, 7}, {6, 8}, {8, 10}, {7, 9}, {9, 11}, {6, 7}, {6, 12}, {7, 13}, {12, 13}, {12, 14}, {14, 16}, {13, 15}, {15, 17}}], 1, 2]]$