Dilated ResNet-22 Trained on Cityscapes Data

Segment an image of a driving scenario into semantic component classes

Released in 2017, this architecure combines the technique of dilated convolutions with the paradigm of residual networks, outperforming their nonrelated counterparts in image classification and semantic segmentation.

Number of layers: 86 | Parameter count: 15,994,691 | Trained size: 64 MB |

Training Set Information

Cityscapes, a collection of 25,000 annotated images for semantic understanding of urban street scenes.

Performance

This model achieves 68% mean IoU accuracy on the Cityscapes dataset.

Examples

Download Example Notebook

Open in Wolfram Cloud

Resource retrieval

Get the pre-trained net:

In[1]:=

$netevaluate[img_, device_ : "CPU"] := Block[ {net, encData, dec, mean, var, prob}, net = NetModel["Dilated ResNet-22 Trained on Cityscapes Data"]; encData = Normal@NetExtract[net, "input_0"]; dec = NetExtract[net, "Output"]; {mean, var} = Lookup[encData, {"MeanImage", "VarianceImage"}]; NetReplacePart[net, {"input_0" -> NetEncoder[{"Image", ImageDimensions@img, "MeanImage" -> mean, "VarianceImage" -> var}], "Output" -> dec} ][img, TargetDevice -> device] ]$

Label list

Define the label list for this model. Integers in the model’s output correspond to elements in the label list:

In[3]:=

In[6]:=

Advanced visualization

Associate classes to colors using the standard Cityscapes palette:

In[8]:=

colors = Apply[
RGBColor, {{128, 64, 128}, {244, 35, 232}, {70, 70, 70}, {102, 102, 156}, {190, 153, 153}, {153, 153, 153}, {250, 170, 30}, {220, 220, 0}, {107, 142, 35}, {152, 251, 152}, {70, 130, 180}, {220, 20, 60}, {255, 0, 0}, {0, 0, 142}, {0, 0, 70}, {0, 60, 100}, {0, 80, 100}, {0, 0, 230}, {119, 11, 32}}/255., {1}]

Out[8]=

$result[img_, device_ : "CPU"] := Block[ {mask, classes, maskPlot, composition}, mask = netevaluate[img, device]; classes = DeleteDuplicates[Flatten@mask]; maskPlot = Colorize[mask, ColorRules -> indexToColor]; composition = ImageCompose[img, {maskPlot, 0.5}]; Legended[ Row[Image[#, ImageSize -> Large] & /@ {maskPlot, composition}], SwatchLegend[indexToColor[[classes, 2]], labels[[classes]]]] ]$

Inspect the results:

In[11]:=

In[13]:=

$NetInformation[ NetModel["Dilated ResNet-22 Trained on Cityscapes Data"], \ "ArraysElementCounts"]$

Out[14]=

$NetInformation[ NetModel["Dilated ResNet-22 Trained on Cityscapes Data"], \ "ArraysTotalElementCount"]$

Out[15]=

$NetInformation[ NetModel["Dilated ResNet-22 Trained on Cityscapes Data"], \ "LayerTypeCounts"]$

Out[16]=

$NetInformation[ NetModel["Dilated ResNet-22 Trained on Cityscapes Data"], \ "SummaryGraphic"]$

Out[17]=

Out[20]=

Out[21]=

Construction Notebook

Download Construction Notebook

Open in Wolfram Cloud

Requirements

Wolfram Language 11.3 (March 2018) or above

Resource History

Date Created: 22 March 2018
Latest Update: 21 June 2018

Reference

F. Yu, V. Koltun, T. Funkhouser, "Dilated Residual Networks," arXiv:1705.09914 (2017)
Available from: https://github.com/fyu/drn
Rights: BSD 3-Clause License