CNN-based semantic segmentation

Typical CNN-based semantic segmentation networks include fully convolutional networks (FCN) [16], SegNet [17], and pyramid scene parsing network (PSPNet) [18].

FCN [16] is an end-to-end semantic segmentation network proposed by Jonathan Long et al. in 2014. Its distinguishing characteristic is the conversion of a fully connected layer into a convolutional layer. It is capable of processing images of any resolution, successfully overcoming the limitation of a fully connected layer—that is, only being capable of processing images of a specific resolution. However, FCN suffers from certain problems, including the loss of details, smoothening of complex boundaries, and poor detection sensitivity in the case of small object. FCN gain a mIoU accuracy 62.2% on PASCAL VOC 2012. SegNet [17] is an efficient, real-time, and end-to-end semantic segmentation network proposed by Alex Kendall et al. in 2015. It identifies a one-to-one correspondence between the decoder and encoder by using the maximum pooling index of the encoder to perform non-linear upsampling to form a sparse feature image. Following this, a dense feature map is generated via convolution. However, the accuracy of SegNet is lower than that of FCN, and it also suffers from the issue of boundary misdetection. SegNet gain a mIoU accuracy 59.9% on PASCAL VOC 2012. PSPNet [18] was proposed by Zhao Hengshuang in 2017, PSPNet exploiting the capability of global context information by different-region-based context aggregation through a novel pyramid pooling module together with the proposed pyramid scene parsing network. PSPNet’s global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction tasks. A single PSPNet yields mIoU accuracy 85.4% on PASCAL VOC 2012.

Atrous CNN-based semantic segmentation

Typical atrous CNN-based semantic segmentation networks include DeepLab [19] and dilated convolution network (DCN) [20], DeepLab v2 [21] and DeepLab v3 [22].

DeepLab [19] was proposed by the University of California and Google in 2015. Fig 2 shows the structure of the DeepLab model, in which the last two pooling layers of CNN are removed and α = 2 and 4 atrous convolution layers are used to replace a convolutional layer and a fully connected layer, respectively. Thus, the network outputs a 1/8 original-resolution feature map (which is more than that of the feature map outputted by FCN) and then upsamples it to restore the original resolution by using a conditional random field to improve boundary prediction accuracy [23]. DeepLab proposes a fundamental paradigm of cascaded atrous convolution architecture. The segmentation of a 480 × 360 image by using DeepLab requires 110.06 ms, which is 73.95% and 65.29% faster than the SegNet and FCN methods of the codec architecture. Moreover, its $$ \overline{IoU}$$ was tested at 67.1% for VOC2012.

Fig 2

DeepLab model structure diagram.

In 2016, Princeton University and Intel Corporation proposed DCN [20]. The atrous algorithm includes signal decomposition at multiple scales, and atrous convolution is applied at multiple scales to extract feature maps. Fig 3 depicts the structure of the DCN model, comprising a set of cascaded context networks with seven 3 × 3 atrous convolutions and one 1 × 1 convolution based on DeepLab; the model uses multiscale information to improve accuracy. The DCN has an $$ \overline{IoU}$$ of 67.6% in VOC2012, 0.5% higher than DeepLab.

Fig 3

DCN model structure diagram.

In 2016, the University of London and Google proposed DeepLab v2 [21]. Fig 4 shows the model of the DeepLab v2 structure using the atrous spatial pyramid pooling (ASPP) instead of the fully connected layer in CNN. ASPP is composed of four atrous convolutions with atrous ratios of 6, 12, 18, and 24, used in VGG and ResNet obtained an $$ \overline{IoU}$$ of 71.6% and 79.7%, respectively, in VOC 2012. DeepLab v2 can consistently detect large object areas.

Fig 4

Model of DeepLab v2 structure.

In 2017, Google proposed DeepLab v3 [22], as illustrated in Fig 5. This model improved the ASPP module by introducing a 1 × 1 convolutional layer and global pooling layer for maintaining features. It modifies the feature fusion method that stiches the output feature map in the third dimension, and then upsamples it to restore the resolution. DeepLab v3 could obtain an $$ \overline{IoU}$$ of 86.9% in the VOC2012 dataset.

Fig 5

Model of DeepLab v3 structure.

ResNet network

He et al. [15] proposed ResNet, in which the “degeneration” problem is solved through the residual block model. The main idea of the residual block structure in ResNet is to add the direct connection channel into the network, i.e., the idea of a highway network. Fig 6 shows the structure of the ResNet network. After the image is input, the short connection of the dimension matching converts from a dotted line into a solid line. When the dimensions do not match, two equivalent mapping methods can be selected, i.e., directly adding zero to increase the dimension or multiplying the W matrix to project into a new space. This allows the network to theoretically always remain in an optimal state, and thus the performance will not deteriorate with depth.

Fig 6

Structural model of ResNet.

When the model becomes more complicated, some problems could arise; for instance, the accuracy could drop rapidly after saturation, resulting in higher training errors or the stochastic gradient descent (SGD) optimization could become more difficult. To solve these problems, the residual structure is used such that the depth of the network model is unlimited in a large range (currently more than 1000 layers). The error rate of ResNet on Top-5 is 4.49%, and the number of network model parameters is less than those used in VGGNet with outstanding results. ResNet is representative of the current deepening model but its deepening increases the network size, and thus more storage space is required.

Our preliminary research [24] found that, replacing the FCN’s VGG16 backbone with ResNet50, $$ \overline{IoU}$$ on VOC2012 increase from 62.6% to 83.6%.Yu et al. [25] replacing the DeepLab’s VGG16 backbone with ResNet50, $$ \overline{IoU}$$ on VOC2012 increase from 67.1% to 75.6%. Lu et al. [26] replacing the DCN ‘s VGG backbone with ResNet50, $$ \overline{IoU}$$ on VOC2012 increase from 67.6% to 81.0%.

The atrous convolutional network has a greater advantage in improving the accuracy of boundary prediction, and it has improved the performance of semantic segmentation in recent years. Currently, ResNet network has the advantages of high accuracy in classification and an excellent $$ \overline{IoU}$$ combined with multiple semantic segmentation networks. Therefore, further research on the fewer-parameters optimization of the model of the atrous convolution network architecture can fulfill the requirements of accuracy and its speed in the online machine vision detection and identification.

Fewer-parameters optimization modeling of atrous convolution architecture and selection of basic architecture with η_seg and η_main ResNet networks

The accuracy of pixel classification and the amount of segmentation can be measured by using $$ \overline{IoU}$$ and T_seg, respectively. The larger the value of $$ \overline{IoU}$$, the higher is the accuracy of pixel classification. The larger the value of T_seg, the larger is the amount of segmentation. In the fewer-parameters optimization of the semantic segmentation network, the aim of fast semantic segmentation technology is to reduce T_seg under the condition that $$ \overline{IoU}$$ meets the requirements of IoU_min.

We can build an atrous convolutional neural network by determining backbone network structure η_main, backbone network depth d_main, and dense prediction network structure η_seg. Therefore, the $$ \overline{IoU}$$ and T_seg of this network can be expressed as $$ \overline{IoU}\left({\eta }_{\mathrm{seg}},{\eta }_{\mathrm{main}},d_{\mathrm{main}}\right)$$ and T_seg(η_seg,η_main,d_main), respectively. The mathematical model of the fewer-parameters optimization based on atrous convolution architecture network can be derived as follows:

To optimize η_seg, η_main and d_main in Eq (3), they are first combined. In addition, a basic architecture with relatively better η_seg and η_main is selected by comparing the values of $$ \overline{IoU}$$ and T_seg of all networks. In this study, the authors used commonly used semantic segmentation networks, such as FCN [16], improved FCN [24], PSPNet [18], and DeepLabv3 [22], available from the representative computer vision deep learning software package, Amazon GluonCV [28]. The $$ \overline{IoU}$$ was obtained from the typical semantic segmentation dataset, PASCAL VOC [1]. The images were segmented using the GeForce GTX 1080Ti GPU hardware environment at T_seg to obtain segmentation results with the same resolution of 1280 × 1024. Table 1 lists the performance indexes of the main semantic segmentation network [28], where the relatively better indexes are represented in squares.

Table 1

Parameters and indicators of the main semantic segmentation CNN.

Number	CNNs	Dense prediction network ηseg	Backbone ηmain	Depth dmain	PASCAL VOC $$	execution time Tseg (ms)
1	FCN	FCN	VGG	16	62.2%	347
2	Modified FCN	FCN	ResNet	101	83.6%	404
3	PSPNet	PPM	ResNet	101	85.1%	510
4	DeepLabv3	ASPP	ResNet	101	86.2%	495
5	DeepLabv3	ASPP	ResNet	152	86.7%	646

The following points should be noted:

Among networks 1, 2, 3, and 4 in Table 1, with the same η_seg, the $$ \overline{IoU}$$ values for η_main = ResNet is better than the one for η_main = VGG.

η_main and d_main are the same in networks 2, 3, and 4. The $$ \overline{IoU}$$ for η_seg = ASSP is better than those for η_seg = PPM or η_seg = PSPNet.

η_seg and η_main are the same in networks 4 and 5. The Networks 5 $$ \overline{IoU}$$ shows a slight improvement, and T_seg increases significantly when d_main increases by approximately 50%. Therefore, in this study, the basic architectures of η_seg = ASSP and η_main = ResNet were selected.

Improvement of ResNet based on downsampling residual building blocks and fewer-parameters mechanism of d_mian

Fig 7 shows the parameter and index of the atrous convolution architecture network for semantic segmentation. The figure also shows the η_main hidden-layer type, building block structure, and layer operation parameter of ResNet. d_main represents the number of hidden layers. Fig 8 shows the ResNet model and its bottleneck building block structure. The ResNet backbone network is composed of a 7 × 7 convolutional layer, 3 × 3 maximum pooling layer, multiple residual building blocks, and downsampled residual building block. Each stage (i.e., the convolutional layer, pooling layer, and downsampling residual building block) reduces the resolution of the feature map by 1/2. The main and skipped paths of the residual and downsampled residual building blocks are connected in series with at most three hidden layers (depth = 3).

Fig 7

Parameter and index of atrous convolution architecture network for semantic segmentation.

Fig 8

ResNet model and bottleneck building block structure.

a) ResNet model structure, b) bottleneck block structure, and c) downsampled building block structure.

The residual unit in the neural network is composed of multiple hidden layers and skip connections. As shown in Fig 9, the input is an n-dimensional column vector X; the i-th hidden layer has a $$ n_c^i\times n_c^{i-1}$$ matrix, $$ n_c^0$$ with weights Wⁱ. The output of the $$ n_c^i$$-dimensional column vector Hⁱ. Moreover, the residual unit has three hidden layers (i =3), outputs of the $$ n_c^3$$-dimensional column vector, Y, are represented as follows [15]:

Fig 9

Residual learning function of ResNet bottleneck building block.

Suppose the actual value is Y_GT. The fitting target of the three hidden layers in the residual unit is the residual value between Y_GT and X:

However, (4) and (5) are satisfied only when the Y_GT and X dimensions are consistent ($$ n_c^3=n_c^0$$). When $$ n_c^3\ne n_c^0$$, the X dimension must be transformed to be consistent with the Y_GT dimension. The hidden layer with the W_skip as the $$ n_c^3\times n_c^0$$ matrix can be added to the skip path to make W_skipX as the $$ n_c^3$$-dimensional column vector. The residual unit operation is as follows:

The output of the ResNet downsampling residual building block, f_Net[X,(W¹,W²,W³)], differs from the X dimension. Output Y and input X represent feature maps, the numbers and sizes of which vary. In the process of changing the number and size of feature maps, if the coefficient of the pixels of the feature map constantly equals 0, the image features are not used; this reduces the accuracy of feature extraction.

Note that 75% coefficients in W¹, W_skip of f_Net(X), f_skip(X) are 0. The image-feature utilization is only 25% for input X.

Once the downsampling residual building block is optimized, the proportion of input-image features involved in the calculation is improved as much as possible during the change of the number and size of feature maps; this helps to improve the accuracy of feature extraction. Thus, the requirement of IoU_min is satisfied, and d_main and T_seg are reduced, achieving the improvement of ResNet based on downsampling residual building blocks and fewer-parameters optimization of d_main. The fewer-parameters optimization of the atrous convolution architecture network was then performed (which is the contribution of this article).

Improved ResNet structure based on downsampled residual building blocks

Based on the hidden-layer unit function and image-feature utilization listed in Table 2, the reasonable use of these hidden-layer units can help construct optimized ResNet downsampled residual building blocks.

Table 2

Function and image-feature utilization of general hidden layer structure.

Hidden layer structure	Stride ls	Convolutional	Downsample	Change channel number	Image-feature utilization
1×1 Convolutional layer	ls = 1	——	——	○	100%
	ls = 2	——	○	○	25%
3×3 Convolutional layer	ls = 1	○	——	○	100%
	ls = 2	○	○	○	100%
Pooling layer	ls = 2	——	○	——	100%

1) Improved structure of the main path with the 3 × 3 convolutional layer when l_s = 2.

Fig 10 shows the modified structure of the downsampled building block with stride 2, 3 × 3 convolutional layers. In the improved scheme, the skip path does not change while the main path changes. The l_s = 2 downsampling is moved from the 1 × 1 convolutional layer to the 3 × 3 convolutional layers. Table 2 shows that the dimension of output f_Net(X) of the main path does not change in the improved structure. Moreover, the image-feature-utilization rate of the main path increases from the original 25% × 100% × 100% to 100% × 100% × 100%.

Fig 10

Modified structure of the main path with the downsampled building block and stride 2 of 3×3 convolutional layers: a) Original structure and b) modified structure.

2) Improved structure of the skip path with 2 × 2 pooling layer and l_s = 2

Fig 11 shows the downsampled building block with the modofoed structure of the skip path of 2 × 2 pooling layers. In the improved scheme, the main path does not change, while the skip path changes with the addition of a 2 × 2 pooling layer. The l_s = 2 downsampling moves from 1 × 1 convolution layer to the added 2 × 2 pooling layer. Table 2 shows that the dimension of the skip-path output, f_skip(X), does not change in the improved structure. The rate of the image-feature utilization of the skip path increased from the original 25% to 100% × 100%.

Fig 11

The downsampled building block with the modified structure of the 2 × 2-pooling-layer skip path: a) modified structure I and b) modified structure II.

3) The improved structures of the main path with the 3 × 3 convolution layer and the skip path with 2 × 2 pooling layer l_s = 2.

Fig 12 shows the downsampled building blocks of the main path with stride-2 of 3 × 3 convolutional layers and the modified structure of the skipped path with 2 × 2 pooling layers. The improvement scheme combines the improved structures of Figs 10 and 11. Table 2 shows that the dimensions of the outputs of both the main path [f_Net(X)] and skip path [f_skip(X)] do not change in the improved structure. The rate of the image-feature utilization of the main path increases from the original 25% × 100% × 100% to 100% × 100% × 100%. The rate of the image-feature utilization of the skip path increases from the original 25% to 100% × 100%. Figs 10(B), 11(A), 11(B), 12(A) and 12(B) show the improved ResNet structure based on downsampled residual building blocks proposed in this paper.

Fig 12

Downsampled building block of the main path with stride-2 3 × 3 convolutional layers and modified skip path with 2 × 2 pooling layers: a) modified structure I and b) modified structure II.

Improved model on PASCAL VOC segmentation task

We replace the backbone of DeepLabv3 with the five types of improved structure to obtain the DeepLabv3 improved model. Evaluate the DeepLabv3 improved model with its $$ \overline{IoU}$$ of the PASCAL VOC dataset [1] segmentation task. In the PASCAL VOC segmentation task, IoU_min = 85%. We verified the accuracy of the $$ \overline{IoU}$$ index. Segmentation time T_seg was obtained for a 1280 × 1024 image, and the comparison of the $$ \overline{IoU}$$ and T_seg of the network model of the atrous convolution architecture before and after improving is presented in Table 3.

Table 3

IoU and segmentation time of DeepLab v3 with a modified ResNet in PASCAL VOC segmentation task.

	Method	Backbone ηmain	Depth dmain = 50		Depth dmain = 101		Depth dmain = 152
			$$	Tseg	$$	Tseg	$$	Tseg
1	DeepLab v3 [22]	ResNet [15]	84.7%	285 ms	86.2%	495 ms	86.7%	719 ms
2	DeepLab v3*	ResNet v2 [29]	84.4%	276 ms	85.9%	469 ms	86.7%	677 ms
3	DeepLab v3*	stride 2 3×3 conv layers mainpath modified structure	85.0%	288 ms	86.7%	474 ms	87.2%	680 ms
4	DeepLab v3*	2×2 pooling layers skippath modified structure I	85.4%	292 ms	87.1%	492 ms	87.6%	691 ms
5	DeepLab v3*	2×2 pooling layers skippath modified structure II	85.4%	299 ms	87.0%	502 ms	87.6%	704 ms
6	DeepLab v3*	stride 2 3×3 conv layers mainpath and 2×2 pooling layers skippath modified structure I	86.6%	296 ms	88.1%	513 ms	88.2%	698 ms
7	DeepLab v3*	stride 2 3×3 conv layers mainpath and 2×2 pooling layers skippath modified structure II	86.5%	303 ms	88.1%	523 ms	88.2%	711 ms

From Table 3, we can draw the following conclusions:

1) Replacing the backbone of DeepLabv3 with the five types of improved structure proposed in this paper, the accuracy of DeepLabv3 is improved under the same d_main situation. For example, depth d_main = 50, and each improved structure shows increase in the $$ \overline{IoU}$$ by 85.0%, 85.4%, 85.4%, 86.6%, and 86.5%. However, the values of T_seg also show sight increases from 285 to 288, 292, 299, 296, and 303 ms. This shows that improving the downsampled residual building block structure based on the improvement in the utilization of image features is effective for increasing $$ \overline{IoU}$$, and the adoption of the modified structure shown in Fig 12(A) is relatively better for improving the model structure.

2) The improvement in the downsampled residual building block structure produces an increase in $$ \overline{IoU}$$, while reducing d_main and T_seg. For the same IoU_min, after the improved downsampled residual building block structure is used, T_seg is reduced, i.e., to achieve fewer-parameters optimization of the network model relying on the atrous convolution architecture. For example, in Table 3, when ResNet + ASPP is used, with d_main = 101, we obtained $$ \overline{IoU}$$ = 86.2%, T_seg = 495 ms. In additio, when the structure in Fig 12(A) is used to improve the structure with ASPP, for d_main = 50, we obtained $$ \overline{IoU}$$ = 86. 6% and T_seg = 296 ms. Thus, we observed improvements in $$ \overline{IoU}$$ and T_seg indicators, and the fewer-parameters optimization effect depending on the network model of the atrous convolution architecture is evident. For example, when ResNet + ASPP is used, for d_main = 152, we obtained $$ \overline{IoU}$$ = 86.7% and T_seg = 719 ms. Morevover, when using the improved structure with ASPP (Fig 12(A)), for d_main = 50, $$ \overline{IoU}$$ = 86.7% but T_seg was reduced to 58.8%.

Fig 13 shows semantic segmentation result of the modified DeepLabv3 in the PASCAL VOC task.

Fig 13

PASCAL VOC semantic segmentation by modified DeepLabv3.

Improved model on CITYSCAPES pixel-level semantic labeling task

We trained each DeepLabv3 (with ResNet50 or modified ResNet50) on the pixel-level semantic labeling task of the CITYSCAPES dataset [30]. To assess performance, CITYSCAPES rely on the PASCAL VOC intersection-over-union metric IoU. Owing to the two semantic granularities, i.e. classes and categories, CITYSCAPES report two separate mean performance scores: IoU_class and IoU_category. In either case, pixels labeled as void do not contribute to the score. We also report the segmentation time of each network run on a GeForce GTX 1080Ti GPU and an Intel i7-5960X CPU. Table 4 presents the performances of different methods on the CITYSCAPES pixel-level semantic labeling task. Table 5 presents the individual classes IoU of different methods in the CITYSCAPES pixel-level semantic labeling task. Fig 14 shows some CITYSCAPES pixel-level semantic labeling results obtained with the DeepLabv3 with different backbone.

Fig 14

CITYSCAPES pixel-level semantic labeling by modified DeepLabv3.

Table 4

Performances of different methods on the CITYSCAPES pixel-level semantic labeling task.

Method	IoUclass(%)	IoUcategory(%)	Segmentation time Tseg(ms)
DeepLab [19]	64.5	83.7	——
DeepLab v2 [21]	70.4	86.4	——
DeepLab v3 with ResNet50 [22]	78.7	89.8	494ms
DeepLab v3 with modified ResNet50 [ours]	79.9	91.6	497ms

Table 5

Individual classes IoU of DeepLabv3 with different backbone in CITYSCAPES pixel-level semantic labeling task.

Method	road	sidewalk	building	wall	fence	pole	traffic light	traffic sign	vegetation	terrain
DeepLab v3 with ResNet50 [22]	98.6%	86.4%	92.8%	52.4%	59.7%	59.6%	72.5%	78.3%	93.3%	72.8%
DeepLab v3 with modified ResNet50 [ours]	98.7%	87.0%	93.5%	57.9%	60.4%	70.9%	77.9%	81.4%	93.7%	72.8%
Method	sky	person	rider	car	truck	bus	train	motor cycle	bicycle	average
DeepLab v3 with ResNet50 [22]	95.5%	85.4%	70.0%	95.7%	75.4%	84.1%	75.1%	68.7%	75.0%	78.7%
DeepLab v3 with modified ResNet50 [ours]	95.6%	87.9%	75.3%	96.1%	65.8%	80.5%	78.7%	72.8%	70.8%	79.9%

From Tables 4 and 5, we can draw the following conclusions:

1) The modified ResNet effectively improved DeepLabv3 performance in CITYSCAPES pixel-level semantic labeling task. Replacing the DeepLabv3’s ResNet50 backbone with our modified ResNet50 (stride 2 3×3 conv layers mainpath and 2×2 pooling layers skippath modified structure I), the IoU_class is improved from 78.7% to 79.9%, and the IoU_category is improved from 89.8% to 91.6%. The segmentation time T_seg increased slightly from 494ms to 497 ms.

2) The modified ResNet also improved most of the individual classes IoU of the DeepLabv3. Especially, the wall, pole, traffic light, and rider IoU have increased by more than 5%. The traffic sign, person, train, and motor cycle IoU have increased by more than 2%.

Improved model on machine vision banknote anti-counterfeit segmentation

The following experiment was conducted using the 2019 version of the CNY machine vision segmentation and recognition application. In this task, the quantitative value for IoU_min was 90%. The experimental model includes an improved ResNet structure before and after the atrous convolution architecture network. In the DeepLab v3 semantic segmentation model, the dense prediction network was set as ASPP, the backbone network was replaced with various ResNet structures, the entire semantic segmentation network was trained with the 2019 version of the CNY image dataset. We verified the accuracy of η_seg and T_seg.

1) The machine vision system adopts an industrial camera, MV-CA013-10GC, with an MVL-HF2528M-6MP industrial lens, a bar light source with FOV = 18.33°, and an imaging resolution of 1280 × 1024.

2) For the image-dataset construction, the object plane is perpendicular to the optical axis. The object distance is 400 mm and various denominations, with 25 pieces of front and back renminbi, were collected with various angle. The 2019 version of the CNY image dataset (200 image) comprises the serial numbers on the CNY, magnetic optical variable ink (OVMI), security line denomination numbers, five patterns of visual features, etc..

3) Hardware condition: The hardware is the same as that used in the aforementioned experiment: GeForce GTX 1080Ti GPU.

Table 6 presents the $$ \overline{IoU}$$, execution time, and $$ A{P}^{Io{U}_T=0.1}$$ of DeepLab v3 with a modified ResNet in the segmentation task of CNY anti-counterfeit features. Fig 15 shows CNY anti-counterfeit feature segmentation with diffuse reflection machine vision and modified DeepLab v3.

Fig 15

CNY anti-counterfeit features segmentation by modified DeepLabV3.

Table 6

IOU and execution time of Deeplab v3 with a modified ResNet in CNY anti-counterfeit feature-segmentation task.

	Method	Backbone ηmain	Depth dmain = 50		Depth dmain = 152
			$$	Tseg	$$	Tseg
1	DeepLab v3 [22]	ResNet [15]	90.3%	287 ms	91.3%	722 ms
2	DeepLab v3*	ResNet v2 [29]	89.9%	282 ms	91.1%	714 ms
3	DeepLab v3*	stride 2 3×3 conv layers mainpath and 2×2 pooling layers skippath modified structure I	91.3%	295 ms	$$	735 ms

From Table 6, we can draw the following conclusions: The modified ResNet effectively improved DeepLabv3 performance in CNY anti-counterfeit feature-segmentation task. Replacing the DeepLabv3’s ResNet backbone with our modified ResNet (stride 2 3×3 conv layers mainpath and 2×2 pooling layers skippath modified structure I), the accuracy $$ \overline{IoU}$$ is improved under the same d_main situation. For example, in depth d_main = 50, our modified ResNet make $$ \overline{IoU}$$ increase from 90.3% to 91.3%, and T_seg show sight increases from 287 ms to 295 ms.

The conclusions drawn in Table 6 are consistent with the conclusions drawn for the PASCAL VOC dataset (Table 3). That is, the improved downsampled residual building block structure based on the improvement of the image-feature-utilization rate is helpful for improving $$ \overline{IoU}$$. Moreover, by adopting the improved structure with relatively better $$ \overline{IoU}$$ and T_seg, the fewer-parameters optimization of the network model of the atrous convolution architecture is evident.