Deep Learning Architectures for Perception

Pooling is the second major structural component in CNNs. Pooling layers typically come after convolution layers and their nonlinear activation functions. The primary function of a pooling layer is to replace the output of the convolution layer’s activation map at particular locations with a summary statistic from other spatially local outputs. This helps make the network more robust against small translations in the input, helps improve computational efficiency by reducing the size of the input^††margin: This occurs because it lowers the resolution. , and is useful in enabling the input images to vary in size^††margin: The size of the pooling can be modified to keep the size of the pooling layer output constant. . The most common type of pooling is max pooling, but other types also exist, such as mean pooling. A typical max-pooling operation is shown in Figure 9.4.

Computationally, both max and mean pooling layers operate with the same filtering idea as in the convolution layers. Specifically, a filter of width, $m$, and height, $n$, slides around the layer’s input with a particular stride. The difference between the two comes from the mathematical operation performed by the filter, which as their names suggest are either a maximum element or the mean over the filter. If the output of the convolution layer has $N$ activation maps, the output of the pooling layer will also have $N$ images, since the pooling filter is only applied across the spatial dimensions.

Downstream of the convolution and pooling layers are fully connected layers. These layers make up what is essentially just a standard neural network, which is appended to the end of the network. The function of these layers is to take the output of the convolution and pooling layers, which we can think of as a highly condensed representation of the image, and perform a classification or regression. Generally, the total number of fully connected layers at the end of the CNN will only make up a fraction of the total number of layers.

We can say that a CNN learns how to process images end-to-end because it essentially learns how to perform two steps simultaneously: feature extraction and classification or regression^††margin: In other words, it learns the entire process from image input to the desired output. . In contrast, classical approaches to image processing use hand-engineered feature extractors. Since 2012, the performance of end-to-end learning approaches to image processing have dominated and continue to improve^††margin: In some specific applications, hand-engineered features may still be better. For example, we might use engineering insight to identify a structure to the problem that a CNN could not easily learn. . This continuous improvement has generally been realized with the use of deeper networks with more parameters, and also by combining CNN architectures with other techniques such as Transformers.

AlexNet was the first deep CNN to achieve breakthrough performance on ImageNet, popularizing the use of ReLU activations and dropout regularization. It demonstrated that deeper networks could dramatically outperform traditional hand-engineered methods, marking a turning point in computer vision.

ResNet introduced residual connections that allow information to skip layers, enabling the training of much deeper networks—up to 152 layers—without suffering from vanishing gradients. ResNet showed that network depth itself could be a key factor in improving performance, establishing residual connections as a fundamental architectural component.

YOLO pioneered real-time object detection by treating detection as a single regression problem rather than a multi-stage classification task. YOLO demonstrated how CNN architectures can be adapted for various computer vision tasks beyond image classification, proving that speed and accuracy need not be mutually exclusive.

These architectures have not only achieved state-of-the-art results in their respective domains but have also influenced countless subsequent designs and established important principles for CNN development.

Transformers take inputs in the form of a set or sequence of tokens. A token is a numeric representation of the raw input data, expressed as a vector. The process of converting raw inputs into tokens is called tokenization, and the specific method depends on the data modality. For language tasks, a token might represent a word, subword, or character, generated through a learned dictionary lookup. For computer vision tasks, a token typically represents a patch—a square subset of the input image. For example, we can convert an image patch of size $P\times P$ with $C$ color channels into a token vector of size $C{P}^2$ by flattening the patch into a one-dimensional vector.

We then transform each token into a token embedding vector, which is a high-dimensional latent space representation. The embedding process is typically a learned linear operation that maps tokens into a space where semantically or structurally similar tokens have similar representations. We also add a positional embedding to each token embedding to encode information about the token’s position in the sequence or spatial location. This positional information is crucial because the self-attention mechanism itself is permutation-invariant and does not inherently encode order or position. For example, if a sentence places dog before cat, the model needs that ordering to capture the sentence’s correct meaning. Similarly, in vision tasks, knowing which patch came from the top-left versus bottom-right of an image provides essential spatial context^††margin: Positional embeddings can be learned parameters or fixed sinusoidal functions. Learned embeddings are more flexible but require more data, while fixed embeddings can generalize to sequence lengths not seen during training. .

Once we have transformed the raw input into embedding vectors, we aggregate them as rows of an input matrix, $X^0\in {ℝ}^{N\times D}$, where $N$ is the context size^††margin: In general, larger context sizes will give better performance because we can capture more unique information. However, this comes at the cost of increased computational requirements, which scale quadratically with context size due to the attention mechanism. and $D$ is the dimensionality of the embedding space.

The self-attention mechanism is the key component that allows Transformers to learn relationships between all elements in the input. Each embedding vector in $X^0$ initially represents only a single token and does not contain contextual information from other tokens. For example, when the phrase toy car appears in a sentence, the embedding vector for car does not yet encode the modifier toy. The goal of self-attention is to allow the model to modify each embedding vector based on its relevance to all other tokens in the sequence.

Mathematically, scaled dot-product self-attention is defined as:

where $Q\in {ℝ}^{m\times d_k}$ is the query matrix, $K\in {ℝ}^{n\times d_k}$ is the key matrix, $V\in {ℝ}^{n\times d_v}$ is the value matrix, and $O\in {ℝ}^{m\times d_v}$ is the output matrix. The intuition behind this formulation is that the term $Q{K}^{\top }$ computes the dot product between all pairs of queries and keys, where a larger dot product indicates that a particular pair are similar or relevant to each other. We then apply the softmax function to each row of the resulting matrix to normalize the rows so that their elements sum to one^††margin: The division by $\sqrt{d_k}$, where $d_k$ is the dimensionality of the keys and queries, helps stabilize gradients during training by preventing the dot products from becoming too large. . Finally, multiplication by the value matrix $V$ produces an output where each row is a weighted sum of all value vectors, with weights determined by the normalized attention scores.

In the Transformer architecture, we typically define these matrices as $Q=X{W}_Q$, $K=X{W}_K$, and $V=X{W}_V$, where $X$ is the input matrix with each row corresponding to one embedding vector, and $W_Q$, $W_K$, and $W_V$ are matrices of learnable parameters. While we present the mathematical form using matrix notation for implementation clarity, it is often easier to reason about the transformation of a single embedding vector. Returning to our toy car example, the attention mechanism would modify the car embedding vector into a new vector that captures relevant information from other words in the sentence. The value matrix $V$ provides potential modifications corresponding to each word, and the attention scores determine which modifications to apply. In this case, the relevance weighting would likely show a strong match between car and toy, resulting in an updated embedding that represents toy car rather than a generic car.

A single attention mechanism is limited in the types of relationships it can capture, constrained by the finite parameter matrices $W_Q$, $W_K$, and $W_V$. To increase the model’s expressive capacity, Transformers use multi-head attention, which runs multiple attention mechanisms in parallel, each with unique parameter matrices $W_Q^i$, $W_K^i$, and $W_V^i$ for $i=1,\mathrm{\dots },h$, where $h$ is the number of attention heads. The outputs from all heads are concatenated and multiplied by another learned matrix $W_O$ to produce the final output. This allows the model to attend to different types of relationships simultaneously—for example, one head might learn syntactic relationships while another learns semantic relationships.

Each Transformer layer consists of two main components arranged sequentially: a multi-head self-attention layer followed by a feed-forward network, with both components wrapped in residual connections and layer normalization operations, as shown in Figure 9.5.

After the multi-head self-attention layer processes the input, we add the attention output back to the input in what is called a residual connection:

This residual connection allows gradients to flow more easily during training and helps the network learn identity mappings when beneficial. We then apply layer normalization to stabilize training:

Next, we pass the normalized output through a position-wise feed-forward network, typically implemented as a multi-layer perceptron (MLP). This MLP consists of two linear transformations with a non-linear activation function^††margin: The ReLU or GELU activation functions are commonly used. in between:

where the MLP is applied independently to each position (each row of the input matrix). The feed-forward network typically expands the dimensionality in the first layer and then projects back to the original dimension in the second layer, allowing the network to learn complex non-linear transformations of the attention outputs. This component is followed by another residual connection and layer normalization:

A complete Transformer architecture consists of multiple such layers stacked in sequence, where the output of one layer becomes the input to the next. The depth of the network (number of stacked layers) is a key hyperparameter that significantly impacts model capacity and performance^††margin: Modern Transformers can have dozens or even hundreds of layers. For example, GPT-3 has 96 layers, while some vision models use 32 or more layers. . Note that the dimensions of the inputs and outputs of each Transformer layer are typically the same ($N\times D$), allowing for flexible stacking of arbitrary depth.

After passing through all Transformer layers, we must convert the final embedding representations back into a format suitable for the specific task. The unembedding layer transforms the learned embedding vectors into task-specific outputs. For classification tasks, we typically take a single embedding vector—either a special classification token or an aggregated representation of all tokens—and pass it through a linear layer followed by a softmax function:

where $x$ is the selected embedding vector, and $W$ and $b$ are learned parameters. The output $o$ is a probability distribution over the possible classes, where the vector size matches the number of classes and all elements sum to one.

For other tasks, the unembedding layer may take different forms. In sequence-to-sequence tasks like machine translation, we apply a linear transformation to each position’s embedding to predict the next token in the output sequence. In dense prediction tasks like image segmentation, we may upsample the embeddings back to the original input resolution and apply per-position classification. The specific design of the final layers depends entirely on the task requirements, while the core Transformer layers remain largely the same across different applications.

Vision Transformers adapt the general Transformer architecture to process images by treating image patches as tokens. Given an input image of size $H\times W$ with $C$ color channels, ViT divides the image into a grid of non-overlapping patches of size $P\times P$, resulting in $N=\frac{HW}{P^2}$ patches. Each patch is flattened into a vector of dimension $C\cdot P^2$ and then linearly projected to the embedding dimension $D$ through a learned embedding matrix. Common choices for patch size include $P=16$ or $P=32$, which balance the trade-off between computational cost (smaller patches create more tokens) and the ability to capture fine-grained details^††margin: For a standard $224\times 224$ image with $P=16$, this creates $14\times 14=196$ patch tokens. .

In addition to the patch embeddings, ViT introduces a special learnable class token, denoted $x_{\text{cls}}$, which is prepended to the sequence of patch embeddings. This class token has no correspondence to any image patch but serves as a global representation that the network can use to aggregate information from all patches for classification tasks. The class token’s embedding after passing through all Transformer layers is used as the input to the final classification head. Vision Transformers use learnable 2D positional embeddings that encode each patch’s spatial location in the original image grid. Unlike 1D positional embeddings used in language models, these embeddings must capture 2D spatial structure. The standard approach uses a separate learned embedding for each position in the patch grid, which is added to the corresponding patch embedding. Alternative approaches include using sinusoidal positional encodings extended to 2D or learning relative positional biases. The complete input to the first Transformer layer is thus:

where $x_i^p$ are the linearly embedded patches and $e_i^{\text{pos}}$ are the positional embeddings.

A key characteristic of Vision Transformers is their data requirements compared to CNNs. Because ViTs lack the built-in inductive biases of convolution operations—such as translation equivariance and local connectivity—they require substantially larger training datasets to achieve comparable performance with similar model sizes. When trained on smaller datasets like ImageNet-1K (1.3 million images), ViTs typically underperform similarly-sized CNNs. However, when pre-trained on larger datasets such as ImageNet-21K (14 million images) or even larger proprietary datasets, ViTs can match or exceed CNN performance, and their performance continues to improve with dataset scale in ways that CNNs do not.

This observation has led to a standard training paradigm for ViTs: pre-training on large-scale datasets followed by fine-tuning on smaller, task-specific datasets. Pre-training can be done with supervised learning on labeled image datasets or through self-supervised methods that learn representations from unlabeled images. Self-supervised pre-training methods such as masked image modeling—where random patches are masked and the model learns to predict them—have proven particularly effective for ViTs and enable training on massive unlabeled image corpora. After pre-training, the model can be fine-tuned on downstream tasks with relatively small amounts of labeled data, often requiring only the final classification layers to be retrained while the Transformer layers remain largely fixed or are fine-tuned with small learning rates.

For robotics applications, this pre-training and fine-tuning paradigm is especially valuable. Robotic systems often operate in specialized environments with limited task-specific data, but can leverage visual representations learned from generic internet-scale image datasets. A ViT pre-trained on diverse visual data captures general-purpose features like edge detection, object recognition, and spatial relationships that transfer effectively to robotic tasks like grasping, navigation, and scene understanding.

The choice between Vision Transformers and CNNs for robotic applications involves several practical trade-offs. ViTs excel when large pre-trained models are available and when the task benefits from modeling long-range dependencies across the entire image. Their flexibility in handling variable input sizes and their ability to scale to very large model sizes make them attractive for applications where computational resources are available and high accuracy is critical. However, CNNs remain competitive or superior in scenarios with limited data, real-time constraints on resource-constrained hardware, or tasks where strong spatial inductive biases are beneficial.

Computationally, ViTs have quadratic complexity with respect to the number of patches due to the self-attention mechanism, making them more expensive than CNNs for high-resolution images. Recent variants like Swin Transformer address this by using local attention windows and hierarchical structures, combining some of the efficiency benefits of CNNs with the flexibility of Transformers. For robotics practitioners, the decision often comes down to whether pre-trained models are available for the specific visual domain, the computational budget of the deployment platform, and whether the task requires the global reasoning capabilities that attention mechanisms provide. In many modern robotic systems, hybrid approaches that combine CNN backbones for efficient feature extraction with Transformer layers for high-level reasoning have proven effective, leveraging the complementary strengths of both architectures.

PointNet applies multi-layer perceptrons (MLPs) independently to each point in the cloud. Given $N$ points where each point $p_i\in {ℝ}^3$ represents spatial coordinates, PointNet computes:

where $h_i\in {ℝ}^k$ is the learned feature representation. This point-wise processing maintains permutation invariance because the same transformation applies to each point regardless of input order. The MLP consists of fully connected layers with ReLU activations, progressively increasing dimensionality from 3D coordinates to higher-dimensional spaces (e.g., 64, 128, 1024 dimensions). The MLP parameters are shared across all points, similar to parameter sharing in CNN filters, but without spatial locality constraints. The overall architecture is shown in Figure 9.8.

After extracting point-wise features, PointNet must aggregate these features into a single global representation while preserving permutation invariance. This is achieved through symmetric functions that produce the same output regardless of input ordering. The most commonly used symmetric function in PointNet is the element-wise maximum:

where the max operation is applied element-wise across all feature vectors $h_i$. This is provably permutation invariant because for any permutation $\sigma$:

While alternative symmetric functions like summation or mean could be used, max pooling has the advantage of being selective, allowing the network to focus on the most discriminative features across all points. However, this global aggregation approach means that PointNet captures only global features and may miss important local geometric structures, which motivates the hierarchical extensions in PointNet++, which we will discuss later in this section.

To achieve invariance to geometric transformations such as rotation and translation, PointNet incorporates transformation networks (T-Net) that learn to align point clouds to a canonical orientation. The T-Net is itself a mini-PointNet that predicts a transformation matrix $T\in {ℝ}^{k\times k}$:

This transformation matrix is then applied to either the input coordinates (input transform) or intermediate features (feature transform). For the input transform, $T\in {ℝ}^{3\times 3}$ aligns the spatial coordinates, while for the feature transform, $T\in {ℝ}^{64\times 64}$ normalizes the feature space. To ensure the stability of optimization, a regularization term is added to the loss function that encourages the transformation matrix to be close to orthogonal:

where $||\cdot |{|}_F$ denotes the Frobenius norm. This regularization prevents the transformation from becoming degenerate and helps maintain the geometric properties of the point cloud.

While PointNet effectively captures global features, it struggles to learn local geometric patterns due to its reliance on global max pooling. PointNet++⁸8. Qi, Charles Ruizhongtai, Yi, Li, Su, Hao, Guibas, Leonidas J. “Pointnet++: Deep hierarchical feature learning on point sets in a metric space.” Advances in neural information processing systems 30, 2017. addresses this limitation by introducing a hierarchical architecture that learns features at multiple scales, similar to how CNNs build hierarchical representations through multiple convolutional layers. PointNet++ introduces set abstraction layers that recursively apply PointNet to local regions. Given a point cloud with $N$ points, each set abstraction layer samples $N^{\prime }$ representative points (where ), groups nearby points around each representative point, and applies a PointNet to extract local features. This process creates a hierarchical pyramid of features, where early layers capture fine-grained local details and later layers capture broader geometric patterns.

Set abstraction in PointNet++ consists of three key operations: sampling, grouping, and feature extraction. Sampling uses farthest point sampling (FPS) to select representative points that provide good coverage of the entire point cloud. Given a set of points, FPS iteratively selects the point that is farthest from all previously selected points, ensuring diverse spatial coverage. Grouping then defines local regions around each selected point using either ball query (all points within radius $r$) or $k$-nearest neighbors. This creates local point sets of varying sizes that capture the local geometry around each representative point. Feature extraction applies PointNet to each local region to learn features that capture local geometric patterns while maintaining permutation invariance within each region. For tasks requiring point-wise predictions like semantic segmentation, PointNet++ includes feature propagation layers that upsample features from coarser to finer resolutions. These layers use inverse distance weighted interpolation to propagate features from subsampled points back to the original point cloud:

where $f^{(j)}$ represents features at layer $j$, $d(x,x_i)$ is the distance between points, and $p$ is typically set to 2. Skip connections between corresponding abstraction and propagation layers help preserve fine-grained details, similar to U-Net architectures in image segmentation.

An alternative approach to processing point clouds treats them as graph structures, where points serve as nodes and edges are defined based on spatial proximity or learned relationships. Dynamic Graph Convolutional Neural Networks (DGCNN)⁹9. Wang, Yue, Sun, Yongbin, Liu, Ziwei, Sarma, Sanjay E, Bronstein, Michael M, Solomon, Justin M. “Dynamic graph cnn for learning on point clouds.” ACM Transactions on Graphics (tog) 38(5), 1–12, 2019. exemplify this approach by constructing graphs dynamically in feature space rather than just coordinate space. DGCNN applies edge convolution operations that aggregate information from neighboring points:

where $x_i$ and $x_j$ are feature vectors of connected points, $h_{\theta }$ is a learnable function (typically an MLP), and the edge set $ℰ$ is dynamically updated based on feature similarity after each layer. This dynamic graph construction allows the network to capture both geometric and semantic relationships that evolve as features are learned. The edge convolution operation differs from standard graph convolutions by explicitly modeling the edge information $(x_j-x_i)$, which captures the relative geometric relationships between neighboring points. This approach has shown success in tasks like point cloud classification and part segmentation.

3D Convolutional Neural Networks extend the successful principles of 2D CNNs to volumetric data by operating directly on 3D grids of voxels. While 2D convolutions slide filters across height and width dimensions of images, 3D convolutions add depth as a third spatial dimension, enabling the network to capture spatial relationships to understand the 3D scene. Specifically, a 3D convolution applies a filter of size $(k_x,k_y,k_z)$ across all three spatial dimensions of the input volume. Mathematically, for an input volume $X$ and filter $W$, the 3D convolution operation can be expressed as:

where $(i,j,k)$ represents the spatial position in the output volume and $b$ is the bias term. Note that in practice, both the input $X$ and output $Y$ typically have an additional channel dimension for multi-channel feature maps, and the weight $W$ is a 4D tensor that includes both spatial dimensions and input/output channel dimensions, while the bias $b$ is a vector with one element per output channel. The geometric interpretation is shown in Figure 9.10.

Common kernel sizes include $3\times 3\times 3$ for capturing local 3D patterns and $1\times 1\times 1$ for channel-wise feature mixing without spatial aggregation^††margin: Larger kernels like $5\times 5\times 5$ can capture broader spatial context but significantly increase computational cost due to the cubic scaling of operations. . The key advantage of 3D convolutions over approaches that process 2D slices independently is their ability to learn features that span multiple depths, such as the full 3D shape of objects or volumetric textures. The receptive field in 3D grows cubically with network depth, allowing deeper layers to capture increasingly global context, though this rapid growth must be balanced against increased computational cost.

Real-world point clouds exhibit extreme sparsity when discretized into voxel grids. In many applications such as autonomous driving and indoor scene processing, the majority of 3D space consists of empty air or unoccupied regions. Standard dense 3D convolutions waste significant computation on empty space, making them impractical for large-scale applications. Sparse convolutions address this by computing only on occupied voxels and their neighborhoods, using efficient data structures to maintain compact representations of non-empty regions^††margin: Popular implementations include spconv for PyTorch/TensorFlow and Minkowski Engine for general sparse tensor operations. .

VoxelNet¹⁰10. Zhou, Yin, Tuzel, Oncel. “Voxelnet: End-to-end learning for point cloud based 3d object detection.” In Proceedings of the IEEE conference on computer vision and pattern recognition, 4490–4499, 2018. was one of the pioneering architectures for processing voxel-based representations. The architecture addresses the challenge of processing irregular point clouds by first voxelizing them into a 3D voxel grid, then leveraging 3D convolutions to extract features and detect objects.

A core component of VoxelNet is its Voxel Feature Encoding (VFE) layers, which process the variable number of points within each voxel to produce fixed-size feature representations. The full architecture is shown in Figure 9.11. Given a voxel containing points $\{p_1,p_2,\mathrm{\dots },p_n\}$, where each point $p_i=(x_i,y_i,z_i,r_i)$ includes spatial coordinates and optional reflectance intensity, the VFE layers apply point-wise multi-layer perceptrons to each point independently:

To capture contextual information within each voxel, VoxelNet augments each point with the centroid of all points in the same voxel. For a voxel containing $n$ points, the centroid is computed as $\overline{p}=\frac{1}{n}{\sum }_{i=1}^n{p}_i$, and each point is then represented as the concatenation $[p_i,p_i-\overline{p}]$, providing both absolute and relative spatial information. The VFE layers then aggregate features across all points in the voxel using element-wise max pooling, ensuring permutation invariance:

where $v$ is the final voxel-level feature representation. This aggregation step converts the variable-sized point sets within each voxel into fixed-size feature vectors suitable for subsequent 3D convolution operations.

After voxel feature encoding, VoxelNet applies a series of 3D convolutional middle layers to build hierarchical representations of the scene. These layers follow standard 3D CNN design principles, progressively increasing receptive field size while extracting increasingly abstract features. The sparse nature of voxel occupancy makes this stage well-suited for sparse convolution implementations to improve computational efficiency. The 3D convolutional layers aggregate information across neighboring voxels to capture larger geometric structures, build multi-scale representations through progressive downsampling, and prepare features for the final object detection stage.

While VoxelNet processes full 3D voxels, PointPillars¹¹11. Lang, Alex H, Vora, Sourabh, Caesar, Holger, Zhou, Lubing, Yang, Jiong, Beijbom, Oscar. “Pointpillars: Fast encoders for object detection from point clouds.” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12697–12705, 2019. takes a different approach by using vertical pillars that extend through the entire height of the scene. The key innovation lies in the Pillar Feature Network (PFN), which encodes points within each pillar and then converts the resulting pillar features into a 2D “pseudo-image” representation. This transformation allows PointPillars to leverage mature 2D CNN architectures for subsequent processing, rather than computationally expensive 3D convolutions.

The pillar-based approach offers significant computational advantages by reducing the problem from 3D to 2.5D, enabling the use of optimized 2D convolution operations and existing hardware accelerations designed for image processing. This design choice makes PointPillars particularly suitable for real-time applications where computational efficiency is crucial, achieving inference speeds suitable for autonomous driving while maintaining competitive detection accuracy, as shown in Figure 9.12.

Building on the success of VoxelNet and PointPillars, several variants have been developed to further improve performance and efficiency. SECOND (Sparsely Embedded Convolutional Detection)¹²12. Yan, Yan, Mao, Yuxing, Li, Bo. “Second: Sparsely embedded convolutional detection.” Sensors 18(10), 3337, 2018. combines voxel-based processing with sparse convolution techniques for improved efficiency, significantly reducing computational requirements while maintaining accuracy. Voxel R-CNN extends the voxel-based approach with refined detection stages, while VoxelNet introduces architectural improvements that further push the boundaries of voxel-based 3D detection performance. Similar to point-based architectures, after processing through these voxel-based architectures—VoxelNet’s VFE layers and 3D convolutions, or PointPillars’ pillar encoding and 2D CNNs—the networks produce rich feature representations that encode geometric patterns and spatial relationships across the scene. These learned features serve as input to Region Proposal Networks (RPNs) that generate 3D bounding box proposals for object detection. The key contribution of these approaches lies in demonstrating that the entire pipeline—from raw point cloud processing to 3D object detection—can be trained end-to-end, allowing the networks to learn optimal feature representations specifically for the detection task rather than relying on hand-crafted features. We will explore the details of training object detection networks and designing appropriate loss functions in subsequent chapters on object detection and segmentation.