From Scalars to Tensors
Throughout this course, you have worked with scalars, vectors, and matrices. These are actually all special cases of a more general structure called a tensor. Tensors are the core data structure of modern deep learning. Every input, output, weight, and gradient in a neural network is stored as a tensor. Understanding tensors is understanding the language that AI frameworks speak.
Building Up: Scalars, Vectors, Matrices, and Beyond
You already know the building blocks. A tensor is simply the generalization of these familiar objects to any number of dimensions.
| Structure | Rank (Order) | Dimensions | Example |
|---|---|---|---|
| Scalar | 0 | 0D | A single number: 7.5 |
| Vector | 1 | 1D | A list of numbers: [3, 7, 2] |
| Matrix | 2 | 2D | A grid of numbers: 3x4 table |
| 3D Tensor | 3 | 3D | A cube of numbers |
| 4D Tensor | 4 | 4D | A batch of cubes |
| nD Tensor | n | nD | An n-dimensional array |
The rank (also called order) of a tensor is the number of dimensions, or axes, it has. A scalar has rank 0, a vector has rank 1, a matrix has rank 2, and so on. The shape describes the size along each axis.
Shape and Rank in Practice
Every tensor has a shape, which is a tuple of integers describing how many elements exist along each axis.
Scalar: shape = () rank = 0 (just the number 5.0)
Vector: shape = (3,) rank = 1 ([3, 7, 2])
Matrix: shape = (3, 4) rank = 2 (3 rows, 4 columns)
3D Tensor: shape = (2, 3, 4) rank = 3 (2 layers of 3x4 matrices)
In Python, you can always check a tensor's shape. In NumPy: array.shape. In PyTorch: tensor.shape. In TensorFlow: tensor.shape. The shape tells you everything about the structure of your data.
3D Tensors: Batches of Data
A 3D tensor is a stack of matrices. The most common use in AI is a batch of data samples.
Grayscale image batch: shape = (batch_size, height, width)
Example: shape = (32, 28, 28)
This represents 32 grayscale images, each 28x28 pixels (like the MNIST handwritten digit dataset). Each "slice" along the first axis is one image matrix.
4D Tensors: Color Images
Color images add a channels dimension for red, green, and blue.
Color image batch: shape = (batch, channels, height, width)
Example: shape = (32, 3, 224, 224)
This represents 32 color images, each with 3 color channels (RGB) and a resolution of 224x224 pixels. This is the standard input shape for image classification models like ResNet.
5D Tensors: Video Data
Video adds a frames (time) dimension on top of color images.
Video batch: shape = (batch, frames, channels, height, width)
Example: shape = (8, 16, 3, 112, 112)
This represents 8 video clips, each containing 16 frames of 3-channel color images at 112x112 resolution. Video understanding models like SlowFast operate on tensors with exactly this structure.
Tensor Terminology
A few terms you will encounter frequently:
- Axis (or dimension): One of the indices needed to locate an element. A 3D tensor has 3 axes.
- Shape: The tuple of sizes along each axis, e.g., (32, 3, 224, 224).
- Rank: The total number of axes. Also called order or ndim (number of dimensions).
- Element: A single number within the tensor, accessed by specifying one index per axis.
- Slice: A sub-tensor obtained by fixing one or more axes to specific values.
Why AI Needs Higher Dimensions
Neural networks do not process one data sample at a time. They process batches for efficiency. This immediately adds one dimension. Then the data itself may be multi-dimensional (images have height, width, and channels). Combine these and you quickly reach 4D or 5D tensors.
- Batching: Processing 32 or 64 samples at once for faster GPU utilization
- Channels: Color images have RGB channels; feature maps in CNNs have dozens or hundreds of channels
- Sequences: Text data has a sequence length dimension (one position per token)
- Attention heads: Transformers split representations across multiple attention heads, adding yet another dimension
AI Frameworks and Tensors
All major AI frameworks are built around tensor operations.
| Framework | Tensor Type | Example Creation |
|---|---|---|
| NumPy | ndarray | np.zeros((3, 4, 5)) |
| PyTorch | Tensor | torch.zeros(3, 4, 5) |
| TensorFlow | Tensor | tf.zeros((3, 4, 5)) |
These libraries provide optimized operations on tensors that run on CPUs and GPUs. When you hear "tensor" in the context of TensorFlow or PyTorch, it means exactly the multi-dimensional array we have been describing -- the generalization of vectors and matrices to any number of dimensions.
Summary
- Scalars, vectors, and matrices are tensors of rank 0, 1, and 2 respectively
- A tensor is a multi-dimensional array of numbers with a defined shape and rank
- 3D tensors commonly represent batches of data (e.g., a batch of grayscale images)
- 4D tensors represent batches of multi-channel data (e.g., color image batches)
- 5D tensors represent batches of sequential multi-channel data (e.g., video)
- AI needs higher-dimensional tensors because of batching, channels, sequences, and attention heads
- PyTorch, TensorFlow, and NumPy are all built around efficient tensor operations
Now that you understand tensor structure, the next lesson covers how to manipulate tensors -- reshaping, broadcasting, slicing, and the operations that make deep learning possible.
التالي · الدرس 17 من 18
Tensor Operations in Deep Learning
قراءة 4 دقيقة
أنهيت 16 من 18 درسًا

اسأل Byte عن هذا الدرس
محادثة خاصة حول هذا الدرس. قرأ Byte الدرس كاملًا، ويجيب خلال ثوانٍ، ويُظهر القسم الذي تأتي منه كل إجابة. يمكنك السؤال بأي لغة.
الأسئلة والأجوبة
أسئلة عامة حول هذا الدرس. يجيب AI Tutor فورًا بلغتك، ويراجع فريق FreeAcademy الإجابات، ويمكن لكل المتعلمين قراءتها.

