avatar khanh than

SOFTWARE ENGINEER

To keep tracking my skills & accomplisments

07 Nov 2024

● GenAI

Matrix And Pytorch

Matrixes are fundamental in AI, especially in machine learning and deep learning, as they provide a key foundation for creating tensors, which is widely used to represent vast multi-dimensional data. Most tensor computations are based on matrix operations, so understanding of matrixes is essential for buildind or fine tuning AI models.

  1. Matrix in programming:
    • Below is a matrix $A$ with $m$ rows and $n$ columns.
    \[A_{m\times n} = \begin{bmatrix} a_{11} & a_{12} & \dots & a_{1n} \\ a_{21} & a_{22} & \dots & a_{2n} \\ \vdots & \vdots & \vdots & \vdots \\ a_{m1} & a_{m2} & \dots & a_{mn} \end{bmatrix}\]

    • If $m = n$, $A$ is a square matrix, which has only one principal diagonal ($[a_{11}, a_{22}, …, a_{nn}]$).

    • In programming, matrix is two-dimentional 2D array $[[a_{11}, \ a_{12}], [a_{21}, \ a_{22}]]$.

    • In math of linear algebra (Đại số tuyến tính) : matrix is used in set of linear equations (hệ pt tuyến tính) :

      \(\begin{cases} a_{11}x_{1} + a_{12}x_{2} = y_{1}\\ a_{21}x_{1} + a_{22}x_{2} = y_{2}\\ \end{cases}\) $ \ \sim \ $ \(\begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ \end{bmatrix} \times \begin{bmatrix} x_{1}\\ x_{2}\\ \end{bmatrix} = \begin{bmatrix} y_{1}\\ y_{2}\\ \end{bmatrix}\)

  2. Vector space ($\mathbb{R^{n}}$) vs Matrix:

    • Matrix in vector space: a matrix represents a transformation between vector spaces, a kind of a bridge or mapping between vector spaces. This means a matrix can help “move” a vector in one vector space to another vector space. For example, vector $\vec{X}$ in vector space $R_{x}$ is transformed to another vector space $R_{y}$ by matrix $A_{2 \times 2}$. This is a key concept in neuron network to build a bridge (model).

      \[A \cdot \vec{X} = \vec{Y}\]
  3. Tensor ($T$) vs Matrix :

    • Tensor is a data structure used mainly in tensorflow or Pytorch:

      • Tensor rank 1 with only 1 element = scalar: torch.tensor([[2]])
      • Tensor rank 1 with over 2 elements = vector: torch.tensor([[1, 2]])
      • Tensor rank 2 = 2D matrix: torch.tensor([[1, 2], [2, 3]]).float()
      • Tensor rank 3 = 3D matrix: torch.tensor([[0, 0, 1], [0, 1, 0], [1, 0, 0]]).float()
      • Tensor rank n = nD matrix: used to represent for vast multi-dim data.

  4. Identity matrix ($I$) : is a square matrix where all items on principal diagonal are 1, other items are zeros.

    • import torch
      $I$ = torch.eye(3)
      print($I$)
    \[I_{3\times 3} = I_{3} = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}\]

  5. Transpose ($A^{T}$) : is a matrix created by flipping a matrix A over its diagonal (rows become columns).

    • A = torch.tensor([[1, 2], [3, 4]]).float()
      A_T = A.T # [[1., 3.], [2., 4.]]

  6. Matrix Multiplication ($AB$): row(A) x column(B)

    \[\begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} \times \begin{bmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{bmatrix} = \begin{bmatrix} a_{11}b_{11} + a_{12}b_{21} & a_{11}b_{12} + a_{12}b_{22} \\ a_{21}b_{11} + a_{22}b_{21} & a_{21}b_{12} + a_{22}b_{22} \end{bmatrix}\]
    • A = torch.tensor([[1, 2], [0, 1]]).float()
      B = torch.tensor([[1, 3], [0, 2]]).float()
      AB = torch.matmul(A, B) # tensor([[1., 7.], [0., 2.]])

  7. Determinant ($det$): only for square matrices:

    \[A_{2\times 2} = \begin{bmatrix} a & b\\ c & d \end{bmatrix}\]
    • det($A$) = $ad - bc$
    • matrix = torch.tensor([[1, 2], [3, 4]], dtype=torch.float32)
      det$A$ = torch.det(matrix) # output is “-2”

  8. Matrix Inverse ($A^{-1}$) :
    • If det($A$) $\neq 0$, $A$ should has an inverse matrix $A^{-1}$, then multiplying a matrix by its inverse results in an identity matrix ($A \times A^{-1} = I$).
    • A = torch.tensor([[4, 7], [2, 6]]).float()
      A_inv = torch.inverse($A$)
      AB = torch.matmul(A, A_inv) # output is $I_{2}$

  9. Trace ($Tr(A)$) : sum of the elements along main diagonal of a square matrix or along the first main diagonal of a rectangular matrix.
    • matrix = torch.tensor([[1, 2, 4], [3, 4, 1]], dtype=torch.float32)
    • torch.trace(matrix) # tensor(5.)
  10. Rank ($rank(A)$) : The minimum number of linearly independent (độc lập tuyến tính) rows or columns (if rows are multiples of each other, they are not linearly independent).
    • A = torch.tensor([[1, 2], [2, 4], [2, 1], [0, 3], [3, 5]]).float()
      rank_A = torch.linalg.matrix_rank(A) # 2

  11. Singular Matrix (Ma trận đơn):

    • A matrix includes only linearly dependent rows. (phụ thuộc tuyến tính), rows are multiples of each other.
    • Determinant of singular matrix is always zero.

  12. Eigenvectors ($\nu$) : Vectors that are stretched by a matrix without changing direction.
    Eigenvalues ($\lambda$) : are scalar values or amount the eigenvectors are scaled by the matrix.
    • $A \nu$ = $\lambda \nu$
      Rearrange the equation: $(A - \lambda I) \nu$ = 0
      To have non-zero eigenvectors $\nu \neq 0$, so det(A - $\lambda I$) = 0

    • A = torch.tensor([[3, 0], [0, 2]]).float()
      $\lambda$, $\nu$ = torch.linalg.eig(A)
      print($\lambda$) # tensor([3.+0.j, 2.+0.j])
      print($\nu$) # [[1.+0.j, 0.+0.j], [0.+0.j, 1.+0.j]]
    • Complex number : = a + bi with $i^{2}$ = -1 (not mentioned here).
    for i in range(m):
        for j in range(k):
            for x in range(n):
                c[i][j] += a[i][x] * b[x][j]

Định thức (determinant) của ma trận vuông bậc \(n\) \(A\) được định nghĩa đệ quy như sau:

Khái niệm:

Tính chất của định thức:

\[\begin{vmatrix} a_{11}+b_{11} & a_{12}+b_{12}\\ a_{21} & a_{22} \end{vmatrix}=\begin{vmatrix} a_{11} & a_{12}\\ a_{21} & a_{22} \end{vmatrix}+\begin{vmatrix} b_{11} & b_{12} \\ a_{21} & a_{22} \end{vmatrix}\]

Ma trận liên hợp (adjoint matrix) là ma trận chuyển vị của ma trận các cofactor của một ma trận vuông.

Ma trận nghịch đảo:

Share on: