2.4. Taylor Expansion and Approximation

Taylor expansion is a fundamental tool for approximating functions using their derivatives. 

1D Taylor Expansion.

Let \( f(x) \) be a smooth and differentiable scalar function of one variable \( x \) near a point \( x_0 \). The Taylor expansion of \(f(x)\) around \( x_0 \), up to the second order, is given by: \[ f(x) = f(x_0) + f'(x_0)(x - x_0) + \frac{1}{2}f''(x_0)(x - x_0)^2 + \frac{1}{3!}f'''(x_0)(x - x_0)^3+O(|x - x_0|^4). \] The term \( O(|x - x_0|^4) \) (pronounced "Big-O of \(|x - x_0|^4\)") describes the error in the approximation. Specifically: \[ \frac{O(|x - x_0|^4)}{|x - x_0|^4} \text{ is bounded. }  \] This notation simplifies the representation of higher-order terms, which become egligible compared to the lower-order terms when \( x \) is close to \( x_0 \).  

• In machine learning, Taylor expansion is used to approximate non-linear activation functions, such as the sigmoid function: \[ \sigma(x) = \frac{1}{1 + e^{-x}}. \] For small \( x \), the exponential term \( e^{-x} \) can be approximated using its Taylor series: \[ e^{-x} \approx 1 - x + \frac{x^2}{2}. \] Thus, the sigmoid function can be approximated as: \[ \sigma(x) \approx \frac{1}{2} + \frac{x}{4}. \] This linear approximation simplifies computations during training, particularly in resource-limited environments like edge devices or mobile applications.

Taylor Expansion in 3D

The Taylor expansion approximates a scalar field \(f(\mathbf{r})\) near a point \(\mathbf{r}_0 = (x_0, y_0, z_0)\) by using its derivatives. Mathematically, it can be written as: \[ f(\mathbf{r}) \approx f(\mathbf{r}_0) + \nabla f(\mathbf{r}_0) \cdot (\mathbf{r} - \mathbf{r}_0) + \frac{1}{2} (\mathbf{r} - \mathbf{r}_0)^\top H_f(\mathbf{r}_0) (\mathbf{r} - \mathbf{r}_0), \]  where \(H_f\) is the Hessian matrix of second derivatives:  \[    H_f(\mathbf{r}_0) = \begin{bmatrix}    \frac{\partial^2 f}{\partial x^2}(\mathbf{r}_0) & \frac{\partial^2 f}{\partial x \partial y} (\mathbf{r}_0)& \frac{\partial^2 f}{\partial x \partial z} (\mathbf{r}_0)\\    \frac{\partial^2 f}{\partial y \partial x}(\mathbf{r}_0) & \frac{\partial^2 f}{\partial y^2}(\mathbf{r}_0) & \frac{\partial^2 f}{\partial y \partial z}(\mathbf{r}_0) \\   \frac{\partial^2 f}{\partial z \partial x}(\mathbf{r}_0) & \frac{\partial^2 f}{\partial z \partial y}(\mathbf{r}_0) & \frac{\partial^2 f}{\partial z^2}(\mathbf{r}_0)  \end{bmatrix}.  \]  

• In physics, the electric potential \( V(x, y, z) \) near a point \( \mathbf{r}_0 \) can be approximated using Taylor expansion: \[ V(\mathbf{r}) \approx V(\mathbf{r}_0) + \nabla V(\mathbf{r}_0) \cdot (\mathbf{r} - \mathbf{r}_0) + \frac{1}{2} (\mathbf{r} - \mathbf{r}_0)^\top H_V(\mathbf{r}_0) (\mathbf{r} - \mathbf{r}_0), \] where \( \nabla V=\mathbf{E} \) is the electric field and \( H_V \) is the Hessian of \( V \). 
• In solid mechanics, stress and strain fields are represented as functions of position in three dimensions. The Taylor expansion approximates these fields locally, helping engineers analyze small deformations near a point. Let \( u(\mathbf{r}) \) represent the displacement field at position \( \mathbf{r} = (x, y, z) \). The displacement can be approximated near a reference point \( \mathbf{r}_0 \) as: \[ u(\mathbf{r}) \approx u(\mathbf{r}_0) + \nabla u(\mathbf{r}_0) \cdot (\mathbf{r} - \mathbf{r}_0) + \frac{1}{2} (\mathbf{r} - \mathbf{r}_0)^\top H_u(\mathbf{r}_0) (\mathbf{r} - \mathbf{r}_0), \] where \( H_u \) is the Hessian matrix of second derivatives. Engineers use this approximation to analyze strain and stress tensors in beams, plates, and complex 3D structures under load.