📘 Lecture 14 — Solution of Linear System of Equations and Matrix Inversion Relaxation Method
📖 Overview: This lecture introduces the Relaxation Method, an iterative technique for solving linear systems of equations. It is based on systematically reducing the largest residual (error) in each equation to zero by updating the corresponding variable. This method is attributed to Southwell and is particularly useful for achieving fast convergence when equations are properly ordered.
🗂️ Topics Covered
The lecture begins by defining the residual of each equation in a linear system and establishing the core principle of the relaxation method: iteratively reducing the largest residual to zero. It covers the procedure for reordering equations so that the largest diagonal coefficients appear on the diagonal. The lecture then details the formula for calculating the increment to a variable based on its residual and diagonal coefficient. Finally, several worked examples are presented to demonstrate the step-by-step iterative process, from initial residuals to final solution convergence.
📝 Lecture Summary
Relaxation Method
This is an iterative method attributed to Southwell. Consider the system of equations: [ a_{11}x_1 + a_{12}x_2 + \dots + a_{1n}x_n = b_1 ] [ a_{21}x_1 + a_{22}x_2 + \dots + a_{2n}x_n = b_2 ] [ \vdots ] [ a_{n1}x_1 + a_{n2}x_2 + \dots + a_{nn}x_n = b_n ]
Let ( X^{(p)} = (x_1^{(p)}, x_2^{(p)}, \dots, x_n^{(p)})^T ) be the solution vector after the ( p )-th iteration.
If ( R_i^{(p)} ) denotes the residual of the ( i )-th equation, defined by: [ R_i^{(p)} = b_i - a_{i1}x_1^{(p)} - a_{i2}x_2^{(p)} - \dots - a_{in}x_n^{(p)} ] We can improve the solution vector successively by reducing the largest residual to zero at that iteration. This is the basic idea of the relaxation method.
To achieve fast convergence, we take all terms to one side and then reorder the equations so that the largest negative coefficients in the equations appear on the diagonal.
Now, if at any iteration, ( R_i ) is the largest residual in magnitude, then we give an increment to ( x_i ); ( a_{ii} ) being the coefficient of ( x_i ): [ dx_i = \frac{R_i}{a_{ii}} ] In other words, we change ( x_i ) to ( (x_i + dx_i) ) to relax ( R_i ), i.e., to reduce ( R_i ) to zero.
🔑 Definition — Residual: The residual ( R_i ) of the ( i )-th equation is the difference between the constant term ( b_i ) and the current computed value of the left-hand side. It measures the error in that equation at a given iteration. 📐 Formula: ( R_i = b_i - \sum_{j=1}^{n} a_{ij}x_j ) and ( dx_i = R_i / a_{ii} ) → The increment to a variable equals its residual divided by its diagonal coefficient.
Example 1
Solve the system of equations: [ 6x_1 - 3x_2 + x_3 = 11 ] [ 2x_1 + x_2 - 8x_3 = -15 ] [ x_1 - 7x_2 + x_3 = 10 ] by the relaxation method, starting with the vector (0, 0, 0).
Solution: At first, transfer all terms to the right-hand side and reorder the equations so that the largest coefficients appear on the diagonal: [ 0 = 11 - 6x_1 + 3x_2 - x_3 ] [ 0 = 10 - x_1 + 7x_2 - x_3 ] [ 0 = -15 - 2x_1 - x_2 + 8x_3 ] after interchanging the 2nd and 3rd equations.
Starting with the initial solution vector (0, 0, 0), we find the residuals: [ R_1 = 11, R_2 = 10, R_3 = -15 ] The largest residual in magnitude is ( R_3 = -15 ). We introduce a change, ( dx_3 ) in ( x_3 ): [ dx_3 = \frac{R_3}{a_{33}} = \frac{15}{8} = 1.875 ]
The iterative process continues until all residuals are very small. The iteration table is:
| Iteration | R1 | R2 | R3 | Max R_i | dx_i | x1 | x2 | x3 |
|---|---|---|---|---|---|---|---|---|
| 0 | 11 | 10 | -15 | -15 | 1.875 | 0 | 0 | 0 |
| 1 | 9.125 | 8.125 | 0 | 9.125 | 1.5288 | 0 | 0 | 1.875 |
| 2 | 0.0478 | 6.5962 | -3.0576 | 6.5962 | -0.9423 | 1.5288 | 0 | 1.875 |
| 3 | -2.8747 | 0.0001 | -2.1153 | -2.8747 | -0.4791 | 1.0497 | -0.9423 | 1.875 |
| 4 | -0.0031 | 0.4792 | -1.1571 | -1.1571 | 0.1446 | 1.0497 | -0.9423 | 1.875 |
| 5 | -0.1447 | 0.3346 | 0.0003 | 0.3346 | -0.0478 | 1.0497 | -0.9423 | 2.0196 |
| 6 | 0.2881 | 0.0000 | 0.0475 | 0.2881 | 0.0480 | 1.0497 | -0.9901 | 2.0196 |
| 7 | -0.0001 | 0.048 | 0.1435 | 0.1435 | -0.0179 | 1.0017 | -0.9901 | 2.0196 |
| 8 | 0.0178 | 0.0659 | 0.0003 | - | - | 1.0017 | -0.9901 | 2.0017 |
The numerical solution is ( x_1 = 1.0017, x_2 = -0.9901, x_3 = 2.0017 ). The exact solution is ( x_1 = 1.0, x_2 = -1.0, x_3 = 2.0 ).
📌 Example: In iteration 0, the largest residual is -15 for equation 3. The increment for ( x_3 ) is calculated as ( dx_3 = -(-15)/8 = 15/8 = 1.875 ). After updating ( x_3 ) to 1.875, new residuals are computed for iteration 1: ( R_1 = 11 - 6(0) + 3(0) - 1(1.875) = 9.125 ), ( R_2 = 10 - 1(0) + 7(0) - 1(1.875) = 8.125 ), ( R_3 = -15 - 2(0) - 1(0) + 8(1.875) = 0 ). The process continues until all residuals are small.
Example 2
Solve by relaxation method, the equation: [ 10x - 2y - 2z = 6 ] [ -x - 10y - 2z = 7 ] [ -x - y + 10z = 8 ]
Solution: The residuals ( r_1, r_2, r_3 ) are given by: [ r_1 = 6 - 10x + 2y + 2z ] [ r_2 = 7 + x - 10y + 2z ] [ r_3 = 8 + x + y - 10z ]
The operation table is:
| x | y | z | r1 | r2 | r3 |
|---|---|---|---|---|---|
| 1 | 0 | 0 | -10 | 1 | 1 |
| 0 | 1 | 0 | 2 | -10 | 1 |
| 0 | 0 | 1 | 2 | 2 | -10 |
The relaxation table is:
| x | y | z | r1 | r2 | r3 |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 6 | 7 | 8 |
| 0 | 0 | 1 | 8 | 9 | -2 |
| 0 | 1 | 0 | 10 | -1 | -1 |
| 1 | 0 | 0 | 0 | 0 | 0 |
Explanation: (1) In L4, the largest residual is 8. To reduce it, we give an increment of ( 8/10 = 0.8 \approx 1 ) to ( z ). The resulting residuals are obtained by L4 + (1)L3, i.e., line L5. (2) In line L5, the largest residual is 9. Increment = ( 9/10 = 0.9 \approx 1 ) to ( y ). The resulting residuals (= L6) are L5 + 1.L2. (3) In line L6, the largest residual is 10. Increment = ( 10/10 \approx 1 ) to ( x ). The resulting residuals (= L7) are L6 + 1.L1.
The exact solution is ( x = 1, y = 1, z = 1 ).
Example 3
Solve the system by relaxation method: [ 9x - y + 2z = 7 ] [ x + 10y - 2z = 15 ] [ 2x - 2y - 13z = -17 ]
Solution: The residuals ( r_1, r_2, r_3 ) are given by: [ r_1 = 9 - 9x + y - 2z ] [ r_2 = 15 - x - 10y + 2z ] [ r_3 = -17 - 2x + 2y + 13z ]
Operation table:
| x | y | z | r1 | r2 | r3 |
|---|---|---|---|---|---|
| 1 | 0 | 0 | -9 | -1 | -2 |
| 0 | 1 | 0 | 1 | -10 | 2 |
| 0 | 0 | 1 | -2 | 2 | 13 |
Relaxation table:
| x | y | z | r1 | r2 | r3 |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 9 | 15 | -17 |
| 0 | 0 | 1 | 7 | 17 | -4 |
| 0 | 1 | 0 | 8 | 7 | -2 |
| 0.89 | 0 | 0 | -0.01 | 6.11 | -3.78 |
| 0 | 0.61 | 0 | 0.6 | 0.01 | -2.56 |
| 0 | 0 | 0.19 | 0.22 | 0.39 | -0.09 |
| 0 | 0.039 | 0 | 0.259 | 0 | -0.012 |
| 0.028 | 0 | 0 | 0.007 | -0.028 | -0.068 |
| 0 | 0 | 0.00523 | -0.00346 | -1.01754 | -0.00001 |
Then: [ x = 0.89 + 0.028 = 0.918 ] [ y = 1 + 0.61 + 0.039 = 1.649 ] [ z = 1 + 0.19 + 0.00523 = 1.19523 ]
Substituting the values: [ r_1 = 9 - 9(0.918) + 1.649 - 2(1.19523) = -0.00346 ] [ r_2 = 15 - 0.918 - 10(1.649) + 2(1.19523) = -0.1754 ] [ r_3 = -17 - 2(0.918) + 2(1.649) + 13(1.19523) = -0.00001 ] Which agrees with the final residuals.
💡 Why this matters: The relaxation method provides a systematic way to iteratively solve linear systems by focusing on the equation with the largest error, making it efficient for certain types of problems.
⭐ Key Takeaways
The relaxation method is an iterative technique that works by computing residuals for each equation and then systematically reducing the largest residual to zero by updating the corresponding variable. The increment for a variable is calculated as its residual divided by its diagonal coefficient. To improve convergence, equations should be reordered so that the largest coefficients appear on the diagonal. The process continues until all residuals become sufficiently small, indicating that the solution has been reached. This method is particularly effective when the system has a strong diagonal dominance.
🧠 Quick Revision Questions
- What is a residual in the context of the relaxation method, and how is it calculated?
- What is the formula for calculating the increment (dx_i) to a variable x_i during the relaxation process?
- Why is it important to reorder equations in the relaxation method?
- In Example 1, why was the first increment applied to x₃ instead of x₁ or x₂?
- When can the iterative process in the relaxation method be stopped?
📘 Lecture 15 — Solution of Linear System of Equations and Matrix Inversion
📖 Overview: This lecture focuses on methods for finding the inverse of a matrix, which is essential for solving linear systems of equations in the form A = (B). It covers the Gaussian Elimination Method and the Gauss-Jordan Method, two fundamental techniques for computing matrix inverses through row operations and augmented matrices. Understanding these methods is critical for numerical analysis and solving real-world engineering problems.
🗂️ Topics Covered
This lecture introduces matrix inversion as a method for solving linear systems of equations. It explains the Gaussian Elimination Method in two stages: first reducing the matrix to an upper triangular form, then reducing it to an identity matrix to obtain the inverse. It also covers the Gauss-Jordan Method, which directly reduces the matrix to an identity matrix using elementary row operations. Several detailed examples are provided for both methods, including the importance of partial pivoting for accuracy.
📝 Lecture Summary
Solution of Linear System of Equations and Matrix Inversion
Consider a system of equations in the form A = (B). One way of writing its solution is in the form (X) = [A]⁻¹(B). Thus, the solution to the system can also be obtained if the inverse of the coefficient matrix [A] is known. The product of two square matrices is an identity matrix [A][B] = [I], then [B] = [A]⁻¹ and [A] = [B]⁻¹. Every square non-singular matrix will have an inverse. Gauss elimination and Gauss-Jordan methods are popular among many methods available for finding the inverse of a matrix.
Gaussian Elimination Method
In this method, if A is a given matrix, we first place an identity matrix, whose order is same as that of A, adjacent to A, forming an augmented matrix. The inverse of A is computed in two stages. In the first stage, A is converted into an upper triangular form, using the Gaussian elimination method. In the second stage, the upper triangular matrix is reduced to an identity matrix by row transformations. All these operations are also performed on the adjacently placed identity matrix. When A is transformed into an identity matrix, the adjacent matrix gives the inverse of A. To increase accuracy, it is essential to employ partial pivoting.
Example: Use the Gaussian elimination method to find the inverse of the matrix A = [[1,1,1],[4,3,-1],[3,5,3]].
Solution: First, place an identity matrix adjacent to the given matrix: [[1,1,1,1,0,0],[4,3,-1,0,1,0],[3,5,3,0,0,1]].
Stage I (Reduction to upper triangular form): In the 1st column, 4 is the largest element, so interchange R1 and R2 (partial pivoting): [[4,3,-1,0,1,0],[1,1,1,1,0,0],[3,5,3,0,0,1]].
Divide R1 by 4: [[1,3/4,-1/4,0,1/4,0],[1,1,1,1,0,0],[3,5,3,0,0,1]].
Perform R₂ – R₁ → R₂: [[1,3/4,-1/4,0,1/4,0],[0,1/4,5/4,1,-1/4,0],[3,5,3,0,0,1]].
Perform R₃ – 3R₁ → R₃: [[1,3/4,-1/4,0,1/4,0],[0,1/4,5/4,1,-1/4,0],[0,11/4,15/4,0,-3/4,1]].
For the second column pivot, max (1/4, 11/4) is 11/4, so interchange R2 and R3: [[1,3/4,-1/4,0,1/4,0],[0,11/4,15/4,0,-3/4,1],[0,1/4,5/4,1,-1/4,0]].
Divide R2 by the pivot a₂₂ = 11/4: [[1,3/4,-1/4,0,1/4,0],[0,1,15/11,0,-3/11,4/11],[0,1/4,5/4,1,-1/4,0]].
Perform R₃ – (1/4)R₂ → R₃: [[1,3/4,-1/4,0,1/4,0],[0,1,15/11,0,-3/11,4/11],[0,0,10/11,1,-2/11,-1/11]].
Finally, divide R3 by (10/11): [[1,3/4,-1/4,0,1/4,0],[0,1,15/11,0,-3/11,4/11],[0,0,1,11/10,-1/5,-1/10]]. This is the upper triangular form.
Stage II (Reduction to an identity matrix): Perform (-1/4)R₃ + R₁ and (-15/11)R₃ + R₂: [[1,3/4,0,11/40,1/5,-1/40],[0,1,0,-3/2,0,1/2],[0,0,1,11/10,-1/5,-1/10]].
Finally, perform R₁ – (3/4)R₂ → R₁: [[1,0,0,7/5,1/5,-2/5],[0,1,0,-3/2,0,1/2],[0,0,1,11/10,-1/5,-1/10]].
Thus, A⁻¹ = [[7/5, 1/5, -2/5], [-3/2, 0, 1/2], [11/10, -1/5, -1/10]].
Example: Find the inverse of [[2,1,1],[3,2,3],[1,4,9]] using Gauss elimination.
Solution: Augmented matrix: [[2,1,1,1,0,0],[3,2,3,0,1,0],[1,4,9,0,0,1]].
After row operations R₂ – (3/2)R₁ and R₃ – (1/2)R₁: [[2,1,1,1,0,0],[0,1/2,3/2,-3/2,1,0],[0,7/2,17/2,-1/2,0,1]].
Perform R₃ – 7R₂: [[2,1,1,1,0,0],[0,1/2,3/2,-3/2,1,0],[0,0,-2,10,-7,1]].
If the inverse matrix is [[x₁₁,x₁₂,x₁₃],[x₂₁,x₂₂,x₂₃],[x₃₁,x₃₂,x₃₃]], the system is equivalent to three systems. Solving by back substitution:
From the first system: 2x₁₁ + x₂₁ + x₃₁ = 1, (1/2)x₂₁ + (3/2)x₃₁ = -3/2, -2x₃₁ = 10 → x₃₁ = -5, then x₂₁ = 12, x₁₁ = -3.
From the second system: 2x₁₂ + x₂₂ + x₃₂ = 0, (1/2)x₂₂ + (3/2)x₃₂ = 1, -2x₃₂ = -7 → x₃₂ = 7/2, then x₂₂ = -17/2, x₁₂ = 5/2.
From the third system: 2x₁₃ + x₂₃ + x₃₃ = 0, (1/2)x₂₃ + (3/2)x₃₃ = 0, -2x₃₃ = 1 → x₃₃ = -1/2, then x₂₃ = 3/2, x₁₃ = -1/2.
The inverse matrix is (1/2)[[-6,5,-1],[24,-17,3],[-10,7,-1]].
Example: Find the inverse of [[4,1,2],[2,3,-1],[1,-2,2]] using Gauss elimination.
Solution: Augmented matrix: [[4,1,2,1,0,0],[2,3,-1,0,1,0],[1,-2,2,0,0,1]].
After row operations R₂ – (1/2)R₁ and R₃ – (1/4)R₁: [[4,1,2,1,0,0],[0,5/2,-2,-1/2,1,0],[0,-9/4,3/2,-1/4,0,1]].
This gives three systems to solve. Solving by back substitution yields the inverse matrix. The final inverse is (1/3)[[-4,6,7],[5,-8,-2],[7,-6,-10]].
Gauss - Jordan Method
This method is similar to the Gaussian elimination method, with the essential difference that stage I of reducing the given matrix to an upper triangular form is not needed. The given matrix can be directly reduced to an identity matrix using elementary row operations.
Example: Find the inverse of A = [[1,1,1],[4,3,-1],[3,5,3]] by Gauss-Jordan method.
Solution: Augmented matrix: [[1,1,1,1,0,0],[4,3,-1,0,1,0],[3,5,3,0,0,1]].
Perform R₂ – 4R₁ → R₂: [[1,1,1,1,0,0],[0,-1,-5,-4,1,0],[3,5,3,0,0,1]].
Perform R₃ – 3R₁ → R₃: [[1,1,1,1,0,0],[0,-1,-5,-4,1,0],[0,2,0,-3,0,1]].
Perform R₂ + R₁ → R₁ and R₃ + 2R₂ → R₃: [[1,0,-4,-3,1,0],[0,-1,-5,-4,1,0],[0,0,-10,-11,2,1]].
Divide the third row by -10: [[1,0,-4,-3,1,0],[0,-1,-5,-4,1,0],[0,0,1,11/10,-1/5,-1/10]].
Perform R₁ + 4R₃ → R₁ and R₂ + 5R₃ → R₂: [[1,0,0,7/5,1/5,-2/5],[0,-1,0,-3/2,0,1/2],[0,0,1,11/10,-1/5,-1/10]].
Multiply R2 by -1: [[1,0,0,7/5,1/5,-2/5],[0,1,0,3/2,0,-1/2],[0,0,1,11/10,-1/5,-1/10]]. Note: there's a sign error in the text; the correct inverse is A⁻¹ = [[7/5, 1/5, -2/5], [3/2, 0, -1/2], [11/10, -1/5, -1/10]].
💡 Why this matters: Both methods produce the same inverse matrix, but Gauss-Jordan is more direct as it eliminates the need for a separate back-substitution stage, though it requires more row operations.
⭐ Key Takeaways
The most critical concepts from this lecture are: matrix inversion via the augmented matrix method, where an identity matrix is placed alongside the given matrix and row operations transform the given matrix into an identity matrix. The Gaussian elimination method proceeds in two distinct stages: first reducing to upper triangular form, then to identity form. The Gauss-Jordan method accomplishes this in one continuous process. Partial pivoting (reordering rows to bring the largest element to the pivot position) is essential for numerical accuracy. The final inverse matrix appears in the columns originally occupied by the identity matrix.
🧠 Quick Revision Questions
- What is the augmented matrix in the Gaussian elimination method for finding matrix inverse?
- Why is partial pivoting important in matrix inversion methods?
- In Gaussian elimination, what is the goal of Stage I and Stage II?
- What is the key difference between the Gaussian elimination method and the Gauss-Jordan method?
- Given an upper triangular augmented matrix from Stage I, how do you solve for the inverse matrix elements?
📘 Lecture 16 — Eigen Value Problems
📖 Overview: This lecture introduces eigenvalue problems for square matrices and presents numerical methods for computing eigenvalues and eigenvectors. It focuses on the Power Method for finding the largest eigenvalue and its corresponding eigenvector, which is essential when solving the characteristic polynomial becomes computationally prohibitive.
🗂️ Topics Covered
The lecture begins with the definition of eigenvalues and eigenvectors, then explains why numerical methods are preferred over solving the characteristic polynomial. It introduces the Power Method procedure for finding the largest eigenvalue, demonstrates the method through multiple examples with full iterations, and discusses the mathematical theory behind why the method converges. Finally, it mentions using the inverse matrix to find the smallest eigenvalue.
📝 Lecture Summary
Eigen Value Problems
Let [A] be an n x n square matrix. Suppose there exists a scalar λ and a vector X = (x₁, x₂, ..., xₙ)ᵀ such that A = λ(X). Then λ is the eigenvalue and X is the corresponding eigenvector of matrix [A]. This can also be written as A - λI = (O), representing a set of n homogeneous equations with non-trivial solution provided |A - λI| = 0. This determinant, on expansion, gives an n-th degree polynomial called the characteristic polynomial of [A], which has n roots.
Finding the roots of the characteristic equation is laborious, so we look for better computational methods. The lecture discusses Power Method and Jacobi's Method for real and real-symmetric matrices.
🔑 Definition — Eigenvalue: A scalar λ such that A = λ(X) for some non-zero vector X 🔑 Definition — Eigenvector: The vector X satisfying A = λ(X) for the corresponding eigenvalue λ 🔑 Definition — Characteristic Polynomial: The n-th degree polynomial obtained from expanding |A - λI| = 0
Power Method
To compute the largest eigenvalue and corresponding eigenvector of the system A = λ(X), where [A] is real, symmetric, or unsymmetric, the Power Method is widely used.
Procedure: Step 1: Choose the initial vector such that the largest element is unity. Step 2: The normalized vector v⁽⁰⁾ is pre-multiplied by matrix [A]. Step 3: The resultant vector is again normalized. Step 4: This iteration process continues, repeatedly pre-multiplying the new normalized vector by [A] until required accuracy is obtained.
At the end, u⁽ᵏ⁾ = [A]v⁽ᵏ⁻¹⁾ = qₖv⁽ᵏ⁾, where qₖ is the desired largest eigenvalue and v⁽ᵏ⁾ is the corresponding eigenvector.
📌 Example: Find the eigenvalue of largest modulus and associated eigenvector of matrix A = [[2,3,2],[4,3,5],[3,2,9]] using Power Method.
Starting with v⁽⁰⁾ = (1,1,1)ᵀ: 1st iteration: u⁽¹⁾ = [A]v⁽⁰⁾ = (7,12,14)ᵀ, normalize using 14: v⁽¹⁾ = (1/2, 6/7, 1)ᵀ
2nd iteration: u⁽²⁾ = (39/7, 67/7, 171/7)ᵀ ≈ (5.571, 9.571, 24.429)ᵀ, normalize by 12.2143: v⁽²⁾ ≈ (0.456140, 0.783626, 1.0)ᵀ
3rd iteration: u⁽³⁾ ≈ (5.263158, 9.175438, 11.935672)ᵀ, normalize by 11.935672: v⁽³⁾ ≈ (0.44096, 0.776874, 1.0)ᵀ
4th iteration: u⁽⁴⁾ ≈ (5.18814, 9.07006, 11.86036)ᵀ, normalize by 11.8636: v⁽⁴⁾ ≈ (0.437435, 0.764737, 1.0)ᵀ
5th iteration: u⁽⁵⁾ ≈ (5.16908, 9.04395, 11.84178)ᵀ, normalize by 11.84178: v⁽⁵⁾ ≈ (0.436512, 0.763732, 1.0)ᵀ
After rounding, the largest eigenvalue λ ≈ 11.84 and eigenvector X ≈ (0.44, 0.76, 1.00)ᵀ
📌 Example: Find first three iterations for matrix A = [[7,6,-3],[-12,-20,24],[-6,-12,16]]
With v⁽⁰⁾ = (1,1,1)ᵀ: 1st iteration: u⁽¹⁾ = (10,-8,-2)ᵀ, normalize by 10: v⁽¹⁾ = (1, -0.8, -0.2)ᵀ, q₁ = 10
2nd iteration: u⁽²⁾ = (2.8,-0.8,0.4)ᵀ, normalize by 2.8: v⁽²⁾ = (1, -0.2857, 0.1428)ᵀ, q₂ = 2.8
3rd iteration: u⁽³⁾ = (4.8574,-2.8588,-0.2868)ᵀ, normalize by 4.8574: v⁽³⁾ = (1, -0.5885, -0.0590)ᵀ
📌 Example: Find first three iterations for matrix A = [[1,-1,0],[-2,4,-2],[0,-1,2]] with x⁽⁰⁾ = (-1,2,1)ᵀ
1st iteration: u⁽¹⁾ = (-3,8,0)ᵀ, normalize by 8: x⁽¹⁾ = (-3/8, 1, 0)ᵀ, q₁ = 8
2nd iteration: u⁽²⁾ = (-1.375, 4.75, -1)ᵀ, normalize by 4.75: x⁽²⁾ = (-0.28947, 1, -0.2152)ᵀ, q₂ = 4.75
3rd iteration: u⁽³⁾ = (-1.28947, 4.99998, -1.42104)ᵀ, normalize by 4.99998: x⁽³⁾ = (-0.25789, 1, -0.28420)ᵀ
Mathematical Foundation
Let λ₁, λ₂, ..., λₙ be distinct eigenvalues of n×n matrix [A] with |λ₁| > |λ₂| > ... > |λₙ|, and v₁, v₂, ..., vₙ be corresponding eigenvectors. Any vector v in the space spanned by eigenvectors can be written as v = c₁v₁ + c₂v₂ + ... + cₙvₙ.
Pre-multiplying by A: Av = λ₁(c₁v₁ + c₂(λ₂/λ₁)v₂ + ... + cₙ(λₙ/λ₁)vₙ)
After r iterations: Aʳv = λ₁ʳ[c₁v₁ + c₂(λ₂/λ₁)ʳv₂ + ... + cₙ(λₙ/λ₁)ʳvₙ]
The eigenvalue λ₁ can be computed as the limit of the ratio of corresponding components of Aʳ⁺¹v and Aʳv: λ₁ = lim(r→∞) (Aʳ⁺¹v)ₚ/(Aʳv)ₚ for any component p.
Finding the Smallest Eigenvalue
For finding the eigenvalue of least magnitude, apply the Power Method to [A⁻¹]. Since A⁻¹ = (1/λ)(X), the inverse matrix has eigenvalues that are reciprocals of the eigenvalues of [A].
⭐ Key Takeaways
The Power Method provides an iterative numerical approach for finding the dominant eigenvalue (largest in magnitude) and its associated eigenvector without solving the characteristic polynomial. The method requires choosing an initial vector, repeatedly multiplying by matrix A, and normalizing—the scaling factor at each step approximates the eigenvalue. For convergence, the eigenvalues must be real and distinct, with the dominant eigenvalue being strictly larger in magnitude than all others. If the smallest eigenvalue is needed, the Power Method is applied to the inverse matrix instead.
🧠 Quick Revision Questions
- What is the core iterative procedure of the Power Method for finding the largest eigenvalue?
- Why does the ratio of corresponding components of Aʳ⁺¹v and Aʳv converge to the dominant eigenvalue?
- In the example with matrix [[2,3,2],[4,3,5],[3,2,9]], what was the largest eigenvalue after five iterations?
- How would you modify the Power Method to find the smallest eigenvalue of a matrix?
- What condition must the eigenvalues satisfy for the Power Method to converge properly?
📘 Lecture 17 — Jacobi’s Method
📖 Overview: Jacobi’s method is an iterative technique for computing all eigenvalues and eigenvectors of a real symmetric matrix. It works by systematically annihilating off-diagonal elements through a series of orthogonal (rotation) transformations until the matrix becomes diagonal, from which the eigenvalues can be directly read. This method is highly recommended for its numerical stability and ability to produce the complete eigen-system.
🗂️ Topics Covered
The lecture introduces Jacobi’s method for eigenvalue computation of real symmetric matrices, starting with the definition of orthogonal matrices and the underlying diagonalization principle. It details the construction of rotation matrices to zero out the largest off-diagonal element, including the formula for the rotation angle. Three complete worked examples demonstrate the iterative process, including how to compute the eigenvectors from the product of the rotation matrices.
📝 Lecture Summary
Jacobi’s Method
An n×n matrix [A] is said to be orthogonal if [A]ᵀ[A] = [I], meaning [A]ᵀ = [A]⁻¹. For computing all eigenvalues and eigenvectors of a real symmetric matrix, Jacobi’s method is highly recommended. It is based on the property that if [A] is an n×n real symmetric matrix, its eigenvalues are real, and there exists an orthogonal matrix [S] such that the diagonal matrix D = [S⁻¹][A][S]. This diagonalization is carried out by applying a series of orthogonal transformations S₁, S₂, ..., Sₙ.
Let A be an n×n real symmetric matrix. Suppose aᵢⱼ is the numerically largest element among the off-diagonal elements of A. We construct an orthogonal matrix S₁ defined with:
- sᵢᵢ = cosθ, sᵢⱼ = -sinθ
- sⱼᵢ = sinθ, sⱼⱼ = cosθ
- All remaining off-diagonal elements are zero, and remaining diagonal elements are unity.
After the transformation D₁ = S₁⁻¹ A S₁ = S₁ᵀ A S₁, the elements at positions (i,j) and (j,i) are annihilated. The new diagonal elements dᵢᵢ and dⱼⱼ and the off-diagonal element dᵢⱼ are computed from:
- dᵢⱼ = 0 only if aᵢⱼ cos 2θ + (aⱼⱼ - aᵢᵢ) sin 2θ / 2 = 0
🔑 Definition — Orthogonal matrix: A matrix [A] is orthogonal if [A]ᵀ[A] = [I], or equivalently [A]ᵀ = [A]⁻¹. 📐 Formula: tan 2θ = 2aᵢⱼ / (aᵢᵢ - aⱼⱼ) → This determines the rotation angle that zeroes out the off-diagonal pair. 📌 Example: For matrix A = [[1, 2], [3, 2], [2, 1]], the largest off-diagonal element is a₁₃ = a₃₁ = 2. Then tan 2θ = 2(2)/(1-1) = 4/0 = ∞, giving θ = π/4. The orthogonal matrix S₁ = [[1/√2, 0, -1/√2], [0, 1, 0], [1/√2, 0, 1/√2]] annihilates d₁₃ and d₃₁.
However, though it creates a new pair of zeros, it also introduces non-zero contributions at formerly zero positions.
Example 1 — Full Jacobi Iteration
Find all eigenvalues and corresponding eigenvectors of A = [[1, 2], [3, 2], [2, 1]] by Jacobi’s method.
Step 1: Largest off-diagonal element is a₁₃ = a₃₁ = 2. With θ = π/4, S₁ = [[1/√2, 0, -1/√2], [0, 1, 0], [1/√2, 0, 1/√2]]. Then D₁ = S₁⁻¹ A S₁ = [[3, 2, 0], [2, 3, 0], [0, 0, -1]]. Sum of diagonal elements of D₁ equals sum of diagonal elements of A = 3+2+1=6.
Step 2: Largest off-diagonal element of D₁ is d₁₂ = d₂₁ = 2. Again tan 2θ = 2(2)/(3-3) = ∞, giving θ = π/4. S₂ = [[1/√2, -1/√2, 0], [1/√2, 1/√2, 0], [0, 0, 1]]. D₂ = S₂⁻¹ D₁ S₂ = [[5, 0, 0], [0, 1, 0], [0, 0, -1]]. This is diagonal, so eigenvalues are 5, 1, -1.
The eigenvectors are columns of S = S₁S₂ = [[1/√2, -1/√2, -1/√2], [1/√2, 1/√2, 0], [1/√2, -1/√2, 1/√2]].
💡 Why this matters: The sum of diagonal elements (trace) remains invariant under each rotation, providing a check for numerical accuracy.
Example 2 — Multiple Rotations Required
Find all eigenvalues of A = [[2, -1, 0], [-1, 2, -1], [0, -1, 2]] by Jacobi’s method.
Step 1: Choose a₁₂ = -1 as largest element. tan 2θ = 2(-1)/(2-2) = ∞, giving θ = π/4. With cosθ = sinθ = 1/√2, S₁ = [[1/√2, -1/√2, 0], [1/√2, 1/√2, 0], [0, 0, 1]]. D₁ = [[1, 0, -1/√2], [0, 3, -1/√2], [-1/√2, -1/√2, 2]].
Step 2: Largest element is d₁₃ = -1/√2. tan 2θ = 2(-1/√2)/(1-2) = -√2/(-1) = √2, giving θ = 27°22′41″. S₂ = [[0.888, 0, -0.459], [0, 1, 0], [0.459, 0, 0.888]]. D₂ = [[0.634, -0.325, 0], [-0.325, 3, -0.628], [0, -0.628, 2.365]].
Step 3: Largest off-diagonal is d₂₃ = -0.628. tan 2θ = 2(-0.628)/(3-2.365) = -1.256/0.635 = -1.978, giving θ = -31°35′24″. S₃ = [[1, 0, 0], [0, 0.852, 0.524], [0, -0.524, 0.852]]. D₃ = [[0.634, -0.277, 0], [0.277, 3.386, 0], [0, 0, 1.979]].
Further rotations would fully diagonalize, but approximate eigenvalues are 0.634, 3.386, and 1.979.
Example 3 — Real Application
Find eigenvalues and eigenvectors of A = [[1, 1/2, 1/3], [1/2, 1/3, 1/4], [1/3, 1/4, 1/5]] by Jacobi’s method.
Step 1: Largest off-diagonal is a₁₂ = a₂₁ = 1/2. tan 2θ = 2(1/2)/(1 - 1/3) = 1/(2/3) = 3/2 = 1.5. θ = tan⁻¹(1.5)/2 = 28.155°. S₁ = [[0.882, -0.472, 0], [0.472, 0.882, 0], [0, 0, 1]]. D₁ = [[1.268, 0.000, 0.412], [0.000, 0.066, 0.063], [0.412, 0.063, 0.200]]. Sum of diagonals = 1.53, matching original trace.
Step 2: Largest off-diagonal is d₁₃ = d₃₁ = 0.412. tan 2θ = 2(0.412)/(1.268 - 0.200) = 0.824/1.068 = 0.772. θ = tan⁻¹(0.772)/2 = 18.834°. S₂ = [[0.946, 0, -0.323], [0, 1, 0], [0.323, 0, 0.946]]. D₂ = [[1.408, 0.020, -0.001], [0.020, 0.066, 0.060], [-0.001, 0.060, 0.059]]. Sum of diagonals = 1.53.
Eigenvalues are approximately 1.408, 0.066, and 0.059. The eigenvectors are columns of S = S₁S₂ = [[0.8343, -0.472, -0.2848], [0.446, 0.88, -0.1524], [0.323, 0, 0.946]].
⭐ Key Takeaways
Jacobi’s method transforms a real symmetric matrix into diagonal form through successive orthogonal (rotation) transformations, each zeroing the largest off-diagonal element. The rotation angle θ is determined by tan 2θ = 2aᵢⱼ/(aᵢᵢ - aⱼⱼ), which ensures the chosen pair gets annihilated while preserving the matrix's trace. Although each rotation creates new non-zero off-diagonal elements, continued iterations drive all off-diagonal elements toward zero. The eigenvalues appear as diagonal entries of the final matrix, and the eigenvectors are columns of the product of all rotation matrices.
🧠 Quick Revision Questions
- What condition must the matrix satisfy for Jacobi's method to be applicable?
- How is the rotation angle θ computed in Jacobi's method?
- Why does the sum of diagonal elements remain unchanged after each rotation?
- How are the eigenvectors obtained at the end of the Jacobi iteration?
- What happens to the off-diagonal elements that were already zeroed when a new rotation annihilates a different pair?
📘 Lecture 18 — Finite Differences
📖 Overview: This lecture introduces the fundamental concept of finite differences, which are crucial for numerical interpolation and extrapolation of tabulated functions. It defines and explores the relationships between forward, backward, and central difference operators, as well as the shift, average, and differential operators, establishing the symbolic foundations necessary for deriving interpolation formulas.
🗂️ Topics Covered
This lecture begins with the definition of interpolation and extrapolation for tabulated functions with equally spaced abscissas. It then introduces the forward difference operator (Δ), shows how to construct a forward difference table, and derives expressions for function values in terms of leading differences. The backward difference operator (∇) and central difference operator (δ) are similarly defined, with their own difference tables. Finally, the shift operator (E), average operator (μ), and differential operator (D) are introduced, along with crucial symbolic relationships linking all these operators (e.g., Δ = E - 1, E = e^(hD)).
📝 Lecture Summary
Introduction
Finite differences are essential for numerical techniques involving tabulated values of functions. For a function (y = f(x)) with given values ((x_k, y_k)), the process of estimating (y) for an intermediate (x) is called interpolation, while estimating (y) for an (x) outside the table is called extrapolation. The concept of finite differences and the symbolic relations between different difference operators are fundamental for establishing various interpolation formulas.
Finite Difference Operators
Forward Differences
For a table of values ((x_k, y_k)) with equally spaced abscissas, the forward difference operator (\Delta) is defined as (\Delta y_i = y_{i+1} - y_i), for (i = 0,1,...,(n-1)). These are called first differences. The differences of first differences are second differences, defined as (\Delta^2 y_i = \Delta y_{i+1} - \Delta y_i). In general, the r-th difference is (\Delta^r y_i = \Delta^{r-1} y_{i+1} - \Delta^{r-1} y_i). These are systematically displayed in a forward difference table (or diagonal difference table), where each difference is placed midway between the elements of the previous column, and the subscript remains constant along each diagonal. The first term (y_0) is the leading term, and the differences (\Delta y_0, \Delta^2 y_0, \Delta^3 y_0,...) are leading differences.
🔑 Definition — Forward Difference Operator (Δ): (\Delta y_i = y_{i+1} - y_i), where (y_i = f(x_i)) for equally spaced (x_i).
📐 Formula: (\Delta^n y_0 = y_n - \ ^nC_1 y_{n-1} + \ ^nC_2 y_{n-2} - \ ^nC_3 y_{n-3} + \cdots + (-1)^n y_0) → The coefficients in the expansion of (\Delta^n y_0) are binomial coefficients with alternating signs. 📌 Example: Express (\Delta^2 y_0) and (\Delta^3 y_0) in terms of the values of the function (y).
- (\Delta^2 y_0 = \Delta y_1 - \Delta y_0 = (y_2 - y_1) - (y_1 - y_0) = y_2 - 2y_1 + y_0)
- (\Delta^3 y_0 = \Delta^2 y_1 - \Delta^2 y_0 = (y_3 - 2y_2 + y_1) - (y_2 - 2y_1 + y_0) = y_3 - 3y_2 + 3y_1 - y_0)
🔑 Definition — Leading Term ((y_0)): The first value in a forward difference table. 🔑 Definition — Leading Differences ((\Delta y_0, \Delta^2 y_0, ...)): The first differences in each column of a forward difference table.
Backward Differences
For a given table of values with equally spaced abscissas, the first backward differences are expressed using the backward difference operator (\nabla) as (\nabla y_i = y_i - y_{i-1}), for (i = n, (n-1),..., 1). The second backward differences are (\nabla^2 y_i = \nabla y_i - \nabla y_{i-1}), and in general, the k-th backward differences are (\nabla^k y_i = \nabla^{k-1} y_i - \nabla^{k-1} y_{i-1}). These are arranged in a backward difference table, where the subscript remains constant along every backward diagonal. Any value of y can be expressed in terms of (y_n) and its backward differences, leading to symbolic relations like (y_{n-1} = (1-\nabla) y_n).
🔑 Definition — Backward Difference Operator (∇): (\nabla y_i = y_i - y_{i-1}), where (y_i = f(x_i)) for equally spaced (x_i). 📐 Formula: (y_{n-r} = y_n - \ ^nC_1 \nabla y_n + \ ^nC_2 \nabla^2 y_n - \cdots + (-1)^r \nabla^r y_n) → The value of y at a point can be expressed in terms of a base value and its backward differences. 📌 Example: Show that any value of y can be expressed in terms of (y_n) and its backward differences.
- From (\nabla y_i = y_i - y_{i-1}), we get (y_{n-1} = y_n - \nabla y_n).
- From (\nabla^2 y_i = \nabla y_i - \nabla y_{i-1}), we get (\nabla y_{n-1} = \nabla y_n - \nabla^2 y_n).
- Then (y_{n-2} = y_{n-1} - \nabla y_{n-1} = (y_n - \nabla y_n) - (\nabla y_n - \nabla^2 y_n) = y_n - 2\nabla y_n + \nabla^2 y_n). Symbolically, (y_{n-2} = (1 - \nabla)^2 y_n).
Central Differences
In some applications, the central difference notation is more convenient. The central difference operator (\delta) is defined by (\delta y_i = y_{i+(1/2)} - y_{i-(1/2)}). Higher order central differences are defined as (\delta^2 y_i = \delta y_{i+(1/2)} - \delta y_{i-(1/2)}) and (\delta^n y_i = \delta^{n-1} y_{i+(1/2)} - \delta^{n-1} y_{i-(1/2)}). In a central difference table, odd differences have fractional suffixes, and even differences with the same subscript lie horizontally.
🔑 Definition — Central Difference Operator (δ): (\delta y_i = y_{i+(1/2)} - y_{i-(1/2)}), where the subscript is the average of the subscripts of the two y-values.
Shift Operator, E
The shift operator (E) is defined by the property (E f(x) = f(x + h)). Applying it n times gives (E^n f(x) = f(x + nh)). The inverse operator is (E^{-1} f(x) = f(x - h)). This operator is fundamental for deriving symbolic relations between difference operators.
🔑 Definition — Shift Operator (E): (E f(x) = f(x + h)), where (h) is the spacing between consecutive x-values. 📐 Formula: (E = e^{hD}) → The shift operator is equivalent to the exponential of the differential operator times the step size. 📌 Example: Prove that (hD = \log E).
- This stems from the Taylor series expansion: (E y_x = y_{x+h} = y_x + h y'_x + \frac{h^2}{2!} y''_x + \cdots = (1 + hD + \frac{h^2 D^2}{2!} + \cdots) y_x = e^{hD} y_x). Therefore, (E = e^{hD}) and taking logs gives (hD = \log E).
Average Operator, μ
The average operator (\mu) is defined as (\mu f(x) = \frac{1}{2} \left[ f(x + \frac{h}{2}) + f(x - \frac{h}{2}) \right]). It gives the average of the function values at two symmetrically placed points.
🔑 Definition — Average Operator (μ): (\mu f(x) = \frac{1}{2} \left[ f(x + \frac{h}{2}) + f(x - \frac{h}{2}) \right]). 📐 Formula: (\mu = \frac{1}{2}(E^{1/2} + E^{-1/2})) → This shows the relationship between the average operator and the shift operator.
Differential Operator, D
The differential operator (D) is defined as (D f(x) = \frac{d}{dx} f(x) = f'(x)), with higher powers representing higher-order derivatives, e.g., (D^2 f(x) = f''(x)).
Important Results Using {Δ, ∇, δ, E, μ}
By defining the operators on a common variable (y_x), the following fundamental relationships are derived:
- (\Delta = E - 1)
- (\nabla = 1 - E^{-1} = \frac{E-1}{E})
- (\delta = E^{1/2} - E^{-1/2})
- (\mu = \frac{1}{2}(E^{1/2} + E^{-1/2}))
- (E = e^{hD})
- Therefore, (hD = \log E = \log(1 + \Delta) = -\log(1 - \nabla) = \sinh^{-1}(\mu \delta)) 💡 Why this matters: These symbolic relations allow one to express any difference operator in terms of another, which is key to deriving various interpolation formulas and converting between them.
📌 Example: Prove the following results:
- (1) (1 + \mu^2 \delta^2 = \left(1 + \frac{\delta^2}{2}\right)^2)
- (2) (E^{1/2} = \mu + \frac{\delta}{2})
- (3) (\Delta = \frac{\delta^2}{2} + \delta \sqrt{1 + \frac{\delta^2}{4}})
- (4) (\mu \delta = \frac{\Delta}{2} + \frac{\Delta}{2E})
- (5) (\mu \delta = \frac{1}{2}(\Delta + \nabla))
- Solution (1): (\mu \delta = \frac{1}{2}(E^{1/2} + E^{-1/2})(E^{1/2} - E^{-1/2}) = \frac{1}{2}(E - E^{-1})). Then (1 + \mu^2 \delta^2 = 1 + \frac{1}{4}(E^2 - 2 + E^{-2}) = \frac{1}{4}(E + E^{-1})^2). Also, (1 + \frac{\delta^2}{2} = 1 + \frac{1}{2}(E^{1/2} - E^{-1/2})^2 = \frac{1}{2}(E + E^{-1})). Therefore, (\left(1 + \frac{\delta^2}{2}\right)^2 = \frac{1}{4}(E + E^{-1})^2 = 1 + \mu^2 \delta^2).
- Solution (2): (\mu + \frac{\delta}{2} = \frac{1}{2}(E^{1/2} + E^{-1/2}) + \frac{1}{2}(E^{1/2} - E^{-1/2}) = E^{1/2}).
- Solution (3): From (\Delta = E - 1) and the proof in (2), we have (E^{1/2} = \mu + \frac{\delta}{2}). Noting that (\mu = \sqrt{1 + \frac{\delta^2}{4}}), we get (\Delta = E - 1 = (E^{1/2})^2 - 1 = (\mu + \frac{\delta}{2})^2 - 1 = \mu^2 + \mu \delta + \frac{\delta^2}{4} - 1 = (1 + \frac{\delta^2}{4}) + \mu \delta + \frac{\delta^2}{4} - 1 = \frac{\delta^2}{2} + \mu \delta). Since (\mu = \sqrt{1 + \frac{\delta^2}{4}}), we have (\mu \delta = \delta \sqrt{1 + \frac{\delta^2}{4}}), so (\Delta = \frac{\delta^2}{2} + \delta \sqrt{1 + \frac{\delta^2}{4}}).
- Solution (4): (\mu \delta = \frac{1}{2}(E - E^{-1}) = \frac{1}{2}(1 + \Delta - E^{-1}) = \frac{\Delta}{2} + \frac{1}{2}(1 - E^{-1}) = \frac{\Delta}{2} + \frac{1}{2}\frac{E-1}{E} = \frac{\Delta}{2} + \frac{\Delta}{2E}).
- Solution (5): (\mu \delta = \frac{1}{2}(E - E^{-1}) = \frac{1}{2}(1 + \Delta - 1 + \nabla) = \frac{1}{2}(\Delta + \nabla)).
⭐ Key Takeaways
A student must understand the definitions and graphical representation of forward (Δ), backward (∇), and central (δ) finite difference operators for equally spaced data. The ability to derive symbolic relationships between these operators, particularly using the shift operator (E) as a bridge, is absolutely critical. The key formulas to memorize are Δ = E-1, ∇ = 1-E⁻¹, δ = E^(1/2) - E^(-1/2), μ = (E^(1/2)+E^(-1/2))/2, and E = e^(hD). From these, one can derive that functions of one operator (like log(1+Δ), sinh⁻¹(μδ)) can be expressed in terms of others, which is the foundation for building different interpolation formulas. Finally, the skill of expressing a function value in terms of its leading differences (e.g., (y_n = (1+\Delta)^n y_0) and (y_{n-r} = (1-\nabla)^r y_n)) is a core technique.
🧠 Quick Revision Questions
- Define the forward difference operator Δ and the backward difference operator ∇ for a function y = f(x) with equally spaced abscissas.
- Show that (\Delta^3 y_0 = y_3 - 3y_2 + 3y_1 - y_0).
- Using the shift operator E, derive the symbolic relationship (\Delta) = E - 1.
- Prove that the differential operator D and the shift operator E are related by the formula E = e^{hD}.
- Derive the expression for the average operator μ in terms of the shift operator E.
📘 Lecture 19 — Interpolation Newton’s Forward Difference Formula
📖 Overview: This lecture introduces Newton’s Forward Difference Formula, a powerful method for interpolating function values at equally spaced points. The formula expresses the value at any point using the initial function value and its forward differences, making it especially useful for estimating values near the beginning of a data table.
🗂️ Topics Covered
The lecture begins with the derivation of Newton’s Forward Difference Formula using the shift operator E and forward difference operator Δ. It then presents the complete formula with binomial coefficients, explains its alternative form using y notation, and discusses its application for interpolation and extrapolation near the start of tabular data. Several worked examples demonstrate the formula’s use for evaluating specific function values, finding interpolating polynomials, and estimating missing table entries.
📝 Lecture Summary
Introduction and Formula Derivation
Let y = f(x) be a function that takes values at equally spaced points x₀, x₀ + h, x₀ + 2h, ... with spacing h. To evaluate f(x₀ + ph) for any real number p, we define the shift operator E such that Eᵖ f(x) = f(x + ph). Using the relationship E = 1 + Δ, we expand (1 + Δ)ᵖ as a binomial series to obtain Newton’s Forward Difference Formula.
The formula expresses f(x₀ + ph) in terms of f(x₀) and its leading forward differences:
f(x₀ + ph) = f(x₀) + pΔf(x₀) + [p(p-1)/2!] Δ²f(x₀) + [p(p-1)(p-2)/3!] Δ³f(x₀) + ... + [p(p-1)...(p-n+1)/n!] Δⁿf(x₀) + Error
An alternative expression uses y notation: yₓ = y₀ + pΔy₀ + [p(p-1)/2!] Δ²y₀ + [p(p-1)(p-2)/3!] Δ³y₀ + ... + [p(p-1)...(p-n+1)/n!] Δⁿy₀ + Error
Here, p = (x - x₀)/h. If we retain (r+1) terms, we obtain a polynomial of degree r agreeing with y at points x₀, x₁, ..., xᵣ.
🔑 Definition — Newton’s Forward Difference Formula: A formula for interpolating function values at equally spaced points using forward differences, primarily used for interpolation near the beginning of a set of tabular values and for extrapolation a short distance backward from y₀.
📐 Formula: f(x₀ + ph) = f(x₀) + pΔf(x₀) + [p(p-1)/2!] Δ²f(x₀) + [p(p-1)(p-2)/3!] Δ³f(x₀) + ... → This gives the function value at any point x = x₀ + ph by using the initial value and successive forward differences.
💡 Why this matters: This formula is fundamental for numerical interpolation; it allows us to estimate function values between known data points when the data is evenly spaced.
Example 1: Evaluating f(15)
Given the table of values: f(10)=46, f(20)=66, f(30)=81, f(40)=93, f(50)=101.
First, construct the forward difference table:
- Δy₀ = 20, Δ²y₀ = -5, Δ³y₀ = 2, Δ⁴y₀ = -3
Here, x₀ = 10, y₀ = 46, h = 10, and p = (15 - 10)/10 = 0.5
Using Newton’s Forward Difference Formula: f(15) = 46 + 0.5(20) + 0.5(0.5-1)/2 + 0.5(0.5-1)(0.5-2)/6 + 0.5(0.5-1)(0.5-2)(0.5-3)/24
f(15) = 46 + 10 + 0.625 + 0.125 + 0.1172 = 56.8672
So f(15) = 56.8672 correct to four decimal places.
📌 Example: Using the formula with p=0.5 gave f(15)=56.8672, where each term contributed: 46 + 10 + (-0.625) correction + 0.125 + 0.1172.
Example 2: Finding Newton’s Forward Difference Interpolating Polynomial
Given data: x=0.1,0.2,0.3,0.4; y=1.40,1.56,1.76,2.00
Construct the forward difference table:
- Δy₀ = 0.16, Δ²y₀ = 0.04 (third and fourth leading differences are zero)
Since Δ³y₀ = 0, we have: y = y₀ + pΔy₀ + [p(p-1)/2] Δ²y₀
Here x₀ = 0.1, h = 0.1, p = (x - 0.1)/0.1 = 10x - 1
Substituting: y = 1.40 + (10x - 1)(0.16) + (10x - 1)(10x - 2)/2
This is the required Newton’s interpolating polynomial.
📌 Example: When differences beyond the second are zero, the formula reduces to a quadratic polynomial in terms of x.
Example 3: Estimating the Missing Figure
Given table: x=0,1,2,4; y=2,5,7,32 (missing at x=3)
Since we have four entries, the function can be represented by a polynomial of degree three, so Δ⁴f(x) = 0 for all x.
Using Δ⁴f(x₀) = 0, which means (E - 1)⁴ f(x₀) = 0, expanding: E⁴ - 4E³ + 6E² - 4E + 1 = 0
This gives: f(x₄) - 4f(x₃) + 6f(x₂) - 4f(x₁) + f(x₀) = 0
Using the given values: 32 - 4f(x₃) + 6(7) - 4(5) + 2 = 0 → 32 - 4f(x₃) + 42 - 20 + 2 = 0 → -4f(x₃) = -56 → f(x₃) = 14
The missing value at x=3 is 14.
📌 Example: By using the property that fourth differences are zero for a cubic polynomial, we derived a linear equation to solve for the missing value.
Example 4: Finding f(0.36)
Given table: x=0.2,0.3,0.4,0.5,0.6; y=0.2304,0.2788,0.3222,0.3617,0.3979
Forward difference table:
- Δy₀ = 0.0484, Δ²y₀ = -0.005, Δ³y₀ = 0.0011, Δ⁴y₀ = -0.0005
Here x₀ = 0.2, y₀ = 0.2304, h = 0.1, p = (0.36 - 0.2)/0.1 = 1.6
Using formula up to fourth difference: y = 0.2304 + 1.6(0.0484) + 1.6(1.6-1)/2 + 1.6(1.6-1)(1.6-2)/6 + 1.6(1.6-1)(1.6-2)(1.6-3)/24
y = 0.2304 + 0.07744 - 0.0024 - 0.00007 - 0.00001 = 0.3053
So f(0.36) = 0.3053
📌 Example: With p=1.6, each term was calculated: 0.2304 + 0.077441 - 0.0024 - 0.00007 - 0.00001 = 0.3053.
Example 5: Finding a Cubic Polynomial
Given data: x=0,1,2,3,4,5; y=-3,3,11,27,57,107
Construct the difference table: Δy₀ = 6, Δ²y₀ = 2, Δ³y₀ = 6 (fourth and higher differences are zero)
Here h = 1, p = x - 0 = x, x₀ = 0
Using Newton’s Forward Difference Formula up to third difference: f(x) = -3 + 6x + x(x-1)/2 + x(x-1)(x-2)/6
f(x) = -3 + 6x + x(x-1) + x(x-1)(x-2)
f(x) = -3 + 6x + x² - x + x³ - 3x² + 2x
f(x) = x³ - 2x² + 7x - 3
This is the required cubic polynomial.
📌 Example: Since third differences were constant (6) and fourth differences zero, the polynomial was cubic, and substituting p=x gave the interpolating polynomial.
⭐ Key Takeaways
Newton’s Forward Difference Formula expresses f(x₀ + ph) = f(x₀) + pΔf(x₀) + [p(p-1)/2!]Δ²f(x₀) + [p(p-1)(p-2)/3!]Δ³f(x₀) + ... where p = (x - x₀)/h. This formula is primarily used for interpolation near the beginning of tabular data; it works only for equally spaced points. When higher-order differences become zero, the polynomial degree reduces accordingly. The formula is derived from the binomial expansion of (1+Δ)ᵖ, and it can be used to find missing values by setting appropriate higher differences to zero.
🧠 Quick Revision Questions
- What is the value of p in Newton’s Forward Difference Formula when x = x₀ + 2.5h?
- For a cubic polynomial, which forward differences are zero?
- How is the shift operator E related to the forward difference operator Δ?
- When interpolating at x = x₀ + 0.3h, what does p equal?
- If Δ⁴y₀ = 0 for all tabular points, what is the degree of the interpolating polynomial?