Deep Learning can solve differential equations (theory & pytorch implementation)

Hey! Welcome to an other math article! In a previous article I described about my thesis approach which is a method inspired by neural odes and the motivation is how neural networks can be used for extrapolation, https://sevent-christina.medium.com/a-different-approach-inspired-by-neural-odes-extrapolation-of-neural-networks-9766f846bd02 here it is to take a look! One thing that I use, that is very interesting, is the solution of an ode using a neural network.

Artificial Neural Networks for solving ordinary and partial differential equations

The method was firstly proposed in the paper https://www.cs.uoi.gr/~lagaris/papers/TNN-LLF.pdf by A. Likas, E. Lagaris, and D. Fotiadis and it was a cutting edge approach that inspired a lot people . Especially now that the topics about scientific machine learning are very “hot” and we have in hands powerful tools to implement and test, this paper is must read. Here I explain and implement this paper in pytorch in a simple way! Let’s dive into the method!

Mathematical Background

Here I will introduce an example of first order ODE, however it is easy to work also for bigger order ODEs or for PDEs.

• General :

In numerical analysis we express an ODE with the following form:

So, for a first order ode we have

We will work on example 2 of the paper:

The analytical solution of this first order ODE is:

• Euler’s method:

There are a lot known numerical methods for solving a differential equation. The most popular is the euler’s method. The basic idea in euler’s is that we discretize the domain [α,b] in a uniform way in N points,

[a=t0, t1, t2, t3, … tN=b] and we find the solution by a formula, Euler’s is:

In our example, we have t in [0,5]=[α,b] and the final formula is:

A two minute implementation method above gives an extra good result! This is not always the case, if we have for example a more complicated differential equation . Fortunately, there are also methods that we can use in such cases, like Runge Kutta or upgraded Euler’s.

The proposed Method

In theory: We suppose a trial solution that is depended on a neural network. This neural network will be trained such that the trial solution fits the analytical solution of the ode.

• According to equation (11) of the paper , for the specific example, we have the y trial below. N(x,theta) is the neural network which will be trained.

• Loss function: Neural Networks are continuous and differentiable functions, so we can have the derivative of a Neural Network.

Implementation: Suppose that N(t, theta) is a neural network with simple architecture. It has one input layer with one neuron that corresponds to each point ti of [a,b], one hidden layer with 10 neurons and an output layer with one neuron that corresponds to the trial solution y(ti), we use sigmoid as activation function.

I define the network and the derivative of the network by hand. Below is also the same network using the Sequential module of Pytorch and then use autograd package for the derivative. It is also very handy! I did it from scratch in order to dive deep into the paper’s method! The derivative comes from formulas (5) and (6) in the paper.

The definition of y trial, f function(the ode) and the loss function.

training of this simple Neural Network achieves great loss.

final result

Implementation using Autograd. Instead of defining the neural network and the derivatives by hand we can use pytorch Sequential module.

The result using Sequential module, autograd and Adam or BFGS optimizer is the same and also very fast 👐

Conclusion

We can use neural networks to solve an ODE or PDE and we can achieve great results very fast. For bigger order odes or pdes the idea is the same but some things change in the implementation.

Thank you for reading! Contact me for comments and more discussion.. :)

Mlearning.ai Submission Suggestions