Kalman Filters

Term 2 begins with some ominous signs, Sebastian the founder of the course and pioneer of self driving cars takes us through some advanced theory used in sensor fusion. It is immediately clear that this is a step up from term 1 and a move into some advanced physics and math. There is a C++ checkpoint to make sure your skills are good enough and then you meet this crew. Udacity instructors  joined by 3 Mercedes-Benz engineers. Constantly in the course I am amazed at the level of industrial partnership that is on show. It is inspiring to learn from true leaders in their field. They quickly get to the point. This is serious aspect of the course, sensors need to work and work together and the data has to be processed very quickly and reliably. Technically this is very challenging and it is now time to really embrace advanced physics and math and implement solutions in C++.

Andrei , Sensor Fusion Engineer, does what every teacher needs to do when tackling tough material. He uses structure to put some order onto the material and help us figure out what we are doing. Essentially we are going to make a complex mathematical process with various steps and creates and overview which he regularly refers back to and this gives us context for what he is saying. Good job Andrei, I would have been totally lost without it.

These are some screenshots of supplementary materiel to the lectures. You don’t need to look to closely to see some huge equations with matrices and differentials all over the place. This stuff is not for the faint hearted. I will try in principal to explain that we are doing. 

Think about how you actually drive for a moment. When you are deciding how much to steer you are taking in to account where you are, also what you can see in front of you and also where you think you need to be in a couple of seconds. We do this all the time without thinking and we generally try to make nice smooth curves in our driving so the corners are comfortable for us.

How can we do this mathematically? A car has sensors instead of eyes so we need to check our sensors do something and then check our sensors again and again and again. So we have some kind of cycle where we are constantly checking sensors. Also we have 2 different types of sensor and they express information differently so we need to translate the data into useful and comparable information so we can work with both at the same time. With both sets of data we now have to factor in the fact that sensors are not perfect so we use probabilities and then based on the inputs we make some conclusions we update our idea about the car and how to drive based on all this process information and then we repeat the whole process.

That really doesn’t do the process justice but just imagine every sentence I have used in this explanation is implemented with some of the math on this page and you start to get an idea of the complexity of what I am talking about.

After quite a lot of lecture material that might have taken a couple of weeks to get through Sebastian introduced the project section with the slightly terrifying phrase ” OK are you up for some torture?…..” When one of the most accomplished people in the industry says that, then you really have to wonder are you going to say yes. It’s one of those moments that any student that takes on something challenging understands, sometimes you just need to make a pot of coffee and get stuck in because it is going to be a long night of study.

In fairness it’s important to bear in mind that you don’t need to be a physicist or a mathematician for this course. A good Computer Engineer needs to understand math and physics but not to the extent as the others. In order to program the code we need to understand what to do but we do not need to have the complex equations off memorized or be able to derive our own math theory. We are applying someone else’s math and we just need to know how to use the math to solve a problem. This is important other wise we would need to do a postgraduate in math. Once we get the code working we can take a “Black Box ” approach. We know what goes into the code and we know how to use what comes out but what exactly happens inside we may not be too familiar with.

Project screenshot

You can see the code here on Github

Close up of screen shot

These screenshots don’t look too amazing for sure and in fairness to Udacity they have updated the course already before I started and now updated it again so it is evolving and this particular graphical part of this project I believe has been improved. Having said that it is useful to consider that the computer in your car that would be actually doing the sensor fusion would be behind the dashboard somewhere and possibly not showing any graphics at all, so actually just numbers and no picture would be realistic and this graphic is just to give us students some easy to understand presentation.

So what is happening here is that there is a little blue car and lot s of dots. The red dots are sensor points from LIDAR and the red dots are points from RADAR. We can see that the dots are not arranged in a nice smooth line because this is realistic. Sensors do not give perfect results and they don’t give results the same as other types of sensors either. So we have a bit of a mess of sensor points and we need to translate all that data into a smooth line for us to follow as quickly and reliably as possible. The testing program has certain limits and we need our results to stay within those limits to pass. This take many many attempts but eventually you will get there.

All in all this was a very tough project. This was some of the most difficult math I have ever seen. I did Discrete Cosine Transform for image compression for my Masters of Computer Engineering and this math was even harder. I actually didn’t realize this kind of math exist outside of NASA or maybe the aerospace industry. It really is an example of how much cars have very recently changed. Most cars on the road have no sensors like this and pure hydraulic power steering never mind a steering system that can adjust itself. So self driving cars are really pushing new frontiers in the industry and with that the sensor and electronic requirements have  increased exponentially. When I had passed this project I felt like I had made big progress as a programmer and engineer.