JohnReilly.tech http://johnreilly.tech John Reilly's Portfolio Tue, 15 Dec 2020 16:29:40 +0000 en-US hourly 1 https://wordpress.org/?v=6.4.4 http://johnreilly.tech/wp-content/uploads/2019/02/cropped-mpc-banner-1-32x32.jpg JohnReilly.tech http://johnreilly.tech 32 32 PyCon_Limerick_2020 http://johnreilly.tech/pycon_limerick_2020/ Tue, 03 Mar 2020 19:50:23 +0000 http://johnreilly.tech/?p=1072

PyCon Limerick 2020

" Adding Computer Vision to Robot Operating System (ROS)"

This article is about the workshop “Adding Computer Vision to Robot Operating System” as presented at the Python Ireland, PyCon Limerick 2020 event.  The main points of the workshop will be sumarised here. The workshop material including Virtual Machines and source code etc are linked on this page.

The objectives of the workshop were :

  • To provide a functioning ROS environment with everything tested and working together.
  • To demonstrate 2 nodes communicating in a ROS environment in a realistic way.
  • To demonstrate and explain how to use object detection in a way that could be expanded upon by the attendee for their own projects.
  • To provide many useful links and resources to allow the attendee to continue learning if they so wished.
"Carla" the Self Driving Car Running on ROS

What is ROS?

ROS is basically a messaging system that allows multiple sensors and programs to communicate with each other. It could be described as middle-ware rather than an operating system. ROS was born out of the need for university researchers to stop reinventing basic robotic functionality for every robotic project they worked on. This allowed them to focus on more important aspects of robotic system design. Many groups of researchers had the same needs and so ROS is now an emerging standard in the field of Robotics research. From this base ROS is expanding into commercial and industrial applications as a universal Robotic standard. A milestone in the development was when Sebastian Thun developed a full self driving car using ROS.

Environment Setup

ROS  only runs on UNIX/Linux. A typical setup may have many extra dependencies. This workshop required, ROS on Ubuntu Linux, with Tensorflow for Object Detection, OpenCV for image manipulation and a Test to Speech API with various other Python dependencies such as Numpy etc. Furthermore everything must be tested to ensure everything works together before starting. To avoid problems  a Virtual Machine was distributed with the environment tested and working, all the source code ready for use in Virtual Studio Code, slides and resources saved on the VM and some useful links in the web browser for reference. There are tutorials on the ROS website these are useful for learning but also for testing the setup to ensure it works. The files and folders from the ROS tutorials are in place on the virtual machine to create a usable tested environment and a starting point for the workshop. The Virtual machine and workshop slides are available here on a shared Google Drive. Be advised the folder is over 4 Gb in size. Import the VM into Oracle Virtualbox with the command “Import Appliance”. The username is “pycon” and the password is “pycon”. 

ROS , twelth edition aka "Melodic"

Basic Concepts of Operation

A ROS system is made up of nodes each node is a program or script capable of running independently.Nodes communicate using messages. There are many standard types and new types can be created. Messages can be posted to topics, similar conceptually to  a person posting a note on a physical notice board rather than directly speaking to someone. A node that is posting a message to a topic is a publisher. A node that is waiting for messages on a topic is called a subscriber to a topic. A node can be both a publisher and a subscriber to multiple topics at the same time.

"Speak.py" - Text to Speech

To demonstrate nodes communicating a node called speak.py was created. This node listened to a topic that another node was publishing to. Initially a basic node sending a message “Hello World” was used but then a much more sophisticated node was used to publish objects detected in an image. The messages were deciphered against a list of object classification numbers and their descriptions. Eg if the message coming in was “53” this was checked against the list to find “53” corresponded to “Orange” and the word “Orange” was read out by the node. The list of objects was from the dataset known as COCO and contains 80 common objects from traffic lights to bananas. In this way the node could read aloud 80 different objects that might be sent by the other node.

"Finder.py" - Object detection

A node called finder.py was used to demonstrate object detection being run on random images and to create a message to send the other node. This was to represent what might happen in a realistic scenario where object detection might be run on images from a camera at up to 25 frames a second with the object classifier looking for people, cars or road signs in the images etc. In principal there is not much difference between using image detection on one image or many images so the workshop focused on how to process one image, extract useful information and send an appropriate message.

The object detection process was the subject of several questions and a detailed section by section explanation of the code was given. The object detection was run using a pretrained classifier using Tensorflow. The outputs were a collection of detected objects, a number to represent how confident the classifier was about each object and coordinates to provide for a bounding box to be drawn around the detected object. For simplicity only the object with the highest level of confidence was selected from each image. A bounding box was drawn around this object to clearly illustrate what the object detection was picking out. The number corresponding to this object was put into a message and published for the other node to decipher and speak aloud.

You can see the code here on GitHub

Finder.py detecting and marking objects with Speak.py output in terminal

The Program running

In the above screenshot the Ubuntu Virtual Machine is shown running ROS through Visual Studio Code. The terminal output in the lower part of the screen shows the output “Object Detected: orange” . This is the screen output from speak.py. There is also an audio output where the word “orange” is spoken aloud. The two images displayed are before and after the object detection/image classification process. Here the original image is shown and the orange that was selected is marked with a bounding box.

You can download the Virtual Machine here from a shared Google drive here. You can see instructions for running the program on the GitHub page linked here.

Important Commands

ROScore : This is a program that must be running in the background the whole time you are running the ROS system.  Open a terminal and type “roscore” and just let it run.

source devel/setup.bash : This command need to be typed into every new terminal to ensure the ROS in that terminal can see everything it need to run.

Catkin_make : This is similar in concept to CMAKE. This command is used after changes are made and you need to “recompile” the project.

ROSrun your_script : To execute a script or node in ROS type ROSRUN my_script.py 

]]> Pycon_Dublin_2019_preview http://johnreilly.tech/pycon_dublin_2019_preview/ Mon, 30 Sep 2019 15:15:31 +0000 http://johnreilly.tech/?p=1004

PyCon 2019 Dublin

Image Classification Workshop

Preview!

In this workshop we will explore key technologies to help us tackle some of the worlds big problems right now. When you think about it, we have electric motors, robot arms, batteries, wireless networks etc so why do we not have robots doing a lot more useful things? Especially things we are struggling with such as food production and waste recycling.

The answer is partially  due to the fact that it is very difficult for a computer to look through a camera and figure out what it is seeing. Recent developments such as GPU’s and image classifiers have at last given us the break through to allow robots to recognize what they are looking at and finally start doing some useful things outside of the lab.

I will present processing techniques from the self driving car industry and apply them to 2 different scenarios. The general technique will be to use a 2 stage pipeline that will apply object detection with pretrained classifiers and then a second stage where image classification with a trainable classifier is used to make some additional determinations and return annotated video.

Only one apple found, not sensitive enough
Better results, we can work with this
Sensitivity too high, everything's an apple!

The first scenario  I call RoboFarm. I will demonstrate a pipeline that can find apples in a stream of video, extract and pass the apples into a second classifier and then determine which apples are ripe and annotate the video.

The second scenario is called RoboRecycling. This will be a classifier to make decisions about which items can be recycled. I will write a blog about this soon.

I will walk you through all the code in a Jupyter Notebook. There are many points in the code where you can make adjustments to see how it effect performance. These points will be annotated and I will focus on these points not the boiler plate code that you can’t change.  The code should provide you with more than enough to base yoru own work on.

More specifically I will cover finding and selecting pretrained classifiers. How to use object detection categories and extract what you need form images. I will also cover how to create and train image classifiers including data preparation and augmentation to improve you results. Trouble shooting and how best to filter spurious results and deal with over and under sensitivity will be covered.

You will see how to prepare images to be used for training and here are 2 images that have been cropped and reduced in size specifically to be fed into a classifier for training. These images were taken by me and are hundreds of pictures of apples, ripe and unripe, branches from apples trees and leaves from apple trees. These images can teach a classifier to determine apples from branches and leaves. You can use the same images with your own classifier to compare results.

The plan is to introduce the ideas for several minutes while USB sticks etc get passed around. Then I will walk though the classifiers explaining and showing where you can experiment with your own ideas. About half way through I will stop presenting and switch to workshop mode. You then can either experiment with an existing classifier as presented or make a new one based on skeleton code and provided datasets. You should finish with a good idea of how to make a pipeline, at least one pipeline working and enough blocks of code to make your own in future.

This workshop will use Python 3, Jupyter Notebooks, Keras on top of Tensorflow. You will need your own laptop if you have a GPU you can manage video if not use single images on a CPU.

Please be advised, this is an advanced workshop. I will not cover basic concepts. This could be considered postgraduate level material.

Link to PyCon Dublin 2019 website here

Recent Posts

PyCon_Limerick_2020

March 3, 2020

PyCon Limerick 2020 ” Adding Computer Vision to Robot Operating System (ROS)” This article is about the workshop “Adding Computer Vision to Robot Operating System”

Read More »

Pycon_Dublin_2019_preview

September 30, 2019

PyCon 2019 Dublin Image Classification Workshop Preview! In this workshop we will explore key technologies to help us tackle some of the worlds big problems

Read More »

Capstone-Project

August 25, 2019

Capstone Project Programming A Real Self Driving Car The Capstone project is where after 9 months of online lessons and projects we finally get to

Read More »

Functional Safety

May 2, 2019

Functional Safety Functional Safety is one of the electives in the course. It is the only project that does not require programming. The Nano-Degree has

Read More »

]]> Capstone-Project http://johnreilly.tech/capstone-project/ Sun, 25 Aug 2019 19:32:23 +0000 http://johnreilly.tech/?p=387

Capstone Project

Programming A Real Self Driving Car

The Capstone project is where after 9 months of online lessons and projects we finally get to write code for a real self driving car. This project is big, so big it’s a team project. That brings a new set of problems. I had to form and lead a team in 4 different countries in 3 different time zones. Communication with Slack, videos calls with Zoom and coordination of multiple branches of code with GitHub became indispensable.  The basic structure of the project is based on something called ROS (Robot Operating System). This allows different component parts to communicate, for example information from the camera goes to one part that in turn tells another part how to steer the car. Python is the main language used but some parts are in C++ for speed.

The Udacity instructors present an overview of the project and some advice on how to proceed. The project  is organised in 3  sections. The first “Perception”, takes in images from a video camera, finds traffic lights and determining what colour they are. The second part “Planning”, takes in the status of traffic lights and adds that information to a series of way-points it is following. This is like following a GPS, you know the route but you still have to figure out where and when to stop at traffic lights. Gradually speeding up, slowing down and breaking are also done in this section. The last section “Control”, is for steering the car and following the planned way-points. Engaging and disengaging the system is handled here so the driver can take over the vehicle if necessary in a manual override. Finally in addition to the main sections are the messages going between each section as they communicate to each other. Messaging  is managed by ROS as is the logging process whereby you can get the  ROS to display on the screen or record on file what is happening for later review. These are vital for troubleshooting.

Udacity provide some amazing tools to support students. There is a complete online programming environment that you can use including a simulator to test your code as shown. You can also download the simulator and run it on your own machine if you have a good graphic card. Most people use the online work space for convenience but also most laptops even new ones would not have a good enough graphic card for this project so the online platform is very important.

Here you can see some green dots ahead of the car. These are way-points, they are similar to how a car might follow a GPS route. They are useful for steering but only part of the solution. We need to take into account traffic light etc and also make sure we speed up and slow down gently. We also need to return gradually to the route if we go off route for a while. So if we go into manual mode and change lane then go back to automatic we need to gently return to the route without a jarring sideways motion, We created mathematical functions to smoothly steer and adjust speeds for all of this. We used something called a PID controller to help with this and a lot of experimentation went into this to give nice smooth steering and speed changes.

Generated Training Data
A trained object detector can take this as input
Successful Output from object detection process

You may have heard terms like Artificial Intelligence or Machine Learning. What exactly that means is a bit hard to explain but essentially various companies have produced programming systems that can find and learn patterns in information. In our case we used something called Tensorflow from Google. This is a library of tools that can be used to find objects in images. In our project we used a 2 step process. The first step used a Tensorflow object detection process. (called ssd_mobilenet_v1_ppn_coco  if you really want to know) . This was able to find the traffic lights and extract them from the video feed one image at a time. The smaller images of traffic lights were then fed into a second stage where the another machine learning part decided what color the traffic light was. The complete process was very technical and will have to be the subject of another article as it is too long to explain here.

However one aspect that is important and easy to explain is the idea of training. Every system of pattern recognition or machine learning needs to be trained on sample data to teach the program what to look for. In order for this learning to be done well you need as many test images as possible. In our case we made special test images from the simulator. We added a function to our program that would generate the images. During normal use this function was disabled. In the above images you can see that we generate over 6000 files for this purpose. In addition we used ready made images from a data set called the BOSCH data set with many thousands more.  We also used more sample images from some sample data of a real self driving car going around the test track. This is one of the things about machine learning. You very quickly start to eat up RAM and Disk space in your computer and this project was enough to stretch even a high end machine.

Coordinating the work between everybody was done with a special code sharing website called GitHub.com This is a picture of GitHub in action, here you can see changes in code being approved and merged with the main group of code called the master branch. One of the great things about the Udacity projects is that they encourage the use of GitHub and provide free courses on how to use it. This is something that is very popular right now in coding and using this system allows us to develop useful skills that are needed for work even though they are not directly part of a programming language.

These are some images of output logs. We had some information from ROS and some we created with messages displayed to the screen. When you are trying to figure out what is going wrong you need to closely look at this kind of thing to find the problem. For example one problem I found was the car would detect and correctly classify a RED traffic light but then it might mistakenly classify a RED light as an UNKOWN light. This would throw off the decision making of the car. So we had to adjust the program to account for occasional imperfect traffic light classification and that became a major focus of the project towards the final submission.

You can see the code here on Github

Finally after months of team work we got to the point after a few re submissions and many hours of looking at logs and retraining classifiers and everything else we get back this video. You can see the car is slowing down before the RED light and about to stop then the light goes green in earlier attempts the car failed to recognize the Red light and motored past quite quickly so this was great. Maybe the car could have moved towards the light faster and stopped before the light changed to Green but I think that would be unrealistic , you don’t accelerate to a Red light you just gently move forward so I was happy with this. 

Looking back on the whole Nano-Degree Program it really is extraordinary. The team behind it and the concepts they teach through projects are as demanding as any postgraduate course I have ever seen. The technologies involved such as Tensorflow and ROS are up to the minute and it is very difficult to find any teaching material in this niche never mind to this level. I had a few objectives in mind when I started out on this path more than 20 months ago. I wanted to update my programming skills first and foremost. With the prerequisite courses in Python and C++ I was able to do get to where many undergraduate college courses might take you but the Nano-Degree is easily postgraduate level. I learned things in this course that I didn’t know existed! Machine Learning, object detection and image classification are truly amazing topics to learn about but also many projects used C++ and very advanced mathematics, I will never forget the Kalman Filter! I had done Digital Signal Processing (DSP) at Masters level and some advanced math such as Discrete Cosine Transform but that was nothing compared to the Kalman filter’s  linear algebra. As for the future I did this course to take myself in a new direction and be part of the autonomous vehicle revolution that is sweeping the world. What my place in that will be remains to be seen but certainly I feel like I have learned enough to make a real contribution to companies in this space and really that’s what I was hoping for. I would absolutely recommend the course to anyone.

]]> Functional Safety http://johnreilly.tech/functional-safety/ Thu, 02 May 2019 14:57:22 +0000 http://johnreilly.tech/?p=525

Functional Safety

Functional Safety is one of the electives in the course. It is the only project that does not require programming. The Nano-Degree has been shortened from 9 months to 6 months. A couple of projects got removed and I am sorry to say this was one of them. As someone with a background in education I can understand that 6 months is better as it fits the regular school or college year better and its easier to keep your calendar in sync with everybody else. You can do a few months break at Christmas and then do a few more months and be done for the summer. But I am sorry to see this project get removed. I will tell you why. My background is varied but back there somewhere I did a Degree called Production Management which was focused on manufacturing. It was only after some time working that I went back to university and did 2 post-grads in Computer Engineering. This means I have a lot more understanding of quality control and ISO standards than most computer engineers. This project implements ISO 26262. I didn’t know anything about it as it is not a manufacturing standard but one especially for automotive functional safety. I knew this would be useful in the kind of companies I wanted to work in and I was keen to see what things had been added to the ISO schemes since I last studied them. In the end we made some very nice professional looking documents and I glad I choose this elective.

Elektrobit (EB) is an award-winning and visionary global supplier of embedded and connected software products and services for the automotive industry.

As usual Udacity have teamed up with an industry leader for their content. This company Elektrobit has been in operation for more than 30 years with more than one Billion devices in more than 100 million vehicles ! That must make them the biggest company that you have never heard of! What many people may not realize is that many parts of cars are made by different companies and the computer systems in cars are often so.

We are lucky to have the nice people at Electrobit take us through their processes and use samples of their actual documentations to teach us how to do functional safety the right way.

 

Dheeraj, Stephanie, and Benjamin from Elektrobit.

For those unfamiliar with ISO or quality control systems they usually involve a defined process with some kind of diagram with lots of jargon to create a standardized approach so different people across a company or industry can read each others documentation and be sure they understand what they are reading. This is so important in an industry where small mistakes could cause loss of life in a car crash.

Outline of the Hazard Analysis and Risk Assessment

The team from Electrobit do a good job of conveying a few key concepts. Safety starts with company culture and the need for safe procedures built into the way a company works and thinks.

Also they are trying to get us to think in a “functional safety” kind of way. This means thinking clearly breaking down concepts into sub-concepts and logically working through implications of designs or design changes throughout an entire design process

They also teach by example. The example they choose is a lane assistance function. Basically a function built into the car that will give some kind of warning to a driver that they are about to leave a lane. Sounds easy enough maybe something to add to the power steering systems. 

If you have not done something like this before you might be shocked at how complex a function can be and how many other subsystems might be involved and how many and what type of requirements we might have to ensure such a function is safe to use.

Here you can see the formal “V model” which is part of the ISO standard. The picture is linked to the ISO website if you want to see their information on it. Basically each step that is described on the diagram is a sub-process that requires thought and documentation.

Summary of ISO 26262 V Model

The documentation that we work through becomes the basis for the project. The project includes the following documents. We are given some templates from Electrobit so they look really nice and professional and are based on real documents.

  • Safety Plan
  • Hazard Analysis and Risk Assessment
  • Functional Safety Concept
  • Technical Safety Concept
  • Software Requirements
Template Cover

You can see the code here on Github

In the end the project proved very insightful about the kind of process we have to work within in the industry and explores the question how do we make cars safe when working in large teams. It was well worth doing and I am proud of the documentation I produced.  If I were writing this course I definitely would want my students to have a good understanding of how to build safety into processes. Having said that I can understand how some people would do the course wishing to focus only on programming and for them there was another elective.

]]> Path Planning http://johnreilly.tech/project-11-path-planning/ Mon, 22 Apr 2019 16:28:25 +0000 http://johnreilly.tech/?p=320

Path Planning Project

In this project there was a lot to accomplish, so I had to make a check  list. A year ago this would have been totally beyond me so I am just delighted to be working at this level right now even though it is really just the beginning, but it’s still a big milestone for me. To get us started Udacity provide us all that we needed to allow us to focus on a set of problems without having to create from scratch a whole virtual world. We get a very nice 3D car simulator that looks like a computer game but with more sophistication that previous simulators on prior projects. This one has traffic in both directions and the traffic is semi random mostly other cars driving at various speeds but they also occasionally simulate real drivers quite well by cutting in front of you very close without warning or slowing down in the fast lane for no apparent reason.

Our car has to react to this environment and just going slow will not do as their is a requirement to make progress through the traffic.  We also get data from simulated sensors. In a real world scenario we would have LIADR or RADAR sensors to help us and we get data to represent other cars detected by sensors on the road from the simulator. Other projects focus on whats called sensor fusion which is combining sensor data into usable information. This project is about making good driving decisions after we have our sensor data sorted out.

The Big Green Line is the cars predicted course

Project To Do list:

  • Stay on the correct side of the road
  • Keep good position in while in lane
  • Don't Collide with other vehicles
  • Don't leave the road
  • Don't break the speed limit
  • Do go as fast as possible within the speed limit
  • Figure out the best lane for the car to be in
  • Safely move into the best lane available
  • Do not speed up or slow down too suddenly
  • Do not move side to side too suddenly
  • Do all the above for at least 6 simulated miles
  • One single violation restarts the clock

I have to say I loved this project, some of the other projects had a huge emphasis on mathematical processes to transform sensor data into useful information. In this project I got to think a lot about the behaviors of a car on the road and how to describe that in words.The beauty of software programming is that if you can express an idea in words you are well on the way to  getting it done in code. So I got to really think about describe the real wold and implementing that in C++.

So how do we drive down a 6 lane highway? Bearing in mind the points on the checklist. We want to move along at near the speed limit because its a highway but not over the speed limit. We need some kind of number in the program to keep track of speed and if possible increase the speed but not above a certain point, but also not to accelerate or decelerate too suddenly. So we make some variables for speeds and rates of acceleration. Then we need to think about  which lane to go into, well the 3 on the wrong side of the road are out and after that we should only go into a lane if it is safe to do so. But what does that really mean. After some thought and with some experimentation I decided safe would be no vehicles in the lane 50 meters ahead of me and 20 meters behind me.

What about when to change lanes? Well if you can do the speed limit you might as well stay where you are but if there is a open lane and you are doing less that the speed limit you should move into that lane. In this project we were able to create a loop that would do checks every 20th of a second and it would check the senors input, we used that to find space to move into.

Debugging in C++ or any language can often involve getting the program to display text outputs to a screen so you can see what it is doing. I got the program to display outputs telling me how far the car ahead was so I could test how it was reacting. As these outputs grew and became more complex I decided to express the outputs from the cars point of view. In a nod to Artificial Intelligence I got the outputs to say ” I am in the Center Lane” , “the right lane is clear” etc. This told me what the car was “thinking” and actually made the debugging information more clear and easier to interpret. I was quite proud of my little innovation!

Of course it was not so simple and plenty went wrong. For example our car (called “the ego car” in a touch of Udacity humor) would often crash into the back of slow moving traffic. This was because on one hand we wanted to avoid sudden acceleration and deceleration but on the other hand  not too crash. When we set up our program to do that we created the possibility that the car might not stop fast enough if there was very slow moving traffic ahead.

So I  thought about what we actually do in real driving. What I do is try to look far ahead and make small adjustments early in plenty of time, maybe just easing off the throttle a little bit. If I get closer to something I want to avoid I slow down a bit harder maybe braking or dropping gear. If I am really near I will brake and if it is an emergency I will do an emergency stop.  I took those ideas and implemented them in C++.

What that involved was doing an number of checks using sensor input to determine how close something was ahead and setting so status to “within 40m” , “within 30m”, “within  15m” or “within 5 m”. With 4 levels of distance I created 4 corresponding levels of speed adjustment . There was a penalty from the simulator if we collided or decelerated too hard but I decided it was better to get penalized for braking hard than colliding and this was the whole point of the project, to think about what decisions do you have to make and what trade offs are involved if you want to be safe but still make progress on the road.

There were other problems like the car sometimes would sit between lanes, it turned out when two adjacent lanes were equally good for progress it would alternate between them  every split second but the smoothing effect of the steering meant this created an average setting that kept the car between two lanes until traffic made one lane decisively worse than the other then it would move to the better lane. These kind of things took a lot of time to decipher what was going on but they were great learning experiences.

After a lot of work and experimentation the car was able to drive for 20 minutes without any penalties for collisions or sudden movements. We only needed to do it for about 5 minutes but I wanted to be sure. I did a number of over 15 minute tests and called that a success.

You can see the code here on Github

Some screenshots of the car in action below

It was very satisfying to get this feedback from the project reviewer

]]> PID http://johnreilly.tech/pid/ Wed, 02 Jan 2019 15:04:17 +0000 http://johnreilly.tech/?p=606

PID Controller

David Silver introduces Uber's Andrew Gray

The PID controller project is all about generating steering angles that create a safe and comfortable driving experience. Imagine a beginner driver and maybe they have been told “try to stay in the middle of the lane” and they see they are a little to the left so they steer right but too hard. Now they need to move a little left so they steer left but too much and so on back and forth creating a really annoying and unsafe driving experience. A PID controller system is  about taking a steering system that might know how to steer well and giving it parameter to work with so it produces nice smooth lines that gradually bring a car to where it is desired without over or under steering.

Once again Udacity team up with the best this time Andrew Gray from Uber is here to teach us all about how PID works.

PID is a method for controlling that actually goes way back. Wikipedia have a great page on it that goes into the history if you are curios. First credited use of PID or similar was Christiaan Huygens in the 17th century to regulate the gap between millstones in windmills . Wow I love when something cutting edge technology has old roots. The PID  system has also been used in cruise control in cars and steering control in ships all before the modern electronic era!

source: Wikipedia article, PID_controller

So what exactly is it? PID stands for Proportional, Integral and Differential. These are 3 related but different values calculated to take into account a desired heading, the current heading and the steering that needs to happen to get them to match. If we just used P for steering we get over-steering and over correcting, in other words oscillating around a desired heading but never staying on it. The other two terms mitigate this effect and generate an effect where steering far away from a desired point is large and then reduces as we get closer to the desired heading. For example if you were far from a center of a lane you might move the steering wheel a lot to quickly get back to centre but then as you get closer to the centre you reduce the steering wheel input to more slowly adjust the car and near the centre you almost completely straighten the wheel. So you do this the whole time but getting a computer to do this is actually quite hard. The animated graph here is from the Wikipedia page and shows what happens if we start adjusting the amount that we factor in the PID terms.

This is a image for the simulator running with a good PID setup , basically it stays near the middle of the road but still have smooth corners. Here you can see the car near the center and in a good position for the gradual bend in the road ahead. As the road straightens out it would return to the very center. If your PID setup is off it will go too wide a the corners or bounce around left to right a lot. This is based on the same simulator as the behavior cloning project but it is used in a different way. 

Really what we have achieved in this project is not too obvious from a description and a few screenshots. We had tutorial material in Python and we had to implement the project in C++. This is great practice because that is what you do in industry the whole time. Also we had to learn by experimentation. This is so important because often we don’t know the best way until we experimented to find it. Even though there are only 3 terms to adjust. But until you developed some sense to what levels are used to there is a lot of guessing so initially I did a series of experience with large variations. I picked the best result and did smaller variations and picked the best results and did smaller variations and then some more experiments  to confirm before I was happy. All that gave me a good understanding of PID and how seemingly small adjustments can make big differences. Having the simulator helped show this very clearly. Eventually I could get the car to go quite quickly but still corner well. Another thing achieved here is a set up for the final project. We have a lot of freedom about how we might implement the final project but PID is a common and useful mechanism for steering and throttle and brake controllers. Eventually I did use these methods in the final project so this PID controller project even though it was one of the smallest projects in the course was still a valuable set up for the final project and indeed “real world” solutions.

You can see the code here on Github

]]> Particle Filter http://johnreilly.tech/particle-filter/ Mon, 31 Dec 2018 20:04:58 +0000 http://johnreilly.tech/?p=653

Particle Filters
and the Kidnapped Vehicle Project

Tiffany Huang and Maximilian Muffert from Mercedes-Benz

Once again Udacity team up with the best to bring us top quality teaching. In this section we learn from Tiffany Huang and Maximillian Muffert from the localisation team in Mercedes-Benz. In other words they are on the team that help figure out where you car is located as opposed to other aspects of autonomous driving such as controlling steering angles or speed. 

 

As usualy there is quite a bit of theory and Sebasitan founder of Udacity also weigh is in this section to give a background in the localisation theory which also applies to robots in general.

Particle Filters are a bit difficult. It’s easier to see them in action and then explain. So I will ask you to look at the series of 5 pictures below. Just have a look at all of them and see what you notice for a moment. Then below the pictures continue reading.

These images are a bit small but it is useful to have them side by side for now. Notice that they all in have the same dark gray areas. This represents a map of some rooms in a building. Notice the first one on the left has many many red dots, the next also has red dot but less of them and they are clustered in some areas. The next images have  dots and the dots are clustered in fewer and fewer areas.

What you saw above is a particle filter in action. Imagine if you were to have 1000 guesses of where you were on a map. Each dot represents a guess. After you scatter your guesses around randomly you can compare your guesses to any measurement you have. So you might say well I am sure that I am 10 meters from a wall and this guess has no walls within 20 meters therefore this guess must be wrong. If you did this for all your guesses you could eliminate many of them. This is what happens in the first couple of images. Then as your robot or car moves it sensors pick up more information and you can continue to evaluate your guesses inst more information. Eventually you get more certain of the location. The blue lines in the above images are measurements that the robot (in this case) is using to eliminate guesses. Each guess is a point called a particle and that’s where the name comes from

Above is the process flow for the program. Not nearly as complicated as the Kalman Filter but plenty for what we needs.

Some times our mearurements and the map don’t quite work perfectly. Our sensors might say there is a lappost 10 meteres away but the map might say 11. In such cases we need to adjust and select the nearest credible point as being the right one. THis is becuase both sensors and maps can be inaccurate. If the map says there is a lamppost every 10 meters and your sensor sees one 9 meters from another and nothing that looks like a lamppost nearby then sometimes is is acceptable to make adjustments and assumptions.

Similar to the Kalman Filter and Unscented Kalman filter project the graphical illustration aspect to the project is not amazing but it is adequate. And similar to the other projects what is shown here are representations of calculations that normally would not be shown at all in a real car so its the calculations that matter and the graphic presentation of them less so.

Having said that there is enough here to explain what is happening. The scenario represents a kidnapped vehicle and it needs to locate itself. We can see a little blue car and there are green lines extending out from the car. There are also black circles. The lines represent measurements to the map points that are the black circles. Here also you can see in the left side of the image there is another dark blue circle. This represents where the program thinks the car is. So clearly this is not working.

 In the next picture you can see the blue car and the blue circle are in the same place. In this successful version of the program the car drove around a course and the blue circle followed it quite well showing that the computer was locating the car correctly. What you see is a simulator that is testing the code and it tell us when it works which is very handy for Udacity to automatically correct work. But there is something else too, many companies use a system called build to test where a test for a program to pass is written and then when the program or program section like a function is finished it is tested against a test to check it. This is important for quality control and is known as “build to test”. And so the Udacity C++ projects are being done in the “build to test” way and this is a useful experience for us to work in this way.

In the end I found this easier than Kalman and Unscented Kalman filters, I also found the concept behind it fascinating. At this stage I was really starting to feel more confident with my C++ and that was major point in doing this course. 

You can see the code here on Github

]]> Unscented Kalman Filter http://johnreilly.tech/ukf/ Sat, 08 Dec 2018 14:43:56 +0000 http://johnreilly.tech/?p=626

Unscented Kalman Filter

Dominik Nuss from Mercesdes-Benz

The Udacity and Mercedes-Benz team from the Kalman Filter project are back to teach an important variation of the concept.

The strangely named Unscented Kalman Filter is as the name suggests closely related to the Kalman Filter. There are a few stories going around about the origin but basically some people working with the Kalman fitlers decided to address some of its limitations especially in relation to dealing with non-linearity also known as curves! The new method removes one aspect of the Kalman filter that is very challenging known as the Jackobian Matrix and adds a method of estimation known as sigma points. Also the overall process is simpler. Despite simplifications and estimations it produces great results and in some circumstances better results than a Kalman filter. I wondered was the name given what the method was actually called in industry and it turns out that it is and Wikipedia have an article on it here.

Andrei passes the teaching baton to Dominik as we go from Kalman filter to Unscented Kalman. Both are great teachers and uses overviews to keep context for what they are describing.

We try to achieve similar things with the Kalman filter and Unscented Kalman Filter processes. We two sensors and we wish to combine the information from both sensors (sensor fusion) but the different sensors produce different types of data and at different rates with different accuracy.

As you can see the UKF road map is a lot simpler than the Kalman Filter process. 

 

The essential difference between Kalman and unscented is the use of sigma points. I will attempt a short expatiation. Imagine you knew exact where something was and you plotted the exact spot on a map , you might say you pin pointed it. Meaning you can put the location with a high degree of certainty to a very small point. What if you are not so sure? Instead of a pin point you might have a bigger circle and say I am sure but the location is within this circle but I am not sure exactly where. The more uncertain you are the bigger the circle would be. Sigma points are made by taking the circle that represents the uncertainty and selecting a few points on it and using those instead of a perfect match to the entire circle surface or perimeter. This means we only have to do complex math for a few points and actually this is sufficiency accurate. The disadvantage of some loss of theoretical accuracy is offset by the fact that this technique works better on curves so it can outperform the Kalman filter in such circumstances. 

The above images show screenshots for the lectures as you can see plenty of hefty math to work through. Similar to the Kalman Filter we need to understand the equations enough to work with them and get them set up in a program but we don’t need to memorize them. We are still acting as Computer Engineers not Mathematicians. 

Also here we see the screen output when the program is running and similar to the Kalman filter project we have a little blue car and many coloured dots around it. The blue and red dots represent Radar and Lidar point and the  Green dots represent a smoothed line that combins the detecions from Radar and Lidar. 

Again like the Kalman filter I would say the picture is not as pretty or as dramatic as I would like. I feel like with all the effort I put in I would like to see something more spectacular but again like the Kalman filter this represents calculations that happen inside your car and may not have any usual output in normal use. Also I think the new version of the course has a nice output screen to look a little more exciting.

My outstanding impression of this project is that is was slightly easier than the Kalman Filter but both of these filters together represent the most challenging mathematics I have ever had to work with. It was great to be working on really tough material and to implement it in C++ was a great learning experience.

You can see the code here on Github

]]> Kalman Filter http://johnreilly.tech/kalman-filter/ Mon, 19 Nov 2018 10:19:43 +0000 http://johnreilly.tech/?p=491

Kalman Filters

Term 2 begins with some ominous signs, Sebastian the founder of the course and pioneer of self driving cars takes us through some advanced theory used in sensor fusion. It is immediately clear that this is a step up from term 1 and a move into some advanced physics and math. There is a C++ checkpoint to make sure your skills are good enough and then you meet this crew. Udacity instructors  joined by 3 Mercedes-Benz engineers. Constantly in the course I am amazed at the level of industrial partnership that is on show. It is inspiring to learn from true leaders in their field. They quickly get to the point. This is serious aspect of the course, sensors need to work and work together and the data has to be processed very quickly and reliably. Technically this is very challenging and it is now time to really embrace advanced physics and math and implement solutions in C++.

Andrei , Sensor Fusion Engineer, does what every teacher needs to do when tackling tough material. He uses structure to put some order onto the material and help us figure out what we are doing. Essentially we are going to make a complex mathematical process with various steps and creates and overview which he regularly refers back to and this gives us context for what he is saying. Good job Andrei, I would have been totally lost without it.

These are some screenshots of supplementary materiel to the lectures. You don’t need to look to closely to see some huge equations with matrices and differentials all over the place. This stuff is not for the faint hearted. I will try in principal to explain that we are doing. 

Think about how you actually drive for a moment. When you are deciding how much to steer you are taking in to account where you are, also what you can see in front of you and also where you think you need to be in a couple of seconds. We do this all the time without thinking and we generally try to make nice smooth curves in our driving so the corners are comfortable for us.

How can we do this mathematically? A car has sensors instead of eyes so we need to check our sensors do something and then check our sensors again and again and again. So we have some kind of cycle where we are constantly checking sensors. Also we have 2 different types of sensor and they express information differently so we need to translate the data into useful and comparable information so we can work with both at the same time. With both sets of data we now have to factor in the fact that sensors are not perfect so we use probabilities and then based on the inputs we make some conclusions we update our idea about the car and how to drive based on all this process information and then we repeat the whole process.

That really doesn’t do the process justice but just imagine every sentence I have used in this explanation is implemented with some of the math on this page and you start to get an idea of the complexity of what I am talking about.

After quite a lot of lecture material that might have taken a couple of weeks to get through Sebastian introduced the project section with the slightly terrifying phrase ” OK are you up for some torture?…..” When one of the most accomplished people in the industry says that, then you really have to wonder are you going to say yes. It’s one of those moments that any student that takes on something challenging understands, sometimes you just need to make a pot of coffee and get stuck in because it is going to be a long night of study.

In fairness it’s important to bear in mind that you don’t need to be a physicist or a mathematician for this course. A good Computer Engineer needs to understand math and physics but not to the extent as the others. In order to program the code we need to understand what to do but we do not need to have the complex equations off memorized or be able to derive our own math theory. We are applying someone else’s math and we just need to know how to use the math to solve a problem. This is important other wise we would need to do a postgraduate in math. Once we get the code working we can take a “Black Box ” approach. We know what goes into the code and we know how to use what comes out but what exactly happens inside we may not be too familiar with.

Project screenshot

You can see the code here on Github

Close up of screen shot

These screenshots don’t look too amazing for sure and in fairness to Udacity they have updated the course already before I started and now updated it again so it is evolving and this particular graphical part of this project I believe has been improved. Having said that it is useful to consider that the computer in your car that would be actually doing the sensor fusion would be behind the dashboard somewhere and possibly not showing any graphics at all, so actually just numbers and no picture would be realistic and this graphic is just to give us students some easy to understand presentation.

So what is happening here is that there is a little blue car and lot s of dots. The red dots are sensor points from LIDAR and the red dots are points from RADAR. We can see that the dots are not arranged in a nice smooth line because this is realistic. Sensors do not give perfect results and they don’t give results the same as other types of sensors either. So we have a bit of a mess of sensor points and we need to translate all that data into a smooth line for us to follow as quickly and reliably as possible. The testing program has certain limits and we need our results to stay within those limits to pass. This take many many attempts but eventually you will get there.

All in all this was a very tough project. This was some of the most difficult math I have ever seen. I did Discrete Cosine Transform for image compression for my Masters of Computer Engineering and this math was even harder. I actually didn’t realize this kind of math exist outside of NASA or maybe the aerospace industry. It really is an example of how much cars have very recently changed. Most cars on the road have no sensors like this and pure hydraulic power steering never mind a steering system that can adjust itself. So self driving cars are really pushing new frontiers in the industry and with that the sensor and electronic requirements have  increased exponentially. When I had passed this project I felt like I had made big progress as a programmer and engineer.

]]> Vehicle Detection http://johnreilly.tech/project-5-vehicle-detection/ Thu, 26 Jul 2018 16:45:57 +0000 http://johnreilly.tech/?p=152

Project 5 Vehicle Detection

Term 1 Project 5 Udacity Self Driving Car Nanodegree

Project 5 in Term 1 is where we pick out cars from images and then videos. We had been working on lane lines and traffic signs up until this point but lane lines and traffic signs all look similar and fairly consistently. How does a car look? What about different styles and colours of cars not to mention nears cars are big far away small. In this project we grapple with these issues as we extract and mark vehicles from a movie stream. We use Scikit Image and Scikit Learn Libraries to help us.

So how do we teach a computer what a vehicle looks like when it is not something clearly defined like a lane line? We take many examples of cars and many examples of things that are not cars and train the computer on those images. I used about 30,000 images in all. Fortunately I didn’t have to take them myself. Udacity do a great job of introducing us to academic and industry standard data sets and sources. There are two used for this project.The KITTI Vision Benchmark Suite which is a project between Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago and also the GTI Database from the Universidas Politecnica de Madrid. Above you can see 5 examples of cars and 5 or things that are not cars from the data sets.

We we have here is a HOG! A Histogram Of Gradient that is. This is a techniques from the SciKit Image Library and it marks where there is rapid change in contrast or color in an image and so give a kind of outline. Various parameters for the techniques are experimented with here to give above results. These parameters effect the overall performance in terms of speed and accuracy in the project.

What about near and far? Near cars are big, far cars small , how do we account for that. In the above image you can see many small rectangles. This is because I am looking for far away targets in the middle of the image which is where a far away car would be. As it turns out there are loads of false matches as the computer thinks everything from the grass to the crash barrier is a match. So this particular experiment did not work but it is a good example of how you can learn from poor results. I did have a good idea to search only a section of ht image for small targets but the search criteria was not accurate enough so I changed the criteria not the whole idea.

Here I am using larger search windows to generate results. Some are accurate some not. Why did I look up in the trees this time? When  a vehicle like a truck is very close potentially it could take up from the lower edge to the upper part of the image so that’s why I am searching with big  squares high and low in the image. There are some good results around the black and white car, but still too many false matches. Also of note here is the fact that there is lots of information above the image. What I am doing here is displaying all the settings I am using so that I can keep track of what I am doing. While experimenting like this there are some many variables to adjust and permutations to keep track of that it can get confusing so its better to display everything and save it for reference.

The next image shows the same setting except one called the scale is changed, this makes the boxes a different size. It seems to work much better. And so these projects progress by lots of experimentation .  Eventually I decide to use different box sizes the medium and the large discarding the smallest as it is too unreliable and when it does work the other sizes catch the same vehicles as shown below.

You can see the code here on Github

You can read a pdf report here

Small
Medium
Large

How many test videos do you think I made?

Over 86 attempts !

So here it is my final video, not perfect but good.

]]>