If was tasked with testing Self Driving Car Algorithms in the “Real World”, I would go about it in the following simple way:
I’d equip a test-car with all the sensors required for the algorithm to test, and in addition I would add sensors to record the steering wheel position and the positions of the gas and break pedals. I would not add any servos to control and actually “do” the driving the algorithm would suggest.
Instead I’d put two test drivers into the car, one at the steering wheel, doing the driving like a human driver would do. And in the passenger seat, I’d put an “instructor” that would get a screen, that cannot be viewed from the driver position. On that screen the following data would be displayed: The Steering wheel position the algorithm would suggest, and the difference to the current steering wheel position. And the same difference for the speed the algorithm would suggest, and the difference to the current speed.
The “Instructor” in the passenger seat would then get the task of conveying the desired direction and speed that the algorithm suggests to the driver verbally. In the beginning this might be awkward, inefficient and error prone, but over time, as the driver and the instructor adjust to their roles, they will become more and more in tune, like a duo of musicians improvising together. As a pair they would do the task of conveying the information the computer provides to the physical machine, the car.
Then of course all the data from these test drives would be collected, together with the actual driving data collected from the sensors in the steering wheel, pedals, velocity meter, etc.
This data could then be used to evaluate the efficiency and practicality of the algorithms. The goal would then be to optimize in a way that difference between the suggested direction and velocity is minimal in regards to the measured values.
This approach would provide several benefits. First of all the “human” part of this interaction is a model that has been already proved to work well, as this is usually the constellation in a rally car. There is a “Driver” and a “Navigator”.
Additionally, this approach can be used without any legal requirements to testing on actual roads. The driver is actually driving and focusing on traffic, and is not distracted by a screen or anything.
But above all, this approach provides the fast iterations necessary to improve an algorithm incrementally, or quickly test a new idea. Because as the human safety drivers are fully in control and actively focusing on the driving there is no risk if an algorithm proves to have bugs, as this case will simply show up in the data as a greater deviation.
There is no need to have an emergency stop button, and there is no risk of the car suddenly stopping abruptly, or accelerating rapidly, if the algorithm runs in a completely wrong direction.
Thinking about the problem at hand, this was the first and most obvious solution to quickly get to a stage to experiment, without putting anyone at risk and without slowing down the experimenting.