The questions are: which codebases do we use? How clean is the code and how active is the community?
In general we should be looking for the biggest community. The one with 2x as many developers will win, unless both have achieved critical mass.
(Jeko) Quickgraph: http://www.codeplex.com/quickgraph
(Jeko) NPack: http://www.codeplex.com/NPack
(Jeko) dnAnalytics: 
Here is a set of math functions that the torcs robot needs. http://keithcu.com/wiki/index.php/Image:Unitlinalg.cpp
Note it doesn't use operators, and our code should to look much nicer.
We should be binary compatible with torcs, but we can change it to match our data structures.
AForge AI Library written in C# Image processing, neural networks, machine learning, and vision. There is a decent amount of traffic on the forums, and the releases have thousands of downloads. The guy is from Latvia and has linked-in profile says he is looking for consulting work: http://www.linkedin.com/pub/4/410/A72
Article about library: http://www.codeproject.com/KB/recipes/aforge.aspx?display=Print
Found a .Net wrapper! http://code.google.com/p/opencvdotnet/
Intel's code written in C++ in 1999-2000. Not too much going on with it, but it was used by Stanford in the Darpa Grand Challenge. It has code that should be thrown away, but it is comprehensive. I think we should start with it, especially that there is a .Net wrapper
Great background doc, with lots of links: http://www.red3d.com/cwr/steer/
OpenSteer port to C#: http://code.google.com/p/opensteerdotnet/
Motion Strategy Library, http://msl.cs.uiuc.edu/msl/
Motion Planning Kit, http://ai.stanford.edu/~mitul/mpk
Darpa contestant (will try to get the code) http://www.torctech.com/services/software.html
Urban driving simulator http://portal.acm.org/citation.cfm?id=1296777
(Jeko) Mono Tao Framework http://taoframework.com/
Ogre: http://www.ogre3d.org/ A good and popular one with a big online community. Used by many "serious" project, it is well documented. A friend of mine worked with and recommended it.
http://www.ogre3d.org/wiki/index.php/OgreDotNet (Supports mono, not that used)
http://www.ogre3d.org/wiki/index.php/MOGRE (Popular, doesn't support Mono)
I've been using openscenegraph for 2 years (that was 2 years ago) and it has been a pain! Lack of documentation and a few bugs. However it is quite popular, it is still not very mature. Its good points however: there are many import of 3D formats supported and it offer good real-time performances.
(Jeko) More: http://www.freeprogrammingresources.com/gamelib.html
The physics engine we will use is likely this: http://www.ode.org/
There are .Net wrappers we will use so we can program in C#: (Jeko) http://sourceforge.net/projects/odedotnet
PDF intro: http://www.ode.org/slides/parc/dynamics.pdf
API overview: http://www.ode.org/ode-latest-userguide.html
(Jeko) We also need a physics simulator GUIs. I noticed SimulatorBob above but it is written in MFC, so we can do better.
Here is a robot simulator: http://playerstage.sourceforge.net/index.php?src=gazebo
http://www.racer.nl/, nice looking, source code available but not open source (restrictive license).
http://www.ultimatestunts.nl/, require art work to look better. The track format supports junctions.
Removed from consideration
All in C#, but going through a rewrite, and they don't have a community yet.
Delta 3-D http://www.delta3d.org/
PLIB, Portable game library: http://plib.sourceforge.net/whats_inside.html Features: sound effects, music, a complete 3D engine, font rendering, a simple Windowing library, a game scripting language, a GUI, networking, 3D math library and a collection of handy utility functions. Each library component is fairly independent of the others - so if you want to use SDL, GTK, GLUT, or FLTK instead of PLIB's 'PW' windowing library, you can.
This is a cool component for people making simple games, but we don't need most of it, and the stuff we do need isn't that powerful or popular. Torcs uses it, but that will change :-)
Irrlicht: http://irrlicht.sourceforge.net/, also has .Net bindings
Looks like a good engine, but some have complaints about it: it is less pluggable, less powerful, etc.
http://axiomengine.sourceforge.net/wiki/index.php/Main_Page This is a port of Ogre that is clean, rich, and supports Mono. It doesn't have the community of Ogre and Irrlicht. Using a native C# component is awesome, but using something with a community is more important.
This is missing a lot