AI-based Virtual Race Engineer incl. Pitstop Handling

Hello all,

with some support from this community (thx to @Brutten), I finally integrated my software package for sim racing rigs, which also include an AI based fully voice dialog capable race engineer chat bot, which can handle a pitstop without any user interaction, to the rFactor 2 simulation. Beside the race engineer the software comes with full supprt for controlling all the sim racing aspects with fully configurable and automated handlng of controller devices like button boxes or stream decks, and all the other technical aspects of the ingame immersion.

The software is free, open source and is fully supported.

You will find a download on GitHub: https://github.com/SeriousOldMan/Simulator-Controller#simulator-controller

Full documentation also on GitHub: https://github.com/SeriousOldMan/Simulator-Controller/wiki

If you have questions or found something, that needs a fix or enhancement, please feel free to contact me...

And now have fun!

 

Hello TheBigO
Very very interesting and big job.
Thank you for sharing

 

github is confusing, which file do we click on/download??

 

I think that
https://www.dropbox.com/s/2ax7h0ezbz6o9xs/Simulator Controller 2.8.2-release.zip?dl=1

 

Yes, that's correct. You will always find the latest download in the section "Latest release builds". The software will tell you, when it's time to download a new version, which typically arrives in a one or two week schedule.

Please note, that the confguration is quite complex. Please read the "Installation & Setup" guide in the Wiki carefully. If you only want to use the "Virtual Race Engineer", there is a prebuilt configuration for you, which makes things much easier, but you will miss out all the other funny stuff...

 

By te way, this is a typical session with the virtual race engineer:

---------------------------------------------------------------------------------------------------------------------

Jona: "Hi, here is Jona, your race engineer today. You can call me anytime if you have questions. Good luck."

(You hear this or a similar phrase, whenever Jona is ready to interact with you. Typically this is at the beginning of the second lap in a race. From now on Jona might call you, when important information are available, or you can call her/him anytime using one of the key phrases - see the section about [phrase grammars](https://github.com/SeriousOldMan/Simulator-Controller/wiki/Virtual-Race-Engineer#phrase-grammars) on that.)

Driver: "Tell me the tyre temperatures."

Jona: "We have a blue screen here. Windows XP is crap. Give me a minute."

(This answer, or a similar one, means, that Jona does not have enough data at the moment to answer your question. Typically, Jona needs to laps at the beginning of the race or after a pitstop, to be completely up and running)

(A lap later...)

Driver: "Tell me the tyre temperatures."

Jona: "We have the following temperatures: Front left 87. 85 Degrees in the front right tyre. Rear left 93. 91 rear right."

(You can ask for other information as well. See the section about [phrase grammars](https://github.com/SeriousOldMan/Simulator-Controller/wiki/Virtual-Race-Engineer#phrase-grammars) for more information.)

(A few laps later...)

Jona: "Warning. Only 3 laps before you will run out of fuel. Should I update the pitstop strategy now?"

(With this question, Jona asks, if you want to start the preparation process for the upcoming pitstop.)

Driver: "No thank you."

(We are brave and will take another lap.)

Jona: "Okay. Call me when you are ready."

(As you can see, Jona always acjknowledges what you said. This helps you to check, whether Jona did understood you correctly.)

(A lap later...)

Jona: "What are you doing? You are running out of fuel. We should prepare for a pitstop, Okay?"

Driver: "Yes go on."

Jona: "Okay, give me a second."

(A few moments later...)

Jona: "Jona here. I recommend this for pitstop number one: ..."

(Jona gives you a complete overview over all the settings, that were derived for the upcoming pitstop, refueling, tyre compound, pressures, repairs, and so on.)

(Maybe you have some corrections...)

Driver: "Can we decrease front right by zero point two?"

Jona: "The pressure of the front right tyre shall be decreased by 0.2 PSI, is that correct?"

Driver: "Yes please"

(A moment later...)

Jona: "I updated the pitstop plan like you said. Anything else?"

Driver: "No thank you"

Jona: "Understood. I am here."

(The pitstop is now planned, but still not locked in. We are brave and stay out for the last lap...)

Jona: "Warning. You will run out of fuel in one lap. You should come in immediately. Shall I instruct the pit crew?"

(With the last question, Jona aks, whether the pitstop plan shall be locked in.)

Driver: "Ok, let's go on."

Jona: "Okay, I will let the crew prepare everything immediately."

(A few moments later...)

Jona: "We are ready for the pitstop. You can come in."

(The Pitstop input area comes to life and all the data is input automatically. See the section about the [pitstop handling](https://github.com/SeriousOldMan/Simulator-Controller/wiki/Virtual-Race-Engineer#the-pitstop) for more information.)

(You enter the pit and bring the car to a stop.)

Jona: "Okay, let the crew do their job. Check ignition, relax and prepare for engine restart."

(And now you are ready for your next stint...)

 

Hello,
Without using Jona, with what I understood we must be able to: create a virtual box, with buttons that reproduce certain actions and functions that cannot be displayed on our steering wheel?
I put an image as an example.

 

Yes, you can create a virtual button box only, but the real purpose is to model your physical button boxes and define several layers (called modes) of functions for the control elements. These layers can be activated according to current context or simulation game. The visual representation shows you what functions are currently active for the buttons, switches and dials. Very similar to Elgato Stream Deck, but more versatile, since you can define rules, which layer will be active when... For exmple, activate the Pitstop mode during a race session, or the Launch mode, when on the Windows desktop, and so on.

 

And there are a lot of plugins available for different aspects of your sim rig. For example, you can control all aspects of a motion rig from the button box or you can change the calibration curves of your Heusinkveld pedals from the button box, and ...

And you can easily code your own plugins, if you have some scripting skills.

 

@TheBigO I'm glad you got the project out successfully.
Maybe, in the future, I will propose you a project for a plugin.

 

Another week has passed. This time the following is included:

1. The virtual race engineer can be instructed not to change the tires.
2. Building on the previous change, Jona decides independently not to change tires in a Splash n Dash pit stop.
3. The treatment and analysis of accident damage very late in a race or before an upcoming pitstop has been improved.
4. In RaceRoom Racing Experience, Jona can now plan, prepare and carry out the pitstop completely on its own.
5. A controller mode is now available for rFactor 2 and RaceRoom Racing Experience, with which all pit stop settings can be controlled via a button box or another controller, but also via voice commands. As in Assetto Corsa Competizione, this can be used in conjunction with or without the help of Jona. A possible configuration can look like this:



6. A first integration is available for iRacing, which allows you to start and stop iRacing via the button box. In the next releases the whole "rest" will come
7. (For developers): Again extensive generalizations for the development of simulator plugins.

And now, happy racing and a successful pitstop ...

 

With todays release 2.8.6, another cycle in the development of the Simulator Controller Suite is slowly coming to an end. With Assetto Corsa Competizione, rFactor 2, RaceRoom Racing Experience and now also iRacing, 4 simulations are fully supported, including integration for Jona, the virtual race engineer.

The next cycle is again dedicated to artificial intelligence. If everything goes as planned, in the end we will see the driver fully supported by a virtual racing strategist. Just as Jona supports you as a race engineer with technical questions and automatically determines the settings for an upcoming pitstop, this virtual racing strategist will calculate the ideal time for a pitstop based on the positions and likely strategies of all other drivers. Completion of the automated setup database is also on the plan for the coming cycle.

 

With version 3.0 - as announced - the next development cycle begins, the further expansion of support for the driver. For some time now, the virtual race engineer has been collecting data from the various sessions and, of course only with consent, transfers it to a central server for consolidation. Release 3.0 now brings a first tool with which this setup information (tire selection and air pressure) can be queried depending on the vehicle, route and weather conditions. An extrapolation method helps to determine setup instructions for conditions that have not yet been "experienced".



Another note at this point: You always have access to your own data, you only get access to the (tire) setup information of the community if you also contribute to the development of the database. Therefore it may be time to reconsider your own consent.

 

This week's release brings many changes "under the hood" to prepare for further steps in the development of the setup database and the virtual race engineer. The most important visible change concerns the virtual race engineer, who now also asks the setup database when the tire pressures are calculated for an upcoming pit stop. This is particularly advantageous in the case of drastic changes in the environmental conditions, because then the dynamic calculations based on the data from previous laps reach their limits. When querying the setup database, extra and interpolation methods are used and a factor is used to consider how resilient the derived tire pressures are if no exact data is available. The query of the setup database can of course be configured.

 

This week's release brings more features to the setup database. In addition to information on the correct cold air pressures and the tire compound depending on the track and weather conditions, which have been collected for a long time already, the setup for mechanical and aerodynamic settings can now be uploaded to the setup database. If consent is given, this information will also be shared with the community. Consents can now be given separately for air pressure and setup information. For this reason, consent will be requested again with this release. In addition to the information mentioned, private notes can also be stored in the setup database, which are never shared with the community.

 

*sigh*

From the License file:

-------------------------------------------------------------------------------
Parts of the RF2 SHM Reader are based on code by TheIronWolfModding
(theCrewChief.org). The following license terms apply:
-------------------------------------------------------------------------------

From the code:

-------------------------------------------------------------------------------
/*
RF2 SHM Reader entry point.
Based on the work of: The Iron Wolf (vleonavicius@hotmail.com; thecrewchief.org)
*/
-------------------------------------------------------------------------------

Furthermore there are several links in the README and the documentation, that points to the GitHub page of IronWolfModding, for example here:



I have removed some comments from files, which contain > 80% of my own code, though. Anyway, I will include the original comments back there as well in the next release. Shouldn't be an affront, the work on crew chief is great,

 

This week, too, there are further improvements to the setup database and the associated integration with the virtual race engineer. The settings for a current session can now be automatically saved in the setup database and automatically loaded from there at the beginning of a new session. Another page has been added to the general configuration tool, with which the dynamic decisions and statistical processes of the virtual race engineer can be influenced or configured.

 

Release 3.1 took two weeks to mature. But a lot has happened, even if much of it is not yet fully visible. All changes serve to integrate a new virtual assistant into the platform. In addition to Jona, the virtual race engineer, the driver is now also supported by Cato, a future expert in racing strategy. Cato can currently hardly do anything, but when fully expanded he will have a complete overview of all drivers, their lap times and the development of the positions, even with different pit strategies. Cato will furthermore be able to develop appropriate pit stop strategies if you are stuck in traffic and he will be able to react to unforeseen events such as sudden weather changes and severe damage - all hand in hand with Jona, the race engineer. All of this will come step by step over the next few weeks / months.

Since two different assistants are now available as conversation partners in Voice Chat, the speech recognition has been fundamentally revised. It now allows several simultaneous dialogues and can switch between the conversation partners with an activation phrase. In addition, many internal changes were made to make further development more flexible and the rule system for the AI kernel was modularized. As always, details can be found in the release notes.

 

This week brings the first usable functionality of the virtual race strategist. The knowledge base now includes a model for the positions and current lap times of all drivers on the track as well as a complete history of all past laps. On this basis, the functions of Cato are now being developed. At the moment, on the basis of this information, Cato can provide information about your own current position in the race and provides information about the distance to the cars in front and behind, as well as their lap times compared to your pace. Corresponding information is also available for the leading vehicle on request. These functions are sure to please those of you who drive with VR glasses.

With this release, data connections have been integrated for iRacing, rFactor 2 and RaceRoom Racing Experience. The data provider for Assetto Corsa Competizione has also finshed implemented, but has not yet passed the final tests. This integration will therefore follow with the upcoming release.

Beginning with this release the sources are no longer part of the release package in order to save some space. But the sources are still openly available on GitHub.

By the way, I am currently building a Discord server for the Simulator Controller community. Feel free to join: https://discord.gg/5N8JrNr48H

 

The next step in the development of the virtual racing strategist comes with this release. Cato, the AI-based virtual racing strategist, is now able to project the development of the positions in the current race into the future using a probability model. This is based on the weighted lap times of all drivers in the last race laps. When overtaking, it is taken into account that the vehicles involved are held up for a certain period of time; this discount can be configured. The model is also able to predict the loss of time during a pit stop and thus the position in the race after the stop.

You: "Hey Cato"

Cato: "Yes, I hear you"

You: "Can you simulate the race in 6 laps?"

Cato: "Understood, give me a few seconds."

(After a few seconds...)

Cato: "You will most likely be on P 7."

In addition, the data interface for Assetto Corsa Competizione was finally tested and approved and a new activation procedure was developed for Push-To-Talk as soon as several "dialogue partners" are active.

Note: In some cases there are still problems with the position data in ACC, when a multiplayer session is active. This will be fixed in one of the next releases.