Logging Out-of-Realtime Physics Freezes [public-test]

Here we go, another VEC Silverstone freeze, about 3s this time at replay position 7055.


Zip archive with all the info, plus a README:
https://www.dropbox.com/s/48jpkhednl6sggy/VEC-S11R7-FreezeLogs-20190406.zip?dl=0

~3s Freeze during my first stint after a swap.

Reference into replay file: 7055
Reference into timeline log file NO_REPLAY_0_RealTimeLogging_PT.csv: 01016662
Reference into trace.txt: ~2732.0.

Code: 
  01016657, 0.0029741931, 0.0011021493, 0.0010425093, 7054.5875000000, 0.0095196470
  01016658, 0.0020078731, 0.0009490486, 0.0007208314, 7054.5900000000, 0.0000000000
  01016659, 0.0019859179, 0.0009341191, 0.0012278878, 7054.5925000000, 0.0000000000
  01016660, 0.0030274710, 0.0014566521, 0.0012191442, 7054.5950000000, 0.0000000000
  01016661, 0.0019818196, 0.0009104075, 0.0012376250, 7054.5975000000, 0.0093718128
  01016662, 3.0301422204, 3.0301413422, -3.0274754220, 7054.6000000000, 0.0000000000
  01016663, 0.0141209772, 0.0141203918, -0.1365969847, 7054.6025000000, 3.0338780464
  01016664, 0.0026662353, 0.0026659426, -0.1242640983, 7057.5075000000, 0.0000000000
  01016665, 0.0008855250, 0.0008846468, -0.1226499161, 7057.5225000000, 0.0000000000
  01016666, 0.0008577151, 0.0008574224, -0.1210082167, 7057.5250000000, 0.0000000000
  01016667, 0.0008331254, 0.0008328326, -0.1193419275, 7057.5275000000, 0.0000000000
  01016668, 0.0009914953, 0.0009909098, -0.1178337155, 7057.5300000000, 0.0000000000

I didn't have full realtime logging enabled, since that is too much data
to have during a ~2h stint.

Tried to cross-reference this into the trace file and came up with the timestamp ~2732.0s.
Here's what's there:

Code: 
  2585.31s steward.cpp  7621: SetSessionName( "Race" )
  2591.25s JNILog.cpp     32: DataPlugin: Received 12000 packets with 15732 bytes of data and counting.
  2651.25s JNILog.cpp     32: DataPlugin: Received 12300 packets with 15732 bytes of data and counting.
  2711.25s JNILog.cpp     32: DataPlugin: Received 12600 packets with 15732 bytes of data and counting.
  2774.04s JNILog.cpp     32: DataPlugin: Received 12900 packets with 15732 bytes of data and counting.
  2834.04s JNILog.cpp     32: DataPlugin: Received 13200 packets with 15732 bytes of data and counting.
  2852.95s NetComm.cpp 14709: === NumAI=1 NumPlayers=33 NumVehicles=26 Session=10 ET=7175.9 ===

Hope this is useful!
These continuous problems are making lots of people very unhappy.
 
hello there, just to be sure about my recent issues, could I consider several screen freezes over the whole session (including rf2 process stalling in task mgr) while being in the monitor mode, waiting for the multiplayer session to move on, as part of this bug reporting campaign?
 
I'm gonna have to add a precision here, and I think it may be important for the whole shabbang, because it would seem that there are 2 different cases of "screen freezes" in multiplayer sessions. Concerning the ones I got, when they happen (anytime, anywhere during the session, whether on track or in monitor), i confirm that the rf2 process is stuck in the task manager, and what I mean is no more CPU process time advancing, BUT no Windows message saying the process has crashed. It would seem other people who have this kind of issues, get screen freezes that last a certain time [could anyone confirm this point plz?] but they end up getting back to their session and the game is still on. These ones may provide CSV files that are relevant, but in my case, they aren't because as they are generated at the end of every session's step, when I got a "process-stuck freeze", the file I would need has never been generated.

Some friends told me I must have a bigger issue than what this post is intended for, however, as everything else works perfectly fine, as I checked the game files OK on Steam, launched a chkdsk on my rf2-only 500Go SSD disk which found nothing wrong on it, ans as I never get a screen freeze when I'm in single mode, I kind of wonder: wtf is going on there....? :) I just mean it in very practical terms, I don't rant or anything... just a baffled computer scientist dude who reached the end of his ideas on how to solve this one...
 
I'll just leave this here:
Logs/replay: https://www.dropbox.com/s/312s8nsa4rwdf79/2019-04-15 Trace.zip?dl=0

First log file:
First freeze, leader disappears just ahead of me at 1872.208 (42:32 in livestream below)
Second freeze, 3rd place disappears just ahead of me at 1929.171 (43:32 in livestream below)
I believe there were other freezes with the other drivers but I don't know the positions.

Then for me i get "lost connection" message approx 2484.495, I get kicked to the server main screen, click "race", rejoin
Livestream at the point where connection to the dedicated server is lost (52:50).
Now that I've watched the stream again I just noticed RF2 was giving me a warning in the lower right about being ahead of the car I was to suppose to follow a handful of seconds before the connection was lost to the server. Very strange that warning was not in the LSI.

Second log file:
After rejoining I get a drive thru penalty for "accelerating too quickly" trying to catch up to the field and the admin command for clearing the drive through wouldn't work.

So frustrating
 
Last edited:
@cosimo do you have any plugins on?
If so which ones?
Do those stalls happen without any plugins or with plugins off?
In your case does it happen in single player as well?
Thanks
 
Hello, with the release of the new build we are also opening a steam beta branch dedicated to the physics timing issues [rubberbanding/out-of-realtime] that we're currently resolving.

This branch [build ID: 3750946] at the moment contains the following changes:

- More reliable physics timing, this should improve recovery and add stability to the physics thread.
- RealtimeLogging is outputting only 2 .csv files now, *PT.csv (this gives info about PhysicsTiming) and *PSPFP.csv (sorry for the name, this records the most computationally expensive areas during the execution of a physics frame).
- RealtimeLogging records now only 2 hours (but it's way more precise)
- The *PT.csv file is the same size, while the *PSPFP.csv size has been reduced significantly, now it should be in the order of some MB per session so PLEASE TURN RealtimeLogging ON! in your player.json file.
Having RealtimeLogging turned on will make rFactor generate the *PSPFP.csv file and we will know which area is causing issues in your machine and we can act directly on it.

Please join this branch if you're having issues and let us know if the situation for you is the same, got worse or improved.
More updates will follow, we need to proceed incrementally making sure step by step that our changes do not aggravate the physics timing on a wide variety of PCs.

branch name: public-test
password : UpERYPhIblEHorTiO
verify your files via Steam after joining the branch

1. Select rFactor 2 from steam game library, right click and select properties.
A dialog box with a number of tabs will appear, your looking for one that says "BETAS".
2. select the "BETAS" tab it by clicking on it.
You will see 2 things now, a drop down menu and a text box to enter a beta access code.
3. enter the password provided,
4. select public-beta from the dropdown menu

post your logs here even if you do not encounter issues, we will look at them all.
 
Last edited:
4thworld said:
Hello, with the release of the new build we are also opening a steam beta branch dedicated to the physics timing issues [rubberbanding/out-of-realtime] that we're currently resolving.

This branch [build ID: 3750946] at the moment contains the following changes:

- More reliable physics timing, this should improve recovery and add stability to the physics thread.
- RealtimeLogging is outputting only 2 .csv files now, *PT.csv (this gives info about PhysicsTiming) and *PSPFP.csv (sorry for the name, this records the most computationally expensive areas during the execution of a physics frame).
- RealtimeLogging records now only 2 hours (but it's way more precise)
- The *PT.csv file is the same size, while the *PSPFP.csv size has been reduced significantly, now it should be in the order of some MB per session so PLEASE TURN RealtimeLogging ON! in your player.json file.
Having RealtimeLogging turned on will make rFactor generate the *PSPFP.csv file and we will know which area is causing issues in your machine and we can act directly on it.

Please join this branch if you're having issues and let us know if the situation for you is the same, got worse or improved.
More updates will follow, we need to proceed incrementally making sure step by step that our changes do not aggravate the physics timing on a wide variety of PCs.

branch name: public-test
password : UpERYPhIblEHorTiO
verify your files via Steam after joining the branch

post your logs here even if you do not encounter issues, we will look at them all.
Click to expand...
Just to try it out, i have made a short online test with the public-test beta and had have a freeze that lasted around 1 second.
But i can not see it in the replay, what is a bit strange, i guess.
Anyway, i have uploaded the generated PT and PSPFP file from that session (RealTimeLogging was set to '1')
I hope that helps.
https://www.dropbox.com/s/42sjdmpnc1aa21i/Bernd Nordschleife RealTimeLog.zip?dl=1

EDIT Short update. I have repeated the same thing, because after joining the beta, i had backupped my userdata folder and have let the game generate a new one.
The new one i have used with the original network setting then, that uses 256 KBPS as up and download rate. With that i have had the freeze.
Now, when i have made the second try, i have changed it to my usual setting 'LAN' with 5000 KBPS up and download rate.
With that i have had no freeze.
Maybe it generally could help, if the default connection speed gets set higher.
 
Last edited:
@Bernd I don't think the issue is bandwidth related, in order to prove that it would require more tests with the 2 different bandwidth settings.

Your logs show a peak in the gfx thread time (of 0.5sec), so it's not physics.
The gfx peak happens more or less at 50% of your session (674 seconds in):

I have the following question anyways:
-> Was damage enabled in the server?
-> At the point of the freeze was any car pitting / joining / was there an accident or something else out of the normal?
-> do you have any external plugin activated? This includes additional post processing libs etc..
-> which cars were available on the server?
-> can you post your specs (you can also write them down in your signature, it's the same).
-> can i have the replay of the session?
thanks
 
Last edited:
4thworld said:
@Bernd I don't think the issue is bandwidth related, in order to prove that it would require more tests with the 2 different bandwidth settings.

Your logs show a peak in the gfx thread time (of 0.5sec), so it's not physics.
The gfx peak happens more or less at 50% of your session (674 seconds in):

I have the following question anyways:
-> Was damage enabled in the server?
-> At the point of the freeze was any car pitting / joining / was there an accident or something else out of the normal?
-> do you have any external plugin activated? This includes additional post processing libs etc..
-> which cars were available on the server?
-> can you post your specs (you can also write them down in your signature, it's the same).
-> can i have the replay of the session?
thanks
Click to expand...
Thank you for the reply.
Here is the info:
- damage was enabled, yes
- i was alone on the server and normally driving on the track, no AI
- the only active plugins have been SteamPlugin and TrackIR_rF2_Plugin
- here is a screenshot of the car selection screen (i hope that's OK)
The server is : ~IsR~Nordschleife Hot Lap


ISR_Nordschleife-Fahrzeuge.jpg


My specs are:
Windows 7 Ultimate 64-bit
MB: Z170X-Gaming 7
Proc: i5-6600K @ 3.5 GHz, 4core
Ram: 8GB
Grafics Card: AMD Radeon R7 370 4 GB Ram (maybe that is a bit weak, but it normally worked without freezes)
"Monitor": TV, 60 Hz, 1024x768

I hope i didn't forget something.

And here is the replay file:
 

Attachments

@Bernd thanks, since your problem seems different from what we're looking at atm (because it's on the GFX thread) can you send me a PM
with the car you used, track and your userdata folder zipped?
so we don't pollute this thread too much.
also I would like to know if this happens to you in single player.
(for some reason i can't start a conversation myself, the forum admin is looking into it)
 
Small update new build ID [3753005]
- you can now enable/disable visual damage from the player.json file.
this has been made to test some possible cause for GFX thread stalls.

branch name: public-test
password : UpERYPhIblEHorTiO
verify your files via Steam after joining the branch
 
Last edited:
Can we also have an option to disable moving objects like helis please? You see, in Donington, there are 2 helis. I know, its not a s397 track, but: One is moving, the other not or even more scary, only partly. Offline it moved sometimes, then stopped, moved, stopped... online it was just hanging in the air. This really scares me.... and it was a track where I encountered freezes (however I think I was on agressive threading)
thx
 
@Stefan_L_01 , thanks, that looks like an animation issue only, it shouldn't be anything that could harm physics computation time with stalls etc..
If you are experiencing issues, please post your logs so we can determine what is causing harm on your machine with more precision.
 
@4thworld I have not had a rubber-banding issue before the update, just had some micro freezes. However, after the latest update I now have flashing of the screen whilst in the pit menu and rubber-banding [This was offline with 40 cars using the VEC Le Mans track]. Further, when I exit from the sim, the sim hangs and does not exit to the launcher - I have to close SteamVR to get out of the sim. I have attached a couple of logs from this morning.

I note that you said "The *PT.csv file is the same size, while the *PSPFP.csv size has been reduced significantly, now it should be in the order of some MB per session". My PSPFP.csv files from very short sessions are 300MB and 600MB in size!! Hope that this helps [PC specs should be in my signature].

I also note that my build is showing as 3750893 and not 3750946

https://1drv.ms/u/s!AtJBkmguK43g1gLGp3iCjBW4nTTg
 
Last edited:
@Mark Fuller , thanks, from the logs i can see that you're not in the public-test steam branch.
Can you try to join the branch and verify your steam file and try again?
Leave the previous log in your message i'll have a look at them anyways
 
You must log in or register to reply here.
Back
Top