Logging Out-of-Realtime Physics Freezes [public-test]

Discussion in 'Technical & Support' started by Christopher Elliott, Mar 19, 2019.

  1. McFlex

    McFlex Registered

    Joined:
    Feb 23, 2012
    Messages:
    1,031
    Likes Received:
    317
    Had two hard lags during our last league race. Lap 11 and Lap 29. You can also see them in my livestreams. See attached links

    Code:
    https://youtu.be/JMIFND3trKQ?t=3801
    https://youtu.be/JMIFND3trKQ?t=5540

    Here are also my log files
    https://www.dropbox.com/s/rmf7nxj5m5f0zq3/Log.7z?dl=0
     
  2. cosimo

    cosimo Registered

    Joined:
    Apr 14, 2013
    Messages:
    827
    Likes Received:
    99
    Here we go, another VEC Silverstone freeze, about 3s this time at replay position 7055.



    Zip archive with all the info, plus a README:
    https://www.dropbox.com/s/48jpkhednl6sggy/VEC-S11R7-FreezeLogs-20190406.zip?dl=0

    ~3s Freeze during my first stint after a swap.

    Reference into replay file: 7055
    Reference into timeline log file NO_REPLAY_0_RealTimeLogging_PT.csv: 01016662
    Reference into trace.txt: ~2732.0.

    Code:
      01016657, 0.0029741931, 0.0011021493, 0.0010425093, 7054.5875000000, 0.0095196470
      01016658, 0.0020078731, 0.0009490486, 0.0007208314, 7054.5900000000, 0.0000000000
      01016659, 0.0019859179, 0.0009341191, 0.0012278878, 7054.5925000000, 0.0000000000
      01016660, 0.0030274710, 0.0014566521, 0.0012191442, 7054.5950000000, 0.0000000000
      01016661, 0.0019818196, 0.0009104075, 0.0012376250, 7054.5975000000, 0.0093718128
      01016662, 3.0301422204, 3.0301413422, -3.0274754220, 7054.6000000000, 0.0000000000
      01016663, 0.0141209772, 0.0141203918, -0.1365969847, 7054.6025000000, 3.0338780464
      01016664, 0.0026662353, 0.0026659426, -0.1242640983, 7057.5075000000, 0.0000000000
      01016665, 0.0008855250, 0.0008846468, -0.1226499161, 7057.5225000000, 0.0000000000
      01016666, 0.0008577151, 0.0008574224, -0.1210082167, 7057.5250000000, 0.0000000000
      01016667, 0.0008331254, 0.0008328326, -0.1193419275, 7057.5275000000, 0.0000000000
      01016668, 0.0009914953, 0.0009909098, -0.1178337155, 7057.5300000000, 0.0000000000
    
    I didn't have full realtime logging enabled, since that is too much data
    to have during a ~2h stint.

    Tried to cross-reference this into the trace file and came up with the timestamp ~2732.0s.
    Here's what's there:

    Code:
      2585.31s steward.cpp  7621: SetSessionName( "Race" )
      2591.25s JNILog.cpp     32: DataPlugin: Received 12000 packets with 15732 bytes of data and counting.
      2651.25s JNILog.cpp     32: DataPlugin: Received 12300 packets with 15732 bytes of data and counting.
      2711.25s JNILog.cpp     32: DataPlugin: Received 12600 packets with 15732 bytes of data and counting.
      2774.04s JNILog.cpp     32: DataPlugin: Received 12900 packets with 15732 bytes of data and counting.
      2834.04s JNILog.cpp     32: DataPlugin: Received 13200 packets with 15732 bytes of data and counting.
      2852.95s NetComm.cpp 14709: === NumAI=1 NumPlayers=33 NumVehicles=26 Session=10 ET=7175.9 ===
    
    Hope this is useful!
    These continuous problems are making lots of people very unhappy.
     
    The Iron Wolf and lonestar29 like this.
  3. scalino

    scalino Registered

    Joined:
    Oct 21, 2013
    Messages:
    52
    Likes Received:
    2
    hello there, just to be sure about my recent issues, could I consider several screen freezes over the whole session (including rf2 process stalling in task mgr) while being in the monitor mode, waiting for the multiplayer session to move on, as part of this bug reporting campaign?
     
  4. scalino

    scalino Registered

    Joined:
    Oct 21, 2013
    Messages:
    52
    Likes Received:
    2
    I'm gonna have to add a precision here, and I think it may be important for the whole shabbang, because it would seem that there are 2 different cases of "screen freezes" in multiplayer sessions. Concerning the ones I got, when they happen (anytime, anywhere during the session, whether on track or in monitor), i confirm that the rf2 process is stuck in the task manager, and what I mean is no more CPU process time advancing, BUT no Windows message saying the process has crashed. It would seem other people who have this kind of issues, get screen freezes that last a certain time [could anyone confirm this point plz?] but they end up getting back to their session and the game is still on. These ones may provide CSV files that are relevant, but in my case, they aren't because as they are generated at the end of every session's step, when I got a "process-stuck freeze", the file I would need has never been generated.

    Some friends told me I must have a bigger issue than what this post is intended for, however, as everything else works perfectly fine, as I checked the game files OK on Steam, launched a chkdsk on my rf2-only 500Go SSD disk which found nothing wrong on it, ans as I never get a screen freeze when I'm in single mode, I kind of wonder: wtf is going on there....? :) I just mean it in very practical terms, I don't rant or anything... just a baffled computer scientist dude who reached the end of his ideas on how to solve this one...
     
    Neawoulf and Rui Santos like this.
  5. Brent

    Brent Registered

    Joined:
    Nov 5, 2015
    Messages:
    236
    Likes Received:
    111
    I'll just leave this here:
    Logs/replay: https://www.dropbox.com/s/312s8nsa4rwdf79/2019-04-15 Trace.zip?dl=0

    First log file:
    First freeze, leader disappears just ahead of me at 1872.208 (42:32 in livestream below)
    Second freeze, 3rd place disappears just ahead of me at 1929.171 (43:32 in livestream below)
    I believe there were other freezes with the other drivers but I don't know the positions.

    Then for me i get "lost connection" message approx 2484.495, I get kicked to the server main screen, click "race", rejoin
    Livestream at the point where connection to the dedicated server is lost (52:50).

    Now that I've watched the stream again I just noticed RF2 was giving me a warning in the lower right about being ahead of the car I was to suppose to follow a handful of seconds before the connection was lost to the server. Very strange that warning was not in the LSI.

    Second log file:
    After rejoining I get a drive thru penalty for "accelerating too quickly" trying to catch up to the field and the admin command for clearing the drive through wouldn't work.

    So frustrating
     
    Last edited: Apr 17, 2019
  6. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    @cosimo do you have any plugins on?
    If so which ones?
    Do those stalls happen without any plugins or with plugins off?
    In your case does it happen in single player as well?
    Thanks
     
  7. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    Hello, with the release of the new build we are also opening a steam beta branch dedicated to the physics timing issues [rubberbanding/out-of-realtime] that we're currently resolving.

    This branch [build ID: 3750946] at the moment contains the following changes:

    - More reliable physics timing, this should improve recovery and add stability to the physics thread.
    - RealtimeLogging is outputting only 2 .csv files now, *PT.csv (this gives info about PhysicsTiming) and *PSPFP.csv (sorry for the name, this records the most computationally expensive areas during the execution of a physics frame).
    - RealtimeLogging records now only 2 hours (but it's way more precise)
    - The *PT.csv file is the same size, while the *PSPFP.csv size has been reduced significantly, now it should be in the order of some MB per session so PLEASE TURN RealtimeLogging ON! in your player.json file.
    Having RealtimeLogging turned on will make rFactor generate the *PSPFP.csv file and we will know which area is causing issues in your machine and we can act directly on it.

    Please join this branch if you're having issues and let us know if the situation for you is the same, got worse or improved.
    More updates will follow, we need to proceed incrementally making sure step by step that our changes do not aggravate the physics timing on a wide variety of PCs.

    branch name: public-test
    password : UpERYPhIblEHorTiO
    verify your files via Steam after joining the branch

    1. Select rFactor 2 from steam game library, right click and select properties.
    A dialog box with a number of tabs will appear, your looking for one that says "BETAS".
    2. select the "BETAS" tab it by clicking on it.
    You will see 2 things now, a drop down menu and a text box to enter a beta access code.
    3. enter the password provided,
    4. select public-beta from the dropdown menu

    post your logs here even if you do not encounter issues, we will look at them all.
     
    Last edited: Apr 19, 2019
    Dookie, Rui Santos, MaD_King and 4 others like this.
  8. Daniele Vidimari

    Daniele Vidimari Registered

    Joined:
    Apr 20, 2016
    Messages:
    659
    Likes Received:
    593
    Are we able to join standard version servers after joining this beta?
     
  9. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    yes you should be able to join standard servers.
     
    Daniele Vidimari likes this.
  10. cosimo

    cosimo Registered

    Joined:
    Apr 14, 2013
    Messages:
    827
    Likes Received:
    99
    Hi,
    just had the steam plugin on, nothing else.
     
    The Iron Wolf likes this.
  11. Bernd

    Bernd Registered

    Joined:
    Jul 9, 2013
    Messages:
    1,509
    Likes Received:
    755
    Just to try it out, i have made a short online test with the public-test beta and had have a freeze that lasted around 1 second.
    But i can not see it in the replay, what is a bit strange, i guess.
    Anyway, i have uploaded the generated PT and PSPFP file from that session (RealTimeLogging was set to '1')
    I hope that helps.
    https://www.dropbox.com/s/42sjdmpnc1aa21i/Bernd Nordschleife RealTimeLog.zip?dl=1

    EDIT Short update. I have repeated the same thing, because after joining the beta, i had backupped my userdata folder and have let the game generate a new one.
    The new one i have used with the original network setting then, that uses 256 KBPS as up and download rate. With that i have had the freeze.
    Now, when i have made the second try, i have changed it to my usual setting 'LAN' with 5000 KBPS up and download rate.
    With that i have had no freeze.
    Maybe it generally could help, if the default connection speed gets set higher.
     
    Last edited: Apr 19, 2019
  12. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    @Bernd I don't think the issue is bandwidth related, in order to prove that it would require more tests with the 2 different bandwidth settings.

    Your logs show a peak in the gfx thread time (of 0.5sec), so it's not physics.
    The gfx peak happens more or less at 50% of your session (674 seconds in):

    I have the following question anyways:
    -> Was damage enabled in the server?
    -> At the point of the freeze was any car pitting / joining / was there an accident or something else out of the normal?
    -> do you have any external plugin activated? This includes additional post processing libs etc..
    -> which cars were available on the server?
    -> can you post your specs (you can also write them down in your signature, it's the same).
    -> can i have the replay of the session?
    thanks
     
    Last edited: Apr 19, 2019
    Rui Santos likes this.
  13. Bernd

    Bernd Registered

    Joined:
    Jul 9, 2013
    Messages:
    1,509
    Likes Received:
    755
    Thank you for the reply.
    Here is the info:
    - damage was enabled, yes
    - i was alone on the server and normally driving on the track, no AI
    - the only active plugins have been SteamPlugin and TrackIR_rF2_Plugin
    - here is a screenshot of the car selection screen (i hope that's OK)
    The server is : ~IsR~Nordschleife Hot Lap


    ISR_Nordschleife-Fahrzeuge.jpg

    My specs are:
    Windows 7 Ultimate 64-bit
    MB: Z170X-Gaming 7
    Proc: i5-6600K @ 3.5 GHz, 4core
    Ram: 8GB
    Grafics Card: AMD Radeon R7 370 4 GB Ram (maybe that is a bit weak, but it normally worked without freezes)
    "Monitor": TV, 60 Hz, 1024x768

    I hope i didn't forget something.

    And here is the replay file:
     

    Attached Files:

  14. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    @Bernd thanks, since your problem seems different from what we're looking at atm (because it's on the GFX thread) can you send me a PM
    with the car you used, track and your userdata folder zipped?
    so we don't pollute this thread too much.
    also I would like to know if this happens to you in single player.
    (for some reason i can't start a conversation myself, the forum admin is looking into it)
     
  15. Bernd

    Bernd Registered

    Joined:
    Jul 9, 2013
    Messages:
    1,509
    Likes Received:
    755
    I start a private conversation, thank you.
     
  16. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    Small update new build ID [3753005]
    - you can now enable/disable visual damage from the player.json file.
    this has been made to test some possible cause for GFX thread stalls.

    branch name: public-test
    password : UpERYPhIblEHorTiO
    verify your files via Steam after joining the branch
     
    Last edited: Apr 19, 2019
  17. Stefan_L_01

    Stefan_L_01 Registered

    Joined:
    Nov 18, 2012
    Messages:
    595
    Likes Received:
    386
    Can we also have an option to disable moving objects like helis please? You see, in Donington, there are 2 helis. I know, its not a s397 track, but: One is moving, the other not or even more scary, only partly. Offline it moved sometimes, then stopped, moved, stopped... online it was just hanging in the air. This really scares me.... and it was a track where I encountered freezes (however I think I was on agressive threading)
    thx
     
  18. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    @Stefan_L_01 , thanks, that looks like an animation issue only, it shouldn't be anything that could harm physics computation time with stalls etc..
    If you are experiencing issues, please post your logs so we can determine what is causing harm on your machine with more precision.
     
  19. Mark Fuller

    Mark Fuller Registered

    Joined:
    Dec 10, 2016
    Messages:
    343
    Likes Received:
    99
    @4thworld I have not had a rubber-banding issue before the update, just had some micro freezes. However, after the latest update I now have flashing of the screen whilst in the pit menu and rubber-banding [This was offline with 40 cars using the VEC Le Mans track]. Further, when I exit from the sim, the sim hangs and does not exit to the launcher - I have to close SteamVR to get out of the sim. I have attached a couple of logs from this morning.

    I note that you said "The *PT.csv file is the same size, while the *PSPFP.csv size has been reduced significantly, now it should be in the order of some MB per session". My PSPFP.csv files from very short sessions are 300MB and 600MB in size!! Hope that this helps [PC specs should be in my signature].

    I also note that my build is showing as 3750893 and not 3750946

    https://1drv.ms/u/s!AtJBkmguK43g1gLGp3iCjBW4nTTg
     
    Last edited: Apr 19, 2019
  20. 4thworld

    4thworld

    Joined:
    Dec 17, 2016
    Messages:
    286
    Likes Received:
    950
    @Mark Fuller , thanks, from the logs i can see that you're not in the public-test steam branch.
    Can you try to join the branch and verify your steam file and try again?
    Leave the previous log in your message i'll have a look at them anyways
     

Share This Page