A suggestion re: faster load times within the game - ram-disk for shader caches

Discussion in 'General Discussion' started by muz_j, Nov 30, 2020.

  1. muz_j

    muz_j Registered

    Joined:
    Aug 20, 2014
    Messages:
    264
    Likes Received:
    188
    I just randomly stumbled on to a pretty good tip if you're wanting to reduce load times when starting up a race that didn't occur to me, but someone else was trying to reduce load times on linux and I've just happily confirmed it results in a significant speed boost on windows too - re-direction of the game's shader cache files to a ram-disk.

    The tip came from this post on reddit:
    https://www.reddit.com/r/rfactor2/comments/k45j6h/nordschleiffe_load_times/

    In terms of instructions - if you've redirected any directories on windows using sim-links (junction points), then this is just a repeat of that, but to a ram disk, rather than an SSD etc.

    In terms of what ram-disk tool to use - I've used imdisk for multiple years and that's my preferred ram disk utility on windows.
    https://sourceforge.net/projects/imdisk-toolkit/

    It's got a command line interface, so the process of moving the original folder, creating the ram-disk, copying the directory (or multiple if you want) is easy to script.

    Here's a rough guide regarding imdisk's command line:
    https://forums.guru3d.com/threads/g...in-windows-with-no-limit-on-disk-size.356046/
    This is more concise: https://gist.github.com/stokito/19e377c872dd85ee4445eabce97fa2e8
    ...and there's official documentation for it too.

    I might script something later if I can be bothered and to make things easier, I've redirected the "...\Steam\SteamApps\common\rFactor 2\UserData\Log" directory, which contains the shader cache files in sub-folders.
    Alternatively you could obviously just redirect the "Cbash" & "Shaders" sub-folders.

    >>>>> By doing this you are going to reduce your available system memory by the amount that is used to create the ram-disk.
    So for some people this is likely to not be useful (as you need to balance available system memory, versus the size of the shader cache folders).

    Lastly you should be keeping a backup copy of anything you redirect using sim-links, as a ram-disk is temporary. Losing shader cache files is obviously not important, but "Log" folder contains race result logs and other useful game logs that should be kept (unless you want to trash them for some reason).

    I haven't put a stop watch on it yet, but the time reduction was obvious and you can see the time figures for person who suggested it in the reddit thread that I linked above.
     
    makan, 4F0X4, Bernat and 1 other person like this.
  2. Bernat

    Bernat Registered

    Joined:
    Jun 15, 2020
    Messages:
    757
    Likes Received:
    585
    Hi. I wrote the post on Reddit.

    With a ram disk of only 50MB for Shaders the loading time shrinked to 50% for Nordschleiffe. This directory doesn't need to be persisted to disk and 50MB seems to be more than enough. It's the easiest one to configure and the impact on ram usage is very small.

    CBash improves loading times even more but it's a bit more work, eats more ram and it should be persisted to disk.

    I'm left wondering why the file system cache isn't already keeping these directories in cache and speeding up loading.
     
    makan and muz_j like this.
  3. muz_j

    muz_j Registered

    Joined:
    Aug 20, 2014
    Messages:
    264
    Likes Received:
    188
    Yeah there's a little bit of maths involved to work out the correct sizing of it and I saw your comment on reddit.
    I realised that as soon as I'd done the speed test, then thought the size of the cache directories was small and found I had only 3 tracks cached in those folders since the last time I cleared them (which I think I did after the last update to the Nurburgring, as there were lighting changes etc).

    I am thinking I might script something to specific to the two folders and see if the load times increase much with the game recreating the files each time and if not, I might just leave it like that - getting most of the benefits, but with the least amount of hassle (from my perspective).
    By script - I mean the end result would be a single icon, pointing to a script to create and config the ram-disk and folders on it.
    It's a minimal amount of effort to do it manually, but it could be made even easier.
    Sim-links would only need to be modified if the drive letter being used for the ram-disk changed.
    Ram-disk size would need to be big enough to handle as many tracks as you want to be able to load in a single session.

    As is - it's been confirmed by both the OP on reddit, me and at least a couple of others (eg you) that there's a definite & significant performance benefit.
     
  4. Bernat

    Bernat Registered

    Joined:
    Jun 15, 2020
    Messages:
    757
    Likes Received:
    585
    In my testing, starting with an empty CBash negates the gain from moving it to a ram disk. The first time we load a track it would be slower due to shader recompiling or whatever is doing. Ideally, only the file that is going to be used would be moved automatically to the ram disk, then removed when it's finished. That's pretty close to what the disk cache should be doing but for some reason it doesn't.

    I've also thought about having a ram disk with the slowest loading tracks shader files (CBash content) and start it only when using these tracks. Switching the ram disk on and off should be possible when in the game menu.
     
    Last edited: Dec 1, 2020
    muz_j likes this.
  5. muz_j

    muz_j Registered

    Joined:
    Aug 20, 2014
    Messages:
    264
    Likes Received:
    188
    Interesting re: the comment regarding having an empty CBash folder. I did actually try that yesterday, but as I haven't put a stop watch on it (before & after) and it wasn't a large track, I'm really not sure if it made any difference.

    Yeah your last comment would be nice to have, but stating the obvious - I think the only ways you could do this would be modify the game's code to include that sort of functionality, or alternatively write a memory resident program to do it "on the fly" while the game is running.
    Symbolic Links can be created on a per file basis, rather than per directory (you can obviously use both too), so you could take that approach, and re-direct to the ram-disk for some tracks and not others.

    There are a couple of features of imdisk that might be of interest or relevant if people want to play around with them.
    1) you can have it automatically start with the OS
    2) you can have the content be persistent across multiple reboots
    3) you can enable dynamic allocation of space

    I'm just going to leave it to others to decide if it's worth the effort or hassle, as there's a couple of ways I can see that doing this could catch people out and cause issues - low memory being one and running out of space on the ram-disk if it's not dynamic being another.
    The most obvious application is with tracks that are very slow to load - Nordschleife / Nurburgring being the prime candidate for me.
    It's an interesting find though.
     
  6. stonec

    stonec Registered

    Joined:
    Jun 19, 2012
    Messages:
    3,399
    Likes Received:
    1,489
    I'll test this once I have more time. But just to be clear, you are saying that moving a small 50 MB folder reduced loading times by double on Nordschleife? That doesn't seem right unless there are fundamental flaws how the application was written. My understanding is that most of the delay comes from the CPU decrypting MAS files and any I/O that's relevant comes from reading those files.

    Did you attempt this multiple times to be sure? I'm only asking because rF2 has this well-known feature where the first loading of track is much slower and even subsequent loading of the same track is sometimes still too slow until you do a full restart of the game. If you have any time to repeat this, could you time 5-10 loading attempts of the same track with and without shaders mounted to RAM disk?
     
    Kahel likes this.
  7. muz_j

    muz_j Registered

    Joined:
    Aug 20, 2014
    Messages:
    264
    Likes Received:
    188
    Read the original reddit thread (which has the OP's own times). I've personally tested and it confirmed it.
    Why don't you test it yourself if you want to verify it ?
    And I'm not being a smart @ss. But if you have any interest - test it out. If not, then either choose to accept it or not.

    Your comment:
    "I'm only asking because rF2 has this well-known feature where the first loading of track is much slower and even subsequent loading of the same track is sometimes still too slow until you do a full restart of the game."

    ...doesn't make sense to me.
    How can a game have an issue with a slow load on the first attempt and then subsequent attempts are still slow, until the game's restarted, which takes you back to step 1 ?
     
  8. muz_j

    muz_j Registered

    Joined:
    Aug 20, 2014
    Messages:
    264
    Likes Received:
    188
    I'm going to admit that I just put a stop watch on it back to back and the time difference is actually a hell a lot smaller than I had thought.
    By "hell of a lot smaller" I'm meaning around 8 seconds.

    Whatever was going on with slow loading within the game in the past, isn't occurring now and I think this is a classic instance of a placebo effect and (me) not actually testing things objectively to verify someone else's claim and probably confirmation bias.
    There clearly is a difference, but not even close to what I had thought was going on - and after looking at the results I just got, I wouldn't bother with using a ram-disk for this.

    Apologies for any inconvenience caused as a result.
    Facts are facts and this isn't worth the time or the effort.
     
    Last edited: Dec 2, 2020
    atomed, makan and Lazza like this.
  9. Bernat

    Bernat Registered

    Joined:
    Jun 15, 2020
    Messages:
    757
    Likes Received:
    585
    I did several tests re-starting the game every time. Since I've read countless posts about slow loading times without real numbers so I wanted to know what did slow mean. I first measured my loading times to confirm they were very high, then I wanted to know how they compared to other ones.

    It seems to be a problem on my part, not in the testing but something in the system is making it very slow. I'll test again trying to factor in some more variables.

    Thanks for the comments.
     
  10. Bernat

    Bernat Registered

    Joined:
    Jun 15, 2020
    Messages:
    757
    Likes Received:
    585
    Well, I'm trying again and now I get good loading times without using ram disks. Around 1m 40s for Nordschleiffe.

    I tested several times with a stopwatch and a friend was having the same issue, in fact, I was testing it in the first place because of him telling me about the very high loading times. It was the first time I used a stopwatch and I assumed the load times had always been high but didn't really know by how much and lately maybe got even higher.

    Now I'm trying to reproduce the higher load times by filling the SSD and shutting the internet conection down to try some things that could have failed but nothing.

    Could it be that after some update the load times went up significantly not only for the first load but a few of them? Maybe it's slower the first 9-10 times then it's faster for all subsequent loadings? That would be weird too but...

    At least I hope this thread is useful in case anyone gets hit by this weird issue.
     
    Last edited: Dec 2, 2020
  11. stonec

    stonec Registered

    Joined:
    Jun 19, 2012
    Messages:
    3,399
    Likes Received:
    1,489
    Exactly this, it doesn't make sense to me either, but I confirmed it a couple of months ago. Whenever you empty CBash/Shaders, you first have one really long loading (close to five minutes on Nordschleife), followed up by around two minute loading times if you reload the same track without exiting the game. After exiting and restarting the game you get back to those 1m30sec loading times, which seem to be the best case scenario for this track. I haven't found a way to improve loading times further from that (a new Ryzen 5000 series CPU might bring maybe 10% improvement over my Ryzen 5 3600). But yeah, thanks for confirming @Bernat, this is indeed very tricky to test with the fluctuations we are seeing.
     

Share This Page