Server disappearing from the lobby and/or crashing

Discussion in 'Technical Archives' started by Marco Bijl, Mar 3, 2013.

  1. Marco Bijl

    Marco Bijl Registered

    Joined:
    Dec 23, 2011
    Messages:
    283
    Likes Received:
    8
    @Diablo,
    THe advanced options button would be a fine sollution indeed. Prevent "regular clients" from mistakes, and allowing us (server side) to use the features sounds fair enough.

    @Lazza,
    How sure are you about your connection? Like I said, I have just about the same install running, and have not had any drop offs since the new build.

    @Jeremy Miller,
    Is it possible to have the dedicated server.exe make a decent, informative record in the Windows Event Viewer? That would make troubleshooting a lot more easy with drop offs. Not sure how that works though, so forgive me if I ask something stupid here :).
     
  2. Lazza

    Lazza Registered

    Joined:
    Oct 5, 2010
    Messages:
    12,345
    Likes Received:
    6,572
    Haven't done any explicit checks, but have an rF1 server on the same box without problems and sometimes multiple drivers sitting on the server when it falls off, without any interruption or problems for them. Plus the firewall (Server 2008 R2) log contains no problems.

    Twice now I've refreshed the matchmaker multiple times and the server's not there... then 5 mins later I try again and it's reappeared. So that's a strange one.

    I'm prepared for the possibility it's something else we've got going on with the server that's doing it, so if it's fine for everyone else we'll have to assume that's the case.
     
  3. Diablo

    Diablo Registered

    Joined:
    Jan 20, 2013
    Messages:
    404
    Likes Received:
    0
    Lazza, about the issues you´ve mentioned, with build 134 I saw the server dropping off the matchmaker quite often. With 146, I cannot say for sure, because everytime I logged in I saw it was either connected and running or had already crashed. So far I haven´t seen either dropping off or crashing with our server.

    And one more thing I´ve just noticed, maybe only interesting for Jeremy. My original installation wouldn´t let me run anything anymore, even though, the data.path pointed to the correct location. For testing purposes I had manipulated data.path to point to <rfactor 2 root>\Core and I had forgotten that when I just tried to start a client session. I have changed it back with the Launcher not running at the time but I still got the message "no mods installed". I had to go to the advanced tab to click change Data Path, even though it already pointed to the correct location, and select the same path again. So there must be a location, where this value gets cached, haven´t found it yet though.

    Edit: In my opinion an internet connection that occasionally looses the odd packet should not cause the total loss of connection to the matchmaker. The internet being what it is can cause such packet loss anywhere during transit, so the application should take care of keeping the connection. If it still falls of the radar it is a bug to be fixed.
     
  4. Radar

    Radar Registered

    Joined:
    Oct 20, 2010
    Messages:
    687
    Likes Received:
    60
    Thanks for that.. That makes sense.. :)

    Lazza / JB. One of mine dropped last night of Matchmaker, the other didn't. I went through restarting the weekend and it still didn't re-appear. Only way I could get it back on matchmaker, was to close the dedi.exe, and restart it.

    FYI. My connection is fibre, so I tend to aggree with Diablo on the packet losses.

     
  5. Lazza

    Lazza Registered

    Joined:
    Oct 5, 2010
    Messages:
    12,345
    Likes Received:
    6,572
    Yep, one thing you could certainly say about 146: you knew what it was doing ;)

    Our server is a dedi box in a data centre, so I'm inclined to think the connection is pretty good. But when it's off the list it pretty much always says it's updating fine on the server, so unless its own check isn't quite right (so it can't reconnect because it thinks it already is) which probably only ISI can judge, the problem is potentially more client-side.

    I haven't really seen the same unreliable list updates we sometimes got in rF1 (where a particular server might need several list refreshes to show up, despite the server-list-setting-increase fix), I've had a couple now where upon booting up the rF2 servers aren't quite sorted right, and possibly once it seemed like a short list and a refresh showed everything, but for the most part the list has looked 'right' and several refreshes haven't shown the server. And as I mentioned above, along with tens of restarts to make it show up I've twice had it reappear by itself (I was waiting for a driver to finish a run before restarting the server) which is a very curious thing; temping to think it might do that every time but the first time 156 went down I left it up for an hour and a half before restarting with no luck.

    Plus when it's had issues the dedi exe doesn't shut down cleanly, but that's an area I'm inclined to blame on our content or a plugin.

    Anyway, I'm going to again try a completely clean install on separate ports and standard ISI content only and see if it still happens.
     
  6. Radar

    Radar Registered

    Joined:
    Oct 20, 2010
    Messages:
    687
    Likes Received:
    60
    I had this issues also Lazza, the dedi doesn't shut down properly.. I'm now having problems where people can connect to one server, but not the other..

    EDIT:.. Found the problem as to why no one could join the other server.. With what you said Lazza, regarding the process still running, I thought I would check mine and sure enough, it was still running.. Funny thing is, when you run 2 instances, it doesn't open a third when you re-run the server, it appears to open over the top, but not doesn't close the server when you shut it down..

    Close the instance, people can join again.. Another thing to watch out for now :(
     
    Last edited by a moderator: Mar 4, 2013
  7. Diablo

    Diablo Registered

    Joined:
    Jan 20, 2013
    Messages:
    404
    Likes Received:
    0
    Reminds me of server berserk mode combined with the not so obligatory binding of ports.
    Since the process is still running, the ports cannot be used by the new instance and the server selects different ports to bind to, contrary to what ISI once said on this forum. The HTTP server port fails to bind, since that one obeys what was said to be the new behaviour in rF2. The problem then seems to be, that the new server instance, despite using different ports, reports the ones configured in Multiplayer.ini to the matchmaker. At least it´s a possible explanation.

    Edit: I linked to the wrong bug thread regarding the binding of ports, that´s corrected now.
     
    Last edited by a moderator: Mar 4, 2013
  8. Marco Bijl

    Marco Bijl Registered

    Joined:
    Dec 23, 2011
    Messages:
    283
    Likes Received:
    8
    I believe that can be tested by us as well.
    What if we close all ports on the firewall, other then those defined in the multiplayer ini, and enable logging incase a port is addressed outside the opened range? We should be getting some kind of log file, and indication when the server drops off, or tries to get access over a closed port.

    I do seem to remember that the matchmaker port is equal to all server instances. So, in principle, that port should not be locked by one, as others cant connect to match maker then.
    If the connection to matchmaker over that port is successful, and the WAN join port is locked out by a running process, would that not give the same result as described above Diablo?

    If so, ISI might want to look if indeed the code sticks to the ports defined in multiplayer.ini, or that in case of no access through these ports, it finds another way, and not "know"it as in the status report.
     
  9. Diablo

    Diablo Registered

    Joined:
    Jan 20, 2013
    Messages:
    404
    Likes Received:
    0
    Just a quick reply this time. First, I think we should open a new thread, since we are way off the original topic now. :) At least I think we are, because the matchmaker issues and crashing happened on our server as well, and we have never had the error message of the OP.

    Plus, I need to think about, what you just wrote, Marco. ;)

    Cheers,
    Marcus
     
  10. Marco Bijl

    Marco Bijl Registered

    Joined:
    Dec 23, 2011
    Messages:
    283
    Likes Received:
    8
    Sorry, you are right.... We are indeed offtopic here.

    Perhaps someone from ISI can make a split for us? I do think this is something to look deeper at.
     
  11. Diablo

    Diablo Registered

    Joined:
    Jan 20, 2013
    Messages:
    404
    Likes Received:
    0
    The thread We (that is, Jim Beam and I) always ran the ModMgr and Dedicated exes directly (either actually direct, or shortcuts with the same target and working ("Start In:") folders).

    To cover recent history:

    - In build 134, the server was dropping off the matchmaker sometimes very often, while continuing to display "Server updating successfully".

    - In build 146, the server would sometimes drop off the matchmaker, display a message indicating it had lost connection, then crash.

    - In build 146+ (test .exe from Jeremy), the server continued to sometimes drop off matchmaker, usually displaying a message saying so. When attempting to Exit to restart the "Shutting down server" message wouldn't go away; the process had to be killed manually. Usually it was only using 14MB of memory at that point, though sometimes 170ish.
    With this same test version I tried a completely clean install on another user account on the server box and ran it completely default with a single standard ISI mod installed (the F2s I think), with the only change being the port settings. This fell off the list as well. We knew a new build wasn't far off so waited to see what happened.

    Now Build 156: as above JB had issues running the server using the 'direct run' method which it seems was never the right way to go about things.

    So we've now got a fresh install, data.path points to the data folder from installation, running both the ModMgr and Dedicated exes via shortcuts that set the 'Start In:' folder to the main folder from installation (the one that contains 'Core' and 'Plugins' folders). Both the mod manager and the dedicated server install and run fine.

    But it's still dropping off the matchmaker list.

    So my question right now is: is anyone else getting this? If not it seems likely something we're doing... so just trying to work that out to start with :)[/QUOTE]

    I hope we can have the other thread split and the relevant part moved over here.

    Edit: Thanks that was quick, but I guess I should have said copied instead of split and move. :) But what of it? The link to the original one is there, so...

    Best regards,
    Marcus
     
    Last edited by a moderator: Mar 4, 2013
  12. Diablo

    Diablo Registered

    Joined:
    Jan 20, 2013
    Messages:
    404
    Likes Received:
    0
    OK I have just checked with my home PC. I deleted all prior rules for rFactor 2 and created 3 new rules for the default WAN join port (UDP), WAN join port +50 (TCP) and HTTP server port (TCP). Now Windows did not ask if I wanted to allow the "rFactor2 Dedicated.exe" to connect to the internet, because those ports were already allowed. Starting a second instance with the same installation and profile caused the usual permission prompt to come up.
    I think that would be the first step to getting a notice when the dedicated server uses ports outside its configuration. But since I have seen crossed over ports, this would only be guaranteed to always work, if one only uses one server instance.

    The connection to the matchmaker seems to be one way only and that is an outgoing one to port 443 (TCP), since apparently the matchmaker uses SSL encryption. There is no local port that the dedicated server needs to listen on for incoming connections from the matchmaker, if that is what you mean.
    But I am not sure if I am following you, especially the second sentence is not quite clear to me. Or maybe my head has used up its thinking time for today. :p

    It definitely does not stick to the configured ports for WAN Join Port (+50), I have checked with netstat, see the bug report I linked to above. The question is, does the code rely on the values in Multiplayer.ini or does it actually check which ports its using before reporting to the matchmaker. I believe the former is the case, because that crossed over ports thing I have seen a couple of times now, results in people attempting to join one server instance ending up on the other one and vice versa.

    I´ll leave it at that for today, my brain seems to start running on empty right now. :)

    Cheers,
    Marcus
     
  13. Radar

    Radar Registered

    Joined:
    Oct 20, 2010
    Messages:
    687
    Likes Received:
    60
    Nice investigation work Diablo..
     
  14. Marco Bijl

    Marco Bijl Registered

    Joined:
    Dec 23, 2011
    Messages:
    283
    Likes Received:
    8
    Marcus,

    This time a quick reply from me :)
    I have had an unexpected funeral yesterday. Not been online here, and have not looked at anything. Will do so tomorrow, as I have to go out tonight as well.
     
  15. Max Angelo

    Max Angelo Registered

    Joined:
    Oct 5, 2010
    Messages:
    4,958
    Likes Received:
    10
    I am trying to narrow the issue with the help of the italian hosts, and atm (only a clue, not an evidence) people with multiplayer.ini line

    Pause While Zero Players="0" // Whether to pause a dedicated server (and stay in practice session) if no human players are present

    set to "1" doesnt suffer of the disappearing server issue.

    I adviced the italian admin of ISR to set some of his servers with such line set to "1", met him just now online and he told me till now no disappering servers, with that line set to "1".

    I have, in the italian sub forum, few hosts so the sample is not big enough to see that line as the discriminant which opens the bug.

    Confirmations?
     
  16. JGraf

    JGraf Registered

    Joined:
    Oct 5, 2010
    Messages:
    461
    Likes Received:
    13
    our server has always been > Pause While Zero Players="1"
    no issues to report since build-156 was installed Saturday 3/2/2013.
     
  17. Marco Bijl

    Marco Bijl Registered

    Joined:
    Dec 23, 2011
    Messages:
    283
    Likes Received:
    8
    I have both settings running at the moment. Servers have been up constantly, without any disruptions so far. 2 - 3 days now. Haven't had a single dropoff since I updated to the new build.
     
  18. Marco Bijl

    Marco Bijl Registered

    Joined:
    Dec 23, 2011
    Messages:
    283
    Likes Received:
    8
    I mean that the server must get a conformation back from the matchmaker. Its "updating succesfully", or Added succcesfully, MODID invalid, stuff like that. The fact that all server instances use the same port, could "blur" the received status, and explain why we saw a dropoffs from matchmaker, but succesfull updating on the server side.

    Could that last be related to the previous joined server? Seen that before a lot.
     
  19. Diablo

    Diablo Registered

    Joined:
    Jan 20, 2013
    Messages:
    404
    Likes Received:
    0
    OK, maybe the "one way only" was a bit misleading. The connection initiation to the matchmaker is done by the dedicated server. The connection being TCP there is of course a back channel. This is quite similar to a browser sending a request to a web server and receiving the requested web page back. A web server by default listens for incoming unencrypted connections on port 80 and port 443 for encrypted ones. The client (browser) uses a (random) port in the range 1024-65535 as the source port and port 80 or 443 respectively as the destination port for the initiation of a connection. The server in turn uses the client´s source port as destination port for a new connection to send its reply to the requesting client. A stateful (connection tracking) firewall will allow that incoming connection on the client side, because it knows that it´s related to the previous outgoing one, thus one does not need to forward/allow such ports.
    So, to sum it up, all rFactor 2 dedicated server instances use port 443 as destination port for connection initiation to the matchmaker, but they also use a random unprivileged port for the back channel, so they don´t really share the same port.

    What I think, might happen, when the rF2 dedi server says it´s updating but disappears from the matchmaker list anyway, is, that it got stuck at the last message, which happened to be "Server updating successfully". It either doesn´t send any more updates to the matchmaker or it doesn´t realise that it´s not receiving any aknowledgements back.
    Edit: Or we´re looking at the wrong end and the matchmaker itself is the culprit here?

    I don´t know if I understodd you correctly, but when this happend with our two server instances being crossed over, there were people trying to join for the first time after server setup and had not joined any server beforehand. They had just started rFactor 2 Multiplayer mode.

    Cheers,
    Marcus
     
    Last edited by a moderator: Mar 6, 2013
  20. Lazza

    Lazza Registered

    Joined:
    Oct 5, 2010
    Messages:
    12,345
    Likes Received:
    6,572
    Tentative response, after 24 hours with this setting our server has stayed on the matchmaker list. Previously with this set to 0 (not our original preference, but in earlier builds this seemed the better option (!) ) performance varied but always dropped off at least once in a 24 hour period. So another day without issue would look pretty good, though unfortunately in matters of this nature you're never totally sure except when something doesn't work ;)

    I think even in earlier builds some people just had no issues, so even if the server set to not pause is a contributor it may not cause a problem in all cases.
     

Share This Page