PC Dedicated Conclave Servers

[Kontrollo]

@maciejs Can you add some additional lines in the log so it's possible to deduce the information on the end-of-game screen? Kills/deaths are probably already possible, but Oro, ranking/winner/score are not, from what I've heard.

[maciejs]

Sure, will take a look

[maciejs]

OK, next build should have some more info in the log after every match.

[Lord_of_Lords]

Wow I'm glad to see there's still someone looking after this amazing feature after a couple years.

 

Any chance we can host our clan Dojo in the future? 
My friends and I are tired of watching each other glitch in and out of existence (as well as walking on air instead of jumping) and it is difficult to guide new members to our labs without resorting to voice chat.  I can only guess that this would be fixed by using a more high-powered server.

[NoSpax]

Multiinstance may be broken again.

Tried to start a server while being online in Warframe (server is on the same computer for testing reasons), got the dreaded H.Misc.cache Error.

Using this batch:

Spoiler

 

set exec=
set type=LunaroClan

%exec%Warframe.x64.exe -log:%type%.log -cluster:private -allowmultiple ^
-applet:/Lotus/Types/Game/DedicatedServer /Lotus/Types/GameRules/DefaultDedicatedServerSettings ^
-instance:2 ^
-settings:%type% ^
-dx10:1 -dx11:0
pause

 

Using this DS.cfg :

Spoiler

 

+nowarning
+version=5

[LotusDedicatedServerConfig,/EE/Types/Base/Config]

[LotusDedicatedServerConfig,/Lotus/Types/GameRules/LotusDedicatedServerConfig]
App.DedicatedServerFrameRate=60

[60fpsLotusDedicatedServerConfig,/Lotus/Types/GameRules/LotusDedicatedServerConfig]
App.DedicatedServerFrameRate=60

[120fpsLotusDedicatedServerConfig,/Lotus/Types/GameRules/LotusDedicatedServerConfig]
App.DedicatedServerFrameRate=120

[LotusDedicatedServerSettings,/Lotus/Types/Game/DedicatedServerSettings]
version=8
starChart=/Lotus/Types/Game/SolarMap/OriginSolarMapRedux
gameConfig=/Lotus/Types/GameRules/LotusDedicatedServerGameConfig
eloRating=2
missionIdToNode={
    {
        id=SB_Title
        node=PvpNode11
    },
    {
        id=TDM_Title
        node=PvpNode9
    },
    {
        id=DM_Title
        node=PvpNode10
    },
    {
        id=CTF_Title
        node=PvpNode0
    },
    {
        id=DM_Alt_Title
        node=PvpNode14
    },
    {
        id=TDM_Alt_Title
        node=PvpNode13
    },
    {
        id=VT_Title
        node=PvpNode15
    }
}
levelOverrides={
    {
        missionTag=PvpNode0
        levels=/Lotus/Types/GameRules/PVPCTFLevels
    },
    {
        missionTag=PvpNode10
        levels=/Lotus/Types/GameRules/PVPDMLevels
    },
    {
        missionTag=PvpNode9
        levels=/Lotus/Types/GameRules/PVPDMLevels
    },
    {
        missionTag=PvpNode11
        levels=/Lotus/Types/GameRules/PVPSBLevels
    },
    {
        missionTag=PvpNode13
        levels=/Lotus/Types/GameRules/PVPDMLevels
    },
    {
        missionTag=PvpNode14
        levels=/Lotus/Types/GameRules/PVPDMLevels
    },
    {
        missionTag=PvpNode15
        levels=/Lotus/Types/GameRules/PVPVoidTearLevels
    }
}
PVPmodes={
    PVPMODE_CAPTURETHEFLAG=CTF_Title
    PVPMODE_CAPTURETHEFLAG_Alternative=PVPMODE_CAPTURETHEFLAG_ALTERNATIVE
    PVPMODE_TEAMDEATHMATCH=TDM_Title
    PVPMODE_TEAMDEATHMATCH_Alternative=PVPMODE_TEAMDEATHMATCH_ALTERNATIVE
    PVPMODE_DEATHMATCH=DM_Title
    PVPMODE_DEATHMATCH_Alternative=PVPMODE_DEATHMATCH_ALTERNATIVE
    PVPMODE_SPEEDBALL=SB_Title
    PVPMODE_VOIDTEAR=VT_Title
    PVPMODE_VOIDTEAR_Alternative=PVPMODE_VOIDTEAR_ALTERNATIVE
    PVPMODE_DEATHMATCH_ALTERNATIVE=DM_Alt_Title
    PVPMODE_TEAMDEATHMATCH_ALTERNATIVE=TDM_Alt_Title
    PVPMODE_CAPTURETHEFLAG_ALTERNATIVE=CTF_Alt_Title
    PVPMODE_VOIDTEAR_ALTERNATIVE=VT_Alt_Title
}
matchmakingRegionOverride=EUROPE

[Lunaro,LotusDedicatedServerSettings]
missionId=SB_Title
motd=Lunaro FFA, WARNING, this is a test dedicated server, expect sudden d/c. If you get lag, please /w NoSpax or PM in Forums.
enableVoice=1
clanOnly=0
highBandwidth=0
eloRating=2

[LunaroClan,Lunaro]
motd=Lunaro ClanOnly
enableVoice=1
clanOnly=1

[DeathMatch,LotusDedicatedServerSettings]
missionId=DM_Title
motd=DM FFA, WARNING, this is a test dedicated server, expect sudden d/c.
enableVoice=0
clanOnly=0
highBandwidth=0
eloRating=0

[VoidTear,LotusDedicatedServerSettings]
missionId=VT_Title
motd=Void Tear Mode?
enableVoice=1
clanOnly=0
highBandwidth=1
eloRating=2

 

No Crash, just an error, followed by server shutdown. Do I have wrong settings, maybe? It has been years since I last used Dedi servers and want to help debugging.

[maciejs]

Yes, the problem is you're running the game at the same time and they fight for access to cache. Multiple copies of dedicated servers are smart enough to avoid it, but it comes with a minor perf penalty, so game is a bit less careful.

[Kontrollo]

Not sure if this the right thread to ask this, and note that I'm not a network engineer, but here goes anyway.

 

Recently, I've hosted some games and noticed that some specific players had a large amount of dropped packets (5%-10%) which caused rubberbanding (example gfycat clip).

Interestingly, it did not seem to be caused by CPU/GPU, ping or bandwidth issues. We did a few tests on that -- of course, these things are hard to pin down -- but one of the guys with the problem had consistently lowest ping, even, and it also seemed to happen when he has the only one on the server. Symptoms weren't rubberbanding in that case of course, but increased load times and again entries in the logs.

I know you're using the UDP protocol for these connections, and after reading up on the subject, I figured it might be because of lost/dropped packets due to fragmentation. Maybe because some routers on the way drop fragmented UDP packets by default. So what I've tried now is to force an MTU on my Interface like this, with different values for the MTU:

> netsh int ipv4 set subinterface "my Internet-facing interface name" mtu=1300 store=persistent

When looking at the logs, I think it might have resolved it for someone, but as different people may have different per-connection MTUs, it's hard to find one setting that works for everyone.

I can figure out an acceptable MTU for specific people by doing ping tests with varying frame sizes, if they allow pinging (which I got from here, section "The Diagnosis and Fix", example params in the blog are wrong, it's -f -l *size*, not -l -f *size*), e.g.:

> ping ***.***.***.*** -f -l 1272

 

So that's how far I've come already, now for the questions:

• Is there a way to investigate, monitor, debug or report these connection issues?
• Is it possible to set the MTU on the connections themselves instead, i.e. just for Warframe instead of globally on my interface?

 

Naturally, when I'm not hosting with a fixed lower than normal MTU, then I myself might be running into problems as client.

 

 

P.S.: From what I've read, the minimum MTU on the Internet should be 576, i.e. any connection must support that without fragmenting the packets. But if I set it that low, no one can connect to any games I'm hosting anymore (even PvE). Warframe doesn't like that, at all.

[maciejs]

Could you send me a log from a session that suffers from this problem? I don't think hand-tweaking a per-connection MTU is a sustainable solution here. Sadly, this is also mostly beyond the scope of application control. We do have a congestion control system that tries to make sure we don't send too much data if client can't handle it. Our internal MTU starts at around 1300 bytes (Ethernet theoretical guaranteed max is 1500). Forcing it to a significantly lower value can cause issues, however, we rarely send packets that big, we simply do not generate that much data in a frame and can't wait too long for the buffer to fill as it'd affect ping.

[Kontrollo]

Thanks a lot for the reply. I've been reading my post again, and I could imagine that it looks like there's definitely a problem with MTU and how it's being handled.

That's not really the case, it's all just speculation on my part.

Here, I'm unable to create a good test environment. I also know networking is hard, and especially so because it's UDP. It makes basically zero guarantees, I can't use any dev tools, and the game itself is a black box to me. So there's no way for me to know how Warframe deals with problems on the UDP protocol layer, except what little I can gather from the logs.

Also, I haven't tried any capture/inspection software like WireShark, because it might raise some third party tool red flags.

The questions are mainly about whether there's a way for me (or you) to get more insight into or mitigate these problematic packet losses.

 

Btw, earlier I've sent some logs and some more information through a ticket and PMs.

[Matheoryon]

Since your rather knowledgeable about this, can you tell me what tweaks i can use to make myself a decent server if I do this? 

[Kontrollo]

Just wanted to leave a note on two things:

1. It's been a while since the last post, and several others and me had done quite a bit of testing back then, including with tools such as WinMTR. Nothing of that turned up anything that really stood out for it to be worth reporting. And I can imagine that you might've done some changes behind the scenes, too, as it doesn't come up as much anymore (I think there was something in the patch notes at some point, not sure if related, though). That, or the network situation of affected parties has improved, it's hard to tell from here.

TL;DR: No news is good news, I guess. 😉

 

2. Sadly, Shawzin is not usable on Dedicated Servers. I have no idea how hard it'd be to fix that and it's hardly a necessity. Just thought I'd mention it while I'm at it. Here's the start of Golden Brown by The Stranglers which I could only do because I was hosting myself. I thought that was funny:

(Need to enable sound in the clip for that to work. Thanks to Pun-chee for trying to dance to the tune in the heat of battle and the guys who made ShawzinBot happen.)

[(XBOX)yepbeeble]

On 2016-10-20 at 7:39 PM, KIREEK said:

I assume this is mostly CPU based, if a specific machine is tasked with the hosting, how much RAM and processing power will be required? Does each match need 1 thread from the hosting machine? Does each match need a specific ammount of ram? how much of a connection speed and bandwidth do you think is needed?

Its obvious that the best machines and the best connections will be placed 1st, but for how long will it take before the machine is overloaded.

Will this only be usefull for a single match or will the server support multiple matches at the same time?

What I would like to know is if it pre-dedicated the RAM to the server, leaving everything else on your computer to fight for RAM, OR are does it host by using an amount of RAM that raises and lowers as needed? 

 

 

[maciejs]

Thanks for the update, @Kontrollo. We did tweak a few things, but nothing major (as I mentioned in our messages... it's really rare for games to change MTU). I will try to take a look at Shawzin at one point.

As for the CPU/RAM thing -- dedicated server is basically game. It is a bit easier both on the CPU and RAM mostly because it doesn't need to render anything, so A) it does not load textures/shaders/meshes (this is actually a _big_ difference) and does not need to create rendering threads.

[Othokeli]

Do you think it would be possible to port the dedicated server so that it can be run natively on Linux without having to run the launcher GUI? That way we could run them headless. Clans could run one on a VPS and give preference to clan members so that they can benefit from it.

[maciejs]

You don't have to run the Launcher GUI, it supports the -headless argument which runs it as a commandline application. It's not really native, but I know people had some success running it under emulators.

[Jitsuryoku]

Questions:

1. Any incentives to do it? Keeping my PC on and/or remembering to host a server each time I start a PC(or wake it up..... by the way, I hope you handled Sleep Mode) seems like a headache. Anything that's as much of a hassle with no real gains associated with it becomes a chore and is likely to lose popularity, once the novelty wears off.
2. Any requirements for hosts? I can be hosting on a toaster with tuna salad as a router and create even worse lag than you would have with one of players as a host. Any plans to weed out the bad hosts?
3. Responsibilities of hosts? What's stopping me from running the server for 3 minutes and then restarting it over and over again (making it impossible to complete matches)? Same problem as above. If there are no responsibilities, trolls can create worse experience than an average bad host. If there ARE responsibilities, it comes back to the question of incentive.
4. Any extra Security? I might be wrong, but sharing your PC seems significantly riskier than standard match making, because it leaves your IP open and visible for significantly longer than 1 mission at a time and it gives people reason to attack it (PvP servers seem attract more DDOS attacks).

Consider putting passive hosting as an option in game. You log into game, you start hosting you earn some..... schmeckles or something (nothing exclusive, I guess, because not everybody should be allowed to host), while having the option to play or do whatever it is you logged in to do. It could be still running with the same ugly script so that people who want to exit the game can continue hosting and stop when they like. I suppose as an exit strategy when game client closes, the server script continues running, but it could stop accepting new matches and stop when last open match ends.

I also hope you took hackers into consideration. Client-side-controlled servers seems like something that can be abused. I mean hacking for in-game benefits, but of course that also extends to attacks on host.

[maciejs]

These all are valid points, I'll kick them up the chain. What I can comment on are:

- requirements: dedicated server has significantly lower hardware requirements than the game, mostly because it does not need to render anything (or simulate effects, play local sounds and so on). There is nothing preventing you from running it on your toaster, but it should still be a better experience than playing _and_ hosting it.

- as for the security, you're right that since server sits there waiting for connections, it's technically a bit more exposed, but to establish an actual connection you do have to go through our layers first, otherwise it'll simply keep rejecting you. This still could be problematic for DDOS of course but worse case scenario it affects your hosting quality. It should not be any more dangerous than simply playing the game (DIsclaimer: there's many smart people out there, so not claiming this is completely 100%-hack proof... do not treat it as a challenge).

[ProjektKrieg]

great, maybe you should host some dedicated servers?

because p2p is always a terrible idea, even with dedicated servers on P2P basis

[Secris]

I just discovered the ability to run dedicated server. Need to learn to click that gear more often in the launcher. I was wondering if there has been any new word on allowing dedicated servers to host the normal PvE nodes? 

[Kontrollo]

On 2020-12-10 at 6:41 PM, Secris said:

I just discovered the ability to run dedicated server. Need to learn to click that gear more often in the launcher. I was wondering if there has been any new word on allowing dedicated servers to host the normal PvE nodes? 

I'm not very up-to-date on things, but if the devs haven't mentioned it again, they should at least strongly consider that for the reintroduction of Trials. It's not completely without problems, e.g. from what I hear some hosters sometimes aren't that great and there's no real way to select one. But in general, it's fair to say it has been a resounding success for the player experience.

Alone not having any migrations mid-match is worth a lot.

 

 

Now why I'm posting here:

@maciejs We've recently had some discussion about synchronisation, lag compensation, hit detection etc. and someone posted this talk:

Spoiler

 

Thanks a lot for that, it was really interesting to have a look behind the curtains. One of my favourite points:

Quote

... this was probably the first time that I ported a fairly complex system to a multi-threaded environment and it didn't backfire in a very spectacular way ...

😄

Also interesting to know that the Lunaro ball has separate logic from other projectiles to make it behave much better in that environment. (Although from what I hear there are some problems with that these days -- I don't play that mode myself.)

 

Here's my question: The talk is from 2017, so a bit outdated. Is that still how it works in general, or have there been (significant) changes? I mean, things worth knowing when giving advice.

Also, I'm not developing games myself, but I do have some general interest in this kind of thing. So if you've done any other talks in the meantime that are publicly viewable, I'd like to watch that as well. ☺️

[maciejs]

I'd say core of the system is the same. We threaded it a little bit more, including the lower level networking part, but this does not really affect game logic. Also, the introduction of dynamically streamed regions like POE/Venus/Railjack added some complexity, but again, this is limited to these maps. We did fix a thing or two as well, but for the most part, the replication layer is fairly robust at this point and I do not touch it too frequently.

As far as other talks go, if you're interested in networking - this is a very good collection of publicly available talks:

https://github.com/MFatihMAR/Game-Networking-Resources

If you're talking about Warframe, I did another talk a year before, but its' more about fixing bugs and stability issues:

Spoiler

 

[Kontrollo]

20 hours ago, maciejs said:

I'd say core of the system is the same. We threaded it a little bit more, including the lower level networking part, but this does not really affect game logic. Also, the introduction of dynamically streamed regions like POE/Venus/Railjack added some complexity, but again, this is limited to these maps. We did fix a thing or two as well, but for the most part, the replication layer is fairly robust at this point and I do not touch it too frequently.

As far as other talks go, if you're interested in networking - this is a very good collection of publicly available talks:

https://github.com/MFatihMAR/Game-Networking-Resources

If you're talking about Warframe, I did another talk a year before, but its' more about fixing bugs and stability issues:

  Reveal hidden contents

 

Well, I know games have some very hard constraints, several interacting subsystems -- some of which may be competing (e.g. physics vs. replication/prediction), and are more and more built with concurrency. There's no one-shoe-fits-all and always some trade-offs.

Thus, I'm interested in what people come up with to make things work, but also to see how they "cheat" in some areas and just create the illusion of something happening when in fact it's a much simpler system underlying it all. And yes, especially also how it is then synchronised over a network.

Of course, I'm more interested in talks about Warframe because I'm familiar with it "on the receiving end", and I might be able to "connect better" to what is said even without having access to any dev tools.

 

So thanks a lot for these! I've now already watched "Vehicle Replication in Watch Dogs 2" from that list, it was quite intriguing.

[Theroxenes]

@maciejsHey Maciej, the latest update seems to have introduced some weird issue with the NRS servers not being reachable for dedicated servers. It looks like everything was working earlier today but now all my instances are getting this:

I have tried forwarding ports, restarting my gateway. I checked the firewall settings and they appear to be intact, allowing all traffic for Warframe.x64.exe. Not really sure where else to look. I can confirm that one other host is having the same issue (nep_blanc). I can send a full log if you would like.

Weirdly I can connect to at least one of these addresses using netcat from my VPS and locally in a VM. Not sure what that signifies.

[MatNova]

Can confirm I'm also experiencing the same issue. Played with port settings, firewall settings, etc. and nothing seemed to resolve it.

I figured I'd wait a bit since maybe something on DE's side went down due to the patch, but at this point I figure it's probably safe to say it's something else.

[maciejs]

Thanks for letting me know. Can you send me your DS.cfg?

[Edit] Actually I think I see the problem, we'll try to fix live. Can you restart?