Grand Prix Garage - GPL

The workings of GPL Online - Part 2

Disappearing cars, violent collisions, latency, and bandwidth explained.

(Below is a detailed discussion of the topic. See the part 1 for a discussion on the consequences for driving.)

Disclaimer

This document is based on studies from GPL replays and network traces. Also some assumptions were made. So it may not be 100% true or accurate. Any comments that are aimed at improving this analysis are welcome by mail.

First some definitions

So we deal with client/server technology at two levels: the first level is within one system, with a server process for central tasks and a client process for every car; and the second level is in an online game, with one server system and multiple client systems. Some tasks of the server process are delegated by the client systems to the server system in an online game: detection of DQ conditions, keeping standings and recording lap times.

An online race is a cooperation of one server and one or more clients. The clients exchange information with the server. The clients do not exchange information with each other directly.

Disappearing cars

Each client sends information on the local car to the server, where it is handled by the client process that represents that car. The server sends information on the cars surrounding a car back to that car's client system for display. The bandwidth settings in core.ini determine how many surrounding cars are visible to the local car. For clients on modem class connections, a maximum of four cars in front and one car behind will be visible. For lan class connections, these numbers may be higher with a maximum of all cars being visible to all clients. See the section on bandwidth for details. I will refer to the modem class values from here on as they are most commonly used.

Four cars visible in front and one car behind is in terms of relative track positions, not race positions. For example, at the start of the race, the pole position can see the second place car behind him (or next to him) and the cars that are last on the grid, as they are the first he would encounter when following the track.
When the positions on the track change, the server starts sending information on a different set of cars. On the client this has the effect that one car will suddenly appear on the track, and another car will disappear a little later. The overlap is caused by the prediction mechanism extending the life of the disappearing car. The most clear example of this is at the start of the race, when you see cars disappear as new cars are dropped on the grid.

A car is dropped on the grid in 3rd place (far left). After a short while, the 2nd place car disappears as he is now the 5th car in front of the local car (which is not visible in these pictures). The pole is still visible, as he is the first car behind the local car, that had qualified last.

Besides display information, the server also sends information on the positions of all cars. You can see this effect if you watch a replay and the car you are viewing disappears as a result of changes in the positions of the cars. It can be invisible for laps in a row, switching the camera to the pit view, but the position data is still updated and the lap counter for the invisible car keeps incrementing when it crosses the finish line.

Another reason for disapearing cars, besides the limited number of cars displayed, is of course a bad connection somewhere on the path between the two systems involved. If display data for a car is not delivered for a certain period, typically 1 second, the car will disappear after another second. When data comes in again, the car will reappear. When things get too bad, a disconnect will occur.

Violent collisions

Violent collisions occur when two cars are in overlapping positions when the collision is detected. Apparently GPL forces the overlap to be undone the next frame, giving the cars involved a very high speed. As a result the incident will look more like an explosion than like a collision (see my exploding grid for an extemely violent example).

So how come cars can be overlapping in the first place?

• The collision detection for each car with any object is done at its local system. Any system involved in an online race, server or client, will not detect collisions between two remote cars. See here for a demonstration of this. I have seen reports that even collisions between the AI-cars in an off line race are not always detected.
• When a collision is detected for the local car, the prediction of the local car that is done on all other systems involved will deviate considerably from the actual position for a short period. This will also be the case when the driver brakes or steers suddenly. The prediction is based on the position and speed sent a short while ago by the local system. The sudden speed and/or direction changes make the prediction inaccurate. This will result in warping when the prediction of the local car on the remote systems is corrected by new data on the car coming in; it may warp to a position overlapping the local car of a remote system, causing a violent incident on the remote system.
• The remote car colliding with the local car will seem to be affected at the system of the local car, but this is only the prediction by the local system. When new data for the remote car comes in from the server, it is very well possible that it is still based on the situation before the collision, even if the remote system has detected a collision as well. This is due to the latency on the connections. The effect is that the remote car will warp back to its original line. In doing so, it may overlap with the local car, again giving a violent incident.

You can imagine what happens when three or more cars are involved, all reacting to each other with the latency delay.

Here is an example from the 1998 GMSS race at Spa:

My Ferrari collides with Brent Martin's car. We both spin. Off line, this would be a light accident.

Suddenly Brent is going straigth again, while I continue to spin. Data came in that indicated to my system that Brent was still going straight some time ago, and prediction extrapolated that. Parts come off our cars and my car is lifted as Brent's car is put in a position overlapping my car.

These two images are only 1 frame apart. I have done more than a 360 by now. Brent has been going straight on during my spin, and suddenly he is spinning to. Data came in indicating Brent's spin, that was started some time ago on his own system, when it detected the collision with the image of my car.

Bad warping, caused by extemely high latencies, can also be the cause of cars overlapping with the local car when they reappear, giving violent accidents.

Latency

On what basis are the remote cars displayed on the local client system, given that it takes some time for the data on a remote car to reach the local system? Several factors play a role here:

1. The rate at which the remote system transmits data about its local car. This is set in the core.ini parameter net_xxx_client_send_every, which is specified in ticks. (xxx = mdm or lan)
2. The latency of the connection between the remote system and the server.
3. The rate at which the server transmits data about surrounding cars to all clients. This is set in the core.ini parameter net_xxx_server_send_every, which is specified in ticks. (xxx = mdm or lan)
4. The latency of the connection between the server and the local system.

The simplest approach is to start with the moment that the display data for a remote car arrives at the local system. Two questions have to be answered:

1. How old is the data on arrival?
2. For how long will it be needed from then on before new data will arrive?

1. How old is the data on arrival?

Following the path backwards, we have:

1. travel time from server to the local client; best guess is half the latency (ping) reported on the local client
2. wait for server update time; on average, this is half the net_xxx_server_send_every period set on the server or half the net_xxx_client_send_every of the remote client, whichever is less
3. travel time from remote client to server; best guess is half the latency (ping) reported on the remote client

2. For how long is the data needed?

In constant conditions, the data will be needed for the net_xxx_client_send_every period of the remote client or the net_xxx_server_send_every period set on the server, whichever is greater. With varying latencies on the connections, the data may be needed for a longer or shorter time.

As an example, assume the latency on all connections is constant at 200ms (round trip), giving a travel time of 100ms between clients and server. The following table gives how much time the prediction mechanism has to bridge for various settings of client_send_every on the remote client and server_send_every on the server (top row). The row labelled "1 (Arrival)" gives the average lag at arrival of the data on the local system. You cannot be more up-to-date than this (on average that is). The row labelled "1+2 (Extinction)" gives the average lag just before the next data comes in. On average, this is the maximum you will lag behind.
The Max clients @ nn Kbps rows give the maximum number of clients a server connection can handle as derived from the section on bandwidth, assuming that a maximum of 5 surrounding cars are visible on each client and the server_send_size parameter is set to the smallest possible value for the number of clients. The Max nn number depends only on the send_every settings, not on the latency of the connections (although high latencies will prevent you from using high send_every settings)..

Experience in Internet play has shown that the ping on connections can vary between 100ms or below (very fast) and 500ms or above (slow to the point of being unplayable). The influence of the core.ini parameters, especially server_send_every, is relatively small. So if the connections are good, increasing server_send_every may allow more clients to join while maintaining reasonable quality.

The trade off for smaller lags by decreasing the send_every parameters is of course bandwidth, which is discussed next.

Bandwidth

First of all, what data is exchanged between the server and the clients?
We have:

• Data needed for displaying cars
  • For the local car from client to server
  • For a number of cars surrounding the local car from server to client (max. 5 cars on modem class connections)
• Data on every car's position from the server to all clients
• Clock synchronization data

The rate at which the display data is sent, and the amount of data sent each transmission, come from parameters in the core.ini file in the GPL directory. They have the following default values. All frequencies are specified in ticks. All sizes are maxima; when there are not enough clients to fill the packet completely, a smaller packet is sent.

On dialup links tcp/ip is forced to the modem class values. Ipx will allways use the lan class values.
On cable or dsl links, tcp/ip will use the modem class values unless net_use_mdm_bandwidth_for_tcp_ip = 0, in which case it will use the lan class values.

From traces and replays I made using a server and one client in a local network, I noted the following figures. I checked them using an Internet connection between the systems as well. I simulated a multi client race by letting ai cars in. The ones listed here are the most important, look here for all details.

• A client never sends the full 84 bytes to the server, rather it is 36 bytes, occasionally 52.
• The server sends 16 bytes per car plus 4 bytes overhead per packet.
• The number of cars visible behind is 25% of the cars shown (rounded down).
• The protocol overhead for all packets is 54 bytes (based on TCP/IP over ethernet). This has to be added to the packet sizes specified in core.ini when calculating the bandwidth requirements.
• To go from bytes to bits I multiply by 8.

This amounts to the following formulae for bandwidth requirements:

• Upload from client to server: (54+36)*(36/client_send_every) bytes per second.

• Download from server to one client: (54+server_send_size)*(36/server_send_every) bytes per second.

• For load on the server connection, multiply the above by the number of clients, reversing upload and download.
  Some examples follow in the table below. For 6/6 send frequencies, take the half of the 3/3 values, for 2/2 double the 4/4 values.

An interesting setting to try is this. The server send size is changed from 84 to 132, allowing to see 6 cars in front and 2 behind. To compensate for the extra data, the send frequencies are reduced to 4. So a little more latency allows more cars to be visible. For this setting to work for dialup users, GPL must first be patched, see here. The settings must be applied to all systems involved.

Dialup users set:

net_mdm_client_send_every = 4
net_mdm_client_send_size = 84
net_mdm_server_send_every = 4
net_mdm_server_send_size = 132 ; (8 cars visible, 6 ahead, 2 behind)
net_use_mdm_bandwidth_for_tcp_ip = 1

Cable/DSL users set the same as dialup, or alternatively:

net_lan_client_send_every = 4
net_lan_client_send_size = 84
net_lan_server_send_every = 4
net_lan_server_send_size = 132 ; (8 cars visible, 6 ahead, 2 behind)
net_use_mdm_bandwidth_for_tcp_ip = 0

This will give the following bandwidth requirements: