There’s no need to waste family game-night time waiting for the same Steam game to download over the internet! All you need is a local caching proxy server. Repeated downloads from a local cache are faster, and you free up your internet bandwidth for other things (like downloading other games). Steam clients will even auto-discover and self-configure to use your local cache.
Valve Software uses the industry-standard HyperText Transfer Protocol (HTTP) to download games via its Steam gaming client. The downloaded games can be intercepted and cached locally by an intermediary/transparent HTTP forwarding cache proxy.
This setup requires you to run a forward proxy server (e.g., Squid), and configure your network router to redirect outgoing HTTP connections to the proxy server. A forwarding proxy server compares incoming requests and returns cached results from its local store if it has recently seen and has cached the same request. The proxy will fetch all other requests from the origin server over the internet.
However, this setup will cache every unencrypted web connection and not just game downloads. Forwarding cache servers were often marketed as “web accelerators” in the years before every website switched to using encrypted and uncacheable connections. While most of the web is uncacheable by a proxy server, Steam downloads can still be cached.
The Steam client downloads games from many regional data centers; each of which under a unique domain name. It won’t get a response from the cache even though the rest of the download address — and indeed the downloaded file — is identical. Two people downloading the same game through the same proxy might not get it from the same server or data center. This causes what’s known as a cache miss, and the cache needlessly needs to re-download the same game twice.
Steam also supports a custom locally auto-configuring HTTP-based proxying protocol. You don’t need to configure the individual clients to use the local caching proxy. All you need is a specially configured caching proxy server and a local Domain Name System (DNS) server.
By default, the Steam client auto-discovers the presence of a local proxy by querying for a special domain name: lancache.steamcontent.com. You can configure your local DNS server to spoof this address and return the internet protocol (IP) address of your local cache server. This setup is similar to how an ad-blocking DNS server works.
Steam will treat the local proxy server as if it was a transparent forwarding proxy server. As opposed to using the industry standard HTTP proxy protocol. Clients typically use this protocol when configured to use a specific proxy server. Steam doesn’t switch over to the industry-standard HTTP proxy protocol when talking to your local caching proxy. Instead, it will treat it exactly as a transparent forwarding proxy server. So, it uses the HTTP Host header to inform the proxy server where it can download the data.
Each download from Steam, regardless of which data center or domain it was downloaded from, has a globally unique path (file name). The caching proxy should be set up to canonicalize (normalize) the request domain name and return cached downloads when identical download paths are requested from different upstream servers. This method will significantly increase the cache hit ratio.
Steam divides each download into file chunks of roughly 810 Kilobytes. This transfer strategy results in tens of thousands of transfer requests for a typical multi-Gigabyte game. The proxy server must implement HTTP connection pipelining to both the back-end servers and the local clients. Without it, it won’t achieve high transfer speeds. HTTP pipelining lets clients reuse server connections for multiple requests instead of opening a new connection per file transfer. This feature is optional in HTTP version 1.1. Not all servers support nor enable it by default.
Steam doesn’t blindly trust a local caching server, however. Each downloaded chunk is accompanied by a cryptographic checksum. The Steam client uses it to verify that it has downloaded the correct file.
The Steam client will bypass the local caching proxy if the client can’t contact to it, or it repeatedly sends back error messages or corrupt/wrong downloads. Steam could not as easily have bypassed the proxy server in an environment with a transparent proxy server. It gives Valve more control over the user experience.
If you don’t want to roll your own setup, you can get a ready-to-deploy solution from LANCache. It includes a pre-configured DNS server and caching proxy server. I’ve got experience running DNS and caching proxy servers. (I don’t need any more and possibly conflicting DNS servers on my network.)
My home network is setup for 2,5 Gigabit per second internal networking with a 1 Gbps internet connection. The first computer on my network to download a game or update get speeds up to 0,8 Gbps over the internet. The second computer to download the same title (within the next 60 days) gets it from the cache at nearly 2,5 Gbps!
It’s a complete over-kill for our two laptops and two dual-booting game stations (six total clients). However, it was fun to set up, and it’s a more environmentally friendly to keep everything up to date.
I would have preferred a solution where Steam used an auto-configuring peer-to-peer (P2P) solution. Wargaming, a competitor to Steam, uses BitTorrent to deliver games and updates. It automatically finds other running instances on the local network without the need for an always-on dedicated server. This would be the most preferable and easiest solution for home users.
Both Windows Update and Microsoft Store also use a similar delivery method when you enable Delivery Optimization. Delivery Optimization is enabled by default for Windows Home edition.