Web server

[edit] Load limits
A web server (program) has defined load limits, because it can handle only a limited number of concurrent client connections (usually between 2 and 60,000, by default between 500 and 1,000) per IP address (and IP port) and it can serve only a certain maximum number of requests per second depending on:
its own settings;
the HTTP request type;
content origin (static or dynamic);
the fact that the served content is or is not cached;
the hardware and software limits of the OS where it is working.
When a web server is near to or over its limits, it becomes overloaded and thus unresponsive.

[edit] Overload causes

A daily graph of a web server's load, indicating a spike in the load early in the day.
At any time web servers can be overloaded because of:
Too much legitimate web traffic (i.e. thousands or even millions of clients hitting the web site in a short interval of time. e.g. Slashdot effect);
DDoS (Distributed Denial of Service) attacks;
Computer worms that sometimes cause abnormal traffic because of millions of infected computers (not coordinated among them);
XSS viruses can cause high traffic because of millions of infected browsers and/or web servers;
Internet web robots traffic not filtered / limited on large web sites with very few resources (bandwidth, etc.);
Internet (network) slowdowns, so that client requests are served more slowly and the number of connections increases so much that server limits are reached;
Web servers (computers) partial unavailability, this can happen because of required / urgent maintenance or upgrade, HW or SW failures, back-end (i.e. DB) failures, etc.; in these cases the remaining web servers get too much traffic and become overloaded.

Overload symptom

The symptoms of an overloaded web server are:
requests are served with (possibly long) delays (from 1 second to a few hundred seconds);
500, 502, 503, 504 HTTP errors are returned to clients (sometimes also unrelated 404 error or even 408 error may be returned);
TCP connections are refused or reset (interrupted) before any content is sent to clients;
in very rare cases, only partial contents are sent (but this behavior may well be considered a bug, even if it usually depends on unavailable system resources).
Anti-overload techniques
To partially overcome above load limits and to prevent overload, most popular web sites use common techniques like:
managing network traffic, by using:
Firewalls to block unwanted traffic coming from bad IP sources or having bad patterns;
HTTP traffic managers to drop, redirect or rewrite requests having bad HTTP patterns;
Bandwidth management and traffic shaping, in order to smooth down peaks in network usage;
deploying web cache techniques;
using different domain names to serve different (static and dynamic) content by separate Web servers, i.e.:
http://images.example.com
http://www.example.com
using different domain names and/or computers to separate big files from small and medium sized files; the idea is to be able to fully cache small and medium sized files and to efficiently serve big or huge (over 10 - 1000 MB) files by using different settings;
using many Web servers (programs) per computer, each one bound to its own network card and IP address;
using many Web servers (computers) that are grouped together so that they act or are seen as one big Web server, see also: Load balancer;
adding more hardware resources (i.e. RAM, disks) to each computer;
tuning OS parameters for hardware capabilities and usage;
using more efficient computer programs for web servers, etc.;
using other workarounds, especially if dynamic content is involved.

Historical notes

The world's first web server.
In 1989 Tim Berners-Lee proposed to his employer CERN (European Organization for Nuclear Research) a new project, which had the goal of easing the exchange of information between scientists by using a hypertext system. As a result of the implementation of this project, in 1990 Berners-Lee wrote two programs:
a browser called WorldWideWeb;
the world's first web server, which ran on NeXTSTEP.
Between 1991 and 1994 the simplicity and effectiveness of early technologies used to surf and exchange data through the World Wide Web helped to port them to many different operating systems and spread their use among lots of different social groups of people, first in scientific organizations, then in universities and finally in industry.
In 1994 Tim Berners-Lee decided to constitute the World Wide Web Consortium to regulate the further development of the many technologies involved (HTTP, HTML, etc.) through a standardization process.
The following years are recent history which has seen an exponential growth of the number of web sites and servers.

Market structure
Given below is a list of top Web server software vendors published in a Netcraft survey in April 2008.
Vendor
Product
Web Sites Hosted
Percent
Apache
Apache
83,206,564
50.22%
Microsoft
IIS
58,540,275
35.33%
Google
GWS
10,075,991
6.08%
Oversee
Oversee
1,926,812
1.16%
lighttpd
lighttpd
1,495,308
0.9%
nginx
nginx
1,018,503
0.61%
Others
-
9,432,775
5.69%
Total
-
165,696,228
100.00%
There are hundreds of different web server programs available, many of which are specialized for very specific purposes, so the fact that a web server is not very popular does not necessarily mean that it has a lot of bugs or poor performance.
See Category:Web server software for a longer list of HTTP server programs.