Why Smart Agents Are A Dumb Idea
 
Smart agents are a dumb idea. Like several of the other undead ideas
floating around (e.g Digital Cash, Videophones), the idea of having
autonomous digital agents that scour the net acting on your behalf seems
so attractive that, despite a string of failures, agents enjoy periodic
resurgances of interest. A new such surge seems to be beginning, with
another round of stories in the press about how autonomous agents
equipped with instructions from you (and your credit card number) are
going to shop for your CDs, buy and sell your stocks, and arrange your
travel plans. The primary thing smart agents seem to have going for them
is the 'cool' factor (as in 'This will work because it would be cool if
it did.') The primary thing they have going against them is that they do
not work and they never will work, and not just because they are
impractical, but because they have the wrong problems in their domain,
and they solve them in the wrong order. 

Smart agents -- web crawling agents as opposed to stored preferences in
a database -- have three things going against them: 

 Agents' performance degrades with network growth
 Agents ask people to do what machines are good at (waiting) and machines
  to do what people are good at (thinking). 
 Agents make the market for information less efficient rather than more

These three barriers render the idea of agents impractical for almost
all of the duties they are supposedly going to perform. 

Consider these problems in context; the classic scenario for the mobile
agent is the business trip.  You have business in Paris (or, more
likely, Peoria) and you need a flight, a hotel and a rental car. You
instruct your agent about your dates, preferences, and price limits, and
it scours the network for you putting together the ideal package based
on its interpretatino of your instructions. Once it has secured this
package, it makes the purchases on your behalf, and presents you with
the completed travel package, dates, times and confirmation mumbers in
one fell swoop.  

A scenario like this requires a good deal of hand waving to make it seem
viable, to say nothing of worthwhile, because it assumes that the
agent's time is more valuable than your time. Place that scenario in a
real world context - your boss tells you you need to be in Paris
(Peoria) at the end of the week, and could you make the arrangements
before you go home? You fire up your trusty agent, and run into the
following problems:

 Agents' performance degrades with network growth

Upon being given its charge, the agent needs to go out and query all the
available sources of travel information, issue the relevant query,
digest the returned information and then run the necessary weighting of
the results in real time. This is like going to Lycos and asking it to
find all the resources related to Unix and then having it start
indexing the Web. Forget leaving your computer to make a pot of coffee -
you could leave your computer and make a ship in a bottle. 

One of the critical weaknesses in the idea of mobile agents is that the
time taken to run a query improves with processor speed (~2x every 18
months) but degrades with the amount of data to be searched (~2x every 4
months). A back of the envelope calculation comparing Moore's Law vs.
traffic patterns at public internet interconnect points suggests that an
autonomous agent's performance for real-time requests should suffer by
roughly an order of magnitude annually. Even if you make optimistic
assumptions about algorithm design and multi-threading and assume that
data sources are always accessible, mere network latency in an exploding
number sources prohibits real-time queries. The right way to handle this
problem is the mundane way - gather and index the material to be queried
in advance. 

 Agents ask people to do what machines are good at (waiting) and
  machines to do what people are good at (thinking). 

The usual answer to this problem with real-time queries is to assume
that people are happy to ask a question hours or days in advance of
needing the answer, a scenario that occurs with a frequency of
approximately never. People ask questions when they want to know the
answer - if they wanted the answer later they would have asked the
question later. Agents thus reverse the appropriate division of labor
between humans and computers -- in the agent scenario above, humans do
the waiting while agents do the thinking. The humans are required to
state the problem in terms rigorous enough to be acted on by a machine,
and be willing to wait for the answer while the machine applies the
heuristics. This is in keeping with the Central Dream of AI, namely that
humans can be relegated to a check-off function after the machines have
done the thinking. 

As attractive as this dream might be, it is far from the realm of the
possible. When you can have an agent which understands why 8 hours
between trains in Paris is better than 4 hours between trains in
Frankfurt but 8 hours in Peoria is worse than 4 hours in Fargo, then you
can let it do all the work for you, but until then the final step in the
process is going to take place in your neural network, not your agent's. 

 Agents make the market for information less efficient

This is the biggest problem of all - agents rely on a wrong abstraction
of the world. In the agent's world, their particular owner is at the
center, and there are a huge number of heterogenous data sources
scattered all around, and one agent makes thousands of queries outwards
to perform one task. This ignores the fact that the data is neither
static nor insensitive to the agent's request. The agent is not just
importing information about supply, it is exporting information about
demand at the same time, thus changing the very market conditions it is
trying to record. The price of a Beanie Baby rises as demand rises since
Beanie Babies are an (artificially) limited resource, while the price of
bestsellers falls with demand, since bookstores can charge lower prices
in return for higher volume. Airline prices are updated thousands of
times a day, currency exchange rates are updated tens of thousands of
times a day. Net-crawling agents are completely unable to deal with
markets for information like these; these kind of problems require the
structured data to be at the center, and for a huge number of
heterogenous queries to be made inwards towards the centralized data,
so that information about supply and demand are all captured in one
place, something no autonomous agent can do. 

Enter The Big Fat Webserver

So much of the history of the Internet, and particularly of the Web, has
been about decentralization that the idea of distributing processes has
become almost reflexive. Because the first decade of the Web has relied
on PCs, which are by their very nature decentralized, it is hard to see
that much of the Web's effect has been in the opposite direction,
towards centralization, and centralization of a particular kind -
market-making. 

The alternative for the autonomous mobile agent is the Big Fat
Webserver, and while its superiority as a solution has often been
overlooked next to the sexier idea of smart agents, B.F.Ws are A Good
Thing for the same reasons markets are A Good Thing - they are the best
way of matching supply with demand in real time. What you would really
do when Paris (Peoria) beckons is go to Travelocity or some similar
B.F.W. for travel planning. Travelocity runs on that unsexiest of
hardware (the mainframe) in that unsexiest of architectures
(centralized) and because of that, it works well everywhere the agent
scenario works badly. You log into Travelocity and ask it a question
about plane flights, get an answer right then, and decide. 

 B.F.Ws performance scales with database size, not network size

The most important advantage of B.F.Ws over agents is that BFWs acquire
and structure the data before a request comes in. Net-crawling agents
are asked to identify sources, gather data and then query the results
all at once, even though these functions require completely different
strategies. By gathering and structuring data in advance, B.F.W.s
remove the two biggest obstacles to agent performance before any
request is issued.

 B.F.W.s let computer do what computers are good at (gathering,
indexing) and people do what people are good at (querying, deciding). 

Propaganda to the contrary, when given a result set of sufficiently
narrow range (a dozen items, give or take), humans are far better at
choosing between different options than agents are. B.F.W.s provide the
appropriate division of labor, letting the machine do the coarse-grained
sorting, which has mostly to do with excluding the worst options, while
letting the humans make the fine-grained choices at the end. 

 B.F.W.s make markets.

This is the biggest advantage of B.F.W.s over agents -- databases open
to heterogenous requests are markets for information. Information about
supply and demand are handled at the same time, and the transaction
takes place as close to real time as database processing plus network
latency can allow. 

For the next few years, B.F.W.s are going to be a growth area. They
solve the problems previously thought to be in the agents domain, and
they solve them better than agents every could. Where the agents make
the assumption of a human in the center, facing outward to a
heterogenous collection of data which can be gathered asynchronously,
B.F.Ws make the asusmption that is more in line with markets (and
reality) - a source of data (a market, really) in the center, with a
collection of humans facing inwards and making requests in real time.
Until someone finda  abetter method of matching supply with demand than
real-time markets, B.F.W.s are a better answer than agents every time.