Yahoo! Pipes and The Web As Database

Age : 45 Joined : 08 Mar 2008 Posts : 1850

One of the central concepts in Complex Systems is Emergence. It is this automagical process through which elements of a system give rise to a higher order system. Emergence is how physics becomes chemistry and chemistry becomes biology. It is how web 1.0 evolved into web 2.0, and how that, in turn, will become the next web.

While the exact mechanics of emergence is complicated and far from being completely understood, scientists know that a new system emerges as a combination of its elements and their interactions. In other words, complex systems are really networks - where elements interact with each other and give rise to a new system.

Perhaps today we are witnessing one of the most vivid examples of emergence - the remixing of the world wide web. The parts of the new web have crystallized - blogs, photos, video, audio, maps, RSS, social network profiles and even plain old HTML pages have formed an impressive network, that now can be mined and remixed. Mashups are really nothing new, the web has been a programmable oyster for at least a few years now.

What is new though is the recent systematic thinking about the web as a database. A few companies, including Dapper, have been working on the problem. But with the recent launch of Yahoo! pipes, we are beginning to see the real power of remixing.

Ye Olde Relational Databases
The Web is just a vast database of information. Everyday, we interact with it without thinking about that too much. We simply take our best query tool, usually called Google, and fire away. Yet decades before the web made its way into our lives, a different kind of database revolutionized our lives. The Relational Database qualifies as one of our best computer science inventions. Lesser known to the non-techie crowd, it nowadays quietly stores terabytes of information behind most familiar ecommerce and corporate sites.

But Relational Databases are remarkably simple. They are collections of tables (structured data) that can be joined (mixed) together via keys to produce a new set of results. For example, the table of sales can be joined with the table of employees to produce a report of who sold what. By combining the tables in various ways, programmers are able to bring seemingly hidden information into the spotlight (think emergence). For example, by combining the sales information with employee records and their geographical locations, one can determine the best sales people in each country.

Another thing that Relational Databases are famous for is visual query and UI tools. Because databases are so simple, and the data is well structured, people have created GUI builders like Visual Basic or Power Builder to automate the UI for fetching and exploring the data. We got so good and so perfect at mapping the databases to the UI, that it's become quite a boring thing to do since about 1997.

Well, now Yahoo! is making this whole business cool again, by changing the rules of the game - the Web is now the new database.