All revolutionaries need a fashion line. Maybe it’s a red headscarf or a black armband or a fez or maybe it’s bell-bottom jeans, but there is some strange intangible need for each revolution to have it’s own clothing to set it apart. The problem is that sometimes the only thing useful that comes out of the revolution IS the fashion.

Software, it seems, has also had its share of revolutions. Here are just a few (in no particular order): Client-Server, EDI, Dynamic Typing, Test Driven Development, Relational Databases, P-Code, HTML, Object Oriented Programing, CORBA, Service Oriented Architecture, XML…etc. Now only a few of these were big enough to actually achieve the status of being a revolution, most were more like movements or perhaps just trends.

Frankly, that’s a lot of movements in the very short span of about 40 years and it’s not even close to a complete list. Many of the so-called revolutions (like real revolutions) have actually made things worse for programmers (CASE based development for example). Others, like Relational-Databases, have really and truly made a positive difference in what we do (although not too many people can still remember the days of hierarchical databases). As a programmer with a long memory I occasionally find myself bemoaning the slow pace at which software engineering has progressed despite it’s almost continuous efforts to reinvent itself. I see hundreds of thousands of programmers all over the world writing the same CRUD applications that were being written twenty years ago. It’s times like these that I begin to worry that many of the software detractors may be right when they claim that software change has become nothing more than trend chasing.

Still, I am not so sanguine about our past as this. While progress has been slow it has not stopped. Recently, however, we’ve come to a very strange place where people address the symptoms of our software dilemmas with increasing skill and capacity while at the same time ignoring the most critical roadblocks that are keeping us back. In particular, two very artificial problems hamper the development of today’s web based applications: first, HTML is wholly inadequate for a host of applications (even with DHTML and AJAX), and second relational databases have passed their prime as a persistence mechanism but there is no successor in sight.

How can you make a phone call if you can’t speak?

There is a very simple reason that developing complex web applications is hard: HTML wasn’t designed for complex applications. I won’t tear away at HTML too much; if you’ve worked in it you know exactly and precisely how limited it is. If you add a bunch of DHTML and AJAX the best you can do is a reasonably sophisticated interface, but the cost to produce it is enormous. Up until recently even modest DHTML and Javascript was basically a no-no. Of course, it’s not just HTML and Javascript; it’s also the notion of state in an HTTP based application. HTTP driven applications are intrinsically designed to be stateless, but many real applications actually need a substantial amount of client application state. Adding that state at the server side is possible but creates complexity that should not be necessary (and can create resource problems as well). An ideal thin client SHOULD keep UI state information to itself since it is the arbiter of that data. Of course this is really just the tip of the HTML/HTTP iceberg when it comes to application development in web land. You also get constraints on the ability of servers to communicate directly (peering) and on servers sending messages asynchronously to the client, etc. It’s a hostile UI environment that we have not yet tamed adequately.

You may be tempted to ask, “So what, don’t write your complex application for the web, use the older technologies.” The answer is that it’s not just good to be egalitarian; it’s a requirement. One way or another you’ll probably have to tackle the cross platform issue. Let’s say, for example, that you write an in house application entirely on windows and it gets popular and everyone loves it. Now the marketing staff needs to use it but they are running Macs and also the laptop users need to use it but it’s client server and IT wont let it in through the firewall and some of the users aren’t allowed to have VPN access, etc. Looks like you’ll need to put it up on the intranet too. Thinking about it as a software vendor, if you’re selling software people may not be able to try your stuff out if they can’t install it locally (IT restrictions) or if it’s a complex multi-user application. You’d better put it up as a web application so people can try it out. People have gotten used to this egalitarian software model and so for multi-user applications we just expect things to work that way. Really this is a good thing. Things should be easy for the user, but that doesn’t mean they need to be hard for the developer.

The matrix has you

How is it that fifteen years in to the web revolution we’re just now getting used to style sheets and we can just begin to see dynamic content becoming available? One answer is that when Netscape died (or rather when Microsoft and Netscape no longer competed for mind share) all progress ended. Looking back it was practically a “pull the plug on the web” kind of moment. Eventually Microsoft stopped even pretending to try. As long as there was competition we kept getting new goodies like CSS and Cookies and IFRAMES and DOM improvements and innerHTML and blah blah blah. Sure, some of these were complete hacks, but over time if progress had continued the hacks would have gotten replaced with solid alternatives. Unfortunately, the doors of innovation slammed shut when it became apparent to Microsoft that they had won (or would win). At that point any improvements were basically bug fixes and security enhancements.

Applets were one of the features that never got working before the spigot got turned off. Concept wise they were fine but try to get them working without forcing the user to download a plug in and forget it. Today, of course, you can download a reasonable plug in for most systems and get good applet support, but who can rely on people being willing (or able in corporate offices) to install the plug in? Apparently not very many people since it’s pretty hard to find a mainstream web application that uses applets in a substantial way. It’s that least common denominator problem again.

Can I lay all the blame for the end of web progress on Microsoft? Nope, they had (and still have) a very able ally: namely big IT departments. That’s right; many of the people reading this may actually work for the enemy of the web. While Microsoft let web technologies die on the vine the instant they gained enough control to do so, many large IT departments have been actively working against them from the beginning. Of course they didn’t do it on purpose, but they did it nonetheless.

Let’s say there is some cool feature I need on a web site and all I have to do is download it and try it out, what do you think I’ll do? I’ll download it and try it out. People are curious, people like toys, people do unsafe things with their lives. IT departments are not people. People who work in IT departments get fired when bad things happen. They can’t keep bad things from happening (and they know they can’t) but they’re willing to keep their users from doing almost anything to try to keep bad things from happening. Hence, IT in enough large organizations became all about what you can’t do and not what you can. You can’t install that plug-in or have local administrative privileges, or browse unapproved sites. Anything that can be prevented will be prevented. The end result is a plain vanilla looking installation that practically fell off the Microsoft applecart. Not surprisingly it isn’t exactly designed to help out people building web applications.

Get up Neo

One of the amazing things about open source is that you can’t kill it. A company that has 2% market share (in most markets) may as well pack it in and go home since they aren’t going anywhere. Open Source projects couldn’t care less. Mozilla never died, it just got better and better and better until it emerged in its Firefox incarnation as a far better browser than IE. At 10% market share it may not be the dominant browser, but there is a very real possibility that if it continues to press forward it may well become the dominant browser (a prospect that has awakened Microsoft). At very least it has created new competition in the market. The question now is, will it stop paying attention to the old standards that it already implements fully and start blazing its own trail as a standard setter?

With the exciting growth of more advanced web applications I would not be surprised to see that happen. Perhaps we will see real modal windows or multi-edit controls or tree controls or the ability to maintain client side state, or an alternative non-HTML format designed for enhanced applications. Maybe we’ll see a Javascript component management framework or perhaps a simple facility that would allow client side applications access to a confined local disk space. Maybe there will be client side Ruby support or a TK implementation for web applications. Perhaps markup will be in a meta-language that lets you mix everything from wiki-syntax to HTML to pure XML to private hybrid DSL that is parsed by some meta-parser. There are a thousand things that could be done to turn the web into a more developer friendly space, and the emergence of competition makes them all possibilities just waiting for someone who feels a need to scratch one of these particular itches.

Sure, all of these things COULD happen, but will they? Here I’m not so sure. I’ve been doing an AJAX oriented project for the last nine months or so and it’s amazing what you can do using this approach, but it’s a LOT more work that writing the same application using a more traditional UI technology. For large scale applications like we have at StorePerform this adds a lot of cost to your project when compared to not using the web at all (although it’s probably actually easier that not using AJAX but still using the web). There are a ton of potential applications that probably can’t be done at all using this approach. For example, building a REAL word processor or spreadsheet is probably just a silly idea (I’ve seen AJAX versions of both of these but I wouldn’t want to use them). AJAX then has two possibilities: either it reawakens our desire for something better, or it fills in as just good enough to do stuff. My hope is that people will stop feeling like they have to develop to the inadequate target platforms we have today and start re-inventing the Internet itself. The other option is that we will build upon the web and extend and extend expending thousands of person-years of effort at the edges rather than solving the problem once at the core. I hope this isn’t the outcome but my psychic powers can’t assure me one way or the other.

No one has ever stood up to an agent and lived

So, on the front-end we’ve been squished by standards that have come to a standstill, but what about the back end? Why haven’t we seen tremendous progress on the server side of the equation? My answer, in a word: Oracle. Now, unlike the front end where Microsoft is directly to blame, Oracle is more of a symbol for the status quo. The one thing that almost any multi-user application has in common is that it needs persistent data (a database), and databases have changed how much since 1980? Virtually not at all! There are some cool “new” things like new index variants and materialized views and OLAP support tools but basically these go largely unused and we’ve still got the same SQL that we inherited from DB2 ages past. Since the big move from hierarchical to relational databases we’ve gone so close to nowhere that I’m not sure I could find the progress with the Hubble telescope.

Unlike the browser space, it seems like the database space has competition so why is there so little progress in basic database technology. My answer is that to make progress in this space you need to fundamentally rethink it, and that’s not something that happens easily in large companies supporting mission critical data. It turns out that it costs real money to play big in the database arena.  So when Versant and Object Store and a dozen other startups went after the object database market a decade ago they ended up delivering some cool technology that almost worked but not well enough for the data you really care about. Naturally people moved on and we were left with Object Relational Mapping which (frankly) isn’t nearly as good. Even object databases aren’t the right answer completely, we need to rethink databases more fundamentally in a world where drive space is free, memory is abundant and huge clustered systems are easy and cheap. We need to look beyond the limitations imposed by the semi-declarative SQL language and find a way to hide the larger problem space.

Today if we want a table that manages a fully cached set of data we do it ourselves. If we want a copy on write table that is garbage collected we do it ourselves. If we need real time access to a cross-table data aggregation we do it ourselves (can you say ETL and OLAP?). If we need to do migration between versions of our application we do it ourselves. Object Relational mapping can’t overcome limitations that are fundamental to the persistence model itself and so we get progress that is glacial. It’s that darned least common denominator issue yet again.

While I may see light at the end of the HTML tunnel, the database problem has been around so long that I’m a little less optimistic. I fear that this may fall in to the category of software that no one can write. My suspicion is that to fix this problem someone will need to dedicate a number of years to an open source project without much in the way of apparent progress before it will come to light.

There is no spoon

If you look at these two problems together they share a common thread. They stem from people accepting the software world in which they find themselves. We accept that the web is as it will be and we shape ourselves to its design. We accept that SQL and relational databases are all that persistence is and we bend to fit it. When we do this we forget the infinite malleability of software. We ignore the fact that the constant refreshing rain of hardware progress has turned the meandering stream into the mighty Amazon. As individual programmers most of us feel (justifiably) that the problems are outside of our domain, but sooner or later someone needs to step outside of that role and start the revolution. I do not know if web 2.0 will ever mean anything or if it will remain another fad word. I do not know if I will ever be able to program persistent data without having to remember what “group by” means. I do not know, but I believe that we will find the one, and I know that if we find him he will not wear a black suit. He will wear black leather.