Moore’s Law And The Future Of Software…

Moore’s law has been an effective a predictor of technology pricing and performance…

My first real experience with the power Moore’s law came when I upgraded from my old IBM PC with an Intel 8086 to the new PC AT with a 286 processor. I paid about the same as I did for my earlier computer, but found myself looking at an incredible boost in performance. The improvement wasn’t just just significant - it was transformational.

I could see creating programs for the new system that I never would have considered viable on the old. Several years later, this experience repeated itself with the introduction of the Intel 80386 - another incredible leap in speed and capability. We seemed to be on a incredible performance trajectory.

But then, something happened…

Software started to get more complex.

The ‘old guard’ programmers that grew up writing in assembly language on systems with 1K of memory began to fade away. With new, powerful - and lower cost - systems entering the marketplace, it became ‘bad form’ to focus on counting bits and cycles. This new wave of applications became more ambitious in their scope and functionality, and more accessible and creative in their interactivity and interface design. Anyone writing commercial or corporate applications started moving their development over to some version of ‘C++’, a language better suited than most to deal with the increasing demands being placed on software developers. DOS - the lightweight operating system that dominated the marketplace - found itself being replaced by Microsoft’s Windows 386, which offered a more robust framework for delivering graphically based applications. And with Windows came HAL - a hardware abstraction layer that hid the details of the machine a program was running on.

This marked the start of one of the most significant transition in technology history.

Software replacing hardware as the driving force for industry progress…

With software in the driver’s seat, computer based technology started to appeal to a much wider demographic, resulting in the arrival of new suites of applications. And with hardware prices continuing to drop at the rate predicted by Moore’s Law, the appeal of these newer applications broadened significantly. Devices that once required a corporate level budget (or serious geek commitment) now became accessible to a more mainstream, less technologically literate audience. This deflation in hardware prices created a virtuous loop that fed an explosion of new and more diverse applications which in turn resulted in increasingly pervasive market penetration.

The arrival of the internet in the mid-1990 only accelerated the process, moving software into what has come to be known as a “continual beta” state, and pushing harder for the eventual commoditization of PC’s - and hardware platforms in general.

That commoditization has become a two edge sword…

The upside is that the drop in hardware prices has made technology of all kinds almost universally accessible. We have gigabytes of storage in our phones and cameras, and processing power in our PDA’s that would have been enough to run an office 15 years ago.

The downside here is that hardware is now a dwindling component of any investment made in a technology based service or solution. It is no longer the main factor in what we pay for our technology. That distinction now rests with software.

And unfortunately, software doesn’t follow Moore’s Law at all…

Traditional software development continues to be an increasingly complex undertaking, and this complexity makes it resource and time intensive - and thus very expensive. There are two very distinct reasons for this increasing complexity. One reason falls in to the ‘Apple’ software camp - it isn’t easy to create exceptionally intuitive software to do relatively sophisticated tasks (examples are the iPhone and their iLife productivity suite). It requires a significant investment, innovative thinking, and iterative refinement. The other reason is commercial pressure. In most application genres, all of the basic functionality people need is already implemented in the software they own. To remain viable businesses, application developers are forced to create new ‘must have’ features that people will be willing to pay for - features that their competitors do not have. In effect, complexity is being added in a bid to escape commoditization. A good example of this is Microsoft’s Office 2007 productivity suite, a incredibly complex and expensive piece of software in what is essentially a twenty year old product category.

Their biggest competition is often the previous version of their software…

As technology consumers, we have become addicted to the historically declining pricing curve fueled by rapidly falling hardware prices. Getting more for less isn’t something we are willing to give up on, and that reticence is creating a real upheaval in the industry. Some recent software upgrades - most noticeably Microsoft’s Vista - have failed to gain any real consumer traction due to their cost and complexity. This marketplace backlash has called in to question the traditional way software has been developed and sold, and has many firms scrambling to reassess and refocus.

So how is the industry adjusting?…

Some firms see this as an opportunity, and are starting to extend their franchises with Software as a Service (SaaS) business models. Companies like Google and Yahoo are entering the application arena using web based services to replace traditional thick-client categories. These applications already offer many of the basic functions people expect from them, and are continuing to grow in sophistication. Being web based, they add a new social/collaborative dimensions not found in their traditional counterparts, but they also fall way short in disconnected use. If you aren’t online, their capabilities are often limited or non-existent. These services are largely ad supported, and offer consumers both free and subscription based options

Others like Amazon are providing rich platforms for new applications to develop against. These web service and infrastructure providers are looking to jumpstart the development of new applications and services by bringing efficiency and scale to the more commodity based capabilities most development projects depend on.

Traditional application vendors aren’t sitting like dinosaurs waiting to be hit by the meteor. Some like Microsoft and Adobe are experimenting with bringing their ‘franchise applications’ on to the web, while at the same time looking at ways to transition their traditional application base over to a ’software rental’ model. Their goal is to create a revenue annuity that no longer depends on continually pushing updates into the marketplace, though it is unlikely that they will be able to do that in a revenue neutral way

The wild card in all of this is open source. There are some fantastic applications in all genres coming out of the open source community. Most people are familiar with with Linux and Firefox, but there’s a whole range of tools and applications beyond that - Open Office, GIMP, Audacity, and VLC, Lucene, MySql, Miranda, Wordpress, PDFCreator and many more.

This model transition in the software industry is really just beginning and has a long way to go before things settle down again. New players will appear and ascend, and some of the big names today may diminish or disappear completely. Whatever the new equilibrium point for the industry ends up being, it will no doubt involve a combination of all of the initiatives discussed here. And I believe it will also reflect a more distributed approach to code craft and a more federated approach to service delivery.

As a result, the economics going forward will absolutely be different then what’s in place today.

They’ll probably operate a lot closer to Moore’s Law…

Feel free to share this post These icons link to social bookmarking sites where readers can share and discover new web pages.
  • digg
  • del.icio.us
  • Reddit
  • Furl
  • co.mments

Disconnected…

I caught this video clip over on Fred Wilson’s blog and had to post it. It captures exactly the way I feel when I have no easy access online.

Enjoy!

Feel free to share this post These icons link to social bookmarking sites where readers can share and discover new web pages.
  • digg
  • del.icio.us
  • Reddit
  • Furl
  • co.mments

Is The Party Over?…

This primary season may be a signal of changes in our political landscape…

The power of the ‘long tail’ is being felt in many aspects of our lives. We increasingly want things in our life to be ‘unbundled’ - broken into smaller bits that we can pick and choose from. We Tivo television to watch it on our own terms, and prefer iPods to radio stations. We’re far more inclined to donate to very specific causes than to general charitable organizations. We buy cars today that can be personalized in hundreds of detailed ways. We are becoming a society composed of an incredibly large number of incredibly small demographics.

Often times, a demographic of just one…

This desire to personalize the world around us touches every aspect of our lives - from the media we enjoy, to the news we follow, to the faith we embrace, to the causes we believe in, to the products we consume. And to the parties and politics we subscribe to.

But that’s only a part of the story…

In past elections, it was possible for candidates to “tune” a message for different demographics. When they stumped across the country for votes, their message, language, and tone would change as they moved from area to area. They always had some common themes, but based on the audience, some subjects would be ignored or glossed over, while others would be placed front and center. They could treat each of these stump speeches as local, targeted events.

But not anymore…

In a world where everything gets recorded either officially or unofficially, nothing is local - or even private. Unlike traditional media, bloggers will compare notes and cooperate to track down inconsistencies or fabrications. If a candidate changes positions, or says something controversial, or does something that’s plain dumb, someone - somewhere - will have a record of it. It may be a grainy cell phone video from someone in the audience, or even some archive news footage that someone digs up.

And it will be viral on YouTube before a campaign can spin it…

What we are starting to witness is the nexus of two significant trends - the increasing number of narrow, ‘issue-based’ voter demographic segments, and the rapid decline of effective narrow targeted messaging.

It’s the ‘Long Tail’ meets the ‘Global Village’…

We are starting to see these elements play a big role in the contest between Barack Obama and Hillary Clinton. Each side has been beaten down over seemingly contradictory stances they have taken on issues like NAFTA, immigration reform, taxes, campaign finance, or the war effort. They have been haunted by various videos from their past, almost blind to the fact that the same web tools they use to raise cash can be used to raise issues they would rather not face. Each ‘owns’ an extremely loyal core demographic, that may never embrace the other candidate if he or she wins. And party officials are becoming increasingly worried that these demographic fractures may never heal.

We see statistics after every primary contest about how a matrix of demographic combinations voted - religion, race, income, age education, martial status, and gender all come in to play. When you look at all of this in its totality, something becomes very clear.

There is no singular ‘Democratic Party’…

From looking at the conflicting and diverse interests and agendas of the various demographics that make up the party, it seems to be more of a loose affiliation of people that don’t identify themselves as Republicans rather than adherents to a broad, unifying Democratic platform. It’s not a single political body, but a small core of beliefs with many agendas in orbit around it.

And the exact same can be said about the Republican party…

At the ‘outer orbits’ of these political parties are what the media refers to as The Independents. These independents are becoming increasingly important. Neither party can win without attracting them in significant numbers. They are courted heavily by both parties and end up being the swing votes in just about every election.

And that fact isn’t lost on them…

It isn’t hard to imagine that this broad swath of voters could break off from both parties at some point if a credible political figure were to unify them into a true third party. If anything, it’s hard to understand why this hasn’t happened already. The current political parties are really just a facade masking a great deal of frustration and discontent on many issues.

They persist more by momentum than shared fundamental beliefs…

Like it has in every other aspect of society, the long tail will make itself know in the political arena. We are a diverse society confronting significant and complex issues. There is a growing consensus that our current political structure is failing us, and that the problems we face run deeper than simply the failings of one party or the other. The ideas and solutions that rise to the top of our political system are not the best our country has to offer. They are simply the ones that are least threatening to the status-quo. They are designed to sidestep any hard choices and to avoid forcing us to face unpleasant realities.

The process and outcome of this Democratic primary may force us to look more closely at these issues and decide where we will turn next for leadership. The political pundits have all said that this isn’t a campaign about experience.

It a campaign about change.

And they may have been more right than they realize…

Feel free to share this post These icons link to social bookmarking sites where readers can share and discover new web pages.
  • digg
  • del.icio.us
  • Reddit
  • Furl
  • co.mments

Incorrect Feed Link…

When I upgraded the Digital Edge to WordPress 2.5, I placed an incorrect link for the “All Posts” feed. The incorrect link was for the old .MAC version of this blog, and hasn’t received an update in quite a while. This has now been corrected.

The current and correct link to All Posts on this blog is:

http://feeds.feedburner.com/thedigitaledgeblog

If you subscribed to The Digital Edge over the past two weeks, please verify that you have the correct link in your feed reader.

Sorry for the mix-up & thanks for subscribing!…

Feel free to share this post These icons link to social bookmarking sites where readers can share and discover new web pages.
  • digg
  • del.icio.us
  • Reddit
  • Furl
  • co.mments

Just The Facts? Maybe Not…

The emergence of semantic search technologies holds a great deal of promise…

One of the biggest benefits anticipated by the wide adoption of a semantic based approach to content discovery is the ability to ask a basic question to a “search engine” and get back a specific answer.

Not a list of sites, but an actual specific answer.

While that ’search experience’ is very appealing on a conceptual level, it starts becoming somewhat muddled when it comes to delivering practical implementations. There are three aspects to this new world of search that present challenges and will require greater thought and discussion.

First - Many questions don’t have simple answers:

Factual information can have a deeper context that is difficult to express in a simple question/answer framework. Consider the question “What is the population in New York City?” You may end up with several different answers - and all of them could potentially be correct.

How?…

One site may quote numbers reported directly from the most recent census (e.g. - the ‘official’ numbers). Others may be more recent estimations of the same, and potentially more ‘accurate’. Others may include or exclude unofficial demographic segments - like the homeless or illegal immigrants - or estimate them using different formulae. They all contain a dimension of ‘truth’, and but you’d need to understand the context each came from to appreciate it.

But none of that subtly is easily express via a simple specific answer…

Second - It’s not clear what the correct answer is:

The fact that a source provides an answer to a specific question doesn’t mean that it is the best answer or even a correct answer (What? There’s inaccurate information on the web?!?) That means that these new “search engines” will need to choose - from potentially many different sources and many different answers - a ‘correct’ answer to return. Current methods for site ranking don’t translate well into ranking factual accuracy. They were designed to measure site relevance based on popularity, not accuracy - and there is at best a weak correlation between the two.

Another approach that could be suggested as a solution here is the application of a weighted model based on a ‘wisdom of crowds’ philosophy. This model would hold that the correct answer is likely to be the most repeated answer. While that may have some rational basis behind it in a more random selection of sources, it may not apply to analysis of content on the web. For a ‘crowd’ based model to work well, the individual sources should not be influenced by one another. They need to remain discrete contributing entities to the final answer, or you end up with “group think”. Unfortunately, the web is a giant echo chamber, and answers on one site -right or wrong - can propagate to hundreds or thousands of other sites. This will give that particular ‘answer’ a disproportionate influence in the aggregated determination of a response.

As I said before, ‘popular’ isn’t necessarily ‘accurate’…

Regardless of the general approach taken to discriminate between multiple potential answers, it will also need to be able to deal well with ‘disjunctive’ information sets. Disjunctive information is information that breaks from the past in some way. Any process that biases its selection exclusively using historical factors will ignore the dynamic nature of some content. The most current answers will - by definition - have the fewest historical references and links. Answers to questions like “What are the known side effects of…?” may have highly relevant updates that will be important to include in a response, but would be deemphasized using a purely backwards-look heuristic. Addressing this would be critical in domains with highly dynamic content flow.

Third - It breaks the current commercial foundations of the web:

The current commercial framework on the web is largely built on either a subscription model or an advertising/sponsorship model. Subscription models place content behind a firewall and require payment to access it. Advertising models are based on generating a meaningful and sustainable level of traffic to a specific site. Neither of these approaches fit comfortably in a search engine based ‘question/answer’ model.

Subscription based content isn’t broadly available for mining by search engines. People tend to view the sites they subscribe to as special sources, and will visit them uniquely to access specific types of information. The value search engines bring to subscription sites today is a link back to a login or sign up page based on fairly broad metatags. That wouldn’t work in a question/answer model.

Advertising based approaches depend on driving traffic to a site to generate revenue. Content is created to address a specific audience. They can discover it via search tools and visit the site - generating traffic. Sites can even buy specific search terms to improve their visibility in certain searches and hopefully see an up tick in visits.

Unfortunately, the ‘Question/Answer’ model takes the opposite approach. It attempts to deliver an answer directly to a user without requiring someone to actually visit the site it came from. In fact, if an answer is selected based on a statistical methodology, there may not even be a specific site responsible for providing the answer - it actually may come from ‘everyone’ in the tracked cloud.

Finding a way to share revenue in this model could be difficult. It could end up looking like the (thankfully) failed ‘piracy tax’ that was proposed on DAT tapes and blank CD’s. It would have added a fee to the sale of these recordable media that would then be distributed to specific artists using a vague allocation methodology. The lesson here is that any solution that diffuses the relationship between performance and compensation is inherently inequitable and ultimately unworkable.

Using web content in this way - essentially harvesting and repackaging information from millions or billions of web sites - raises significant copyright/IP issues as well. And these issues, like the web itself, exist on a global scale. Finding a solution will require moving beyond the parochial and politically deficient requirements of individual jurisdictions, and embracing a simpler global framework that is easy enough for everyone to use, but specific enough to address the genuine concerns of content creators. This could end up being a catalyst for the broad adoption of an enhanced version of the current Creative Commons framework - something long overdue in the online world.

—–

While the issues discussed here are not insignificant, there is enormous value in finding broadly acceptable models for working through them. These are foundational components of our next move forward on the web, and there is a great deal we will learn in the process. Determining how to address the ‘answer selection’ challenge will push the boundaries of social search and discovery methodologies, as well as accelerate progress in top down semantic analysis. Establishing a commercial and legal framework for dealing with content sharing at this granular level will create a surge of creativity and innovation in cooperative computing and social interaction that would easily dwarf the accomplishments of social pioneers like Facebook.

The innovation horizon on the web just keeps getting broader and broader…

This post is an expanded consideration of a subject I talked to in a comment on a previous post.

Feel free to share this post These icons link to social bookmarking sites where readers can share and discover new web pages.
  • digg
  • del.icio.us
  • Reddit
  • Furl
  • co.mments

Avoiding The Bidding Bubble…

EBAY is a strange marketplace…

I have been part of the EBAY community for many years, both as a buyer and a seller. Over that time, I have had the opportunity to observe people’s bidding patterns, and have come to a pretty interesting conclusion.

Too many people on EBAY have no rational basis for the way they bid…

For those readers that don’t know how bidding works on EBAY, it’s actually pretty simple. You can enter the maximum amount you are willing to bid on an item, and EBAY will automatically raise your bid in fixed increments as other people bid against you. This is different from bidding in a live auction, where the amount you signal is actually placed as a bid. On EBAY, no matter what the maximum value was that you entered for your bid, you’ll only end up bidding the minimum you need to win an auction.

What I have found, however, is that when people are outbid, they will often come right back and adjust their “maximum” bid amount upward. I doesn’t matter that they might have had days to decide what their maximum bid should be. They suddenly want to bid higher when they see they are losing.

And within seconds they freely abandon reason…

Based on the general pattern of bidding behavior I have observed on EBAY, I would posit that there is only one truly rational bidding strategy that can be employed there.

Sniping…

“Sniping” is a typically pejorative term applied to the practice of placing bids on an auction item at the last possible moment before it ends. On an emotional level, other bidders tend to consider snipers something of a cross between a jerk and a thief. They become angry and indignant when a sniper sweeps in in the final seconds and “robs them of their victory”.

But emotions aside, the fallacy of bidding early is clear…

There is no first mover advantage in placing a bid. While it may appeal to a primal urge we have to “mark our territory”, early bidding does not offer any knowledge or control that can help the bidder create a more beneficial outcome for themselves.

In fact, the opposite tends to be true…

I see two key negative consequences from early bidding:

First, any bids placed before the final moments of an auction can stimulate an emotional competitive response in the market and drive the winning bid higher. In that case, these early bidders simply end up bidding against themselves. Second, auctions with existing bids tend to attract more interest than those without bids. And that increased interest will only work to the seller’s advantage - not to the buyers.

Bidding isn’t a personal competition. It’s a marketplace mechanism for determining the spot value of goods. Unfortunately, the social context of the bidding process can create “Bidding Bubbles” - emotional pockets that drive valuations above the rational worth of the items in question.

It happens with stock markets. It happens with housing markets.

And it happens on EBAY…

The process of bidding exploits our competitive nature to drive us to act against our broader self interest. It injects emotion into a situation best served by dispassion - turning interest into desire, and desire into desperation.

And it leads us to make irrational decisions…

Successful bidding on EBAY requires patience and discipline. You need to determine in advance the maximum amount you are willing to bid for an item. You need to wait to the very last moment, and place that maximum bid at a point in the auction when no one can react to it.

And you need to be willing to walk away without “winning”…

Feel free to share this post These icons link to social bookmarking sites where readers can share and discover new web pages.
  • digg
  • del.icio.us
  • Reddit
  • Furl
  • co.mments


 

Now On DIGITALedge.TV

 

Promo For re:Search

The Show About Search

re-searchpromo.jpg

Click To Learn More