The Internet: Preparing For The Future…

Share

Regardless of how you may feel about about the pervasive coverage of the passing of Michael Jackson, this event has served as yet another reminder of just how dominant a role the internet now plays in the distribution of news and media. It’s the first channel many people turned to to find out what was happening as that story rapidly developed.

mj-memorial2

Ironically, these types of events also remind us of the many limitations that still remain around web content delivery, and the broad challenges the web will face in supporting the kinds of things we are expecting it to support one day.

So how did the web hold up?…

As events unfolded, the LA Times – who broke the Jackson story – had its website crash after several million visitors hit it in less than an hour. Many other major news sites slowed down significantly due to high volume. Even Google, a company used to dealing with access on a massive scale, had problems handling the load of people searching on its Google News section for information about Jackson’s death. And Twitter, a site that has effectively become the web’s real-time “newswire”, was running at least 5 minutes behind in getting tweets posted. While not an infrastructure disaster, this event certainly pushed most news/gossip sites close to the edge of their capacity, and not many had a graceful way to degrade.

Problems seemed to be even worse for the mobile web. Unaware of what was going on that night, I had posted the following on Twitter:

torntweet

This wasn’t the first time I have had problems connecting to the web wirelessly through AT&T, but clearly it was more than just typical AT&T issues that ended up causing it this time. In general, I believe the adoption of web enabled mobile devices is outpacing even the fairly aggressive growth in mobile data capacity. Combine that with both a spike in demand and many unresponsive news sites, and the results were no doubt frustrating for many others as well.

The other big traffic spike happened this past Tuesday.

Many sites decided to set up live streams of the memorial service held for Michael Jackson. Though it didn’t go off completely trouble free, Akamai, the leading video distribution/streaming provider on the web, ended up serving about 20 million live video streams during that time frame. That is nearly 10 times the number of streams they typically handle – by any measure a huge spike. While this number is way short of the 100 Million+ viewers that watch live events like the Superbowl, it still represents around 3 times the audience that watches a typical top rated TV show every week. This was impressive.

Unlike the bit starved mobile web, the wired web didn’t seem to have an issue with overall bandwidth or routing. There were no reports of general slowdowns or serious bottlenecks occurring because of this event, which is great news. What didn’t seem to scale up as well were individual sites. Some of those issues could probably be addressed with a more aggressive adoption of cloud based site deployments. If designed correctly, cloud-based deployments could help these types of sites scale capacity dynamically to better handle unexpected spikes in demand. In effect, this is exactly what all of the major news organizations did by using Akamai to deliver their video, an aspect of their delivery that seemed to work pretty well. They produced and packaged the content itself and then leveraged Akamai’s shared global infrastructure to handle delivery – something they would never be able to do well on their own.

We need to start thinking differently about how to build out the web going forward, especially around the optimal use of shared vs proprietary resources. I also think we are probably getting to the point where we should look more closely at what role fundamental network management technologies like QOS, packet prioritization, deep packet inspection and traffic shaping should play on the internet, and how we can make sure they aren’t abused. This is an issue I am somewhat torn about. I am a big proponent of network neutrality, but recognize the very real negatives of an ‘all packets are equal’ approach to managing traffic. Though it tends to be an issue that evokes passion from all sides, we will need to have a rational, dispassionate discussion about it if we are serious about making the web into a truly global media backbone – something it has the potential to become.

Events like these remind us that we still have a lot more to do.

The Politics Of Power…

Share

There has been a lot of talk about the need for us to develop and deploy a smart power grid in this country. It is often mentioned as one of the pillars of our new 21st century economy, and an area we are going to need to commit some serious capital to.

blog-unitedstatespowergrid

In an excellent article Follow the Money in yesterday’s WSJ Online, three prominent venture capitalists discuss this topic in the broader context of investment opportunities in the clean energy space. While the article is definitely worth reading in it’s entirety, there is one quote from it I’d like to focus on here.

In talking specifically about the need to build out our power transmission infrastructure, Vinod Khosla from Khosla Ventures had this to say:

By the way, we don’t need government to build transmission. If we solve the eminent-domain problem, there’s enough money in transmission that private industry would build all of it. They [governments] have to give you right of way and eminent-domain rights. So, it’s not that difficult a problem to solve, and it’s not about money.

Everything Vinod says about private capital being more than ready to build out the grid is completely true. I’ve verified this with my friend Edward J. Terillo, a former energy analyst who knows this space intimately. Money is sitting on the sidelines waiting for the chance to do this build-out.

What’s holding this money back is that these proposed power lines need to run somewhere – and nobody wants that ‘somewhere’ to be anywhere close to them.

The center of politics when it comes to rebuilding our aging power infrastructure isn’t in Washington. It’s in ‘Anytown, USA’. Most people probably don’t pay much attention to the specifics of this issue. They may not know the difference between traditional power lines and the proposed ‘smart grid’, and have no real understanding of the stress our current national power transmission system is under. But what they will pay close attention to – and in serious numbers – is the mere suggestion that new power lines might be running through their neighborhood. What they absolutely know is that a power line running anywhere near their house will reduce its value in a significant way.

And they will do everything in their power to oppose it.

Mr. Khosla points out that one solution to this is for the government to grant these companies “right of way and eminent-domain” so that they can make these investments unencumbered. Based on that, he believes that getting our power distribution infrastructure upgraded is “…not that difficult a problem to solve”.

I don’t see it that way.

While mandating ‘rights-of-way’ may be sound in theory, what it means in practice is that the federal or state government will need to co-opt the rights of local townships and force them to permit these build-outs on lands within their jurisdiction. It also means that homeowners impacted by these mandated rights-of-way, while probably being marginally compensated for any land taken from them, would be unlikely to receive compensation for any negative impact it may have on their home values. I can’t imagine that there are that many people in Congress or the state legislatures that would want to have their name tied to any legislation like this.

But even assuming that these government bodies could marshal the political will to wrest control of these lands from local municipalities (not a given by any means), they would still need to determine where any new power lines sould be run. And that is where this process would really spin out of control. A broad collation of local rights groups and environmentalists will certainly launch a range of challenges to tie the planning process up in court. Individual homeowners that are impacted by these lines will be able to do the same. Politics will come in to this at every level, with federal agencies going out of their way to present plans that won’t disproportionately impact any specific demographic, while local politicians will look to protect their electoral power base.

And somewhere in that mess, the actual engineers might have something to say as well.

Perhaps a more practical solution, when all costs are considered, is to run the grid underground when it passes near or through any populated area. On a per mile basis, it costs about five times as much to bury the lines as it does to run them on towers. However, taking that approach may stir up far less opposition – utilities lines of all sorts are buried all around us – and might even end up costing less on a full project life cycle basis than delaying implementation by decade or more and dealing with the cost of endless court challenges. We should probably also think more about local/edge based power generation – especially solar – and how that could change the demand profile a national level grid would need to handle.

Figuring out an approach that works for everyone is going to be tough. There are a lot of different interests with a stake in this, and political minefields are everywhere. Attempting to impose a federally mandated solution will meet fierce opposition that will cut across party lines. It will require all sorts of deal-making and back-scratching to get any type of agreement in place.

And it will take time.

As Tip O’Neill once said, “All politics is local.” And that will prove to be especially true when it comes to the politics of power

No More "Cheat Codes"…

Share

My son loves video games, and is actually quite good at them.
cheatcodes-halowars
When he first started actively gaming, he would often look for any “cheat codes” that might be available for the different games he wanted to play. For those not into gaming, cheat codes are special codes that a player can type in to unlock new weapons, cars, levels, etc, without having to earn them by reaching specific milestones in the game. It was a shortcut to gratification – a way to get something immediately without having to earn it.

I would always tell him that there weren’t any cheat codes in real life, and that he needed to work for the things he earned. Fortunately, he took those words to heart, and he now has a real sense of pride around the gaming accomplishments and player rank he has justly earned. And I’m really proud of him too – just less for the gaming part and more for the lesson well learned.

Unfortunately, it turns out that I was wrong about life – there are cheat codes out there.

Just look at the news. We have had a endless string of athletes disgraced by their use of performance enhancing drugs. We have politicians getting caught on a fairly regular basis because they thought they could trade the public trust for personal gain. We have Bernie Madoffs’ springing up all over the country – money managers who bilked investors out of billions of dollars to fund their lavish life styles.

And then we have the most pervasive of all offenders – people living lifestyles beyond their means. It seems like I haven’t met anyone recently that doesn’t know somebody who is way over their head in debt (or who is dealing high debt loads themselves).

How did we get into this situation?

There’s plenty of blame to go around. Partly it came from a culture that embraces consumption and material success as virtues in their own right. Partly it came from people giving in to an environment of easy credit – live the “good life” now, and worry about paying for it later. Partly it came from people leveraging the constantly rising value of their homes to fuel an otherwise unsustainable lifestyle. Some of it came from people who bought houses they couldn’t afford in the hope that they would appreciate enough in value to somehow cover them before they crashed. Part of it was from reckless policies by the government, and a financial industry all too will to oblige them.

While the specific causes may be diverse and complex, all of them have one thing in common – a desire to find a shortcut; a way to “get things” without having to work for them first. And we’re living with the consequences of that right now.

Getting out of this mess will require creativity, sacrifice and hard work. Whatever solutions we come up with, they can’t be based on somehow returning us to the path we were on before. That has proven itself to be bankrupt on so many levels, and will simply set us up for failure once again. Instead, they need to focus on building a culture and an economy that rewards risk, demands personal responsibility, and shuns recklessness.

And we all need to agree that cheat codes won’t cut it anymore…

Words: Going Beyond The Purely Political…

Share

This is one of the most most anticipated elections in the past 50 years…

There is a lot at stake in this election. We are being asked to choose between two very distinct views of where we should head as a Nation, and the path we should take to get there. The choices we make today should be well considered – they really do matter.

It’s easy to get swept up in the rhetoric and passion of the moment, and subsume our better selves to the less-than-civil conversation that passes for political discourse in our society today. We attack each other with sound-bites and slogans instead of debating our future with knowledge and reason.

And in doing that, we fail to honor the principles that our nation was built on…

With that in mind, I would like to share two quotes with you that will go in to the voting both with me today. Both are from John Adams, one of the most inspirational and intellectual of our Founding Fathers.

The first is a reflection on the personal responsibility we each have for the realization of a free society:

The only foundation of a free Constitution, is pure Virtue, and if this cannot be inspired into our People, in a great Measure, than they have it now. They may change their Rulers, and the forms of Government, but they will not obtain a lasting Liberty.

John Adams, letter to Zabdiel Adams, June 21, 1776

The second is a call for us to inspire and guide the next generation:

It should be your care, therefore, and mine, to elevate the minds of our children and exalt their courage; to accelerate and animate their industry and activity; to excite in them an habitual contempt of meanness, abhorrence of injustice and inhumanity, and an ambition to excel in every capacity, faculty, and virtue. If we suffer their minds to grovel and creep in infancy, they will grovel all their lives.

John Adams, Dissertation on the Canon and Feudal Law, 175

I believe these are words that all of us need to hear. They call us as both individuals and a society to become something better, and they challenge us personally to take ownership of making it happen.

Straight Talk about Change We Can Believe In has been around for quite a while…

CNN: Monitoring The News?……

Share

This interview was on CNN this morning…

Here is a 13 second clip from an interview that John Roberts of CNN did with director Kelly Nyks discussing his new documentary “Split: A Divided America“:

The film seems both timely and interesting, and takes a hard look at the state of politics and civil discourse in our country. This is an important film, and one worth seeing.

However, that isn’t the reason I am writing about this interview…

This post is actually about what’s playing during this interview on one of the monitors in the background. Take a close look over John Roberts’ head in this frame grab:

Playing in that monitor is Nickelodeon’s SpongeBob Squarepants – not exactly one of the the critical news sources I would have expected to see rolling in the CNN studio.

Should we be surprised?

Probaby not. Politics and comedy have blurred on shows like Saturday Night Live and The Daily Show – shows that may actually end up having an impact on the coming election.

Maybe something’s going on with SpongeBob we’re just not aware of yet…

Presidential Debate #2: A View From The Clouds…

Share

The second Presidential debate took place Tuesday in Nashville, Tennessee…

As I did with the first debate, here are the “Word Clouds” built from the answers provided by both candidates – Seantors Barack Obama and John McCain:

Word Cloud for answers given by Sen. Barack Obama:
Click to enlarge.

Word Cloud for answers given by Sen. John McCain:
Click to enlarge.

The feedback I received from the post presenting word clouds based on the answers given in the first Presidential Debate was definitely positive, and I appreciate the response.

I hope you find these equally interesting…

Presidental Debate #1: A View From The Clouds…

Share

Politicians use words both to communicate and to obscure…

The first presidential debate last Friday was an excellent showcase for how each candidate struggles to define their beliefs and priorities, and works to create an unflattering image of their opponent.

As an interesting exercise, I have created “Word Clouds” based on the answers given by each of them during the debate. Hopefully, this will provide you with a different, less polished view of what they were looking to communicate – and to hide.

Word Cloud for answers given by Sen. Barack Obama:
Click to enlarge.

Word Cloud for answers given by Sen. John McCain:
Click to enlarge.

There is a certain Rorschach Test nature to looking at these word clouds. It’s easy to see in them what we want to see. In some ways, they can tell us as much about the way we think as they do about the way the candidates think. Ask yourself if you could you have guessed which word cloud belonged to each of the candidates if I hadn’t labeled them.

I plan on creating these word clouds for each of the upcoming presidential and vice-presidential debates. In fact, I would like to start producing these clouds for a wider variety of things that come up in non-political areas as well.

I hope you find them interesting…

NOTES: 1.To help clean up the clouds, I removed references to each of the candidates names, and the name of the moderator Jim Lehrer. 2.Word Clouds were produced using Wordle.

Using Clouds To Cut Through The Fog…

Share

Tag Clouds have been around for a while…

They offer a great visualization of what topics are ‘hot’ within a given framework, and they can work in a variety of settings. You could have a tag cloud of search terms being used on Google, stocks mentioned in posts on Twitter, or tags assigned within a single blog. A tag cloud works by showing the most common items as large and bold while the less common ones are small and somewhat faded.

They give you a good feel for what’s in a content set with just a quick scan…

That why I found these Tag Clouds produced by Thomas Hawk so interesting:

Flicker image by Thomas Hawk

Politics is full of words. They can create a fog of ambiguity that confuses issues and obscures positions.

And sometimes politicians choose words specifically for that purpose…

What Thomas Hawk did is a great way to cut through that fog. The top cloud is an analysis of terms that were used in John McCain’s acceptance speech, and the bottom is of the terms used in Barack Obama’s. It’s great way to see what images each candidate is trying to invoke, and how they are using words to shape perceptions about themselves and their agendas. There are clearly some similarities in these two clouds, but also some interesting differences.

Dig in and enjoy…

NOTE: I am posting this is a completely non-partisan context. The only endorsement going on here is for cool data modeling approaches.

Money Tech 2008…

Share

I had the opportunity to speak yesterday at Money Tech 2008…

The conference, hosted by O’Reily Media right here in New York, was focused on the convergence of Web 2.0 and Wall Street.

blog-moneytech.jpg

I took the opportunity to talk abstractly about the process of information discovery, and how it has changed since the arrival of the internet. It was probably a good audience for this type of talk, though I admit I had trouble fitting everything I wanted to cover into my 15 minute time window.

One of the topics I did touch on there was the changing meaning of scarcity as it relates to information.

Essentially, the value of a piece of information is greatly impacted by how scarce it is – the more people that have it, the less valuable it typically becomes. Pre-Internet, scarcity was all about controlling access. There were only a few sources of information available, and all of them locked up their content and charged people money to access it. Professionals that could afford the high monthly subscriptions got access – and everybody else was out of luck.

But the Internet changed that…

Now, there’s an endlessly growing number of information sources available on every imaginable topic. Most of what is produced today is both free and broadly available. Content has become ubiquitous to the point of being overwhelming – you have access to just about everything. The challenge – the new scarcity – is in discovering the specific content in this sea of information that’s valuable to you.

So how does this impact the marketplace?…

Pre-Internet, when people paid for access to a content source, they got it. There wasn’t a lot of content out there (by today’s standards), and with even the most basic of filtering, everyone that cared about a piece of information was able to discover it right away. That tended to limit how valuable a given piece of information could be, and the only way to gain an edge was to build a very efficient transactional mechanism behind it. Those that could convert the information they had into something actionable and execute on it quickly stood to gain the most.

But the story is different now…

Free or not, you can’t simply look at everything that’s being published anymore – there’s just too much of it. You need to invest more effort into filtering through it to discover information that can bring value to you.

And the value you can get from that information today is best described by this chart:
blog-moneytech-slide.jpg
The green bar in this graph represents the time it takes for you to discover a relevant piece of information. The blue curve traces the value of that information as it becomes less scarce – as more and more other people discover it over time.

Whats obvious here is that the more efficient you are in the discovery process – the quicker you find things that matter to you – the more value you will be able to extract from them. And the maximum value a piece of information can deliver to the most efficient ‘discoverers’ can be significantly greater than it was back when everyone that paid for access got access to it at the same time. Unlike before, during that early discovery window the number of “people in the know” could actually be quite small. And in financial markets, that’s a good thing.

If you think about it, while access to content may have become completely democratic, the process of content discovery is now purely Darwinian.

I guess that’s what makes it such an exciting field to be working in…

Publishers Go On The Offensive…

Share

Keep a very close eye on ACAP…

blog-acap-logo-sm.jpgA broad collection of content publishers have banded together to form a standards group call ACAP – Automated Content Access Protocol. They released the first version of their proposed standard yesterday. The goal of this standard is to give publishers a means of controlling how their content is discovered, indexed, and retained by web search engines. The objective of this group has been to replace ROBOTS.TXT, the current standard used to direct and limit web crawling, with a more robust and comprehensive model that provides more granular content permissioning.

And that’s not all…

ACAP outlines a more comprehensive long term agenda of control that places them in the middle of just about every type of content exchange that happens on the web today. They claim that having a standard like this in place, one which offers more flexible and descriptive rules for how their content can be used, will end up letting them make more content available to users. Their argument is that more content will be available to more people if everyone just goes along with more control.

Unfortunately, that sounds like the same argument used by the big media companies to justify adding Digital Rights Management to just about everything they’ve produced.

And look at where that got them

The media marketplace today is weak, fragmented, and struggling to remain relevant. The promised flood of ‘back catalog’ content never materialized, and prices never came close to the market’s perception of the value they received from digital distribution. Unfortunately, DRM was always more about protecting outdated business models than providing any benefits for consumers. And everyone has suffered because of it – both producers and consumers. The industry might never recover.

The same will be true with the publishing industry and ACAP…

All their marketing spin aside, ACAP is just a money grab by the publishers. Not satisfied with monetizing the significant traffic search engines generate to their own web properties, they now want to sell the right for search engines to index pieces of their content in the first place.

And that mindset has some serious implications for the future of the web…

The free web would start to erode. If they are unable to ‘pay to play’, small and niche search providers will fade away, along with any disruptive innovation they might have brought. The remaining folks – the Google’s, Microsoft’s, Yahoo’s, and Ask’s – will not doubt end up paying, and will likely try to create exclusives of some kind with these publishers to differentiate themselves and maximize their returns. If that were to happen, we could end up with a patchwork of coverage that will balkanize the search space.

And it could even jeopardize open content technologies like RSS…

The entire premise of ACAP is dangerous. It threatens free and open search on the web, and can easily become a slippery slope. The right to link to other sites freely, to create unencumbered feed readers that aggregate content, to aggregate comments left at sites, or to deliver other content centered innovations could all be at risk.

The publishing industry, like their peers in the media industry, remained rooted in the past and failed to adapt their companies and business models to the realities of a digital world. Instead of innovating, they want to go back to the way business was 20 years ago. Back to when they had total control and consumers had little choice.

ACAP is their attempt to reclaim that on the internet.

The entire web community will suffer if they succeed…

You can check the ACAP site at: http://www.the-acap.org/
A good overview document can be found here: ACAP Business Case.