The Black Liszt

When you call a programmer "arrogant," are you committing libel?

The best programmers are often accused of being "arrogant." Are they? When you make the accusation, are you committing libel?

How to respond when accused of Arrogance

You can just tough it out. Dilbert shows us the way here:

In order to figure out how to respond, maybe we should understand just what arrogance is.

What is "arrogant," anyway?

Here's the scoop from the dictionary:

Definition of ARROGANT

1: exaggerating or disposed to exaggerate one's own worth or importance often by an overbearing manner <an arrogant official>

2: showing an offensive attitude of superiority : proceeding from or characterized by arrogance <an arrogant reply>

The second meaning is clear: you're arrogant if people don't like the way you act or talk; they somehow think that you think you're better than they are.

The first meaning is more interesting: it links the way you act to the facts of the case. You're arrogant if you act like you're better than you are. Hmmm...

Does this mean that you're arrogant if and only if you exaggerate how good you are? Sure sounds like it. So your arrogance is real arrogance IF your view of your self-worth is greater than your actual worth. Sounds reasonable, actually. If Eli Manning (the QB in the Superbowl who is not married to Giselle Bundchen) says "I'm a great quarterback," is he being arrogant? I'd say "no."

Arrogance is understandable and justified...

What happens when some aggressive, ignorant fool takes over a meeting, presses his own neanderthal solution and is close to getting it turned into marching orders for the less-aggressive ignorant fools? First of all, I'd say buddy, you're in the wrong place. Bail out! May Day! May Day! Second, I can completely understand getting everyone's attention, perhaps with some edge, and putting out a superior solution.

...Except when it's not!

The other problem is that sometimes the nerd is really wrong. He's just blown it. This is easy to understand. Are all nerds Top Nerds? Of course not! So there are whole lots of nerds that are wrong (or at least sub-optimal) on lots of subjects lots of the time! Yuck!! Even worse, such a nerd is, almost by definition, an "arrogant nerd," even if the nerd is behaving pretty well.

Arrogance and Libel

Suppose you call someone a lying tax cheat. In public. Their reputation is under attack, and they respond by suing you for defamation of character, i.e. libel. IF you can prove that the person in fact has lied about important things and has in fact cheated on their taxes, it's case closed: there is no libel, no defamation of character, when all you're doing is speaking the truth. 

Now suppose you call someone an arrogant programmer. In public.Their reputation is under attack, and they respond by saying they're not arrogant, they're just right and you're wrong -- get over it! IF you can prove that the person in fact writes bad programs, designs them poorly and that there is in fact a better way of doing things, it's case closed: they are arrogant! There is no libel, no defamation of character, when all you're doing is speaking the truth.

Conclusion

There is a great deal to be said about nerds and arrogance. In the end, it's pretty simple. Try to be nice most of the time. When someone's being a fool, be kind. But you still can't let fools determine technical outcomes. Have you missed something? Are you really smarter in this case? If so, get the right outcome. Will you be called "arrogant?" Probably. Let them prove it!

Fundamental Concepts of Computing

There are a small number of truly fundamental concepts in computing. They are not generally taught or talked about, but they underlie most of the smart things you can do in computing.

The fundamental concepts are like "the fundamentals" in a sport, the very basic things you have to do, like dribbling in basketball or blocking in football. It's where the phrase "blocking and tackling" comes from. Woe to the team that puts all its energy into fancy stuff -- it will be beaten by the team that does the fundamentals.

The fundamentals are generally recognized in sport because there are objective measures of scoring and determining which team won. Eventually, people figure out which activities contribute most to winning. But in computing, it's a sad story.

Competition would help us understand what are "computing fundamentals"

If we competed in programming the way we do in sports, teams from different places would take on the same job at the same time. Each would complete the job, roll it into production and run it for awhile. For each team and their product, we would collect a variety of information, including: the size and cost of the team; the elapsed time they spent building, the resources required to build and operate, the number of bugs, the level of user satisfaction, etc.

Who won? Well, we'd take some combination of the information above.

I bet if we did this a lot in programming, the "fundamentals" of programming would gradually become clear to everyone. Everyone would want to know what the winning team did to win, what winning teams had in common, and over time, the programming equivalent of "blocking and tackling" would become obvious.

But we never compete!

Oh, you think we do? Like when there are competing products, or when competing companies have similar computer systems?

Well, sure, at a business level there is competition, but at a programming level? Think about football. How would you feel if the team that won had twice as many players as the other team? What if the winning team got to use a different ball than the other team? What if the winning team was given ten downs per possession, and the other only had three? What if one team always had a goal post that was half as high and twice as wide as the other team? With differences like this, it's obvious that the game is rigged and there's not much to learn from the game. There is as little to learn from examining the programming practices of companies or products that win in business.

So what are the fundamentals of programming?

There is no generally accepted answer to this dead simple but incredibly important question! And given the lack of meaningful competition, there is no objective way to prove what they are!

I've spent way too much time:

• programming, and
• trying to get better at it, and
• wracking my brain to determine exactly what "getting better at programming" means, and
• trying to identify the key factors that lead to better results.

So I've got opinions. I've written about some of the fundamental concepts in some of my private papers, and I intend to post about some of them here.

For this post, it's sufficient to establish the basic concept: in fields that we care about, there are measures of goodness and a not-too-large collection of "fundamentals" that constitute the "blocking and tackling" for the field. And that, sadly, a broad understanding of those things are lacking in the field of computing.

 

(Most) Nerds are Introverts

A book on Introverts was just published. Mostly it states things that are obviously true. But since most people don't know these things and they're not part of mainstream cultural thinking, it's worth reading.

Jobs and Wozniak

One review of the book refers to the hoopla about the "wonderful, creative" Steve Jobs. Here's an excerpt:

If you look at how Mr. Wozniak got the work done — the sheer hard work of creating something from nothing — he did it alone. Late at night, all by himself.

Intentionally so. In his memoir, Mr. Wozniak offers this guidance to aspiring inventors:

“Most inventors and engineers I’ve met are like me ... they live in their heads. They’re almost like artists. In fact, the very best of them are artists. And artists work best alone .... I’m going to give you some advice that might be hard to take. That advice is: Work alone... Not on a committee. Not on a team.”

Introverts

Not every introvert is a nerd (far from it); and not every nerd is an introvert (though most of them are, I think). So this book is worth looking at because of the high overlap. I was particularly struck by the transformation in American society to elevate the extrovert to the image of what is desireable in a human being. To the point of admissions officers at top universities saying they'd rather have someone who was good at sports and slapping backs than someone who (among other things) doesn't put going with the crowd above all else.

Internet Software Quality Horror Shows

Whether the software is a cool social app, an academic website or a real business, there is a common theme: the software is poorly designed and, even worse, it just breaks. As in falls flat on the floor, waves its arms in surrender, and just gives up. And not just once -- it keeps breaking! As I've said before, we really need a revolution in software quality.

Cool Social Apps

Hey, social is where it's at -- how can billions of Facebook users be wrong? Before long, there will be as many FB users as MacDonald's has sold hamburgers (billions and billions)!

Those guys must be great programmers, huh? I mean, just look at their office:

Here's one of them giving a talk at a conference:

See how cool he is? He's just wearing a t-shirt, not even "business casual."

The other social media are just as cool. Here's a "chill" Twitter office:

And Jack Dorsey, the Twitter CEO -- quite the opposite of a buttoned-down financial guy, huh?

It's perfectly obvious that these guys must write just the coolest, most awesome code ever. There's no way people this cool could make elementary programming mistakes, particularly when their application is so very dead-simple, and hardly ever changes -- they could spend practically all their time being cool and polish up some already-faultless code a couple times a day, and still be OK.

Except this little detail, which I scraped from my own screen, and which I personally have seen countless times:

Yes, the famous Twitter fail whale. I think Twitter got tired of all the publicity their "cute" failure message was getting them, so they reverted to something more discrete; here's an example:

FB is just as bad, of course, and they've always tried to minimize the message when they screw up:

Apparently, FB is incapable of keeping even the most recent day's worth of updates on-line -- you should try going back in history and seeing how far you get. Oh, you thought the stuff you wrote was your data, did you?

Naturally, it makes sense to consider that you get what you pay for; all these cool social apps are, after all, free. You can hardly complain when something you didn't pay for is flakey -- return it and demand a full refund!

So let's turn to a more promising field. Everybody's supposed to go to college and learn stuff, so...

Academia

Let's see if the universities do any better. I was just on a local college's website, and it was even worse than Twitter -- Twitter's code knew it was screwing up and put up the fail whale. In this case, any number of links I hit encountered badly broken code:

Oh, alright. The colleges are perpetually underfunded, and putting up a website that works isn't a high priority compared to ... all the other things they spend money on. I guess.

Probably a real business does it better, right?

Profit-making Big Company

Even more so, an essential public service, like the cable company! Those guys have the money, the funding, the experience and the mandate to do it right. Let's pick the case where their motivation is the highest: collecting money.

Oops.

Just a few days ago, I was on my local cable provider's site trying to access my account. Here's what I got:

Not just once, but repeatedly, for hours!

But maybe it's just TW that's got problems -- surely all the other big companies do things great, with their huge staffs and policies and procedures and all, right?

Sadly, no. Here's just one personal example from Verizon:

Summary

There's no getting around it. Software is just bad. Everywhere. We can speculate about why this is the case, but let's agree on the facts: it's bad, and not getting better.

Interviewing Software People

The methods in widespread use for interviewing and selecting software engineers are appalling. It is only because they are so bad that ridiculous methods like those often used at Google in which applicants are given trick puzzles to solve can seem like an improvement. The sad thing is, asking candidates to solve mental puzzles is better than what's usually done, which is not much at all. Come on, people -- we can do better!

Typical Selection Practices

Managers need more programmers. They get a job requisition from finance, then go to HR to get candidates. Since HR knows nothing about the substance of the work that needs to get done, what ends up going into the job requirements are a bunch of motherhood-and-apple-pie blah-blah (self-starter, etc.) and a list of keywords of the technologies in which experience is required.

HR screens candidates based on whatever is in the resumes and may interview candidates, basically to see whether they mouth the expected platitudes when prompted by the HR people. Those who play the game are then passed to the programming department.

Typical Interview Practices

The candidate is typically scheduled for a round of interviews with programmers and managers. Since the managers rarely know much and have usually forgotten what little they used to know, they don't ask questions of substance; they basically find out if they like the candidate and if they believe the candidate will "fit in" and follow orders. The fellow programmers who interview remember their own interviews, in which questions of substance were few and far between, so they basically chat up the candidate and decide whether they like them. At the end of this "rigorous" process, if everyone agrees, the candidate is accepted.

What a joke! When you're hiring a musician, an audition (in which the musician performs) is standard practice. When you're hiring a writer, reading things previously written by the candidate is standard practice. So when you're hiring a writer of software programs, naturally you'd expect that reading programs previously written by the programmer would be standard practice -- but it's not!

Leading Edge Interview Practices

Instead, the leading edge at places like Google is to hit the candidate with trick questions. For example: "Suppose you were suddenly shrunk to the size of a nickel and found yourself at the bottom of a blender. The blender is going to start in a minute. What would you do?"

If I were the size of a nickel, most of my neurons would be gone, so I wouldn't be me anymore. But more seriously, how relevant is the kind of skill questions like this test to writing programs?

I could make an argument that this kind of thinking is relevant to a kind of programming that is important, but very rarely needed: algorithmic design. No one has ever (to my knowledge) measured it, but I would be surprised if algorithmic programming amounted to as much as 1% of all the code in a typical application. The vast majority of code that's written needs different kinds of skills: visualizing user interactions, understanding data structures and data flows, understanding and effectively using complex subsystems, and many other activities. These activities benefit little from the kind of skills and instincts required for solving trick puzzles.

Are there people who can do the puzzles and be great "regular" programmers? Of course. The problem is the reverse: there are many people who would be perfectly adequate programmers who are flummoxed and generally disconcerted by questions of this kind. I've been in groups of trick question masters. They're great for finding what's complicated in basically simple things and other arcane but fundamentally counter-productive skills. Other than that -- you can have 'em! Take 'em all -- please!

What can be Done?

There are lots of simple steps that can be taken to improve the outcomes of sourcing and selecting software people. Any step you take is likely to make things better. I hate to do it, but I have to admit that even trick questions are better than the "Hi, how ya doin'" method of interviewing. But surely we can do better.

Reading code. When hiring writers, we read their past works. Why are we so reluctant to do so for people who write code? I suspect it's because very few people would actually be able to read the code and make a reasonable judgment of the author -- for all too many typically mediocre programmers, that would amount to a tour de force way beyond their limits. That's OK -- find out who can read code with meaning and judgment, and they become your main filtering agent. Maybe, just maybe, you'll end up with .. people who write better code! What a concept!

Auditions. What's wrong with an audition? If someone is supposed to know database design really well, show them one of your current ER diagrams and ask for comments. Tell them a change that is proposed and ask how they'd make it. Pose a recent tough problem you had to solve (which you've already solved) and ask them to solve it.

Detailed archeology. A candidate programmer may not know much about your stuff, but she'd darn well be an expert on stuff she's coded in the past. Find a subject where your experience overlaps hers, and ask for a detailed rendition on what she did, why and how -- and what she learned and would do differently today.

Subject Matter Testing. Yes, testing. Like an audition, only more objective. If someone really is the expert php programmer they claim to be, they'll ace the test. No problem.

Conclusion

Software hiring is an embarassing mess. In a field that is over-the-top exacting with fairly objective pass/fail criteria (the program works or it crashes), the methods we use to ask new people to join us are random, ad hoc, and almost completely unrelated to finding out whether the candidate can actually perform as required. Asking trick questions can actually be an improvement, but that's not saying much. We can and should do better, and even a little better can make a huge improvement in the quality of the people who build our software.

 

Regulations: Goals or Directions?

The sheer bulk of our regulations is exploding. By any reasonable measure, our regulations are obese; our super-size body of regulations cost more to create, feed and implement -- and they aren't getting their intended job done!

When confronted with the huge bulk of our regulations, some people say we need more regulations, while others claim we need fewer. This is the wrong debate.

The real problem is the kind of regulations we have -- our regulations spell out how we're supposed to do things, when they should be telling us what we need to accomplish. 

What is the goal of regulations?

In most cases, regulations are created to assure things that you, I and most sensible people want.

• When corporate officials cook the books or otherwise hide what's really going on, we want them to stop and to be held responsible -- like, go to jail! That's SarBox.
• We want hospitals and doctors to be careful with our medical records, and not pass them out to anyone who asks for them. That's HIPAA.
• We want financial institutions to be careful with our records, and make sure our private account and transaction information are kept safe. That's PCI.
• We want our money to be safe when invested with investment people and the stock market; we don't want people stealing or pulling strings to make themselves richer and us poorer. That's the SEC.

These regulations and many more are sensible. I want them and you probably do too. I want the corporate bad guys to go to jail. I want the medical people to keep my records confidential. I want the banks to keep my finances to themselves. I want my banks to be sound and the financial reports generated by public corporations to not be phony.

How are all those Regulations working out?

Are the regulations and regulators doing their job? How about Bernie Madoff? All sorts of bad things helped create the financial melt-down we're suffering from -- how many of the top execs got in any kind of trouble over it, not to mention went to jail? There are massive losses of credit card data, some of which make the news and most of which don't, causing endless trouble to consumers with identity theft. Who's held accountable?

The one certain thing about regulations is that we pay the price for following them. What is optional is that they do their job. In fact, if you think about it, if your financial records aren't always stolen, it's not because the regulations are effective -- it's because most people are honest, and don't want to steal your financial records!

We're getting more and more regulations, and we're taking more time, trouble and money to create and follow them. But the resulting regulations are doing a poor job of stopping the bad things we want them to stop.

Means vs. Ends

The reason why our ever-growing number of regulations fail to protect us is simple. In the vast majority of cases, they spell out, often in great detail, how to accomplish the goal, instead of plainly and simply defining the goal. The regulators insist on giving us what amounts to detailed, turn-by-turn directions for driving from Lincoln Center in Manhattan to Ridgewood, NJ instead of simply stating that we should drive safely to Ridgewood, NJ, and leave the exact route to us.

I usually like to drive up West End Ave to 72nd Street, turn left, and go up the West Side Highway to the George Washington Bridge, and so on. I wouldn't be surprised if that's the route the regulators would insist that I take.

However, sometimes when there's traffic, I continue north on West End up to 96th St and get on the West Side Highway there. That's not how the regulators or the GPS would tell you to go, but it turns out to be the smart route to take in certain traffic conditions. Given that no one has regulated (at least yet) my route to Ridgewood, I am free to take the route I think best and adapt to changing conditions. I can learn and innovate, so long as I reach the goal safely. Makes sense. But that's not how regulations work!!

If my trip were regulated the way HIPPA, PCI and other things are, I would be slapped with a fine or worse for "violating the regulations" by failing to take the 72nd St entrance to the West Side Highway. Because the regulations give me exact, turn-by-turn directions I have to follow for reaching the goal, rather than simply letting me drive to the goal in the most sensible way.

Are Too Many Regulations a Problem?

No! But it's a BIG problem that we are buried under mountains of micro-managing, often-obsolete, turn-by-turn-directions-type regulations, which are by their very nature bulky. Which wouldn't be so bad if they actually got the job done.

If we had goal-oriented regulations, they would be concise, clear, and not need revision very often. They would be easy to understand and leave room for people to learn, adapt and be creative while still achieving the intended goal of the regulation.

We Need Regulations 

We need government to establish and enforce a common set of rules that make our lives better. In that sense, we need regulations. We benefit from having them and we benefit from having them be enforced. But not if they're costly, out-of-date, stifle innovation and on top of it all don't work!

If regulations were written to define the goals they are intended to accomplish, they would be inexpensive, always relevant, enable innovation and have a much better chance of being effective. We should all want regulations that tell us what to do, and leave it to us to determine how to do it. Don't tell us how -- tell us what.

The Name Game of "Moving to the Cloud"

The most recent in a long string of technology fashion trends, "the cloud" is hot. Like its hot technology fashion predecessors, it mostly consists of old ideas with a little spicy sauce on top and fresh packaging. If you mindlessly follow the fashion and just "go to the cloud," you are likely to end up in the same unhappy place where most mindless followers of fashion trends end up. What is "the cloud?" Simple. Clouds are belated enterprise IT implementations of the consumer internet.

What is the Cloud?
Is the "cloud" a new development? Well, it is a new name...

Ancient Clouds
I first encountered the cloud more than 40 years ago. Before that fateful meeting, my only experience with computers had been up close and personal. You had to get in the room, push buttons, flick switches, feed card decks or punched paper tape, listen to whirring sounds and watch blinking lights. Like with this computer:


But The Cloud! Ahhh, the Cloud...I remember it vividly.

Of course, things were a bit different then. We only had "fat clients," as in "takes two guys to lug it" fat. The principle was identical, however: I was in one place, with a "fat client" like this:


...and in another place was a computer like this:


...and everything worked.
Remember the importance of labelling, however: what we did 40 years ago wasn't "cloud computing," which hadn't been "invented" yet -- it was merely "telecommunications."

Creating the Modern Cloud
Between then and now, lots of iterations of Moore's Law have come and gone. All the hardware has gotten smaller, cheaper and faster, while the software has gotten larger, more expensive and slower -- but, fortunately for all of us, the rate of hardware evolution is greater than the rate of software devolution, giving the impression of net progress.

Where this leaves us, 40 years later, is faster remote computers talking with lighter remote clients over incredibly faster networks, all at lower cost. Sprinkle with a little extra software and drizzle with some marketing hoo-haw, and -- kazaam! -- you've got today's hottest technology fashion trend, Cloud Computing.

Clouds aren't always friendly
When we think of clouds, we're likely to think of this kind of cloud:


friendly, fluffy shapes floating in an otherwise sunny sky. When people think about cloud computing, this is the kind of cloud they seem to have in mind. But as we know, there are other kinds of clouds. There are dark, oppressive clouds that make everyone depressed. And there are really mean clouds, that wreck things horribly, creating the situations for which "disaster recovery plans" are made. Is this just a metaphor? Of course not. But just like financial fraud, the big, juicy examples are usually hushed up in order to protect the guilty.

OK, so What is "the Cloud"

There is a little-discussed trend that is deeply embarassing to IT professionals: there is a wide and growing gap between the use of computing technology in the consumer world and in the corporate, data center world. When the average user of corporate data systems is home, he works in a very advanced computing environment. His local machines and devices are amazingly capable and pretty easy to use. When connected to the internet, he can access a nearly limitless world of cloud computing resources -- which are themselves largely run out of data centers that are remote from the people who set up and administer the software in them, and which contain an ever-evolving mix of dedicated and shared resources and services. The consumer internet has been based on a cloud computing model for a long time.

The corporate world is a whole different thing. The corporate world has been consumed with consolidating their diverse data centers. They are finally beginning to confront the extreme flexibility and ease of use that consumers enjoy every day, and are finding it increasingly difficult to explain why the computing they run with such high capital and operating costs are so cumbersome, error-prone and inflexible.

In this context, there is no way that anyone associated with corporate computing is ever going to plainly admit that what they are basically doing is trying to catch up with the consumer internet. So they must be doing something else. Oh, yeah -- they're evolving to the latest, smartest trend in corporate computing, adopting the latest technologies and being really leading-edge: they're "moving to the cloud," but of course in a "smart" way, with large doses of "private cloud" technology along the way.

Summary
What's a "private cloud?" A corporate data center with a fancier name.

What's a corporation "moving to the cloud?" A corporate IT group trying to play catch-up with the consumer internet, and desperately trying to make it look like something else.

What's new about "cloud computing?" Very little; mostly naming and marketing fluff.

Is anything real happening when a corporation "moves to the cloud?" Sometimes yes! Sometimes, they really are copying a couple proven techniques of the consumer internet, slowly and at great cost and trouble, but nonetheless creeping towards a 21st century computing model.

Thanksgiving Meals and Software Turkeys

Software development is normally conducted the same, common-sense way that Thanksgiving feasts are created. Perhaps this is why software so often resembles a post-Thanksgiving mess.

Requirements

The "requirements" (i.e., the menu and number of guests) for Thanksgiving don't differ all that much from one year to the next, and after all, the menu is a variation on a well-practiced theme: a dinner. But you'd still better be exact in defining the menu: "some kind of vegetable," for example, won't do. Furthermore, the requirements are made by experts for highly experienced users, none of whom will need documentation or training to "use" the resulting product.

The requirements for a typical software project, by contrast, are not made by people who are experienced consumers of the product. They are usually made by people who are experienced at making requirements, which is roughly as effective as having menus designed by people who never eat.

Design and Construction

Once the menu has been planned, recipes are selected (not many people wing it and risk cooking a dish that neither they nor anyone else has ever cooked before), the shopping list is made, the shopping done and finally the dishes are cooked. You can see here a picture of some of the advanced cooking techniques used by experts in 1963.

Software developers try to follow roughly the same method of finding recipes (designs) and getting ingredients (lines of code). But they're always making dishes neither they nor anyone else has ever made, so the recipes they find need severe adaptation, and basically they have to make it up. The same goes for ingredients: they find people, try to get them to understand the made-up recipes, and then have them create ingredients (lines of code) that work together. They try to convince themselves and anyone who will listen that everything will turn out OK because strict project management techniques are being adhered to. Uh huh.

The Finished Product

The finished Thanksgiving meal is often a sight to behold. So are the people assembled to admire and to consume it, who typically have at it with skill, experience and vigor. While some of the younger participants may need talking to (see below: the wise guy with his feet up looks like he's about to get it from the lady on the far right), in the end things work out remarkably well. Everyone knows their job and does it.

Not everything goes perfectly when cooking the Thanksgiving meal, but most of it works out really well, and in the end all the requirements of the end users (that they end the meal being happily full) are satisfied.

In software, once the "meal" is cooked, there is usually an extensive testing, integration and quality process involving labs, staging areas and other things to which the supposed "cooked and ready" meal is subjected for fear that it simply won't be edible. In spite of all these measures, everyone knows disaster is not just possible but likely, and so before being brought to the table, the meal is served to special people who are used to eating half-cooked, never-been-cooked-before dishes. This is called an "alpha" release. It resembles getting some poor fool to eat bites of the meal intended for the king to assure that it wasn't poisoned; the trouble is, in the world of software, it usually has been, if only inadvertantly as a result of the usual chaos of building never-been-built-before software. In the world of software, there is usually no equivalent of the dinner-table picture above.

The Aftermath

In the world of Thanksgiving dinners, the aftermath is pretty typical. Here are typical remains of the kind of meal cooked by the ladies pictured above:

Looking at a mess like this is generally a happy thing, which is why my dad took the picture. You remember how good the meal was and chuckle about what was left.

In software, however, this picture resembles the meal that was actually served: a ripped-apart, cold, coagulated mess that you may be able to pick at. Hey, maybe we can make a turkey sandwich by bringing in some extra tried-and-true ingredients (bread, mayo)! The sad fact is, by the time most software is developed and delivered, the original cast of characters has given up, moved on or descended into open cynicism. Aided by the fact that the software doesn't work and/or the situation has changed so much that the software is no longer relevant, at least as it is.

Summary

Software development techniques, even today, have a remarkable parallel to making a Thanksgiving meal. But Thanksgiving meals have a track record of working out pretty well for all concerned, certainly in the are-you-full-afterwards department. And software development techniques have a track record of not working out so well, except for the turkeys who run the projects, who rarely seem to be fired for the messes they so consistently deliver -- after all, we learned alot from this, and things are going to be different next time! Sure!

If you like hearing gobble-de-gook babble about project management and late software that doesn't work, by all means continue to model your software development after Thanksgiving. But if you look forward to legitimately associating the concepts of "software" and "grateful" together, without sarcasm, then I suggest you leave the turkeys to Thanksgiving and try something else for software.

Developers, Designers and Project Managers at War

There is a natural conflict between the various groups that create computer products. This graphic captures it pretty well.

Credit:

http://alextoul.posterous.com/the-war-between-developers-designers-project

Status in Software: Silliness and Stupidity

In all too many software groups, you get higher status by being more removed from actual customers, their needs and concerns. This is bass-ackwards. It's silly. It's perverse. It is profoundly stupid and counter-productive. If this is how your software group works, change it or leave. Now.

The Inward Flow: Support

In most organizations, here is the perverse flow:

• Customer has problem. Contacts Customer Service.
• L1 customer service takes the call or e-mail. Eventually. They try to do something, but don't have much knowledge or power. So after wasting some time, it's off to...
• L2 customer service, which is backed up failing to handle other things L1 already kicked up to them. After wasting some of their own time, and often some of the customer's as well, it's off to...
• L3 customer service, which is the place where the really experienced L2 agents are promoted. Life is messy in L3. All the nasty problems end up there, often with the customer already being (understandably) mad, but too frequently lacking the skills and resources to even reproduce the problem, much less fix it. After spending some time here, the worst problems of the most upset customers migrate to...
• Sustaining engineering. This is the death-watch group in engineering. Two types of characters are typically confined here: ignorant entry-level people who hope to move up and out; and experienced engineers who missed the cut for working on the new stuff. If it's an easy bug, they may be able to fix it. Otherwise...
  
• ...it may actually be necessary to interrupt an exalted person who wrote the code that caused the problem, taking him away from the important business of writing code that has brand-new problems! But this drastic measure is avoided if at all possible.
  

There are actually more layers to march through, but the pattern should be pretty clear by now: the "most important" people are protected from the consequences of their past mistakes by layers and layers of carefully arranged bureaucracy designed to deflect and defuse any contact with real customers and the problems those customers may be having. The more you know, the more distant you are kept from having your august presence sullied by the trivial annoyances of mere customers. It doesn't need to be this way.

The Outward Flow: Development

When new products are created, it is all too often the case that the higher your status, the more removed you are from contact with the people who will ultimately use the product you create.

In very large organizations, the remote peak of the status hierarchy is occupied by research groups or labs. These are truly hilarious. Why do they have ultimate status? It is a given that they see no customers, hear no customers and talk with no customers; but even better. they produce nothing tangible at all -- unless you count academic papers and research reports. Those people are sure important! Their ground-breaking work will (pick your favorite) "lay the foundation for," "create the basis of," or "make the discovery on which" generations of future products will be built. Sure.

Smaller organizations would love to have such a group -- it's prestigious! -- but instead make do with a few exalted individuals who think deep thoughts and create "architectures" that "solve" a wide range of present and future problems.

High level design people then take over to create a "design" within the "architecture." This is not easy! It's important to fend off the constant pressure to produce something practical that works for today's customers, in favor of doing the design "the right way," i.e., spending lots of time thinking about problems some customers may have in some unspecified future, and "creating a framework" that will supposedly make them easy to solve.

At this point, software development splits into an alphabet soup of competing creeds, each of them certain of their unique virtue and access to software heaven. There is the much-maligned waterfall, agile, SCRUM, extreme, and on and on. The details of what happens next vary. The status relationships and ultimate outcomes are pretty much the same: the more important you are, the less likely you are to have meaningful contact with customers. This remains true as the software staggers through phases that may various include integration, testing, staging, documentation and roll-out.

Finally -- finally! -- the software is inflicted on the customers for whose benefit it was built. All I can say is that the chorus of complaints, however loud it may be, is rarely loud enough to penetrate the excellent sound insulation of the rooms in which the company's "brain trust" festers.

Conclusion

If you want to run a charity organization for egotistical, self-absorbed and self-important programmers (OMG! Did I just use the demeaning term "programmers," implying these people might actually lower themselves to doing actual, like, work!? I meant to use a more elevated term like "intergalactic systems architect" or "chief scientist.") -- like I was saying, if it's your goal to provide welfare to high-minded computer scientists, by all means employ a staff of "elite" techies and help them avoid being interrupted by the hoi polloi. Their deep pondering is way too valuable to be sullied in any way by the mundane concerns of the common people. If, on the other hand, you have real work to do and want your best people to lead, then make sure that the closer people are to customers the more status they have. Building a product or service that real people value and want to use requires -- gasp -- contact and interaction with those same real people.

 

Three Most Important Factors in Storage: Performance, Performance and Performance

We all know about the importance of location in real estate. What's the equivalent in storage? Performance. It's the one thing that you can't fix. When you look at storage, it should be what you look at first, second, and last.

Location and Performance

The three most important things in real estate are location, location and location. Real estate agents may talk about the attractive paint job, the great landscaping and the new roof. But if you didn't like them, you can fix them. The one thing you can't fix? The house's location.That's why it gets the top three slots.

What about storage? You'd never know it from storage vendors, but the three most important things in storage are performance, performance and performance. You can fix most everything else with server-based software. You need replication? Your database can do it all by itself. You think thin provisioning is great? It's cheaper and better to get it with a VM. But performance? You want that go-cart to do 100 mph ... uphill??? Fuhhgeddahbouddit, buddy. However fast you're going is as fast as you're going to go.

The Storage Performance Problem

We know there's a performance problem because of the fundamentals of spinning disks. We know there's a problem because vendors are coming out with expensive solutions that emphasize performance, and companies are going public based on storage performance; oddly enough, in the case of Fusion IO, they don't even deliver real storage, just a board that goes into a server! But people are so desperate for performance, they try it anyway. One company has even come out with an affordable solution that just screams performance.

The biggest thing that convinces me there's a problem comes from the leader in server consolidation and virtualization, VMware. I went through their best practices in configuring virtual storage, which tells you all you need to know. They have four best practices. All four best practices amount to the same thing: make sure you get enough performance from your storage! Their best practices are explicit: you should buy storage not based on capacity, but on performance.

Here they are:

• Configure and size storage resources for optimal I/O performance first, then for storage capacity. --> Don't buy TB, buy iops (i/o's per second).
• Aggregate application I/O requirements for the environment and size them accordingly. --> When you buy iops, make sure you look at all your applications.
• Base your storage choices on your I/O workload. --> In case you didn't get it yet, pick storage based on iops!
• Remember that pooling storage resources increases utilization and simplifies management, but can lead to contention. --> Remember that using a classic SAN can make storage performance worse, so don't be fooled.

According to VMware, there are four most important factors in storage: performance, performance, performance, and performance!

Does anything but performance matter?

Of course it does. Do you want to lose your data when a server fails? You'd better not buy server-based storage. Do you want your performance to drop to a crawl when there's a disk fault? You'd better ask how frequently that happens, how badly RAID re-builds impact performance, and for how long (hours of severely degraded performance taking place weekly is not unusual in a large system). But performance still takes the top 3 slots. It's just like location: if there are two equally-well-located houses, you avoid the shack with the outhouse and buy the comfortable, modern house. With storage, if you have two systems with enough performance to meet your current and future needs, you pick the one that isn't a board stuck in a server, and the one that has enough affordable capacity.

Conclusion

Performance is more important, by far, than any of those silly features the SAN vendors love to rattle on about. But in a post-SAN world, performance is front and center. The bigger disks get, the worse performance gets. The more you virtualize and consolidate your servers, the more performance you need. In a word, you need SSD's, because they're fast storage. But they're expensive. So an appropriate blend of SSD's and spinning disks would be great, fast but affordable, if they really were in a seamless pool of storage. That's the Xio Hybrid ISE in a nutshell. In a performance-starved world, it's food for the hungry -- food you can actually afford to buy.

Storage: The KISS Principle in a Post-SAN World

The dominant model in storage today is the SAN (Storage Area Network), a.k.a. "storage mainframe." While "SAN" makes you think of a storage version of a LAN (Local Area Network), it is far from it. In fact, SAN's are monolithic, mono-vendor, administratively heavy-weight, burdensome beasts. They are laden with "must-have" features that sound good, but which are mostly crippled versions of functions performed more effectively, at lower cost, by server software.

Most storage vendors make it clear what they mean by the KISS principle: "Keep it SAN Storage." Why? more revenues, more profits, more high-margin maintenance -- in general, more for the vendor and less for the buyer. It's time for buyers to revolt. It's time to enter a post-SAN world of simple effectiveness. It's time for storage to be fast, scalable and affordable. It's time for a return to the original meaning of KISS: "Keep it Simple, Stupid." In other words, it's time for the Xio ISE storage blade.

The Controller is the Problem

Storage buyers buy, well, ... storage. Duhh. If they didn't need storage, they wouldn't be talking with storage vendors. And it's true that storage vendors deliver storage. But what do they sell? Anything but storage. It sounds strange, but it's not -- since every storage vendor sells storage, how can you tell one storage vendor from another? Only by talking about something else. Today it is standard practice for storage vendors to emphasize the importance of features that are somehow related to storage, but aren't actually storage.

This brings us to the controller. Every traditional storage vendor has a monolithic controller. The controller is an expensive box that sits between the servers and the actual storage. The controller is where all these storage-related features are implemented. The game every storage vendor plays is to make you want what's in the controller, because whatever it is, only that vendor has it. The controller is what makes you buy one vendor's terrabytes rather than another vendor's. The controller is where vendor differentiation is. Last but not least -- the controller is where vendor profits are.

What about those "must-have" features in the Controller?

I would love it if someone de-constructed them all, publicly and effectively. But let me start from a simple observation. In my job, I get to closely follow the technology decisions and deployments of dozens of growing, leading-edge companies, and I get a quick look inside many more. What I find says volumes about the status of all those "value-adding" features of storage systems: nobody uses the fancy, "value-adding" features of storage systems -- they just use storage! As in plain old storage, like reading and writing.

The reason all these leading-edge people just use plain-old KISS storage, is pretty simple: they focus on bulding technology that supports their business. The value is delivered by applications that use files and databases. Files and databases need storage. Give them storage and you're done!

Just this morning I talked with some terrific folks who operate a leading edge internet advertising service. They already handle monster volumes out of multiple data centers. When orders are placed, the orders need to get out to all the ad servers. A perfect application for that popular feature of SAN controllers, volume mirroring, right? Wrong. There are at least a handful of reasons why this would be a terrible solution. But it doesn't matter, because they get the job done, effectively, quickly and well, with mysql's replication facility. Their application puts the ad opportunity into the master database, which replicates it to read-only slaves. Problem solved.

I see this pattern everywhere: the problems that SAN vendors use to sell their controllers are better solved, more simply and with less expense, with applications and server software.

Introducing the no-controller SAN: post-SAN Storage

What does that mean? If you remove the controller in a SAN, what are you left with?

With the old SAN vendors, you're left with a big, expensive pile of storage you can't use. In the post-SAN world, exemplified by the Xio ISE, you've got what amounts to storage blades. You can direct-connect a storage blade to a server or to a couple of servers, or you can network a set of blades together with a set of servers.

Each ISE storage blade comes with a full complement of storage capacity and performance. If you need a truly giant pool of storage, you can combine any number of them into a single volume using server-based software. But more likely you'll want to share them among a pool of servers, which can easily be automated using RESTful calls from a UI or script.

Then there's the issue of storage performance. Bigger disk capacities equals shrinking performance. That's why Fusion IO and similar companies are so hot. Note that Fusion IO doesn't have controllers or any of the fancy (= useless) features that come with them. People are snapping them up anyway. Maybe that post-SAN storage is worth looking into ... you could save a bunch of money, keep things simple and above all deliver the performance your business demands. If you've got the kind of problem Fusion IO says they solve (a storage performance problem), you should do yourself a favor and discover how Hyper-ISE delivers the performance you need at price you can afford. And Hyper ISE gives you performance while keeping your data safe and providing full fail-over, unlike Fusion IO, which isn't really "storage" at all -- it's just a board in a server, so when anything about that server goes wrong, your data goes wrong with it..

Conclusion

Most computer storage today is anything but simple, scalable or affordable. Intelligent storage buyers are increasingly buying the storage that they need and only the storage they need. They are saving money, time and trouble by refusing to buy expensive controllers that are laden with features they already have in the server, features they just don't need. These buyers have effective, simple, scalable storage that costs less, performs better and lasts longer than old-style SAN's. Welcome to the post-SAN world, where the sun shines, things are simple and life is so good, it makes you want to KISS someone in a new way.

Better Software Lets You Compete Against Giants and Win

There are lots of personal and emotional characteristics that help winners build terrific new companies. They're important. But when it's a software company, guess what? The software really matters!

There are a number of elements that contribute to success, including concentrating your forces, targeting poorly defended or new territory, and rapid iterations in response to new knowledge and changing conditions. However, there's nothing like superior weapons to help your cause.

Your competitors are typically GIANTS compared to you, so giant that they darken the sky...

Darkening the sky is bad enough, but what's really scary is when they notice you, pay attention to you, and get in your face...

Now you have one choice and one choice only -- pull out your weapons. If your weapons (software) are truly, fundamentally superior (and you don't screw up the other things too badly), then you've got a good chance of vanquishing a larger, established, well-equipped foe...

Which feels awfully good and definitely deserves raising your arms in victory.

Conclusion

If giant competitors cover the sky with dirigibles, you had better be dive-bombing them with powered, fixed-wing airplanes. When invading a place that is well-defended by armies with swords and war horses, you'd better have guns and armor. When the giants in your industry dominate with a kind of software, your software had better have the same advantage that guns have over swords and airplanes have over balloons...and that smart little girls have over giant adults.

 

"Top Nerd:" Nerd Values and Attitudes

Though the subject of unapologetic humor, nerd values and attitudes are wonderful. Society would be better off with more nerds.

When other groups have gotten status and prestige, or just respect, in the past, they have had to either (1) exercise aggressive dominance, or (2) do the victim thing. Nerds are getting respect like never before by just coming out, valuing who they are and what they do, making a bit of fun of themselves, and by the simple fact that people actually know stuff and can get stuff done with passion and complete dedication are really valuable, admirable people who tend to enjoy what they do! 

I found a delightful blog post by Liz Andrade, a self-described nerd, who describes some nerd values that also illustrates why nerds are so valuable.

Here are a couple excerpts:

Nerds are Inspiring!

Part of being a nerd has to do with having some strong opinions on whatever it is you’re nerdy for — be it Star Wars, video games or typography — nerds pride themselves on knowing a lot about what they are into and your opinions on the matter are part of your identity.

...

This past year I shopped for glasses ... and the experience I had ... made something that in the past was nothing more than a necessary task into a remarkable experience! How? The people at these locations were total eye wear nerds!

This is a sect of nerd I was not even aware existed ... They were able to suggest ideas based on my face shape and style, they knew about eye wear designers, frame shapes, materials, vintage styles and their enthusiasm for the subject was infectious!

When you are passionate about what you do, you inspire the people around you – and who doesn’t want to work with someone inspiring!?

Nerds are Authentic

Part of being nerdy is accepting yourself for who you are and what you are into even if isn’t what fits into the status quo or flow into the mainstream. Those who are able to embrace their nerdisms and not be ashamed of them have this obvious badge of honesty.

Whether it is real or imagined, if someone can be totally open and honest about their Red Dwarf obsession, you feel they are probably transparent about other things in their life, like business practices and ethics.

Nerds are Memorable

Nerds are usually stand out from the crowd… and being unique makes you easier to remember, as simple as that. It is each of our unique experiences and abilities that make us valuable individuals, blending in has become a liability to any business trying to be remarkable!

 

Of course the work of nerdy, remarkable people like Temple Grandin has gotten some attention and helped the cause as well. Temple has had to overcome some obnoxious and inexcusable barriers ... to make the world a better place! Not to mention, to get her job done! More of us should be more like Temple....

 

Software Quality: Theory and Reality

The theory of software quality makes my head hurt; the reality makes me want to cry.

There is a great deal of material written about software quality. It's a HUGE subject. It's also a diverse subject with lots of experts and lots to study. There is one simple reason for this: Software quality is a horrible %^*%^* mess, and it's not getting better!!!

Software Quality Theory Makes My Head Hurt

Just scan through the Wikipedia article on the subject and your head will probably hurt too.

I particularly like this alert at the top of the Software Quality Factors section:

This section needs attention from an expert on the subject. See the talk page for details. WikiProject Software or the Software Portal may be able to help recruit an expert. (September 2008)

Note that they've been seeking this expert for nearly three years!

Big government agencies have whole organizations devoted to the subject. For example, there's DACS, the Department of Defense (DoD) Information Analysis Center (IAC). What does DACS do? Read this (warning: reading this may make your head hurt):

Designated as the DoD Software InformationClearinghouse, specifically aimed to serve as an authoritative source for state-of-the-art software information providing technical support for the software community, the DACS offers a wide variety of technical services and supports the development, testing, validation, and transitioning of software engineering technology to the defense community, industry, and academia. DACS subject areas encompass the entire software life cycle and include software engineering methods, practices, tools, standards, and acquisition management. Also included are programming environments and language techniques, software failures, test methodologies, software quality metrics and measurements, software reliability, software safety, cost estimation and modeling, standards and guides for software development and maintenance, and software technology for research, development, and training.

I could go on and on, but my head hurts, so I'll stop.

Software Quality Reality Makes Me Want to Cry

With all these impressive-sounding things, books, conferences, experts, criteria, methods and certifications, software quality should be totally nailed, right? To the contrary: something is nailed when ... people stop talking about it! Take the disease smallpox, for example. It's nailed! There aren't theories, experts, or much of anything beyond historical references and scare-talk about potential re-emergence.

This is one the better summaries of the reality of software quality that I've seen; ironically, it's from a zombie website for obsolete software written for a long-obsolete machine, that is/had been(?) run by a couple people from some little island in the Carribean.

 

"Top Nerd" Activities: Work Hard, Save the Day and Have Fun

There are nerds. There are super-nerds. And then there are ... Top Nerds. What do Top Nerds do? Simple:

• Top Nerds work hard. Really hard. Why? They like hard work.
• Top Nerds save the day. One of my Top Nerds buddies is doing it as I write this. No "software project" of any kind, however well-run, could possibly pull off what he's pulling off.
• Top Nerds have fun. They're exploring, pioneering, accomplishing, learning. This is the best kind of fun!

I just hosted a gathering of Top Nerds. Having them all together for a long weekend was fun for everyone -- who else spends the first more-than-half of the Fourth of July in a conference room going deep into the details of machine learning techniques and applications, solves leading-edge problems in their practical application and discovers new, transformative uses for them, ... and has a blast? Well, we sure did!

Naturally, Top Nerds scramble and solve obstacles as they arise. One of our number was too busy saving the day for his company to be able to commit a holiday weekend to being in New York City with the rest of us. But he skyped in (that's him on the screen in the picture), told us what he was doing, how and why, and in the interaction that ensued, everyone ended up ahead of the game, and thoroughly entertained as a side-effect!

Because that's what Top Nerds do!

The new "Top Gun" is "Top Nerd"

"Top Gun" is so last-century. Now nerds are on top of the heap, and being "Top Nerd" is best of all.

When the hottest, coolest thing around is fighter planes...

... it makes sense that the coolest dude around would be the best fighter pilot. This is what the 1986 movie "Top Gun" was all about.

Top Gun is macho alpha male behavior to the max. It's competitive guys who look at other capable guys as something in the spectrum of rival to enemy. Everyone else is just stuff to be vanquished in primal combat.

Fast-forward to 2011. When I walk around the streets, everyone who isn't about to be retired (and an amazing number of ones who are) is either plugged in, communicating via their portable device and/or (increasingly on buses and trains) absorbed in their e-readers. Advertising is rapidly shifting to digital media, and similar digital transformations are taking place in other domains. Are fighter planes at the heart of this world-wide, all-pervading transformation? Hardly. It's computers and the software who make them do what we want (mostly). And who's the fighter pilot for the computers? It's the people who write the software; in other words, it's nerds!

This is a really good changing of the guard. With rare exceptions, nerds are much nicer people than "Top Gun" types. Nerds are much more interested in learning and accomplishing things than non-nerds. Cooperation and collaboration are characteristics that are well within nerd-normal behavior. This is illustrated by the fact that when you've got a "Top Gun," you've often got a bunch of bitter, defeated rivals, while "Top Nerd" is normally designated by acclamation by hard-working, admiring fellow nerds.

In my opinion, this is a good thing. Good for nerds, and good for the world.

 

Software Quality Horror Tales: Electronic Diversity Visas

The State Department of the US has inflicted unimaginable pain and suffering on tens of thousands of people throughout the world through their electronic Diversity Immigrant Visa program. It's a highly visible and public example of what's wrong with software development in general, and software quality in particular. Sadly, it's no different in principle from countless numbers of other projects, doomed from inception by inappropriate standards and techniques.

The Facts

The Diversity Lottery Program is just that -- a lottery. More than ten million non-US-citizens worldwide apply for tens of thousands of slots that can lead to US citizenship. Only this time, after notifying the "winners," who started spending money most of them didn't have to comply with the requirements to complete the process, the State Department cancelled the lottery and invalidated the results. Why? A bug in the computer program that chose the winners.

The Human Consequences

From the Wall Street Journal:

Ever since he traveled from his home near Yaounde, Cameroon, on a scholarship to Michigan State University in 2009, Dieudonné Kuate dreamed of immigrating to the United States.

As a visiting graduate student in epidemiology, he marveled at the sophistication of the chemistry labs and the excellence of the teaching. There was no comparison to his university in Yaounde, where he shared a cramped 27-square-foot room with three other students.

One of eight children, Mr. Kuate grew up on a poor farm in the western plateau town of Banjoun. His parents couldn't read or write. Mr. Kuate is the only child in his family to complete university. "My dreams have been to be a top researcher in my field of specialty. The only place I see these goals being realized is the United States," says the 31-year-old Mr. Kuate, who returned to Yaounde last year and finished his Ph.D. in chemistry.

For the past six years, Mr. Kuate has applied for the State Department's annual green-card lottery, and, like 15 million other people, he applied again this year. The 20-year-old program offers about 50,000 people a year a chance to win permanent residence in the U.S.—and a ticket to the American Dream.

Denied six times, Mr. Kuate finally saw his number come up on May 1.

"There is no English word to express my happiness when I discovered that I was selected," said Mr. Kuate, whose first name means "God given."

Emmanuel Tumanjong for The Wall Street Journal

Within days, his older brother sold family land in Banjoun for around $4,400 for Mr. Kuate to use for application fees, medical examinations and to start a new life in the U.S., he said. His mother believed God had intervened: "According to her, I was going to travel to the white man's country and see how to help other family members who have not gone far in book work," he said.

But on May 13, those hopes were abruptly dashed. After logging on to the State Department website, Mr. Kuate said, "I saw a message saying the lottery had been canceled."

Mr. Kuate was among 22,000 people around the world mistakenly informed last month that they had won the lottery. There had been a technical glitch and the lottery would have to be held again, the State Department said, explaining that a computer had selected 90% of the winners from the first two days of the application window instead of the full 30-day registration period.

The software

It's pretty clear that this is one of the more trivial programming jobs on the planet. I shudder to think how much it cost to build, how long it took, and the whole environment that was created that made it (I'm sorry to say) likely that a horrific bug like this would be inflicted on so many innocent people.

Since I have no access to the code or project documents, I will comment on a couple of things that are publicly available.

Take a look at the Department's page that announces the status of the 2012 lottery. Play around with it a bit, as I did.

Did you want to find more information? Did you take advantage of the kind offer to provide more information:

More information is available on our website:

http://dvlottery.state.gov

Perhaps, then, you noticed that the link leads you to the same page you're already reading!! Kafka couldn't have done it better. No doubt this was the careful work of the Division of Self-referential self-reference of the Department of Redundancy Department.

Did you take note of the fact that all entries were submitted electronically between October 5, 2010 and November 3, 2010? Which implies that starting on November 4, 2010, they had all their input data? All they had to do was run the lottery program a couple times on the input, run some checks to make sure it was working properly on the new data set, then run it "for real" and publish the results. To be generous, this should have taken about a day. OK, it's the government, we'll give them a week. Really? Geeez...alright, a month! No. NO WAY. $%&$%^& SIX MONTHS!!!!??? ^&(^^&* MORE THAN SIX MONTHS???!!!

With that much time, this should have been the most proven-to-be-perfect program in history. PhD students should have been able to break new ground in proving the certainty of correctness of this program. It should have been possible to run it a number of times that compares favorably to the number of grains of sand on all the beaches on planet earth.

I love the fact that there's a transcript of a statement on the subject by "David Donahue, the Deputy Assistant Secretary of State for Visa Services." The statement location and date are unspecified. The date of posting is not given. The fact that he made a statement verbally rather than just talking with the public via the web site kind of implies that he's incapable of writing or typing. His "internet department" (or whatever) must be responsible for the web site. And it implies that he still has a job! For some reason, I find that one really annoying! I guess you can screw over incredible numbers of people on behalf of the US Government and suffer no personal consequences. It must be OK to do that!

There's a lot more to be said about this fiasco, but I'm tired.

Conclusion

Software quality. We need a revolution! Stop the Horror! End the Terror!

 

Software Quality: Horror, Failure, Tragedy and Ineptitude

Most software stinks. Too much software is a real horror show. Loads of software is discovered to be rotten to the core before it is widely inflicted on customers, leading to missed budgets, deadlines and crippled businesses. An amazing amount of software, already bad and late, is still inflicted on the world, sometimes creating true tragedies. The only things in software that are more riddled with stupidity and incompetence than designing and building it are the methods used for assuring its quality.

We should be running around in panic, trying to strengthen the levees against the rising flood of software problems; instead, we have smugly promoted standards, careers and processes. In the face of repeated, costly failures, the standard prescription is to do more of the same, except spend more money and take more time.

"Software quality" should be added to the cynical pantheon of "southern efficiency" and "northern charm," words that we know just don't belong together.

I have spent more than 40 years in the software industry, and have made my share of mistakes. I'm no more perfect than anyone else. But through my personal experience of writing a great deal of code over decades, and now from my vantage point of getting an inside look at the work of many software groups, some unmistakeable patterns have emerged.

The key observation is that software development, as it is taught and practiced nearly everywhere, is not a collection of best practices that are constantly being refined and improved over time. It is a disorganized mish-mash of ideas badly adapted from other fields whose practices are simply inapplicable to software, and then patched and elaborated to make them sound good. This observation applies even more strongly to software quality.

I accuse no one of bad intentions. It does, however, make me angry and sad that our field is so burdened with pre-scientific, faith-based dogma that doesn't appear to be getting better. We don't need evolution here. We need revolution.

I hope to illustrate these claims in a series of posts. They deserve a cover like this one:

Are better methods available? Yes. they. are.

Software: Move Quickly While Breaking Nothing

In places that want to succeed (why would you be anywhere else?), the pressure is on the software team. Change more stuff. Move more quickly. Where are the results?

Ok, great you say. I'm already working as hard as I can. The only other thing I can do is relax the process, the safeguards that assure good, high quality results. I'll give it a try.

Maybe you (and your company) get lucky for a bit. Then the inevitable happens. The release that just got pushed had bugs and/or unintended consequences. There are panicked calls, late nights, frayed nerves and angry managers. The managers are extra angry at the programmers who did what the managers said, not what they thought they implied (i.e., don't break anything). The software people fume about management that acts like the ancient regime, making insufferable, impossible demands of the oppressed workers. No one is happy.

"Push More Releases" is NOT the Answer (by itself)

Sometimes when I talk with programmers, I'll give some version of my "releases are stupid" talk or my "dates are evil" talk and some brave soul comes out with the classic "I tried that and got screwed" response, as summarized at the beginning of this post. And I always agree with the brave soul: if all you do is push more releases, you're not just inviting disaster, you're sending a chauffeured limo for disaster and welcoming it with a red carpet and cheering crowds.

The classic, project-management-centric methodology with its infrequent releases is broken. Broken!! So who in his right mind would think that keeping the methodology except for some essential, integral parts is a good idea?

Let me see if I've got this right: you don't like your methodology, so ... you're going to cripple it and hope things will improve??!!

By the way, if you're sitting there with a smug expression thinking "we don't have that problem, we use agile," think again. Agile does not change the situation.

The Answer is a New Methodology, Optimized for Speed

The details of the new methodology are important; I've written about them at great length in my private-distribution papers. It's a seismic shift from the normal way of doing things. Like with any big change, it's useful to start with baby steps. So here's a start:

Grow the Baby

The mainstream process of software development resembles a car factory. There are lots of inputs. The inputs are assembled into sub-assemblies and then assemblies. There is a final integration phase, after which the car rolls off the assembly line. When it rolls off, people hope it works; there is no expectation that it will work prior to the end of assembly; until then, it's "work-in-process." This is typically the way things work whether you release once a month or once a week.

The speed-oriented method resembles growing a baby. There is one and only one overriding rule: Don't Kill the Baby! The baby is grown by tiny incremental additions; each addition takes awhile to get right, but none of them is fatal. It takes a while for it to learn to walk, for example, and spends some time walking poorly. But while this is going on, the baby doesn't lose its ability to crawl, eat, burp, roll over.

Don't Kill the Baby!

When the subject is babies, there is near-universal agreement that killing them is something to be avoided. But when software is developed with the usual methods, it's alive only some of the time -- mostly it's dead! The cornerstones of the speed-oriented method are:

• Small, frequent changes. You should make progress towards your goal every day. Do the best you can and make the most progress you can every day.
• The new stuff doesn't need to work at the beginning.
  
• The old stuff can't be allowed to break. This is usually achieved by some kind of continuous integration and live parallel testing. How you do it isn't important. That you do it is. Your software must not break. Clear?
• Iterate. Don't lay out months' worth of work. Set overall goals, then spend some time looking at what you did yesterday to help decide what to do today.

Baby Yourself!

One of the Oak companies was having trouble meeting all their commitments. The pressure was on to deliver more, and deliver it quickly. But the company was also getting beat up because of serious flaws in recent releases. Talk about being caught between a rock and a hard place!

It took quite a bit of work, but they shifted to the kind of speed-oriented, always-alive method described here. They had some fun with it along the way. Part of the fun was the mascot they adopted, which I got to see during a recent visit.

Grow the baby!

Why Computer Software is So Bad

There are aspects of the theory and practice of computer software that drive me nuts.

I look at a widely accepted theory or practice, and am aghast that the cancer spread and became part of the mainstream. I look at a genuinely good practice and am shocked at the way everyone lies and slithers to make it sound to the unsuspecting that whatever lousy thing they do is a shining example of a good thing. I look at a nearly-universal approach to problem solving that may have made sense many Moore's-Law generations ago, but has long since been rendered irrelevant by advances in hardware. And I look at proven, understandable techniques that improve productivity by whole-number factors that are spurned and/or ignored by most people in the field.

How significant is this stuff I'm complaining about? A minimum of a factor of 10. Sometimes a great deal more. So it matters! This is theoretical, but it has real-world, practical implications. It's the only way, for example, that small companies can beat large, established ones that have software staffs 10 to 1,000 times larger.

I think I understand some of the causes of this deplorable state of affairs. But that doesn't make me feel better. And it definitely doesn't cure the sick or empower the unproductive.

A number of my private-distribution papers go into these subjects in considerable depth. But I thought it might be interesting to summarize a couple of the main observations here.

Among the causes of bad software are:

The blind and deaf leading the blind. In the majority of cases, the people in charge have little to no personal experience of creating software, and no interest in how it's done. This is about as sensible as having someone in charge of a baseball team who not only can't play baseball, but can't see onto the field where it's being played; all he can do is see the score at the end and hear reports from the players about how things are going. So everything turns into politics -- convincing the blind, clueless boss that you're the great contributor, everyone else is a chump. The larger the organization, the more likely it is to be led by such a genius.

Process over substance. The larger and older the organization, the more likely it is to elevate the importance of process, and the more elaborate and all-consuming that process is likely to be. The more process there is, the less code gets written, and the productivity of the innocent few who actually want to work gets ground down to the abysmal norm.

Lawyers. Shoot the lawyers! Shoot them! They and their legal ways are a plague. When lawyers see a problem, they want to write a law or create a regulation that makes the problem go away. Except that instead of saying this in simple, results-oriented terms (e.g. "programmers should not be allowed to die unnecessarily while writing code"), they say it in terms of incredibly elaborate, micro-managing, this-is-how-thou-shalt-do-it regulations (e.g., "programmers should be offered a nutrional meal no later than 12 noon local time each day consisting of no less than..." and 538 similar instructions, constantly growing). If regulations of this kind actually led to good results it would be one thing. All I should have to say at this point is "credit card theft" and PCI.

Fashion over function. Programmers are supposed to be nerds, aren't they? How can programmers let their decisions be made by "fashion," whatever that's supposed to be in the world of software? I refer you to the first point, about software teams being led by people who are clueless. These people want to seem smart in the eyes of their bosses and peers, who know even less than they do. So they make decisions that are guided by C-office fashion trends, which are usually laughably out of step with true optimal decisions.

A preference for bad new ideas over good older ones. How do bad ideas catch on? Some company promotes them. Books are written. Appealing rhetoric is created and repeated. Suddenly a fashion trend is born. Someone can appear to be smart and modern just by advocating the cool new stuff and sounding smart, without actually knowing anything or doing anything. Everyone involved thinks they're helping advance the state of software, when in fact they're digging the hole of despair deeper. It's remarkably like when the dumbest guy in Scotland moves to England, thereby increasing the average intelligence of both places.

A preference for bad old ideas over good newer ones. There aren't many good new ideas; mostly there are old good ideas that are still good. But because conditions change (like processors getting fast and memory getting cheaper), ideas emerge to respond intelligently to the new state of affairs (like this one), instead of blindly and stupidly continuing on as though nothing had changed. Like most people do.

Wrong scope; usually too narrow. This is classic optimization over too narrow a range. Much bad software results from compartmentalism or simply failing to look at things through the eyes of the ultimate consumer. This is an incredibly common mistake. The trouble gets really bad when is the practices that result from the little "silos of excellence" get elevated into industry "best practices." (Whenever I hear the phrase "best practice," it's really hard for me to avoid getting a serious look and pronouncing in deep tones "you folks had best practice a bit longer before you inflict yourselves on the world of paid, professional programming.")

I'm sure there are many more causes of bad software than the ones I've listed here -- we haven't even gotten into bad programmers in their innumerable incarnations! But every item on the list above thoroughly deserves inclusion, even more because most of them are not listed by most people as a cause of software malaise.

Fusion IO and Xiotech Hybrid ISE

Congratulations to Fusion IO for pulling off the most visible, most large-scale event in the storage industry in recent months: their highly successful IPO. And congratulations to Xiotech for pulling off the storage industry's most important event in recent months: the GA release of Hybrid ISE storage blades.

Fusion IO

Fusion IO went public and traded up in an over-subscribed offering. The excitement about the company and its prospects are justified: the storage industry has been basically ignoring its large and growing performance problem for years now. The industry has toyed with a variety of ineffective strategies involving the obvious alternative technology, SSD, but done nothing to move the performance needle.

The brilliance of Fusion IO is to come out with a new category of product; they call it "memory storage." It's a board that you typically put into a server.

 

This is the brilliant part! The people who run applications have a huge performance problem, and the storage "experts" refuse to solve it for them, so Fusion IO gives them something in "their world" -- a server board -- that they can use, along with application changes, to solve their problem. It's a classic strategy: sell to the guy who actually has the problem.

Here's the other thing I like about Fusion IO: their success clearly and unambiguously demonstrates that applications are experiencing storage performance problems, and that these problems are house-is-burning serious.

Xiotech Hybrid ISE Storage Blade

Xiotech's release for GA (general availability) of the Hybrid ISE last week is the most important recent event in the storage industry.

The success of Fusion IO clearly demonstrates that increasing numbers of applications are experiencing house-is-burning performance issues. Most of the traditional SAN storage vendors have responded by simply putting SSD drives into their existing products. In all cases, the result has been modest upticks in performance coupled with drastic reductions in capacity and dramatic increases in cost. Not a good combination, and not popular with customers.

The Fusion IO approach works well for the small number of companies who have just a couple of hugely important applications that are completely under their control, and who therefore don't mind violating the normal principles of storage management and re-writing their applications to take advantage of server-based storage. In fact, I've just described exactly the situations of FaceBook and Apple, who between them account for more than two thirds of Fusion IO's recent revenues!

What about the vast majority of companies who have pressing performance problems and don't want to or simply can't break the bank and madly rewrite their applications? What they need is a new kind of storage they can just plug in and make their problems disappear: storage that is affordable, reasonably priced, many times faster than what they have. In other words, storage that is, well, storage, that looks and acts like normal storage in every way ... except that it is hugely faster, like five to ten times faster.

Hybrid ISE, that's your cue...

The Xiotech Hybrid ISE. The right combination of HDD, SSD and RAM cache in a 3U rack-mount package to provide the excellent performance and capacity your applications need. Just by plugging it in. Check it out!

Business Benefits vs. Speeds and Feeds

As a programmer building computer-based products, I have been subjected to decades of relentless, cruel beating by oh-so-superior marketing types on the question of how to describe a product or service. Do you use technical terms and talk about how much and how fast? (That's mere speeds and feeds. Said with a sneer.) Or do you talk about "business benefits," meaning I suppose lower costs or superior customer service or something else suitably vague and unmeasureable? (Obviously the "right" answer.)

After decades of watching good material being subjected to "marketing polish," which means all the substance is removed, and seeing the ultimate outcome (typically unsatisfying), I finally know the answer. The right answer truly depends on what you're selling and to whom.

Option 1: You are a big, tired company selling essentially the same old stuff, maybe with the latest buzzword pasted onto it. You're trying to sell it to a bunch of people marking time, who mostly want to avoid getting fired.

If you're marketing for the seller, your job is to avoid technical detail and any specific measurements of performance like the plague. You'll just confuse and annoy the buyer, who only wants to be assured that everything is going to be just fine. "Business benefits" or anything else that makes you sound safe and conducive to good naps is the way to go. This case describes the vast majority of seller/buyer combinations, and so of course represents mainstream marketing.

Option 1a: You are a big, tired company selling essentially the same old stuff, maybe with the latest buzzword pasted onto it. You're trying to sell it to a bunch of people who have the kind of problem you solve, people who know stuff and who really care about getting things right.

If you're a normal marketer, you will not like these buyers. They are obnoxious and pushy and care about all sorts of nerdy things. It's pretty clear they're not going anywhere. Talking with them is a sure sign that we're selling "too low" in the organization. We've got to get to the people who care about "business issues," i.e., the people who are ignorant about what they're buying and whether it's any good.

Option 2: You're a hot new company with a brash, ground-breaking product that's going to change the industry. You're trying to sell it to the same firing-avoidance people as in option 1.

You are wasting your time. Talking about "business benefits" isn't going to help. These people just don't care. Your passion is the final nail in the coffin: you have "I'm risky" tatooed on your face. Just leave. Now.

Option 2a. You're a hot new company with a brash, ground-breaking product that's going to change the industry. You're trying to sell it to a bunch of people who have the kind of problem you solve, people who know stuff and who really care about getting things right.

These are people with a problem. They know because they've measured it. They know how fast, how much, how often they're getting now, and they have a real good idea of how fast etc. a product has got to be to make the problem go away. Can you deliver or not? If not (or if you insist on bringing the conversation back to "margin" or some other stupid supposedly "business" virtue), please just leave. If you say you can, give me the numbers, please. The benchmarks. The speeds and, yes, the feeds. Numerically, if you please! If the numbers are good, how soon can I have it?!

Conclusion

If you've got a hot product, don't waste your time talking with anyone who doesn't have a hot problem. When you do, lead with the facts. If they've got the problem and you've got the solution, the air starts smelling like "deal."

Postscript: Cars and Trucks

Now that I've figured it out, I'm realizing just how out of it these clueless marketing types really are. It's all about whether you're selling to buyers who actually do things, know stuff, and have to suffer the consequences of their own choices. Think about ... pickup trucks, for example. Like this one:

Who's more about mainsteam marketing than GM? How do they market this truck, which will largely be bought by people who actually need to drive a truck, and want it to pull and haul stuff, and will be in big trouble if it can't? Well, go and look at GM's own marketing for this vehicle. There you will learn all sorts of things like this:

More maximum hp and torque - 397 horses and 765 lb.-ft. of torque. (Our standard workhorse, the Vortec 6.0L V8 gas engine, generates 360 HP and 380 lb.-ft. of torque.)

Duramax 6.6L Turbo-Diesel

The reliable available Duramax Diesel has improved torque, horsepower and towing numbers, as well as highway fuel economy that was improved by more than 11% over the previous generation.

With a 36-gallon tank (approx.) the Duramax offers up to 680 highway miles on a single tank 

 To better manage larger capacities, the steering knuckle is taller and 66% heavier than before

 The rear suspension includes 20% wider leaf springs with an asymmetric design that reduces wheel hop on acceleration and better handles increased payloads

Speeds and Feeds! And of course, nice pictures and some "business benefits" like how comfortable you'll be sitting in the driver's seat.

 

How Great Software Teams Can Go Wrong

Every once in a while, a miracle happens: a group of smart, hard-working software people latch onto a group of customers with unmet needs, and wonderful things take place.

Once things get up a head of steam and it becomes obvious that good things are happening, the vultures start circling. "That's an inappropriate metaphor!" you might exclaim. "Vultures feed on dead animals, not ones that are alive and well." OK, I concede the point. "Vampires" would be a better metaphor, because everyone knows that vampires attack the young and healthy and turn them into unfeeling parasites like themselves. Let's compromise: when there's great software happening, the vampires start circling.

The vampire/vultures tend to be older than the programmers actually doing the work. Regardless of the facts, they like to complain about things like the lack of order, the lack of predictability, and the lack of control. They ALWAYS find lots of evidence that these supposedly indispensible things are missing. Things are moving quickly and the business is growing quickly, so of course the vampire/vultures find all sorts of evidence that the business, as currently "organized," isn't "scalable." Things may be going OK now, but we have just been lucky! There's too much at stake to leave our future to chance. We need some "experienced, senior" managers to assure the future of our important business.

What happens next is simply awful. I've seen it too many times. I know all the script variations. But let's skip past the mayhem to the final stage, which can reasonably be called "the inmates are in charge."

The Inmates are in Charge

There are sure signs that the inmates are in charge. When this happens, you may as well put this sign over the entry door of your office: "Abandon All Hope Ye Who Enter Here."

Following is a sample list of the reasonable-sounding things that people say in this horrible place, each followed by the reality.

“After you work a certain number of hours, your efficiency simply goes down. You can’t concentrate as well."

Everyone slips out starting at 4 p.m. Which would be OK if they hadn't wandered in around 10am.

“This pushing to make unreasonable deadlines just exhausts everyone. They fight with each other, don’t get any better results, and then, after, everyone is so beat that nothing gets done for weeks.”

Deadlines get established so far in the future that earthquakes, floods and fires can take place, and they can still be met. The meaning of the word “deadline” evolves from “I’ll make it or die trying” to “a dead person could meet the target.”

“People who do nothing but work are unpleasant, narrow, one-dimensional people. They don’t work well in teams, and before long they just crack up and become unproductive anyway.”

HR people start roaming the halls at 4 p.m. suggesting that people go home and spend more time with their families. Anyone who seems stressed is invited to take a week’s vacation.

“Our products are sophisticated, complex pieces of technology. It takes a long time to understand everything. It’s hard to find enough experienced people to do the work.”

The “go along to get along” people are promoted and salaries of senior people advance rapidly.

“There is a huge amount of value in the intellectual property we have built up here over the years and embodied in our people and processes.”

Suggestions for change are rejected without serious consideration.

“Rodney isn’t really a team player. He makes other people uncomfortable, and sometimes says things that make other people feel bad. He’s bad for morale.”

Rodney is smart, hard-working and wants to build a great product. But he doesn’t fit in well, and is asked to look elsewhere for employment.

“Our customers have come to depend on our reliability and commitment to quality. We have to continue to pay attention to what we do, and continue to do it extremely well.”

Don’t bother suggesting doing more, working in parallel, or getting things out more quickly.

“Our customers don’t have the money to chase after every cool-sounding new idea that comes along, most of which fail anyway. They depend on us to incorporate new  technology for them once it actually becomes useful and proven.”

We will continue to defend our approach, using words like “practical” and “proven,” even after it has become dead-and-buried obsolete.

AARRRRGGGHHH! Get me out of here!!!

Conclusion

Thankfully, not all great software efforts follow this tragic script. But here's something I've learned: when things are going well in software terms, there are always vultures and/or vampires eager to attack and do their awful worst. Constant diligence is required to keep them frustrated and hungry.

Single Point of Failure: Logical vs. Physical

People who want to build a highly available computer system tend to focus on eliminating single points of failure. This is a good thing. But we tend to focus only on the physical layer. We don't even notice the single points of failure at the logical layer. Logical single points of failure are just as likely to result in catastrophe as physical ones, and it's high time we started paying attention to them!

Why? Keeping your system up and running is the most important job of an organization's techies.

Physical redundancy

We eliminate single points of failure by having more than one of every component, and a structure that enables the system to keep running with a failed component, while allowing it to be repaired or replaced. For example, here is a typical redundant system diagram (credit Wikipedia).

And here are instructions for replacing a hot-swap drive in a redundant design (credit IBM):

Logical vs. Physical

We are familiar with the concept of logical and physical in computing. All of computing is built on layers and layers of logical structures. Frequently, we call something a "physical" layer which is actually the next layer down the stack of logical layers; we call it "physical" only because it is "closer" to the actual physical layer than the layer we call "logical." A good example of this is in databases, in which it is common to have a "physical" database design and a logical (higher level) one; of course, calling it a "physical" design is a joke, there's nothing physical about it.

Keep eliminating physical single points of failure

I am not arguing that physical redundancy isn't important or that we should stop eliminating physical single points of failure. When I see people running important computer systems that have single points of failure, I tend to wonder how often the people in charge were dropped on their heads onto concrete sidewalks as children, and how they manage to feed themselves.

The principle is simple: if you have just one of a thing, and that thing breaks, you're screwed. For example, if you have your database on one physical machine and that machine breaks, no more database.

What is a logical single point of failure?

I think people don't pay attention to logical single points of failure because it just isn't something anyone talks about. It's not part of the discourse. Let's change that!

A prime example of a logical single point of failure is a program. You create physical redundancy by running the program on several machines. Great. That covers machine (physical) failures. What about program (logical) failures? After all, program failures (i.e., bugs) hurt us far more often than machine failures. But somehow, we don't think of a program as a logical single point of failure. We think that the program has bugs, that our QA and testing weren't good enough, and we should re-double our QA efforts. And somehow, miraculously, for the first time in history, create a 100% bug free program. Ha, ha, ha, ha, ha, ha, ha.

Suppose you have version 3.2 of your program running in your data center. If that program is running on more than one machine, you have eliminated the physical single point of failure. If version 3.2 is the only version of the program that's running in the data center, then version 3.2 is a logical single point of failure. The only way to eliminate it is to have another version of the program also running in the data center!

Eliminate logical single points of failure, too!

Smart people already have a data center discipline that eliminates logical single points of failure.

Suppose there is a new version of linux you think should be deployed. Do you stop the data center, upgrade all the machines, and start things up again? While that may be the most "efficient" thing to do, from a redundancy point of view, it's completely insane. Smart people just don't do it. They put the new version of linux on just one of their machines, and see how it goes. If it runs well, they will deploy it to another machine, and eventually all the machines will have it. It's called a "rolling upgrade."

Same thing with the application. Great web sites change their applications frequently, but using the rolling upgrade discipline, and if they're really smart, with live parallel testing as well.

Getting into the details of application design, the best people go beyond this method to create another logical layer, so that the things that change most often are stored as "data," in a way that changes are highly unlikely to bring down the site. A simplistic example is content management systems, which are nothing more than a way of segregating the parts of a site that change often from those that don't, and keeping the frequently changed parts (the content) in a non-dangerous format.

Conclusion

There is little that is more important than keeping your system available to your customers. Eliminating single points of failure is a cornerstone activity in this effort. Many of us are well aware of physical single points of failure, and eliminate them nicely. It's time for more of us to include logical single points of failure in our purview, and to eliminate them with the same vigor and thoroughness that we do the phsical kind.

Franz Liszt: 200

2011 is the 200th anniversary of the birth of Franz Liszt.

Liszt's music is recognizable by anyone who (like me) watched early cartoons, like this one:

[OOops -- removed from YouTube. Search for Hungarian Rhapsody #2 cartoon]

(Image credit.)

Beyond that, he is significant to me as is obvious from "BlackLiszt," as I briefly explained in the first post of the blog.

Liszt is one of the greatest, and most under-rated composers of music. This article says it well:

BBC Radio 3 is beginning Franz Liszt's bicentenary year with...wall-to-wall Mozart. Nothing could make clearer something that has bugged me for years: the critical, snobbish, misinformed and persistent denigration of a musician who was the very embodiment of Romanticism.

It is possible to admire Liszt simply because you like listening to his music. Here are some of the additional reasons I admire Liszt, with obvious parallels to computing:

• He had complete technical mastery of his instrument, which he achieved by hard work and relentless practice.
• He started with nothing and earned his way to fame and fortune. He was one of the most accomplished and generous teachers, helping countless talented younger people to make themselves better.
• He deeply appreciated the accomplishments of others. He showed the seriousness of his appreciation by promoting the people and works he admired. He showed the depth of his understanding by making transcriptions that are themselves works of art.
• He took the current standards of his day seriously and mastered them; and then smashed the conventions to make things better.
• He felt deeply and acted on his feelings. He felt and expressed the deep connections of music to the human spirit.
• In spite of his position and accomplishments, he spent his life learning and exploring.

Personally, I never need much of an excuse to bring Liszt into a conversation; but I'm glad it's his anniversary year, because it makes performances easier to find and I don't need to work quite as hard to bring the conversation around to one of the greatest performers, teachers, creators, appreciators and pioneers who ever existed, an inspiration to us all.

Chase's Exemplary Handling of Data Theft

I think if Chase had really tried, they could have done a worse job telling customers about the recent security breach.

Background

Apparently being incapable of performing the requisite fairly simply processing and analysis on their own, Chase and other giant financial institutions give their customers' data to Epsilon (among others!) for marketing-related processing. Despite (I assume) conforming to all the odious rules and regulations for keeping the data secure, Epsilon somehow suffered a major data breach; in order to protect the guilty, the details have not been released.

Chase's e-mail

Naturally, Chase and others rushed to assure their customers that everything was really OK, while providing them with helpful hints about avoiding getting scammed by all the crooks who now have the data. Here's the one I received.

Why Chase deserves an award for Badness

Chase provides a wealth of examples not to follow if you want to treat your customers with respect. Here are a few of the highlights.

• Timing. The breach reportedly took place on March 30. It was made public the following day. I received Chase's e-mail on the evening of April 4. Boy, Chase sure fell over themselves getting the word out to their customers, didn't they?
• What was stolen. Epsilon's own press release admits that not only customer e-mail addresses, but also names were stolen. If you read Chase's tardy missive word for word, you notice that they carefully omit to tell their customers that their names were also stolen, while repeating that no "customer account or financial information" was stolen. Surely a customer's name is part of that customer's account! If not, exactly what is it? Why couldn't they just be honest, and tell me that my name was stolen too?
• What was stolen. Epsilon's second press release emphasizes how they have absolutely, definitely, no-kidding determined that nothing but names and e-mails have been stolen. I'm sorry, but this can't possibly be true. Chase isn't using Epsilon just to do e-mail blasts. They are using them for their analysis based on detailed customer information. According to Epsilon itself, this data includes "Comprehensive income, credit, debt and asset data." It is simply not credible to claim that this data could not be deduced by the thieves from what they took. Neither Chase nor Epsilon bothers to mention all the customer-specific information they've got, which also includes "age, marital status, occupation, ethnicity and changes such as a new child, a move, changes in household income or a new driver."
• Disastrous advice. Look at the list of recommendations in the e-mail. Do they once, even once, describe, mention or warn against phishing, which is the real danger of having this information out there? They do not! What do they warn against? Repeatedly, they tell you not to put sensitive information into an e-mail, or to respond to a spam e-mail. When the real danger is phishing!
• Unwanted spam. I can't help pointing out that Chase gives me the incredibly insightful advice to "be on the lookout for unwanted spam." As opposed to the spam I want? After I've identified my spam and put it into "wanted" and "unwanted" piles, exactly what should I do? Since I was told to be "on the lookout" for it, I guess I should spend some time looking at it.
• Follow up. Chase promises to tell me "everything we know as we know it, and will keep you informed..." Simply put, there has been no follow-up. If you're not going to do it, don't say that you will.

Summary

It is clear that Chase

• notified its customers tardily,
• demonstrably lied about what was stolen,
• gave terrible and/or laughable advice about what the customer should do,
• and finally made promises they failed to keep.

Could they have done worse? Probably. Meanwhile, let's use this as an anti-role-model for how to handle situations of this kind.

Bad Customer Service: Whose Fault is it?

When a customer gets bad customer service, who's at fault? The person delivering the bad service, right? In some cases, yes, of course. But in a variety of important cases, the person delivering the bad service is not at fault, but gets blamed (or blasted!) for it anyway, while the guilty party gets promoted.

Sounds strange, right? But it's all too common. I experienced it personally last weekend, and my own experience will illustrate the point quite nicely.

The Cablevision Store

After several years of fine service, my cable modem started showing signs of dementia, wandering off into disfunction with increasing frequency, the only cure for which was a hard re-boot. After several weeks of wishful thinking, I finally accepted reality and took my old cable modem to the cable company's store to trade it in for a newer model last Saturday.

Shortly after starting my transaction, another customer came into the nearly-empty store with a DVR. He went up to the service person next to mine, handed over the DVR, and explained his issue. He said that he had been given the wrong DVR the day before -- he needed one with an HDMI connection. Could he please trade for the right box?

I'm sorry, says the pleasant CSR (customer service representative), we're out of those boxes. You'll have to come back next week. That's when the fireworks started.

"I spend $250 a month for your service...how can you be out of DVR's?? ... I demand to speak to a manager ... I can't come back on Monday, I work in the City and Saturday is my only day for getting stuff like this done ..."

It got worse. Various impolite words started flowing freely...

...inspiring the large fellow helping me to try to calm the guy down, telling him there's no need to use language like that, etc. In the end, he stormed out with his non-functional DVR without a resolution.

Meanwhile, I got a new cable modem with no instructions. I was told to just plug it in and it would work.

Who is Responsible for the Customer's Bad Experience?

The CSR's in the little store-front operation? Hardly. They were pleasant enough. But they didn't have what the guy needed -- and had every right to expect! This is a guy who's headed right to FIOS, in spite of Verizon's well-deserved reputation for equally horrific customer service.

OK, so who is at fault here? It's hard to know for sure, but it is likely to be some combination of simple incompetence in the people and/or systems that manage inventory at the stores...

...which wouldn't be a bit surprising... or it's some "clever" person in management or finance trying to make the corporation some extra profit (and themselves a promotion) by shaving down the inventory levels even further...

(credit: www.bybee.com)

More Where that Came From

I started thinking about all this as I drove home and started installing my new cable modem. I replaced it exactly, and it didn't work! OMG!! It's a new cable modem! What could be wrong? I checked the cables. I rebooted a few times. I replaced all the cables. Could my wireless router have somehow gone bad at the same time? Out to the store, bring home new wireless router, install, no improvement. Probably the new cable modem is simply dead. It's too late to return to the Cablevision store, so I get on chat. The guy asks me to run a test, and by the time I'm back (all of 3 minutes later), my session has been timed out for inactivity! So I log in again (requiring another download of the chat program), and the fellow can't "see" my modem from the network side. No surprise. He asks me to connect it to the place where the cable service enters my house, and he still can't see it. So he agrees that it's bad, and actually gives me an appointment for a service call on Monday.

All I can say is, thank goodness for neighbors who install WiFi without security.

The service guy shows up on Monday, looks at my new cable modem, and immediately says, "they gave you one of those?" It turns out the service guy knows that these devices are obsolete and notoriously unreliable. As I have confirmed yet again. He hooks up a new cable modem he had brought with him and everything works fine. Problem solved.

Whose fault was the bad customer service I received? The people in the store? The people on chat? The guy who came to my house? Of course, the answer is none of the above.

If you tried to track down the person actually responsible for the bad service I (and many, many like me) received, you'd run into something like this:

(Credit: Animation Guild)

Again, the root cause is likely to be some combination of hard-to-fathom incompetence, simple clueless-ness and actual, anti-customer changes made by someone who thinks they can advance in their career by making things worse for customers. Sadly, this is all too often how organizations, large and small, actually work ... unless you take constructive steps like this to make it otherwise.

Software Project Management: "Releases" are Stupid and Out-moded

Tell me, do you use Google by any chance? Yes? OK, now tell me which release of Google you use now, which ones you've used recently, and which one you like best; and, by the way, how was the upgrade process? Oh, you don't know? There's no way to find out, you say?

What the heck is WRONG with those people at Google? Don't they know that modern software development processes demand a disciplined release and planning methodology? There is NO WAY customers will put up with a product when the vendor just shoves new releases at them any time they feel like it, without even proper notification! And what customer is going to sign up with a vendor who won't commit to a future roadmap, with at least a year's worth of features laid out, tied to hard-commit release dates? That Google, they violate every rule of the game, there's no way they're going to make it as a software/service company!

What, how can it be! NOOOOoooooo.... Google is the world's most valuable software company!!?? When they don't even have the most basic element of proper software methodology, releases?? I must be sleeping! This must be a nightmare!!

Get over it!

Here are the facts:

• Classic project management has been a disaster for software development.
• All the heavy-weight process things that people do to make things better ... invariably makes them worse!
• The classic big-bang release is the cornerstone of the temple of evil.
• The classic little-bang release (a.k.a. "agile") is a brick in the temple of evil.
• "Releases" are ... stupid! ... and not only that, they're ... outmoded!

 There is life after "releases." It is a better life. The software is better. The users are happier. Go there. Enjoy it.

HDD Capacity Improves While Slowing Down. Help!

Here's a typical hard disk drive (HDD):


This is where your data spends most of its time.

The amount of data you can put on each HDD has gotten exponentially better over time. For example, this chart (all credits: Wikipedia):


shows hard drive capacity in GB over time.

Back in the mid-1980's, HDD's were sized about 10MB. By the mid-1990's they were up to about 1GB, an increase of 100 times. Now they're up around 1TB, an additional increase of 1,000 times. Amazing, particularly when you consider that the average HDD was also shrinking in physical size over that time:

Right: an older 5 1/4" HDD with 110MB; left: a 2 1/2" HDD with 6.5GB.

Suppose the size of your main customer database is 1TB. In the mid-1990's, it would have required about 1,000 HDD's to store the data, and today you can put it all on a single HDD: it's wonderful, yes? Well, maybe not. The problem is using your data.

The problem is simple to understand. Here:


is the inside of a HDD. Your data is stored on platters. It's read and written by a head, which is at the end of the long arm that ends near the center of the platter. In order to read any piece of data, the arm has to position the head on the right track of the platter (i.e., move it towards the edge or towards the center of the platter), and then has to wait until the spinning platter brings the data to the head.

Here's the killer: these times (called seek times) haven't improved much in the last 25 years! In the mid-1980's they were around 20ms; today, most HDD's are around 10-15MS, and the very fastest are around 3ms, around a 7X improvement at best.

These trends (more data in less space, and unchanging seek time) are likely to continue.

Improvements in HDD's since 1980's:

Speed 7X
Capacity 100,000X

The bad news isn't over yet. Remember how your customer data used to take 1,000 HDD's? That meant that you had 1,000 HDD's all available to read and write your data. Now you've got just one -- and it's hardly any faster than HDD's were 25 years ago!

What does this mean? If you want to not just keep your data, but also access and update it, you've got a big problem today, and it's getting worse.

How are people responding to this situation? Simple: they are turning to SSD's. SSD's solve the speed problem! But, as provided by most vendors, SSD's introduce a whole host of new problems. Today, storage buyers confront an extremely unpleasant choice: buy extremely expensive SSD's that lack fundamental storage features, or buy affordable HDD's that simply aren't fast enough. Yuck.

There is an alternative. It's a good one. It combines everything you like about HDD's with the speed of SSD's in a novel hybrid combination. I can't stop thinking about about it. Check out the Xiotech hybrid ISE.