Basab Pradhan

BRAKING NEWS: The Anti-Tonsuring Law

BRAKING NEWS
Interviews that bring you to a screeching halt

Last week in a forced tonsuring incident in Bangalore two school boys’ heads were forcibly shaved by the school staff. Bangalore police has made an arrest. The government has responded by promising a bill in Parliament that will make forcible tonsuring illegal.

Braking News met PK Dhut, Minister of State for Cultural and Religious Affairs.

Braking News: Good morning, PK.

PK Dhut: Please don’t call me PK. I don’t want to be associated with that Aamir Khan movie. You may call me Purna Kesha, which means One with a Full Head of Hair. I come from a long line of hairful ancestors.

BN: Ummm…that’s hard to pronounce. May I call you Dhut ji?

PK: Yes, that will be fine. Obviously, you didn’t study Sanskrit in school. You must have studied German or French or some useless foreign language instead.

BN: Dhut ji, the government has proposed an anti-tonsuring law which will make forcible tonsuring illegal. Why?

PK: We respect people of all hair conditions, bald and hairful. However, lately, some misguided people have been forcibly tonsuring people. These people are converted to baldness against their will. We cannot allow such forcible conversions.

BN: But aren’t there already laws on the books which prevent the use of force? Why do we need a special one for forcible tonsuring?

PK: You see people are very sensitive about their hair. It requires a special law.

BN: Really? Well, let’s take an example. Right now, you are picking your nose.

PK: No, I’m not.

BN: Well you were…till a moment ago. Let’s say I was strong and you were weak and that I found the sight of you picking your nose in front of me so disgusting that I forcibly stopped you from picking your nose. It would be torture for you, wouldn’t it?

PK: Maybe. But I don’t pick my nose.

BN: In this case, do we need a law against forcible prevention of nose picking?

PK: Forcible tonsuring is happening across the country. It is a matter of national importance. Don’t compare it to an itchy nose.

BN: OK…moving on. Dhut ji, don’t you think the anti-tonsuring law is unfair?

PK: Why?

BN: Well it makes it illegal to convert from hairful to bald, but not bald to hairful.

PK: No worries on that count. When the bill is brought to Parliament, it will ban both forcible tonsuring and forcible hair transplants.

BN: But nobody has ever heard of a forcible hair transplant. This law really just targets forcible tonsuring doesn’t it?

PK: Not at all. We respect all people, regardless of their hair condition.

BN: Dhut ji, how will the authorities determine whether the tonsuring was forcible or not?

PK: Oh that is simple. If the tonsuring victim says that so and so person forcibly tonsured him, that person will be arrested.

BN: But what if the tonsuring victim just shaved his head himself and is lying? What if he has been forced to lie?

PK: Well…there’s no law against forcible lying.

BN: Then do we need a law against forcible lying?

PK: No, no. Of course not.

Taming the Shrew

The march to quash freedom of expression in India continues. All India Bakchod, a comedy group in Mumbai, is now under attack from MNS, a Shiv Sena style, right wing Hindutva party. The reason is that in a roast of Arjun Kapoor and Ranveer Singh, both Bollywood actors, the language used was, in their view, inappropriate. [link]

Ameya Khopkar, president, Maharashtra Navnirman Chitrapat Sena, the film wing of the MNS said to The Hindu “We will not let any film featuring the actors present at the AIB comedy show to be released in Mumbai, until they apologise for their actions.

“What kind of message are we giving to the world outside? What if small children, who follow Ranveer and Arjun, watch this show? Will they not be spoiled after watching such shows?”

While AIB may seem to be under attack, the threat is not against AIB, which is small and can continue to flourish on YouTube. The threat is against Bollywood, the shrew of Mumbai, that is largely liberal and annoyingly espouses liberal views in its movies.
This is just the beginning. Not just Bollywood, every writer, actor, artist and musician in India is watching these events closely. They’ve seen what happened to theatres that screened PK. They’ve seen how Perumal Murugan was forced to abandon writing altogether. How AIB, a small comedy group in a tiny comedy industry in India, is being harrassed. And they are all afraid. “That could be me, if I’m not careful,” they are thinking. And “careful” does not create art.

The modus operandi of the Indian right wing is clear. The government need take no action at all. There are enough Hindutva goons in every state of the country who will gladly, for a few hundred rupees, go break up a theatre or rough up some people. All the government has to do is stand aside and let the goons have their fun. The media faithfully broadcasts it. No arrests are ever made. Everyone gets the message – “Be careful. Or you’ll be next.” The shrew is tamed.

There are people, good thoughtful people, who think that freedom of expression in India is over permissive, or that it is not uniformly applied to all religions. So reining in these Bollywood types, who dare to make fun of our godmen (PK), is a good thing.
But, you see, it is not. Today it is about godmen. Tomorrow it will be the length of your daughter’s skirt. Today it is PK. Tomorrow it could be Delhi Belly. Who is making the decision on what is appropriate and what is not? MNS? Do you trust their judgement?

There are other good, thoughtful people who make this into a development vs freedom debate. They argue that the BJP is all about economic development. And freedom of expression is a much less important thing. So let them curtail it if they want, as long as the country prospers economically.

Why is this an either/or choice? I don’t see how freedom of speech and expression get in the way of economic development. This is a false choice. Do not accept it.

People, wake up and smell the coffee. Your freedoms are slowly, but surely, being fenced in. Wake up, before they are taken away and we become an Iran.

A Secularist New Year Resolution

Ever since the BJP came to power, they’ve been crawling out of the woodwork – RSS ideologues, Hindutva radicals, and random school teachers. They want to remind you in speeches that India is Hindu; that Hinduism is great and any book or movie that says otherwise should be banned; and that their version of Hinduism (no short skirts! no public kissing!!) is the one that everyone must follow. It’s pervasive, relentless and, to the dismay of those of us who disagree, it is slowly moving from being intolerable to being irritating, but “chalta hai.”

There seem to be two aspects of what is happening. Let’s call one – What is Being Said. The other – Real Changes.

What is Being Said is getting crazier and crazier. Should we clamp down on inflammatory speech? Nitin Pai, a public policy commentator with Acorn, makes a good case that protecting freedom of expression is better and easier than restricting inflammatory speech. Why? Because you can’t ensure that nobody is ever offended.

It might be better, but I’m not sure it’s easier. Rioting in India is difficult to control because of the 24 X 7 news cameras and because the rioters tend to be unemployed youth for whom rioting is a source of income. The odds are stacked against the police.

In any case, there are laws on the books that supposedly curb hate speech. So perhaps there’s not much to be done here except murmur to yourself “Sticks and stones will break my bones but words can never hurt me.”

The problem is that things don’t end with What is Being Said. It drives Real Changes. Books are being pulped (Wendy Doniger’s The Hindus, Sekhar Bandopadhyay’s Plassey to Partition). Textbooks are being changed (Dinanath Batra has rewritten the Gujarat state school textbooks where science and history are liberally mixed with mythology). And there are serious extra-judicial bans being placed on all kinds of ‘unacceptable’ behavior. Small, but Real Changes.

What is Being Said is not independent from Real Changes. When What is Being Said goes unchecked it supplies oxygen to Real Changes. It emboldens all the closet Hindutva sympathisers to come out in support of What is Being Said. Which gives the mistaken impression that there is broad-based public support for What is Being Said. And the next thing you know, a book or a movie is banned. Or it becomes OK to harass anybody who kisses in public. Suddenly, Real Change has happened.

Bit by bit, these ideologues and goons are chipping away at the edifice of our beautifully diverse, secular India. And it all begins with letting What is Being Said go unchallenged. While politicians and pundits rant about What is Being Said on TV (and that is important) the secularists among us are not doing our bit in the forums where opinion is formed – among friends and family.

We secularists tend to not engage with the Hindutva-flavored friends of ours on social media. It’s OK, we say. I can’t change the way they think so why engage in a meaningless debate. But that’s misguided. You can change the way people think. Most people don’t put too much thought into these issues. They tend to go with the flow – wherever they think the majority of their family, friends and people who they respect are going. We need to let them hear what we think about What is Being Said.

If the discourse on Hindutva in your homes and chai shops, on Facebook and Twitter, remains one-sided, then be prepared to lose what you love. Years from now you’ll look back and regret that when Real Changes were turning the clock back on secular India, you could have been more engaged and made a difference, but you didn’t.

Are you OK with that? I’m not. Ain’t gonna happen. No sir, not on my Newsfeed. I will engage. And that is my new year’s resolution.

Dark Data is a Big Opportunity for Services Companies

Why is Big Data big? Obviously not because someone invented a new statistical method of data analysis that can explain everything. It’s because there is a lot more data spewing out of business processes where there were none before. The questions haven’t changed – as a business manager I still want to know how to forecast sales or understand which levers to apply to improve my business outcomes. But where there was once no way to answer these questions, today there actually is data that can be analyzed to provide some of these answers.

So far all the attention has been on business problems whose answers lie in the copiously flowing data from associated business processes. Marketing, for example, where all Marketing is getting subsumed into Digital Marketing which, as anyone will tell you is a data gusher.

But there are innumerable questions in business that need answering where the data just isn’t there to analyze. Or rather, the data is there but it is unusable, just beyond reach.

I am part of a non-profit theatre company in the Bay Area. Theatre companies live and die by their ticket sales. All costs are fixed. Once you decide to stage a play your costs are all locked in. Your revenue, however, is completely variable, by the number of tickets you sell.

In such a business, it would be crucial to understand where one stands on ticket sales. In other words, you should be able to answer the following question at all times “Based upon the ticket sales today, X days before opening night, we are on track to fill Y% of seats”.

Almost all ticket sales are online. Which is a good beginning. The online ticket selling service that we use sends us a daily email with our cumulative ticket sales till that day. But, and here’s the nub, they don’t store a time series of daily ticket sales. So if I wanted to draw a graph with number of days to opening night on the X axis and cumulative tickets sold (by show, of course) on the Y axis, I’m out of luck.

This is not a unique situation. I’m sure you can think of many such examples in your business where you know the data is created but it isn’t kept, or it isn’t in the right form.

Then there’s a form of data that is not captured but can be, with just a little work. At Infosys, when I was trying to wrap my head around how to implement CRM in a company which hadn’t used one for years, I thought that it might be too much to expect the field force to make notes after every client meeting. But perhaps, if they could just log every client meeting, that by itself would be very useful. It would be a measure of business activity which we may be able to correlate to deal value and perhaps, could serve as a rough, early warning forecasting system. There are so many opportunities for squeezing a business process for meaningful data. Analyzing this data, typically doesn’t need Hadoop clusters, but the business outcomes could be quite significant.

Think of data as fossils in sedimentary rock. The fossils in the upper layers are newer, better formed and easier to interpret. The ones in the lower layers are just the opposite. But they are just as important to understanding and improving your business.

IT Services companies will see a lot of opportunity in Big Data and Analytics. But software companies will take away most of the value in the top sedimentary layers. The lower layers will be messy. And straightening out messes, is where IT Services companies thrive.

Meanwhile, I’ll be trying to sort out my ‘messy’ ticket sales data using Google Script. If anybody knows of a script that will help extract a number from an email that follows an identical string of text, please send me a note. Thanks.

My Lost Years Without Desi Ghee

From The Guardian

Butter, cheese and even red meat are not as bad for the heart as has been maintained, a cardiologist has said in a leading medical journal, adding that it is time to “bust the myth” of saturated fat…
“Recent prospective cohort studies have not supported significant association between saturated fat intake and cardiovascular risk,” he argues. “Instead, saturated fat has been found to be protective.”

Doctors and scientists are now ready to proclaim that saturated fats are good for you. Not just OK, but good. Better than those sucky unsaturated fats we’ve been using all these years. This is not just the opinion of one diet guru somewhere. There are now many studies and meta-studies (apparently that is a thing where you study other studies) that conclude that saturated animal fats of the kind found in butter and lard do not lead to heart disease.

Now I’m not a science hater. In fact some of my best friends are scientists. My father was one. And I do understand that paradigm-shifting is in the nature of scientific revolutions. (There is actually a book by Thomas Kuhn, sitting on my shelf, that explains all this. It is an important book, though totally unreadable.) For example, at one time Neanderthals were supposed to be cousins of us homo sapiens but from a branch on the family tree that died off before we hit the scene. But with more fossils and genome analysis it has emerged that homo sapiens were contemporaneous and interbred with the Neanderthals and we all now carry Neanderthal DNA. Delightful shift in paradigms, no? Though a little embarrassing for those people who used the word “Neanderthal” to describe certain people at work.

But I have no problem that 2.7% of my genome is Neanderthal. I’m pretty liberal that way. On the other hand, this latest round of myth-busting, paradigm-shifting science about saturated animal fats has gotten me terribly depressed. To think that all my adult life I have substituted margarine for butter and Crisco for desi ghee fills me with great regret.

I grew up in a place called Hisar, in the state of Haryana, India. Hisar was the proud seat of Haryana Agricultural University. My father was a dairy scientist and a professor of animal nutrition in the College of Animal Sciences.

For those of you who aren’t as familiar with Haryana, it is an Indian state neighbouring Delhi that is famous for many things including Haryanavi, a bold, assertive dialect of Hindi that you will often hear in Bollywood movies used by thugs and comedic policemen.

But the thing that Haryana is most identified with is milk. The state poet Uday Bhanu Hans has described it thus, Desan mein des Haryana, jit doodh dahi ka khana. Which is too deep to translate into English, but roughly means that Haryanavis like their dairy products.

In this state, in its only agricultural university, in its college of animal sciences, my father was a professor of animal nutrition. My connection with livestock and milk was visceral. Literally. Besides spending many a Sunday at the University farm (see photo), I consumed copious amounts of milk and dairy products.

Ghee, or clarified butter, is the pinnacle of a Haryanavi’s connection with milk. It is its most refined, celestial form. It goes into the havan fire as an offering to the gods. It also goes into every edible thing imaginable. And if you can’t do without it, mixed with some milk, it can be had straight from the glass, between meals.

In the bazaar ghee was often referred to as Shudh Desi Ghee. “Desi” separated it from that evil concoction of hydrogenated vegetable oils called Dalda. And “Shudh” was well, pure. Some halvai’s would mix in a bit of Dalda since it was much cheaper. But not this halvai.

All through our childhood, we had lots of ghee. But as an adult who made responsible health decisions, I reduced and then completely stopped having ghee. We cooked in vegetable oil and ate sukhi roti. I pined for ghee, but knew that she wasn’t right for me. For twenty years now, I’ve been living a sukhi zindagi, thanks to faulty science. I may find it in me to forgive her in time, but this is not one to forget.

On the bright side, I should still be thankful that I have the rest of my life to enjoy ghee. Just imagine if I had died before the saturated-fat-is-bad paradigm had shifted. To have gone through my adult life without ghee, only to have it poured on my funeral pyre would have been such a travesty.

And now that ghee and butter are good, what about bacon?

Government Mass Surveillance Will Create a Surge in Technology Spend

How the NSA hacked Google in one simple graphic. Photograph: Washington Post

The Guardian has a presentation called The NSA Files that is the most brilliant rich media presentation of a complex subject I have ever come across outside of a museum. So go read it, even if it is just to see the presentation.

As technology races ahead, from time to time public debate and the law of the land must catch up to it. Government surveillance is one of the most important technology issues of our times. In the US, the NSA files have already had a profound impact on the perceptions of Americans about surveillance and civil liberties. Outside the US, Germany is aghast that the US and UK, NATO allies would spy on Angela Merkel and Germany. On the other hand, there is some evidence that the NSA has aided in combating Mexican drug cartels and prevented terrorism.

The political issues surrounding mass electronic surveillance are complex and do not lend themselves to quick fixes. The issues at stake involve civil liberties, anti-terrorism efforts and international spying, among other things. It might take a decade of public education and wrangling in courts, legislative bodies, NATO and perhaps even the UN, before policy, practice and the law around surveillance settles down.

Leaving aside the political issues, the NSA files will feed a long surge in surveillance and anti-surveillance technology. Snowden’s leaked files are like the Trinity test – the first detonation of a nuclear bomb. Prior to that governments knew that a nuclear bomb was feasible and some were feverishly working on making one, but the public did not really know much about it. The Trinity test, first brought the power and potential horror of a nuclear bomb to the attention of people around the world. And just like Trinity set off a nuclear arms race that lasted for decades, the NSA files will set off a “Snoop Brawl” which will lead to a burst of technology spending around the world. But unlike the nuclear arms race, which was a race principally between nations, this Snoop Brawl is going to be multi-faceted; many players and many fist fights.

Country vs Country
Countries now realize, if they hadn’t actually known it all this while, that the NSA gives the US almost unfettered access to their secrets – those of their governments, politicians and companies. Dilma Roussef, Petrobras, Ban Ki-moon and Angela Merkel were all targets of the NSA; none of which can be justified by national defense. Surveillance is not a single-purpose (e.g. protect the country against terrorism) tool. It is an arsenal of weaponry to project a country’s power and further its interests. And no country has a bigger arsenal than the NSA.

Not China for sure. A few months back, China was pilloried for not doing enough to stop Chinese hackers, allegedly sponsored by the Red Army. That doesn’t seem so egregious anymore. If all major countries decide to bolster their electronic espionage and counter-espionage capabilities, that itself is going to be a lot of hardware, software and thousands of tech jobs.

Country vs Citizens
The NSA collectes metadata on all phone calls in the country – who called whom, when and from where. It is easy to see how this data could be very useful in tracking the networks involved in illegal activity. Or political opponents. If information is power, this is the hammer of Thor (Thor 2 was so-so by the way). Which country that isn’t shackled by its laws, can pass up the opportunity to gather this data?

Outside of a few advanced democracies with active civil liberties protection groups, there is no countervailing force to stop a government from collecting and using this data. The only hurdle is the technology and skills to mine a massive data set like this. Which money can buy. Expect countries to spend a lot on this and other Big Brother technology.

While this gives the government a valuable instrument to catch the bad guys, the very same instrument, in the wrong hands could be used to suppress democracy itself. Or give more power to despots. Expect big orders from tin pot dictatorships as well as big nations where ruling classes are trying to quell democracy.

Companies vs Governments
The US is a hub for internet companies and cloud companies that carry the private data of people and businesses around the world. Much of this data travels through pipes in the US or sits in databases controlled by US companies. Courtesy of Snowden now the world knows that US companies have been cooperating with the NSA with no disclosure to the customers.

European and other advanced nations are likely to enact laws that prevent companies like Google and Apple from becoming listening posts for the US. They may require these companies to “fragment their clouds” and keep the data of their, say, German customers in Germany with restrictions on who can this data be shared with.

On the other hand, companies like Google will have to work hard to regain the trust of their customers. Google is already taking measures to prevent the NSA from eavesdropping without them knowing. Apple is making noises that they might challenge the legality of not allowing them to disclose when information has been shared with the government at their request.

Companies vs Companies
The other things that all companies will realize – not just cloud companies with user data – is that their IP is not safe. In the past year, the Chinese hacking incidents being reported have already raised awareness of this issue. But what the NSA files make clear is that ethics and national interest seem to have no intersection at all. Every country may be spying on foreign companies that can bring value to leading companies in that country. Protection of IP and confidential information will become a key concern for companies. Much more than it is today.

Consumers vs Companies and Governments
Consumers are going to increasingly want to know how their communication and confidential information is protected by online services. Now that they know that “Enemy of the State” is for real what steps should they take to protect themselves? Encryption and information security are complex technical issues which most people don’t understand or even care about. Perhaps there is a need for “information security rating agencies” that rate online services on how they protect users’ confidential data – from hackers and governments everywhere. Perhaps people will increasingly want to roll-their-own email rather than use Gmail or Yahoo mail.

Like the nuclear arms race, the Snoop Brawl will create a flood of spending from both governments and companies. And exactly like the nuclear arms race, it leaves no positive impact on the human condition.

Writing Simply

Greg Mankiw writes about a carbon tax in the New York Times. Whether or not you agree with the economic arguments for the tax, it is easy to agree that Mankiw writes very well. Mankiw is a Professor at Harvard and was the Chief Economic Adviser to President George Bush. You would expect him to be quite an authority on economic theory. But in this piece he explains a complex matter in a way that a high-school kid would understand it.

Writing simply is a craft. It requires hard work and practice. James Thurber, an American writer and humorist, was known to go over his writing again and again until he was satisfied that it was clear and simple. His “My Life and Hard Times” was a book that I read and reread, again and again, as a child – and still can, as an adult. It is a timeless classic.

Writing simply requires expertise. You can’t be clear and simple in your writing until you are clear about the subject in your head. Writing simply isn’t the same as writing about simple things. Richard Dawkins has written volumes of erudite stuff about evolutionary biology that is completely accessible to people with no more that high-school biology.

On the other hand, there are all manner of experts who hide behind a facade of jargon and what Thurber calls expression-complexes. Referring to psychoanalysts Thurber says:

…I have discovered that they all suffer from one or more of these expression-complexes: italicizing, capitalizing, exclamation-pointing, multiple-interrogating and itemizing…It is a defense mechanism used to cover up a lack of anything new and sound to say on their favorite subjects, and to make up for an inability to write simply and convincingly, or to think clearly.

Writing simply often requires having an opinion. The reason I like the Economist or Om Malik, is because they generally don’t waffle. You know where they stand on issues. The cover page of the Economist and its headline leave you in no doubt about their position on Syria.

Writing simply is risky. If you are wrong with your facts, you can’t hide behind a possible misinterpretation. If you are wrong with your opinion (like the Economist was, by their own admission, on Iraq) someone will call you out.

When that happens, you go back to your blog post and admit that you made a mistake. Right there, next to where you made it. In italics, so nobody misses it. And then continue to write the way you want to.

Cloud to Enterprise: Resistance is Futile

At the end of its Journey to the Cloud, will an enterprise still be running its own data center?

Perhaps, but if it does, it will be much, much smaller. It will be the Private Cloud part of a Hybrid Cloud that will run only a small portion of enterprise workloads. On the other hand, many enterprises will live entirely in the Public Cloud.

Consequently, most of the enterprise data center capacity in the world will disappear, assimilated by the Borg, that is the Public Cloud.

Why is the gravitational pull to the Public Cloud so strong?

One reason is that the price-performance of the Public Cloud is much superior to that of a dedicated data center, even one with Private Cloud. And the gap is widening all the time.

At the heart of this price-performance advantage is scale and specialization. The kind that takes Public Cloud service providers to places like Prineville, Oregon to build massive chiller-less data centers with custom-built servers.

Facebook’s done exactly that. There is a fascinating, if somewhat dated piece in Wired magazine, which goes into a lot of detail on Facebook’s data center in Prineville. Needless to say, the scale and level of specialization is such that every single element of cost and energy coinsumption is highly optimized. Not something an average enterprise can or should be spending their time on.

Facebook’s not an enterprise cloud service provider (yet?). But Amazon, Google, Salesforce.com and Microsoft will undoubtedly have the same laser focus on cost and performance.

Cost advantage also comes from “pooling” – the simple notion that the capacity required to run a pooled data center is less than the sum of the capacities of the individual data centers. Pooling is a powerful cost driver in many services from electric power to limo services. It has nothing to do with specialization, but the cost savings are quite real.

But it’s not just about price-performance. Over time, specialized cloud service providers will add product features that will simply make them better. Or allow customers to do things they couldnt do before. Big Data is certainly shaping up to be one of those areas.

This has happened before. A close analogy would be how Salesforce.com started out by being the cheaper, easier to implement SaaS competitor to on-premise CRMs, but is now functionally a superior product.

Better price-performance and superior feature-function may be quite enough to explain the exodus to the cloud. But there is actually an even bigger force at play here. Something that doesn’t lend itself well to ROI calculations, but nevertheless, CEOs understand it very well.

Running a data center is not core to any large enterprise. Its business is selling widgets or serving customers. And for it to be the best in the world at what it does, it needs to focus on what makes it good. Running a data center is not one of those things.

Which is why if I was a data center today, I would be saying my prayers. Resistance is futile. Prepare to be assimilated.

I Want My Customer Data

For the last few years, I have been using Mint (mint.com) to keep track of our household expenses. My needs are very simple. I want to be able to answer simple questions like “How much are we spending on regular monthly expenses?” and “How much is going towards discretionary expenses like eating out?”.

But I find it difficult to answer the simple questions above. It takes a lot of time and effort to get to a point where I can say that I am reasonably close to the real answers to these questions. As I describe the problems that make it so time-consuming, it actually throws light on a new battleground for consumer services – customer data in the hands of customers.

Except for a tiny fraction of cash expenses, all of our expenses are in the form of transactions – debit cards, credit cards, electronic transfers, bill pay transactions, and yes, a few, hand written checks. We try to keep things simple so we don’t have too many accounts. All these accounts are hooked into Mint. Mint pulls all these transactions so that I can see everything in one place. So far so good.

To get a handle on our household expenses, each transaction needs to be categorized correctly into categories like Entertainment or Restaurants.

That’s where the problems start. Every time I log into Mint, I have a whole bunch of transactions that are labeled “Uncategorized”. I then have to manually go through each transaction, try to figure out the merchant to whom I made that payment and then categorize it. Often, the name of the merchant is completely garbled. If my wife made the payment, I have to wait for her to be around so I can ask her. It takes time and is annoying as heck.

It shouldn’t be that difficult to get this right. A typical card transaction is passed from acquirer to network (like Visa) to issuer (my bank or credit card company) and then to Mint. On the way, nobody seems to care enough to categorize the transaction intelligently, and ensure that the merchant name is represented correctly. It’s all left to Mint to do whatever it can with the data it has.

Mint tries. It allows you to categorize recurring expenses automatically. But non-recurring expenses are far too high in our family to ignore. Mint uses very little intelligence to extract the meta data from transactions. A typical transaction would show up on Mint like so

CHECK CRD PURCHASE 02/18 LYFE KITCHEN OF PA PALO ALTO CA 434256XXXXXXXXXX 08304975697XXXX ?MCC=5812 on Feb 19

In this case Mint extracted the merchant name as “Lyfe Kitchen Pa” (correct) and tagged it “Uncategorized” (missed opportunity).

It so happens that the string “MCC=5812” refers to the Visa merchant code for “Eating places and restaurants”. A simple google search will tell you that. Why Mint would choose to leave it Uncategorized is difficult to fathom.

In other cases, while extracting the merchant name it applies no intelligence, it appears. It just pulls out the first two or three words that are not numbers. It typically fails for merchants like 23andMe or 37 Signals or 76, the gas station.

Ultimately, no one in the entire chain of merchant-acquirer-network-issuer/bank, all of who are making money because I am spending, care enough to do anything with my data, except the bare minimum to complete the transaction. They don’t recognize the value that I put in my data. Mint does. Which is why I spend a lot more time on Mint than on my bank account website. But even Mint doesn’t do enough.

Next, consider Simple (simple.com), a new banking service that I started using a few months ago. It is still by invitation only but if you can wangle an invitation, you won’t be disappointed.

This is a screenshot of what I might see on the Simple website. Simple extracts the merchant name quite well. And the automatic expense categorization works quite well. I don’t know how it does it, but I have never had to go in and change the category on a transaction. In fact if I had to, I wouldn’t know how to do it.

But that’s not all. It will put the address on a little embedded google map. That can be a big help in identifying where you were. If you went to a restaurant, it will tell you how much you tipped.

It’s not perfect, but I feel like they are putting the information that they have to the best possible use. To do more, they would have to get more information from upstream sources over which they don’t have much influence.

I have been comparing Simple with Mint which may not be a fair comparison. Simple has to deal with its own transactions (bill pay or their own debit card). Mint aggregates across many different sources of transactions each presumably with its own idiosyncrasies.

But if you were to compare Simple with any other bank that I am aware of, the difference in the use of customer data (and user experience) is vast. It is light years ahead.

Consumer services today offer a lot of choice. One of the most important ways in which consumer services will compete with each other is what they let their customers do with their data. For a long time, the focus of these companies has been on the use of customer data to extract insights for themselves – how to cross sell more, how to identify loyal customers to serve them better and so on. But this is different.

Customer expectations are rapidly changing. They are being shaped by companies like Apple and Amazon.com that set the standards, not just in their industry, but across industries. Customer data is now part of the customer experience. This is the new battleground.

So what are these customer expectations? Here are mine:

1. That my service provider will obtain and share the data with me in a timely fashion.

I’ll illustrate this with an example. Today’s smart meter technology allows my utility to obtain the power consumption at my home in at a resolution of 15 mins. At this resolution, the data can tell me a lot more than what my monthly bill tells me, which is almost nothing, other than the fact that it is high or low. (Currently mine is running too high and I don’t know why!)

But deploying smart meters costs money. Lots, in fact. Will it be worth it? It might have been hard for utilities to justify the cost. After all, they are all monopolies. Luckily, regulators in most advanced nations have been nudging utilities in that direction.

In every industry, there will be similar challenges. How do you justify the cost of gathering more data that is useful to the customer? Expecting new revenue from additional services to justify giving customers more data may be too short-sighted.

2. That my service provider will understand that that data is mine.

I shouldn’t have to pay just to get that data. Although I will gladly pay for a service using that data that is of incremental value. I should be free to take that data out myself, or allow another service provider to pull it out on my behalf.

Today, my bank charges me a fee if I want to see a used check image older than 6 months. Tax filing time must be quite profitable for the bank. I don’t have a problem with the bank trying to turn a profit. But not on my data. If your storage costs are too high (really?) allow me to easily export it to my Evernote or DropBox account. (New feature idea – managing check images!)

3. That my service provider will take the utmost care to secure my data

This is generally well understood. Because of laws and the damage that negative publicity around loss of customer data can do, most services try hard to protect it. Try harder! Lately, the hackers seem to be winning.

4. That my service provider will add value to the data

My online brokerage service has always given me a CSV download of my transactions. But till two years ago, they did not have a decent performance analysis of my investment portfolio. So I had to go and put my entire portfolio in a Google spreadsheet which would look up prices from Google Finance and calculate the rate of return. But with reinvested dividends and what not, it took a lot of work to keep the portfolio up-to-date. How you can be an online brokerage and not offer the most basic use of my data – portfolio performance – is beyond me?

Performance analysis, alerts, suggestions – they are all possible. And expected. If my credit card hits me with a foreign transaction fee, I want to know about that in an alert (thank you Mint!).

5. That my service provider will understand that the reward is mostly my loyalty

There was a time, when online retailers did not give you a transaction history. I stopped shopping on those sites. I don’t want to spend the time to search for the same item all over again, if I want to buy another one.

For service providers, this is going to become the cost of doing business. So if you think that you will invest in giving me more value from my data only if I pay you more for this value, your competitors who think differently will get my business.

But, if you are clever about it, you will discover value points that I will pay for. You see, the work that you do to help me get more value from my customer data, in turn helps you understand me better. And when you understand me better, you will be able to design services that I will want to pay for.

Wanted: A Rules Engine for Excel

James Kwak writes about the role of Excel in the JPMorgan 2012 trading loss

After the London Whale trade blew up, the Model Review Group discovered that the model had not been automated and found several other errors. Most spectacularly,

“After subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR . . .”

So @SUM instead of @AVG and boom – $2B in trading losses.

I am exaggerating of course. It didn’t quite happen that way, but it does appear that this Excel error made the trade appear much less risky than it actually was. [You’ll find a full analysis of the JPMorgan trade here. A lot more interesting is this history of Excel bloopers.]

Excel is the weapon of choice for financial analysts of all hues. You could be an analyst evaluating complex derivative instruments or an investor projecting a public company’s future earnings. If you are a financial analyst, you probably spend a big part of your life looking at a grid of tiny grey cells.

During my startup days, we built a product that would pull in data from our library directly into their spreadsheets and keep it current. Since we worked with our users to design the product we got to see many, many spreadsheets.

As a rule, these spreadsheets are massive, multi-MB beasts. They start big. Over time they gather more and more data and complex analyses and become massive. In their full glory, they are inscrutable to anyone except their owners. And often even that is doubtful.

Not surprisingly, we regularly found errors in the spreadsheets. As expected, data errors were common. But errors in formulas, of the kind above, were not uncommon at all.

Now think about it. Companies are making decisions worth tens, sometimes hundreds of millions on the backs of financial analyses done in Excel spreadsheets, that are understood solely by their owners and are impervious to scrutiny by anybody else.

Excel was never designed to be anything more than a personal productivity tool. If the stakes of getting a risk model right are that high, then shouldn’t that risk model be treated like enterprise software – with development standards, commented code, versioning, unit and integration testing?

It’s not as if enterprise software doesn’t have its share of messes. The CIO of a print publication recently told me that they did not think they knew all the different offers on the publication that were available out on the internet. Apparently, there was a link on some forum that was still allowing a special offer and they didn’t know how to turn it off! We recommended re-engineering the whole application and putting a rules engine in front of it.

Perhaps that’s what financial modeling needs. A rules engine that drives all the analysis below. The problem with Excel is that the formulas and the data are all mushed together into a blob of grey cells. They need to be layered. Data in one place, rules in another. If I want to project a company’s revenues by applying the average year-on-year growth in the last four quarters, that’s written in the rules area. The historical revenues are in the data area. If the gross profit is a fixed percentage of revenue, that too is written in the rules area. When I change the rule, the computation changes and not otherwise. There are no computations that aren’t in the rules.

When I review my model with my team members, we are just looking at the rules, confident that the spreadsheet itself is simply a manifestation of the rules and data. Once in a way, we do look at the data sets as well, just to make sure we are using the right ones and that they are current. And we test the model from time to time to see if it gives out expected results.

But this sounds more and more like a database application. And database applications need programmers. In the real world that we live in, no financial analyst will let a programmer stand between him and his model. But perhaps that is the challenge here. Can we build a rules engine that is enterprise strength, does not require a programmer and sits on top of the WYSIWYG goodness of Excel?

Excel is one of the most powerful applications of our times. It is the killer app in MS Office. It is ubiquitous and everyone who will ever build a financial model already knows how to use Excel. Which is why we do too much with Excel. Which is why we end up betting hundreds of millions on the backs of black box Excel financial models. This needs to change.