My book of the year

It wasn't my most enjoyable read of the year, nor the funniest or most moving, but for its sheer impact and clarity my book of 2013 is the snappily titled Big Data: A Revolution That Will Transform How We Live, Work and Think. Jointly written by Oxford professor Viktor Mayer-Schonberger and Kenneth Cukier of The Economist, the book works as a compelling argument for the power and potential of big data analysis.

For someone like me who works in market research, the shift from sample-based methodologies ("we got answers from 2,000 people who represent the whole UK population") to big data analysis ("we looked at all the available data and found these correlations...") is both exciting and challenging. I knew that before I read the book. But Mayer-Schonberger and Cukier provided the clearest and most comprehensive explanation I've had yet of just how different and radical big data is set to be.

So one resolution for me in 2014 is to find out more about big data and take my first steps into it. Mayer-Schonberger and Cukier have told me a lot about what big data is, why it's revolutionary and how it will change the way I work. What they haven't told me is how you actually "do big data", so that's the question I'm currently seeking to answer.

Lou Reed: linger on

Lots of fine tributes to Lou Reed last week. I've been listening to the man's music for 25 years and love it still.

So here's my Lou Reed top 10, in no particular order...

Sunday Morning
Satellite of Love
What Goes On
Pale Blue Eyes
Stephanie Says
Venus in Furs
Coney Island Baby
Dirty Blvd
The Murder Mystery
Last Great American Whale

Six of my ten are Velvet Underground tracks. So I guess I feel - like most people - that the early years of Reed's career were his peak. But you have to love how the guy kept making uncompromising music on his own terms, into middle age and beyond. I remember when New York came out in 1989. To me and my friends at school it was one of the most angry and sharply political albums we had ever heard. It's also a terrific collection of rock songs.

And that for me is the key to Reed's longevity, the real reason I'm still listening to him: superbly-crafted songs that are full of invention. Strip away the drugs and the sleaze and much of what the media has focused on in the wake of the man's death, and at the core of everything is the quality and creativity of the music. That's why he will surely linger on.

Young people full of ability

During recent years, I've been fortunate to work with some amazing and inspiring people. Last week provided yet another example.

The occasion was a discussion group that I facilitated with seven young people, all of whom have a visual impairment (i.e. they are blind or partially sighted to a degree that presents challenges when getting out and about). In many ways the group were typical of Generation Y: sociable, adept communicators, pragmatists rather than idealists, very much online and digitally networked, keen to get out, meet new people and have new experiences.

But what really struck me was how these young people - all of whom have a disability - were willing to consider the needs not only of themselves, but also of others. For them, there are older, more isolated people with a visual impairment who also require support and services but whose needs are distinctly different from their own. Some are even volunteering with charities in their area to provide help to such people.

All this reminded me of a couple of important points. Firstly, we live in a climate where people who claim any sort of disability benefit are liable to be labelled scroungers. But in my experience, young people with disabilities have no desire to scrounge. They want to play a full part in society. They want to fulfil their potential in life, in education and in work. They are also prepared to give back and to help others if they can, as the participants in my discussion group last week showed.

The second point has more to do with age than disability. The cohort of young people that is currently entering - or attempting to enter - the job market has a huge amount going for it. Their ability to communicate and their openness to change are refreshing and full of potential. They are also much more giving than is generally acknowledged. For example, recent stats from the UK Cabinet Office suggest strong levels of volunteering among younger age groups.

But this cohort also faces significant challenges, given the realities of an economy that offers a lack of entry-level jobs and rising costs of living. So let's hope employers and the government work to give young people a break and support them in progressing towards their goals in work and life, whatever their abilities.

Bad attitudes at high altitudes

This week has brought news of fighting between groups of climbers and guides on Mount Everest. My impression is that such high-altitude altercations have been brewing for some time.

Ever wondered what it's like to be on the world's highest peak, without incurring the massive expense and life-threatening risk that such endeavours can entail? If so, I highly recommend Into Thin Air by Jon Kracauer. This gripping first-hand account of the 1996 Everest disaster shows us how Everest has become perilously over-crowded.

It's no surprise that the bad atmosphere up there has got even worse over the years. It's also undeniable that the over-crowding, coupled with the commercial imperative to get paying customers to the top, has made an already hostile environment even more dangerous.

One thing's for sure, I'm going nowhere near Everest. Big money and big mountains simply don't mix.

Statistics under the influence

To anyone like myself who has worked in market research, it comes as little surprise that respondents to surveys have been under-reporting the amount they drink (see here). Come to think of it, to anyone with any experience of human nature, this should come as no surprise.

When quizzed in a survey, many people provide what they feel to be acceptable answers, rather than giving their best shot at the truth. This is the case even when the interviewer is a complete stranger (or when the survey is online) and they've been assured that their answers will be treated anonymously.

The alcohol stats are just one example of this. Other surveys routinely tell us that up to 70% of the UK population give to charity. I doubt the figure is so high.

But my favourite examples are surveys exploring personal relationships. They tend to tell us that men have had up to twice as many sexual partners as women. This is in fact mathematically impossible. If two groups are "interacting", then the average number of interactions is the same in each group*. The reality is that heterosexual men and heterosexual woman have, on average, the same number of sexual partners in a lifetime. All the surveys tell us is that, true to stereotype, women as a group prefer to underplay their promiscuity, men to exaggerate it.

The lesson of all this is not that we should be sceptical of all market research. But thinking carefully about where statistics come from and what they really mean should be standard practice for us all, market researchers included.

* OK, just in case there are any genuine detail freaks reading this.... I'll admit it's possible that other interactions outside UK man + UK woman (e.g. men having relations with other men or with non-UK women) could have some effect in boosting the figures for the male population. But that couldn't possibly explain such large discrepancies in the data. Similarly, we might acknowledge that some alcohol, once purchased, never actually gets consumed. But again the effect is not big enough to explain the big gap between booze sales and what people say they drink.

Deception in cycling: the truth will set you free

Time for a book recommendation. Amid all the hubbub around Lance Armstrong's overdue "confession", I was reading The Secret Race by Tyler Hamilton and Daniel Coyle.

The book has two great strengths. One is Hamilton's candid, exhaustive retelling of the drug-taking and other cheating that he took part in as an elite bike rider. If you want to know what the peloton was really up to, and just how widespread and routine it all was, there are answers here.

The second reason I will remember this book is more surprising. Hamilton regrets what he did, but he is bold enough to pose thorny moral questions about his actions. He describes how, as a rising star in professional cycling and a member of Armstrong's US Postal team, there came a point where he had a choice: he could take performance-enhancing drugs or he could find another job. Such was the extent of drug-taking by the 90s, that you simply couldn't keep up without some EPO, testosterone, or whatever your particular cocktail came to be.

The Secret Race doesn't ask us to excuse the drugs or the blood transfusions or the wholesale deception that took place. It does, however, get us thinking about what we might have done, had we been in Hamilton's position and faced with the same choice. At the same time, Hamilton's account of the anguish he has experienced - physically at times but primarily mentally - as a result of drug-taking and blood transfusions leaves us in no doubt that cheats pay for their crimes, one way or another. And, of course, he was eventually caught, exposed, vilified, banned.

Faced with his fateful choice again, Hamilton would I'm sure opt to "just say no" and stay honest. Ultimately, his story encourages us to do just that, whatever the immediate cost.

Trust Barometer puts high pressure on leaders

The annual Edelmann Trust Barometer was published this week. It received noticeably less attention from the media than in recent years, no doubt because it doesn't show any dramatic year-on-year falls in the trust that the public feels for institutions and business. Nonetheless, the survey continues to underline the high levels of scepticism and mistrust that now characterise our perceptions of political and business leaders.

One finding that inevitably stands out is the attitude towards banks and bankers. Trust in the banking industry in the UK has halved in the past five years, not surprisingly. Of the 18 countries surveyed, only Spain and Ireland have less trust in their banks. The fact that trust levels are twice as high in the US as in the UK must surely say something not just about the way the financial crisis has been reported and perceived on each side of the Atlantic, but also about the healthy degree of scepticism that naturally occurs among the British public. It would certainly be hard to argue that US banks have behaved any more honourably than their UK counterparts (see my post The Flaw: money-grabbing and misery in modern America).

The Trust Barometer also underlines how the events of recent years have cemented the public's mistrust of both political and business leadership, at least in the developed world. CEOs are little more trusted than government officials. And once again, the UK displays one of the highest degrees of scepticism among the countries surveyed.

Edelmann's analysis concludes that, in order to regain the respect and trust they have lost, leaders need to throw out the old mantra of top-down authority and embrace the "new mandate" of "inclusive management".  I believe that such a shift is healthy not only for leaders themselves but also for the performance of organisations. For one thing, many of the damaging leadership mistakes of recent years - from the ridiculous bets on subprime mortgages to insane corporate mergers (RBS-ABN, HP-Autonomy, etc) to the Iraq War - have happened because people with power were able to push through decisions that surely wouldn't have stood up to reasoned, informed and independent group scrutiny.

More diffuse models of leadership will gradually emerge. Chains of command will give way to webs of influence. The challenge then becomes to ensure that organisations don't drift into a sclerotic, directionless mess that stifles decision-making and accountability. Let's hope the finest minds in organisational development are at work on this very issue as I type.

In the dark midwinter

Few people in the world get a darker winter than the one we experience in the UK. We are so far from the equator that the sun for several months of the year has hardly any power in it at all. And as we approach Christmas, it just about gets above the horizon at midday. Unless you live in Northern Russia, one of the barely-populated regions of Canada, Greenland, Alaska or the Nordic countries (or you are stationed in Antarctica), you get a brighter winter than we do here in mainland Britain.

There is more and more acknowledgement of the effects of weak sun on our mental state. Seasonal affective disorder (SAD) is now recognised as a genuine condition, and "SAD lamps" are doing great business this winter.

Meanwhile, anyone who's done biology at school can tell you that a lack of vitamin D, which our bodies generate through contact with sunlight, can lead to rickets. But the list of serious ailments linked to insufficient vitamin D keeps growing. Recent research has even reinforced the link between vitamin D deficiency and Multiple Sclerosis (see the MS Society for more information).

Given that the winter sun in the UK is not strong enough to enable our bodies to create the vitamin D we need, it's vital we get the vitamin from other sources. There's some in oily fish and eggs, as well as fortified breakfast cereals and margarine. But given the overwhelming evidence around vitamin D, surely it should be added to more staple foods in order to ensure the population gets what it needs? It would even make sense to add it to mince pies, yule logs, christmas pudding, advent calendar treats and anything else we tend to guzzle during the dark midwinter. Fortified brussel sprouts anyone? 

Cricket coaching: it's not just cricket

During November I've completed UKCC1, a short course for cricket coaches. The whole experience was impressive and really worthwhile. The standard of tuition for one thing - as delivered by Hampshire Cricket Board in my case - was outstanding. The tutors were passionate and knowledgeable about their cricket, and skilled communicators to boot.

The course content was spot on too. It went right back to basics. So we looked at some fundamental skills for young batters, bowlers and fielders. However, most of the time was spent working on the best ways to explain, demonstrate and impart such skills. All of which helped to remind me of some inescapable rules around effective communication and learning, such as:

• Keep it simple - a young audience can only take in so much in one go, so it's best to focus on a single aspect of a technique (the position of the hands, for example) rather than trying to get everything across in one go
• Sometimes you need to talk a lot less than you might think - keep explanations concise and when demonstrating a technique, keep your mouth closed and let the kids take it in
• Plan and prepare for a coaching session even more than you think is necessary
• Make it fun, obviously
• Make sure everyone is involved and can learn and enjoy, regardless of ability.

First and foremost, I took the course as a first step in learning how to coach young cricketers and get them excited about the game. But I must say that, as you might guess from the list above, I got a useful refresher in how to communicate with any group, no matter the age and no matter the topic in hand.

A fallen idol and lessons we could learn

Lance Armstrong has been deservedly stripped of his titles and reputation. He stands to lose a lot of his ill-gotten money as well. Much has been written about Armstrong's crimes and what cycling and sport might learn from them. But the comment that stood out for me went much further and pointed to implications for us all.

The piece in question, The voodoo cult of positive thinking, was penned by Ed Smith for the New Statesman. It's well worth a read. Smith's point is that Armstrong's demise underlines the folly of the modern world's obsession with willpower, our belief that if we want something enough and focus our energies strongly enough on achieving it, we can make it happen.

Smith compares Armstrong to another tainted exponent of focus and drive, Tiger Woods. I'd say he's also cycling's equivalent of Richard Nixon, Silvio Berlusconi and Bernie Madoff. Such was the force of his personal conviction that the ends came to justify any means necessary. The world and the people around him were just tools to be manipulated in order to secure his victory and power. Many of us have worked with people who share some of these traits. They are often, like Armstrong, charming, charismatic and popular. But they can also be extremely destructive. No doubt some of the bankers who arrogantly failed to spot the flaws in their massive gambles on sub-prime mortgages (see my blog piece on The Big Short) would fit this category too. 

The cult of willpower doesn't only apply to immoral alpha males and females. As Oliver James points out in his much-read book Affluenza, many people in the English-speaking world (and increasingly elsewhere) are dangerously stressed by the gap between what they believe they should achieve in their lives and what they are actually, realistically, able to achieve. Armstrong perpetuated the lie that, by applying our willpower, we can overcome all obstacles and challenges on the path to personal success. His unmasking as a serial cheat should help us to chill out a bit, stop obsessing with over-achievement and, in the words of the serenity prayer, accept the things we cannot change.

Armstrong - like other great deceivers - will not be forgotten. But given what we could learn from his story, perhaps that's no bad thing. The final irony is that an individual once hailed as an inspiration, an example to all, has now become a sobering lesson in how not to live and behave. 

Welcome back KP

We've all heard quite enough about Kevin Pietersen's fallout with the rest of the England cricket team. With an overseas tour about to begin and KP back in the squad, it's time to get on with playing the game again.

Even though I can't defend his behaviour in recent months, I for one am pleased to see Pietersen back in the England fold. This is simply because he is an exciting and high-quality cricketer, the kind the game needs and the kind that can win matches for his team.

The stats stack up favourably for Pietersen. As the chart shows, he has the best batting average (49.48) of any long-term England batsmen of the last 25 years.

If we look at strike rate, it's not hard to see why Pietersen excites the crowds and has the ability rapidly to take a game away from the opposition. Of the players in the chart, only KP and David Gower have a strike rate over 50, meaning they score at better than 3 runs an over in tests. Gower's strike rate was 50.59. KP's is 63.26, so he ticks along at almost 4 an over.

An England team that includes Pietersen scores more runs and scores them faster, and is therefore more likely to win games. Unless KP's presence in the dressing room has a seriously negative effect on the performance of other players, he should be the first name on the team sheet.

The Flaw: money-grabbing and misery in modern America

I have finally watched the 2011 documentary The Flaw. Pulling together interviews with a range of academics and experts, as well as people involved in and affected by the sub-prime mortgage fiasco, the film works as a highly watchable insight - albeit a polemical one - into the causes and consequences of the financial crisis in America.

A few ideas from the film stood out for me:

• Asset markets are naturally prone to bubbles because they fail to obey the normal rules of supply and demand. People tend to buy more of an asset - equities, property, etc - as its price rises, driven by the belief that the price will continue to rise and that they will therefore make a profit when they sell the asset. This fact implies that stock market and property bubbles are pretty much inevitable in market capitalism.

• Asset bubbles and income inequality are intrinsically linked. As assets inflate in price, wealthier people have the spare cash to invest in assets, so they tend to be the ones to profit most from asset price increases.

• Income inequality in turn inflates asset market bubbles and damages the real economy, since the very wealthy spend proportionately less of their income, compared to the average citizen, on real goods and services and proportionately more on assets.

• Over the last 30 years, banks have, as we all know, put more and more of their resources into financial instruments based around personal debt (notably mortgages and credit card debt, and derivatives thereof). This is because these instruments offer a quick return. In other words, bankers have - surprise, surprise - been trying to maximise their profits. The consequence, however, is that banks have come to see the real economy - i.e. factories, small businesses, infrastructure, etc - as a less attractive destination for their capital.

This last point helps to explain not only the untenable rise in obscure financial engineering that led to the financial crisis itself, but also the drop-off in investment in the real economy and the decline in manufacturing. This link holds true, I suspect, not only for the US, which is very much the focus of The Flaw, but also for the UK.

You have to wonder whether the catastrophic events of recent years will convince banks to shift their priorities back from obscure financial instruments to lending in the real economy. Sadly the evidence from lending figures so far suggests that this isn't happening. And herein lies a reason why we all need the bonus culture to be dismantled for good. So long as bankers are motivated to seek out fast profits, they will do so in the way they know best, i.e. constructing and trading credit default swaps and the like. Thus there will be less money available to support businesses and projects that actually employ people in a sustainable way and create wealth for the broader population.

My one cheerful thought to come out of the film was a realisation that, in the UK at least, things could have been worse. The Flaw shows how the bursting of the US house price bubble had miserable consequences for individual home owners burdened with mortgages that they never should have been granted. Many now owe considerably more than their houses are worth. This hasn't happened to the same degree here in the UK because house prices have not crashed as dramatically as they have in the US, not to mention Ireland, Spain and so on. For this scrap of good fortune, we in the UK should, for now, be grateful.

Memories of The Games

It's hard to argue with the widespread view that the London 2012 Olympics and Paralympics have been a resounding success. We've had so many memorable performances, an even better than expected showing from the home team and very few administrative cock-ups or other embarrassments for the organisers: cue a justified feeling of a job well done among all those involved.

For me, a couple of memories stand out. Firstly, there were the crowds and the atmosphere at the two events we attended. The number of people up Box Hill for the men's cycling Road Race was amazing. It's not an easy place to get to, especially when all the surrounding roads are shut for the day. But the great British public turned out in force and thoroughly enjoyed themselves. Even the police joined in the relaxed, celebratory spirit of things, with some of the boys in blue even high-fiving the crowds lining the route from their passing motorbikes.

Similarly at the Paralympics, the main stadium was packed and the support never anything but totally wholehearted. As at the Road Race, the loudest cheers were reserved not just for the winners but for those doing their very best at the back of the field. An Iranian cyclist who trailed the peleton by a bigger and bigger gap on each of the nine laps of Box Hill received a bigger and bigger dose of vocal support each time he passed us. A runner from Djibouti who took over 11 minutes to complete the 1500 metres brought the Olympic Stadium to its feet in applause.

Maybe there is something in the British psyche that makes us love the underdog, or maybe it's just human nature. Either way, it was a warm and rather moving way to show that doing your best is really all you can do, that winning gold isn't everything and that, despite the fervour for Team GB, we welcome all comers to these shores.

One unmistakably British ingredient in the Olympic mix was the Opening Ceremony. My expectations of it were so low, but what an extraordinary thing it was. Thoughtful and thought-provoking, radical and vibrant, I loved it. Beyond the sheer spectacle and ingenuity of it, the show won me over with its focus on some the things that make Britain what it is: the pioneering resourcefulness that kick started the industrial revolution, our singular creative output, our sense of humour, even our national health service.

If the opening ceremony opened a few people's eyes to some of the things that are uniquely British in our past and our present - things that we should treasure or in some cases try to rediscover - then it was a few million quid well spent. Indeed I hope that a bit more national self-confidence will be one positive legacy of The Games. If nothing else, surely we've shown that we can design, build and run a hugely complex event with great panache. And surely the oft-voiced, defeatist assumption that stuff in Britain tends to go wrong now sounds more than a little hollow.

Recommended reading - Michael Lewis on the banking crisis

I read Michael Lewis's excellent book The Big Short while on holiday. Although not as funny as his most famous work, Liar's Poker, it's a compelling, revealing and at times scary read.

For me, Lewis brought much-needed clarity to the reasons behind the cataclysmic - not to mention costly - banking failures of recent years. I would highly recommend the book for anyone who feels they don't completely understand what the financial crisis was/is (which is surely pretty much all of us!).

I finished the book with a number of strong impressions of the convoluted, out-of-control investment banking world that Lewis dissects. The over-riding sense is of the sheer folly - nay gross stupidity - of banks' decision-making and investment choices around sub-prime mortgages and the many exotic financial instruments that were spawned from them.

All this left me more convinced than ever that governments' financial support for businesses like AIG, RBS, Bank of America and so many others was simply wrong. These banks were not victims of circumstance, nor were they brought down by the actions of a few rogue employees. As institutions they made appalling business decisions. They deserved to fail, and the world economy would surely be better off without them.

Lewis is very adept at telling his complex story through a number of fascinating characters. In particular, he focuses on the handful of individuals who were not only smart enough to spot the sub-prime crisis on the horizon but also shrewd enough to bet big on the crisis with their investments (hence the title of the book). What's interesting about these people - the likes of Steve Eisman and Dr Michael Burry - is that they aren't your standard Wall Street/City types at all. They are outsiders, swimming against the tide. They are wired to "think different".

Burry, who stuck to his bearish convictions despite huge pressure from his clients to take more conventional investment positions, has Asperger's Syndrome. Of course when the sub-prime crisis hit, he was proved very much right, and he made fortunes for himself and his doubting clients. What a telling lesson in the value of diversity and different voices within society and within organisations.

If more people had been willing to pay attention to the views of "oddballs" like Burry and Eisman, rather than following the received wisdom emanating from Goldman Sachs, Morgan Stanley, the ratings agencies and indeed governments, perhaps this whole mess could have been avoided.

The age of sport

I was interested to learn that Roger Federer was the first Wimbledon finalist since Jimmy Connors to have passed his 30th birthday. Meanwhile, 2011 Tour de France winner Cadel Evans was the oldest rider to seal the yellow jersey in Paris in the post-war era. This year, at 35 and a half, he looks well past his prime.

So what is the peak age in sport? I looked at details of the current top 15 ranked players across 4 sports (using the final placings at the 2012 Tour for cycling, and official world rankings for the other sports). That analysis produced the averages shown in the chart below. And the optimum age to excel on the international sporting stage, based on my not very scientific analysis, is... 30 years 7 months.

It's surprising to see that there is not much divergence between the mean ages of elite players in these 4 sports, despite their very different physical demands. You might expect golfers to be considerably older than stars of sports that require more speed, reaction and agility. But, at an average age of 33, the world's top golfers are only a couple of years older than their counterparts in cycling and cricket.

In fact, the youngest player in the entire group of sportsmen I analysed was a golfer, Rory McIlroy, at 23 years 2 months. That said, the oldest - Steve Stricker (45) - is also a denizen of the fairways. Clearly golfers can hope for greater longevity than cyclists, tennis players or cricketers.

Tennis stars are the relative babies of the group. Roger Federer, who is 31 in a couple of weeks, is the oldest player in the ATP top 15. The narrow spread of ages in the top 15 - there are only 7 years between youngest player Marin Cilic and the great Rog - underlines the achievement of players like Federer and Sampras in winning grand slams over such long periods.

Cricket presents some interesting contrasts. Batting definitely favours maturity. The world's top ten batsmen are all 27 or over, and their average age is 32. But fast bowlers average a comparatively youthful 29 and a half. And not surprisingly, without the same requirement for pace and aggression, spin bowlers are a tad more senior at just below 32.

Meanwhile in cycling, Bradley Wiggins (soon to be Sir Bradley, then Lord Wiggo, surely) should have a couple more shots at Tour glory in him. But, at 32 already, you wouldn't bet on him winning in 2015 or beyond. In any case, I'm already relishing his battle with comparative youngsters Chris Froome and Andy Schleck - both 27 - in the 2013 race.