Analytics, Big Data, Crosswords and Agatha Christie

28 Jan

Miss Marple
I had a Eureka moment this morning (NO, I did NOT jump out of the bath midway through it – I’ve often wondered what Archimedes’ family’s reaction would have been when he ran out butt naked btw – and anyway on Monday mornings it’s quick showers not luxurious baths!). But it suddenly struck me that I’m like Miss Marple – I kid you not, I am! (For those for whom whodunits belong to the dim and distant past – Miss Marple was the 70 something spinster of Agatha Christie’s who solved many crimes in a as-it-turned-out-not-so-sleepy village called St. Mary Mead) I don’t mean I look like her (well, judge for yourself, though the number of wrinkles on my face will catch up to that on hers soon enough). But, Miss Marple solved difficult crimes because of her shrewd intelligence (yep thats me). The crimes she faced always reminded her of a parallel incident. She used analogies that lead her to a deeper realization about the true nature of a crime. Although she looked sweet (yeah, that’s me too), frail, and old, she feared neither the dead nor the living. She also had a remarkable ability to latch onto a casual comment and connect it to the case at hand.

This realization actually came to me as I was reading yet another novel by current favorite mindless reading – Nora Roberts (husband calls it trash – her credits call her Amercia’s favorite author) – and I suddenly connected dots on what she writes about – always smart sassy women as heroines, almost always older women with marriages in most cases behind them, fun ‘the S word’, almost always an element of the supernatural – but in an understated way, everyone likes to drink beer (yep, that’s part of why I like these books), most occupations are physically oriented – carpenter/ garderener/ dancer – maybe because its more logical to talk greater amounts of body display that way – these are romance novels).

This set me thinking of Miss Marple, and — would you believe it – analytics! And I figured that – what you do in analytics is exactly this – connect dots, draw parallels and then solve problems. (see my earlier blog on this)

Actually while in analytics it kind of becomes the whole point, this aspect of pattern detection is true of all facets of life:

– One of the first activities a pre schooler does is connect dots to make pictures

– All music is actually pattern making – there are notes in specific patterns that are set to specific time periods, and the combination creates something that really pleasing to the eye (on this “note” – did you know that Harley Davidson has actually patented the sound of its exhaust! There’s music for you!)

Kekule, in his dream, saw a snake catching his own tail, connected the dots – drew a parallel – and voila, you have the structure of the Benzene ring!

– In my aerobics class (yes, yet again) – that’s all Niru does, many patterns and combinations of patterns of steps – and when she goes over these patterns of steps again and again – a) they kind of get set somewhere in the nerve cells of the brain , and b) they can be combined in different permutations to afford variety and make the classes more interesting (for the regulars – and more difficult – for novices like me)

– The Fibonacci sequence was a result of someone discovering how fast rabbits multiply – My own Sita and Gita (the two rabbits who I thought were female but turned out clearly to not be so) demonstrated this gleefully – much to our shock – when they gave birth to 7 leverets a few weeks ago, or which – hold your breath – only 2 survived)

– All solvers of cryptic clues on crosswords, while are really well informed people coz they KNOW a lot of stuff, are also good spotters of patterns and solvers of these – the practice of solving them daily makes you really “get” the way in which a typical crossword writer will set his clues…

– Take a look at this picture I saw on facebook the other day that I found fascinating.
It prompted me to read about the Golden Ratio – and its fascinating how many (seemingly unrelated) things are set to that pattern of ratios – that makes for maximum aesthetic appeal

– Even our behavior is set to patterns – infact, apparently the way I write my blog posts has a pattern – my friend Avinash told me my posts remind him of Stephen Jay Gould – the way I weave different things together (OK, so I got to decide who i want to be – Stephen Jay Marple, or Jane Jay Gould!)

I could go on and on – but to come back to analytics, this is the skeleton of the discipline, and this is what all statistical tools and techniques are programmed to do – detect the patterns, in order to show correlations (read evolve the models) and then solve problems. Having said that, what really makes the models work is the intelligence behind it that is giving it the a) context and b) applicability, and that’s really coming from the people who are using it (maybe that’s why a good statistician may not always be a good businessman but a good businessman (should I say businessperson) does in general appreciate the point of statistics!

This then brings me to a term that is getting a lot of hype and a lot of copy – Big Data. While that is a subject that I think I will really deal with in another post (hmm – this list of “will do laters” is getting longer by the day – I should start going back and reading my own posts), in my mind, the hype on Big Data has to be taken with a little bit of salt and lots of common sense. Big Data is the MEANS to an end – NOT the end in itself – it gives you the armoury to deal with unprecedented volume, velocity and variety of data – but the USAGE or APPLICATION of this data is what businesses really need to focus on – specially those that are not tech players. There is a need to, therefore, take heads out of clouds (yes, please note the intended pun 🙂 ) and get down to reality – explore bigdata/ cloud/ hadoop what have you – but remain focused on WHAT you can do with it – HOW MUCH you can use it, and more importantly, HOW can you help your organization use it for what it really needs it for. That’s where most organisations need to focus their energies on as they use BIG Data, not so much on the technology itself.

Meanwhile, for those of you who do like the challenge of detecting patterns and solving problems, here are a few…

—-, —-, 275, 445, 720, 1165, …
516, 1440, 1932, 1611, 2535, 3027, —-, —-, …
516, 1324, 734, 507, 1315, 725, —-, —-, …


4 Responses to “Analytics, Big Data, Crosswords and Agatha Christie”

  1. Avinash March 31, 2013 at 9:51 pm #

    Another interesting post Sangita. I like the patterns in your posts – They are apparently random but actually structured at the same time 🙂 So bring on more Miss. Wrinkle-less Marple

    And yes pattern recognition is what defines intelligent life. Heuristics is something that humans knowingly/unknowingly worked for with thousands of years now. Pattern recognition along with brute force solutions can make computers do great things with ‘Big Data’. The question is will these machines pick up spurious correlations and patterns. Humans do all the time – Stock Market Tech Analysts are prime examples.

    • joshsang April 1, 2013 at 8:14 am #

      I think what makes human analysis score over comp is the Added element of ‘gut feel’/ judgement – despite latest tech, that is really hard to replicate

  2. Avinash April 1, 2013 at 8:28 am #

    Yes and that ‘Gut’ feel is an Art mostly right? It like a doctors diagnosis almost.

    I have seen many spurious corelations being made. Even causality implied for unrelated events.


  1. Stereotypes; and the Bucking of Them (Or Why I #$% Love Science) | joshsang - April 8, 2013

    […] while ago, I wrote a post on patterns – and how solving for those patterns makes for a lot of fun and excitement. This one is […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: