Meta Analysis

National Center on Women and Information Technology is wrapping up its summit today. I’ve attended much of the first two days, and the presentations on research and projects related to women’s opportunities in computing have been some of the best I’ve encountered anywhere. Lots to write about in those presentations! Let me begin with a little story about two remarkable programmers, Lucy Simon Rakov and Patricia Palombo.

Lucy Simon Rakov and Patricia Palombo were the recipients of the NCWIT Pioneer Award, and girl, were they ever pioneers! These two women were programmers for the Mercury space program, the first to send a person into space and home again. They did all of this with about 120 KB of raw computing power! (The next time I hear some would-be Steve Jobs tell me his code is elegant, I’m gonna laugh in his face.)

Mark Guzdial has a nice post on these great women on his Computing Education Blog:

NCWIT Pioneer Awards to two women of Project Mercury: Following their passions http://bit.ly/Lslrio

In the course of every day work, and everyday life, education pays back in ways other than cash. This is true for all sorts of learning, including reading, life experience, professional education and so on, but I’m talking about formal education for the moment.

One of the things that I have found most satisfying about a career in analytics is the opportunity to work with a diverse group of people, many of whom devote their lives to work that benefits all of us, worthwhile work that I might never be able to do myself. So many people invest a career in something others never think about. Indeed, the whole point is that others shouldn’t have to think about what they do. It’s a pleasure to meet these people, and to have something to offer to support their efforts.

At manufacturing plants galore, I’ve helped in the process of making better bottle caps, food wrap and beer, quieter medical diagnostic equipment, and more dependable electrical connectors. It pleases me to open a bottle and know I had a little hand in making sure it will open when I want it to open, and not when I don’t. It also pleases me to know that I have helped surgeons understand factors that may help people avoid amputation, and have greater likelihood of good surgical outcomes. When I see a client opening a new store in my community, it’s nice to know that my guidance for the store’s market research program helped make that possible. I owe all these experiences to formal education.

For every story we hear about a guy (it’s always a man, have you noticed?) who dropped out of school and made a fortune in business, there are 10,000 more guys and gals who dropped out and didn’t make a fortune, don’t have careers that make them proud, aren’t getting by very well, let alone contributing to society as much as they might prefer.

My college education has bought me more than a salary and more than opportunity. It actually does what it is supposed to do – provide me with broad knowledge which helps me to be effective in the workplace, active in my community and adaptable in a changing world. It disappoints me to see educated people downplay the value of education, minimize it to no more than a salary, or suggest narrowing education to purely vocational scope. If your education has meant more than a paycheck to you, speak up about it, speak loud, and don’t let the BS artists drown your voice.

Yesterday I wrote about innovation, with a little jab about the glamorization of dropping out of college to start a business. I have a lot more to say about education.

Lately, I’ve heard so many people talking down the value of formal education. At Ignite Chicago, Paul Pagel, Cofounder of 8th Light, gave a lightning talk on reasons not to go to college, and said he hoped he’d succeed in talking a few people out of it. Paul favors what he calls a “modern apprenticeship.” A young Ivy League graduate I met recently is angry that Ivies don’t top the chart for return-on-investment (greatest pay in relation to college costs.) In fact, Ivy League schools rank quite high in those listings, but the top slots went to Harvey Mudd, Caltech , MIT and Stanford. And then, there’s investor Peter Thiel, who is funding startups for a handful of young entrepreneurs, with a condition that they may not go to college while they are in his program.

Yeesh.

Now I’m as material as the next girl, and I make no bones that financial independence is important to me. I’ll fight for equal pay, and bend over backwards to encourage young women to choose professions in science, technology, engineering and mathematics, largely because these are fields that pay well. I’m also in favor of providing good vocational education in the public schools. But education returns more than just a paycheck!

When I was an undergraduate, many of my classmates were what they now call “non-traditional” students. Traditional students were people who had came in directly after high-school. Everyone else was “non.” My class-mates included people whose motivations were largely practical – people who had taken a break coming back to start or finish a college education, homemakers preparing to support themselves when their inevitable divorces came to pass. There were also many retirees. Unfilled seats were open to seniors to audit (attend to learn but not receive college credit) classes, free of charge. But not all the seniors were auditors.

Those students had already worked a lifetime, could afford retirement, and were spending their retirement earning college degrees for the first time and paying tuition for the privilege. They had worked and sacrificed to provide their children a college, and now, though it would never pay them a cent of cold, hard cash, they were investing time and money to enjoy an education themselves. Their presence added to the richness of my experience, and encouraged me to think of education as much more than just a way to make money.

More on this tomorrow…

I’ve been attending business launch presentations for twenty-some years, and it’s stunning for me to see the trivia that passes for innovation these days. If I hear one more entrepreneur crowing about an iphone app to remind me to remind Grandma to take her medication, my brain will implode. Well no, it won’t, but I will be tempted to stand up and shout, “A mobile phone app is not innovation!” Apps use technology, but they don’t push it forward.

Not every business needs to be innovative. Most do not! Take, for example, Hooters. They have built an empire on the time-tested fundamentals of beer, burgers and breasts. No news there, but they make a lot of money. They prove that a business can be successful with nothing more innovative than chicken wings and hot sauce, served by an attractive woman – the oldest value-add in the business world. If your business isn’t innovative, that’s A-OK, but let’s not kid ourselves, about it, shall we?

When I heard a VC pitch about flat screen technology in the 80s, that was innovation. It was so new and expensive that the only commercially viable applications were military, like use on a submarine, where space is precious. When I heard a prototype interactive voice recognition system just a few years later, that was innovation, though today I might wish to do without it. And let me tell you, this stuff wasn’t developed by college dropouts working in a garage with a little seed money from Mom. It was the result of concentrated teamwork by people with serious formal education.

Now let’s talk about some of the ultimate in modern innovation – the transistor, the vacuum tube and information theory, as well as thousands of other remarkable works of invention, all came out of one great center of innovation – Bell Laboratories. If you want to know a little something about how serious, full-strength scientific and commercial development happens, you must read The Idea Factory: Bell Labs and the Great Age of American Innovation. What a page-turner for business and technology fans. You’ll never look at a tech pitch the same way again.

New piece in a new publication: My trans-analytic Voyage: Text Analytics on Both Sides of the Atlantic contrasts my observations at analytics conferences in the US and Europe.

While in London recently, I attended some of the Big Data Week events. One big draw was a Community Meetup featuring a panel of Big Data celebs. One question that came up: “How important is business knowledge?” To make a long story short, the panel did not band together and rush to the defense of business knowledge. There’s the difference between the philosophy of data mining, which was created to empower the business person, and the emerging culture of data science.

As nearly as I can recall her words, bit.ly’s Hilary Mason said that smart people could solve any problem. She did use the words “smart people,” and that was a real thorn in my side. Hilary is a force in the analytics community and she is much to be admired, yet my experience leaves me at odds with her on this one. It’s not that I doubt that people can adapt to new business situations or unfamiliar issues. On that point, I’m a believer. But it shouldn’t be done in a vacuum.

What’s wrong with letting an analyst dive into a problem without business knowledge? For one thing, it’s inefficient. The patterns that the analyst finds may not be meaningful – like predicting an event based on factors that only happen afterward. Or making assumptions that aren’t reasonable for the situation. And then, there is reinventing the wheel. Just recently I heard about a fabulous new analytic technology, complete with a group of fans, and I was dying to see it in action. After a huge buildup, I finally got to see the stuff in action, and my heart fell. It was nearly identical to something I worked with in the nineties that was a big flop with clients. The developers clearly hadn’t researched the history of their market.

It’s not realistic to expect that every project must be tackled by an analyst who is an expert in the business behind it. Maybe that wouldn’t even be desirable. But if the analyst doesn’t know the business, then working closely with someone who does is a must.

Graph Databases and Analytics

While in London, I attended a talk by Nicki Watt and Michal Bachman, two Open Credo software developers who shared their experience building a recommendation engine based on the Neo4J graph database. I asked why they had chosen that particular platform, and got a simple answer – they didn’t, the decision had been made before they came on to the project. But they did explain some useful things about what graph databases are and what they do well, not to mention what they don’t do so well.

Graph databases are said to be “schema-less”. They don’t have the relatively rigid structure that we expect in relational databases. Instead, they can store a wide variety of information, from numbers to video and more, organized in a relatively flexible structure described by a changeable graph. Neo4J is only one of many such databases, others that you may encounter include MondoDB, AllegroGraph and FlockDB. The advantages of the graph structure include rapid creation of and changes to a database, and excellent performance for many routine operations.

What graph databases aren’t made for is analytics. They don’t lend themselves to operations that might require aggregating large quantities of data, or random sampling, or classical statistical analysis. Analytics can easily bog graph databases down to a standstill.
There are practical situations where you can work around these limitations and end up with good results. So, for example, when making a recommendation, the trick might be to use a relatively small number of easily accessible cases and choose the best among them. Think of how people find partners – getting to know the people in the vicinity and evaluating them as potential partners, rather than traveling far and wide in search of an optimum mate. Another strategy includes serving an old result while waiting for a new one to be calculated, so the user never experiences a long wait for response. Graph databases perform well for transactional applications and those where a quick analysis of a modest number of similar cases fills the bill.

So what about classical statistical analysis, data mining and exploration? What about operations research? My take is that we will still do best to keep as much of that work away from transactional systems as possible, and that planning to create and maintain a relational database for analyst use should be part of the process when architecting new applications.

Text Analytics Summit Europe took place April 23-24, and I had the opportunity to speak there. My presentation, “Cross-lingual Text Analytics: A New Frontier in Linguistic Technology”, was based on my article of the same title that appeared in Multilingual magazine earlier this year. In that talk, I explained the meaning of “cross-lingual” text analytics, the process and why translating text to feed into English-language text analytics tools is undesirable.

The London group was much more motivated to talk about languages other than English than any audience I’ve encountered in the US! There were several other speakers discussing issues related to non-English text analytics, including some case studies. And the discussion during breaks and such was very different from the US. Americans need to smell the coffee and realize that if we don’t rise up and get into customer engagement and text analytics for languages other than English, we’ll be losing business to international competitors who will get there first. Believe me, they have a huge head start!

Back from a long road trip and recovered from jetlag, I must now get back to writing! Just finished a piece for Language Technology News http://langtechnews.hivefire.com/, will post a link when that’s available. In the past few weeks I have given three presentations on text analytics – in San Francisco, London and Chicago – and I’ve heard many other interesting speakers, so I have some new stories to tell over the next couple of weeks.

Next up – I’ll be giving the keynote presentation at Text Analytics Summit Boston in June! http://www.textanalyticsnews.com/text-mining-conference/ You can read the conference agenda here: http://www.textanalyticsnews.com/text-mining-conference/conference-agenda.php. Hope to see you there!

Social Media Analytics Summit, April 17-18, San Francisco
Capitalize on Multi-lingual Social Media Analytics

European Text Analytics Summit, April 23-24, London
Cross-lingual Text Analytics: A New Frontier in Linguistic Technology

Chicago Web, Game and Social Media Analytics Group, May 2, Chicago Free!
Crossing the Language Chasm: Extracting Information from Foreign-Language Text

Predictive Analytics World, June 25-26, Chicago
Cross-Language Text Analytics: Overcoming Language Barriers