I have a dream ...

I have a dream. My dream is that one day those overpaid nincompoops who run many of our companies and organisations wake up to the importance of data, and start working with it accordingly.

If you're not persuaded of the importance of data, try imagining your organisation functioning without data (or its cousin, information, which is usually rooted in data) and see how far you get. No e-mails, no internet, no customer orders, no invoices. No telephone calls, no meetings, no discussions with colleagues, not even to discuss the weather, unless you're one of the very few organisations which is not affected by the weather (really, you'd be surprised).

How long would that situation be able to last? Minutes?

Why can't people understand the importance of data and its quality? Why don't we treat it in the same way that we treat other parts of our business? The very idea of an airline only maintaining its fleet when something went wrong with it would horrify all of us, but that's what we do with data. Few of us do not realise how preventing tooth decay not only saves us costly treatment and potentially a great deal of pain, but leaves us with far better teeth than any dentists ministrations could produce on badly maintained teeth. (Read Jim Harris' blog post on that topic here.)

So why do we wait until the CEO is told that $ 1 billion PROFIT was made instead of the actual $1 billion LOSS, with the resultant chaos, before we take data seriously? Clearly, unmaintained airlines falling from the sky make a greater immediate impact than data quality wrecks, but the results can be equally pernicious. Why must so many people waste so many hours trying to prove return on investment (ROI), when ANY and ALL data quality improvements are beneficial - I am yet to be persuaded that there is no return on any investment (in one form or another) on every improvement of data quality. Sadly, most businesses make money DESPITE their data quality, not because of it. (See Henrik Liliendahl Sørensen's post showing how simple it can be to show ROI here).



I have a dream of a revolution in data quality, where resources and focus are built into the prevention of data quality problems, rather than on trying to resolve them only when their detrimental effect becomes obvious; where as much control is put into data as is put into production, maintenance, finance, human resources and other aspects of organisations.

I have a dream. How long must it remain a dream?

IAIDQ Blog Carnival, April 2010

I've dusted down the blog today to host April's IAIDQ blog carnival for information/data quality bloggers, a look at some of the month's best blog posts.

In keeping with this blog's focus, I've decided concentrate on posts about data quality (as a data issue) rather than on business or other practices, or on personnel issues; so I've largely bypassed posts about persuading executives to invest in improved data quality, data quality tools within businesses, return on investment and the like, though this is no reflection on the quality of those posts.



We can start with Daragh O Brien, a man who rarely utters a word I don't agree with, and he utters them always with great aplomb. His recounting of the difficulties of matching and moving his contact data from 'phone to 'phone in his post Do we have an App for that? shows well how real people have to grapple with data quality issues on a daily basis.

Also on the theme of data everywhere in our environment, and certainly not just within businesses, is the good Jim Harris' post Data, Data Everywhere, But Where Is Data Quality?. Jim, an obsessive compulsive blogger and independent consultant, speaker and writer looks at the avalanche of data we contend with daily, why its quality matters, and how we need to manage it.

I can't let the carnival go on without a mention of Dylan Jones, editor of Data Quality Pro and a prime mover in getting the importance of data quality recognised. Dylan's post is an Expert Interview with Jill Wanless (author of the Data Quality from the Ground Up blog). So I get to mention two data quality scions in a single paragraph.



I won't generally eat anything more exotic than a chicken and mushroom pie, and you'd have to tie me up and use a cattle prod to get me anywhere near IKEA, but Henrik Liliendahl Sørensen has continued his posts about data diversity with Data Quality and World Food. Henrik's work with, amongst others, Omikron, has given him a good understanding of the importance of understanding global diversity, and he blogs about it regularly.



Finally I'd like to make an honorable mention of Julian Schwarzenbach's final entry in his series The Data Zoo - How data personalities interact. Though I'm breaking my rule here of avoiding blog entries which revolve around data quality within businesses rather than as something generic, Julian's work in sifting and identifying the personalities involved in data quality work is a remarkable series, though I'm stuck with the feeling that I actually belong in each one of the nine categories identified, which is a trifle worrying ...

Apologies again to the writers of the excellent blog entries I had to exclude from this carnival, and I'm looking forward to next month's batch already.

Data Quality is a DATA issue

In the blogosphere, on Twitter and at conferences I often come across mentions and discussions of whether data quality is a business issue or whether it is a technical/technological issue. It's normally an either/or question, with no other options considered.

And when I come across such discussions and statements I stick my fingers in my ears, sing la la la and try to think happy thoughts whilst I click on to something more to my liking. But sometimes I feel an overwhelming need to comment ...

Not that data quality CAN'T be a business issue - it can ... in businesses; and it can certainly be a technical issue. What gets me is the tunnel vision that surrounds this discussion. One would think that the only place that data exists is in businesses (and large ones at that), and that no data quality professionals existed outside them; and that data only exists on computers.

We are surrounded by data (and their cousins information and intelligence). It is in businesses but it is also to be found and used in huge quantities outside them in government, public utilities, health services. It is found in large businesses but is also to be found in huge quantities in small businesses where there is no talk of warehousing, OLAP, management buy-in or any other expression we can think of.

When a patient goes into surgery, for example, the purpose of data quality is not to make money but firstly to prevent a death (for example by transfusing blood of the wrong type) and secondly to achieve a health improvement to the patient.

Obviously, for those wrestling with data quality and company politics within large corporations every day data quality can appear to be a business issue. But we need to have a much more generic outlook with data quality.

Maybe we just need to be more careful in our use of language. "Poor data quality has an effect on the economics of a business": absolutely!

Data quality CAN be a business issue.
Data quality CAN be a technical issue.
Data quality CAN be a customer issue.
Data quality CAN be a health issue.

Data quality is ALWAYS a DATA issue.

And that's how I think we should regard it.

A true story of how data quality issues can cripple a business.

I've just been told about a data quality issue which may not have cost millions to resolve, but illustrates very well the effect poor data quality (and lack of information quality) can have at every level.

Call 1: A café reports to its head office that its bank card payment terminal has stopped working. Head office assumes (assumption 1) that it is a technical issue and instructs the café (call 2) to call in the technical support from the equipment provider (call 3).

The equipment provider sends a technician (visit 1), finds no technical faults and assumes (assumption 2) that the problem lies with the telecommunications provider. They leave behind a hefty bill for the visit. Café calls (call 4) the telecommunications company, who check their systems and find no fault, and therefore assume (assumption 3) that the equipment manufacturer is at fault. Call 5 to the equipment manufacturer, who point the finger back to the telecommunications provider; call 6 to the telecommunications provider who .... well, you're getting the picture.

For 6 months this situation remained unresolved. Calls were made, technicians sent, invoices sent. More importantly, the café was losing customers that didn't have the cash to pay and wanted to use their bank cards. Trying to resolve the situation was costing their staff time and money and their good humour.

A new round of calls and eventually somebody at head office actually checked - and found that the bill had not been paid. The equipment was working again within 24 hours, but the costs of this situation were crippling for the café concerned.

Not only do we see how dangerous it is to make assumptions, but here also there's a clear information quality problem in the support department of the equipment manufacturers. When their technical service was called they could check on technical aspects of the equipment installed, but there was no link with the financial system, so nobody could tell the café that the switch had been thrown because the bill went unpaid.

Sometimes it doesn't matter how good your data is - you have to actually go and look at it and make sure you use it properly.

Dear Mr Other ...

The year is ending and two calendars for 2010 arrive from "data quality" companies, each with strange ideas about my postal address.

The first is from a UK data quality exhibition organiser, who addresses me as:

Mr Other Graham Rhind

I can't imagine what is supposed to have been in the field which output as "Other", and I celebrate my individuality, but not in this particular way.

The address block (for my Dutch address) ends with a British postal code and "GB", struck out by the postal services to allow the mailing to reach me - eventually. I recognise the postal code as that I use when I am provided with a web form which does not allow me to add my Dutch postal code. This company is happy to invite foreigners to its exhibitions, it just won't allow them to register without providing false data.

The second is from a Spanish company, who puts my name below the final address line, guaranteed to delay mail because it's where the sorting machines expect to find the country name or the postal code. I am addressed a "D. Graham Rhind" - that's D for Don - and the postal code line reads:

52000 1018 VV Amsterdam

I guess that this company uses a Spanish CRM system that only allows a Spanish postal code to be entered. To allow my (not at all Spanish) details to be entered they have added the least used Spanish code in the postal code field (for Melilla) and then put my Dutch code into the place name field.

Remember, these are both data quality companies. I can see that I still have plenty of work to do to bring the message of our cultural diversity to all in 2010.

Informatica - a step forward in web form quality

Yet another e-mail (from Informatica Netherlands this time) with news of a new white paper. Wearily (why wearily? check this blog entry to find out), I click to check the web form ...



Hey! Hang on a minute. Informatica have actually had the nous (British slang approximately meaning common sense, intelligence ...) to have pre-filled the form with my data (which they indeed already have in their system). And what's this? No state field? And yes, there's Montenegro in the country list, back in all its glory.

Could this be a result in my crusade for better Internet data collection? I gingerly change the country to Canada and yes! The province field appears! Not in a sensible place, unfortunately (if you're going to change fields, don't change them where the customer has already been in the form - the country should be asked beforehand).



But what can I say? RESULT!!

B-Eye Network - another web form of shame

Another invitation to download a white paper, this time from the B-Eye Network, another organisation with data quality at its heart - except when it comes to its web form.



What hit me first with this form is that there is no indication of which fields are required, though you can be sure that they are there. In fact, to find out which fields I should fill in (according to B-Eye) I have to fill in the fields I can complete, hit the send button, and then hope for the best (or pray, depending on your personal preference). Only at that point will the form tell you what is required. And what is required shows a grave lack of understanding of the world out there.



I don't have to fill in a state (or a province), and I am very grateful for small mercies. But I do have to fill in a "last" name (that's a culturally loaded field label, by the way), even if I don't have one, and fill in a Zip/Postal Code, again, even if I live in one of the 60 or so countries or territories without one.

And then we look at the country drop down, always very instructive. I'll gloss over the "Falkland Islands (Islas Malvinas)" ... touchy subject ... and land instead on Serbia and Montenegro, a country that hasn't existed since 2006.



I await eagerly the next invitation to download a white paper. Who dares?