GRC Database Information

Step One: Acquiring the Knowledge

2012-01-20T10:20:00.001+01:00

My guest blog post at PostcodeAnywhere, about the first step in improving international data quality, has been posted at http://blog.postcodeanywhere.co.uk/index.php/step-one-acquiring-the-knowledge/

Effin' obscenity screening!

2011-12-22T12:37:00.002+01:00

You can read my new guest post for PostcodeAnywhere, about obscenity screening in an international environment, here: http://blog.postcodeanywhere.co.uk/index.php/effin-obscenity-screening/

The Speed of Change

2011-10-05T11:52:00.000+02:00

I often warn people about the speed of change. Not just the speed of change of data within your database, but the speed of change of that data's main context - the world in which we live. People know that change happens but tend to underestimate how fast and how far reaching many of these changes are.

The world changes so fast that I have to upload a new version of the Global Sourcebook for Name and Address Data Management almost weekly.

To put some numbers to that change I looked at the past nine years and summarised the speed that change is occurring. On average, there are changes in this many countries every year in these areas:

Telephone number systems: changes in 60 countries per annum
Administrative districts (top level): 14.4 countries per annum
Currencies: 4.7 countries per annum
Postal code and addressing systems: 4.3 countries per annum
Country names: 1.4 countries per annum
New countries: 0.8 per annum

This is a surprisingly fast rate of change. Are you keeping up with our dynamic world?

The Kaleidoscope of Data and Information

2011-09-20T12:06:00.000+02:00

John Owens , whom I much admire, made this comment to a Henrik Liliendahl Sørensen blog post about the difference between data quality and information quality. Taken completely out of context, he said:

“…it is the quality of the information (i.e. data in a context) that is the real issue, rather than the items of data themselves.The data items might well be correct, but in the wrong context, thus negating the overall quality of the information, which is what the enterprises uses. It will be interesting to see how long it is before data quality industry arrives at this conclusion. But, if they ever do, who will be courageous enough to say so?”

I agree entirely, yet disagree profoundly. Data and information are not the same thing yet are inextricably linked – one without the other isn’t possible but they still must not be confused.Data and information are as different as chickens and eggs, but are equally dependent upon each other.

Basically, data is stored information whilst information is perceived data.

Data and information are immutably linked – I have never found data which isn’t stored information nor information that isn’t rooted in data – but as they are different parts of a cycle they need to be defined, understood and managed as two separate entities.The challenge with data is keeping it complete, accurate and consistent.The challenge with information is to perceive the information that the data is a stored version of without alteration and in a way that gives clarity. It is at the information stage that we should be thinking about fitness for purpose, not at the data stage.

Let me give you an example from a recent episode of the BBC’s science program Bang Goes the Theory. A presenter went to a shopping centre and prepared two plates of bacon sandwiches. One was accompanied with the message that regularly eating processed meats increases the chances of getting bowel cancer by 20%. The other was accompanied by the message that regularly eating processed meats increases the chances of getting bowel cancer from 5% to 6%. Though the data underlying both pieces of information is identical, as is the information provided, the audience were understandably worried when seeing the first message but happy to tuck in after seeing the second. The first message would be fit for the purposes of the health authority, the second for the bacon marketing board, but in neither case is the fitness for purposes related to the data - it is related to the information provision.

It is at the points where information becomes data and data becomes information that the potential for corruption and misunderstanding of the data and its perception are at their highest. We also know that once data is stored, inert though it may appear to be, it cannot be ignored as the real world entities to which the data refers may change, and that change needs to be processed to update the data.

Those of a certain age may remember having kaleidoscopes as children. Tubes of tin or cardboard with a clear bottom in which there were chips of coloured glass or plastic, a section of which could be viewed and with mirrors creating a symmetrical pattern from that section. Move the kaleidoscope and patterns form, patterns which change and are always different, though the coloured chips themselves never change their inherent properties when being viewed.Whether your data is a shopping list or a data warehouse containing hundreds of tables and millions of record, working with data and information is much like looking through the kaleidoscope.

Depending on how we view we tend to see something different every time we look. Reports, dashboards, views, queries, forms, software, hardware, your cultural background and the way your brain is wired will all alter the perception of the data for us and thus have an enormous influence on the information we’re receiving from the data.

Like a kaleidoscope we tend to extrapolate what we see to the whole universe. If a report shows a positive result in one part of the operation, the tendency is to assume this result is valid throughout. In these examples square green chips represent accurate data whilst red or other shapes is errant data.

Both human nature and data and information systems tend to filter out the negative and boost the positive, so often data looks better than it really is, and so then is the information derived from it.

But sometimes the data looks entirely bad, though it is not so. The way we look dictates the apparent quality of the data.

Yet data has tangible and innate qualities, its accuracy, completeness and consistency, which together are an indication of its quality. And any data which has these qualities provides a foundation for better information quality because the perversion caused by the view of the data is ameliorated. In these examples the data has been made consistent and accurate – the coloured chips have the same colour and shape.

And regardless how we view that data, we see green square data. Data quality and information quality are different and yet rooted in each other. Data cannot be good if it represents the information that it is the stored version of incorrectly. Information cannot be good if it is based on incomplete, inaccurate and inconsistent data.

Data quality ensures that the data represents its real world information entity accurately, completely and consistently. Information quality is working to ensure that the context in which the data is presented provides a realistic picture of the original information that the data is representing.

There’s a general feeling that only data which has a purpose should be stored. I would not agree as purpose, as with so much, depends on context and our viewpoint. Data which has no purpose now may be required to fulfil an information requirement in the future or be related to occurrences in the past; whilst for the people who are being paid to manage data, whether it is used or not, finds his or her salary is very meaningful!

Ultimately data is used to source information, and information quality is important. But we should not confuse the differences between data quality and information quality. Both are essential, and they are separate disciplines.

Have you checked your country drop down recently?

2011-07-28T11:53:00.003+02:00

When visiting a data quality software supplier’s site recently to download a white paper, I noticed that the country list on the sign up form didn’t contain South Sudan (which became a new country on 9th July 2011) or the new territories which came into existence when the Netherlands Antilles were dissolved on 10th October 2010 (Bonaire, Curaçao, Saba, Sint Eustatius and Sint Maarten).

I shot off a tweet to the company concerned and they told me they were using the United States’ Department of State list. That list has added South Sudan but, shockingly, at the date of writing this blog, has failed to make the changes required by the dissolution of the Netherlands Antilles.

As I mentioned in my post here most companies rely on external sources for their country names and code lists, such as the World Bank and the United Nations, both of which use lists which exclude most of the world's territories) ; or the ISO (International Organization for Standardization), which still has not added South Sudan to its list.

Relying on other organisations for your country lists is problematic. To start with, unless you are aware of global changes (and too many people aren’t), or you check the list every day, you won’t notice changes as they happen, as with the data quality company I mentioned above. Secondly, maintenance of country code lists is not the core business of the United Nations, the World Bank and so on. They maintain a list in order to facilitate their own business – and that is rarely likely to coincide with your business. Many of these organisations are very heavily politically dependent or influenced, such as the United Nations or ISO (which doesn’t include Kosovo in its listing, for example), whereas you are likely to need to manage the reality of the situation on the ground, with less emphasis on political niceties. Finally these lists are often updated only long after a country has come into being – it can take ISO many months to assign a country code – whereas you will ideally need to be ready to make changes to your data before the country comes into existence.

When you’re managing international data your country code is likely to be linked to other data, such as currency, international telephone number access code, address format or postal code structure, which is not taken account of in country name lists being maintained purely for political purposes. Using lists which exclude Kosovo, for example, which is a de facto entity and has language, currency and addressing differences with Serbia, will cause problems for your data quality.

Maintenance of country lists and codes needs to be given more thought and more attention. If you’re not in a position to manage your own lists, take a look at the one we offer: http://www.grcdi.nl/countrycodes.htm . It may not suit your needs, but it is one of the few lists created without a political agenda, which is updated ahead of requirement, and with name and address data management specifically in mind. Using a correct and up to date country lists will improve your data quality and can save you from considerable embarrassment.

Data silos - learn to live with them.

2011-03-29T13:06:00.004+02:00

Mention data silos (separate data, databases or sets of data files that are not part of an organization's enterprise-wide data administration) and the blood pressure of any self-respecting data quality grafter will rise. Data silos are unconnected to data elsewhere in an organisation and thus are unable to share common types of information, which may create inconsistencies across multiple enterprise applications and may lead to poor business decisions.

One thing, though, that we need to think about: in almost all cases data is in an organisation to mitigate the work that that organisation does. The organisation is not there to improve data quality. My proviso is that every organisation has an obligation to the data owners (usually its customers) to manage that data correctly. Thus any improvement in data quality must serve the organisation and its customers, not the data in and of itself.

Data silos are inevitable, and that will never change. Organisation-wide data management systems are overcomplex and put too great a distance between the staff and the data they need. An employee's main focus is (or should be) to do the work that they are being paid to do as well as they can. If an organisation cannot provide ease of access to the tools that they need (hardly possible unless every member of staff has an accompanying IT employee at their beck and call) then they will reach for the tools that they can use - pen and paper, local spreadsheets, documents and database tables. Few employees (or work groups or departments) are going to wait around and do nothing while an attempt is made to hook their requirements into the cumbersome company database systems.

So whilst a CEO depending on a central database system to provide his or her business intelligence may suffer from data silos (only may, because it depends what is being stored in those silos), the troops doing the real work find having the data they need at their fingertips a distinct advantage.

I've been on both sides on the silos debate. I dug my heals in hard when attempts were made to integrate my data island into the company database because I knew it would not fulfil requirements, it would not enable me to do my job, it would be a waste of money, and I would not use it. I was right on all counts. At the same time I was trying to get the sales staff to give up their data for integration into a central marketing database. I failed for the same reasons (and because the staff were not going to give up the data which gave them the edge on their colleagues, improved their commission and enabled them to spend two weeks every year in the Caribbean as a reward from the company - incentives can be very bad for data quality).

Naturally data silos in certain parts of a company are very damaging, without dispute. You can't have a call centre employee jotting a caller's details onto a piece of paper or into an Excel spreadsheet instead of into their centralised system, for example. But not all data silos are evil.

As Jim Harris points out here, it's not so much having data silos that is the problem: it's what is done with that data. A much more open and accepting attitude to data silos is required, because we may wail and gnash our teeth as much as we like - they're not going to go away.

Are technology advances damaging your data quality?

2011-03-20T11:28:00.003+01:00

Way back on a Friday evening in 1989 I took home a computer, a software package called Foxpro and a manual. By Monday morning I had written a package to manage the employees and productivity of the telephone interviewing department I was running at the time.

I've been using Foxpro ever since. It's fast, powerful, easy to use and it has unparalleled string manipulation capabilities. Unfortunately, Microsoft have stopped developing the product, and will stop supporting it in a few years time, so I recently started looking for a good replacement.

At first I thought I wasn't getting it. I checked site after site, product after product, expert after expert, and instead of finding products which were easier to use: more accessible, more flexible, more agile, more data-centric, I found products which were technically challenging: over-complex, cumbersome, which put wall after wall between me and my data, which required reams of coding to do the simplest action like add a field, remove a field, expand a field, and so on. And most of which were priced to suit the budgets of companies with turnovers matching the GDPs of a large African country. Yes, there are some applications (very few) that try to make process easier, but they are primitive and clunky.

Most are based on SQL (that's a query language, ladies and gentlemen - the clue's in the name - and really very difficult to use to do any type of string processing) and based on client-server setups that required a high level of technical knowledge. You can get a record and display a record and store the modifications (what SQL was designed to do), but if you want to do more than that it gets tough.

I tried to work with some of them and just couldn't get my head around them. Never mind a whole application in a weekend - creating a database in a weekend was a challenge. My synapses may not be working at the speeds that they did in 1989, but I'm not in my dotage just yet.

Many support packages - data profilers and so on - have been created to work solely with these high-end packages, even those free and open source variants, cutting out a huge chunk of the data market.

I wasn't getting it. And then I realized I was getting it. This is the state of play in the data industry at the moment. A chasm has grown between easy to use, cheap but less scaleable products (Microsoft Access, Visual Foxpro, FileMaker and so on) and those scaleable but complex (and far too expensive) client-server applications.

So how does this work in practice? Joe from sales wants to create a database of his contacts. He can't manage the complexity of the SQL system in which the company is working so he'd have to send e-mails, set up meetings to request this database, get the IT department to put it into their planning, wait six months and watch his sales plummet. Or he could open Access or Excel and start typing. Guess which is the option most people take?

These systems encourage the creation of data silos (more about those in my next post).

Data quality is adversely affected by these databases also because they put a distance between the user and their data. Being unable to actually browse through the raw data is a handicap in data quality terms. Data which is filtered through systems, even to the extent of any SQL query other than SELECT *, will be less useful and have less quality.

The data software industry needs to take a close look at what they're up to and ask themselves of they should really be producing data products which can only be used and understood by highly trained technical staff, because they're not giving the data quality industry an iota of help. They should open up their programs to those common file formats that they're currently ignoring, such as .dbf. They need to be made easier, more flexible, more agile and a good deal cheaper

As for me, I'm back with Foxpro, and I have decided to stop apologising for using it. It allows me to produce top quality data, and that's what it's all about.

Data quality and information perception

2011-02-23T13:48:00.006+01:00

One of my mantras for ensuring data quality is to look at the data. Not at profiles or analyses or graphics but at the raw data. Browse through it and the best data quality tool that there is - your brain - will highlight data quality issues in seconds.

Information is rooted in, and derived from, data. When information is based upon data which is poor in quality the information will mislead.

This was highlighted by an attempt to bring together opinions about the current tensions in North Africa and the Middle East from blogs, feeds, social media and so on in a graphic form on The Guardian website (here).

We see Muammar Gaddafi there, nice and large, and .... but hang on a second. There's Moammar Gadhafi too. And Muammar Qaddafi. And then just plain old Gaddafi. The same person represented in four different places on the graphic because of transliteration issues. If these 4 entries had been brought together in a single place the graphic would look different.

As I mentioned, this issue is mainly caused by different transliterations of a person's name from Arabic, but this sort of variance within data is very common. Place names, for example, are often found written in many different ways within the same databases. Basing decisions on such variant data would be unwise. Yet decisions are based on data like this, from profiles and graphics like this, every minute of every day.

Go on, grab a coffee and go and look at your data. You'll be amazed.

I see bubbles

2011-02-10T12:31:00.005+01:00

We're humans, by nature short-termist and usually not capable of learning from history.

There was a time when advertising was a blanket phenomenon - you broadcast your message on television, the radio or in print media and hoped that it was seen by as many people as possible. If they were people who had an interest in you message, so much the better.

Then along came direct marketing as a way of showing your message only to those who might be interested. Inevitably, through various shortcomings including poor data quality, it wasn't long before direct marketing got a bad reputation - "junk mail" and so on.

And now we have the Internet. Yet, despite its opportunities, we seem to have regressed back to the blanket coverage approach prominent before direct marketing.

When somebody hits your page they tell you a remarkable amount about themselves: where they are, the language of their browser and operating system, how they reached your page and so on. Unless their system is infected with a virus, you shouldn't know any more amount them as individuals. But what is also known is the contents of the page being viewed and where it is hosted.

How is this information being used by the Internet advertising giants? In a non-scientific but pretty revealing study, I analysed over a period of some weeks those advertisements being shown on the various pages that I visited. What I found was:

- 35.6% of advertisements were in the language of the website and suitable for somebody living in the country where the site was being hosted, but not for me. Adverts for German school reunion sites or the American Girl Guides association are just not correctly targetted.

- 38.8% were in Dutch (the language of the country in which I reside, though not the language of my browser or operating system) and were aimed in the same scatter gun way as broadcast adverts are - adverts for holidays, electricity companies and so on.

- 19.4% were those deliberately misleading or downright criminal adverts asking you to count how many bouncing balls you see, or informing you that you are today's lucky 1 millionth visitor. If I won prizes on each of these clicks I'd have a GDP larger than China's.

- Shockingly, as least to me, only 4.4% were relevant to the contents of the website being viewed - baby clothing adverts on baby name sites, for example. Almost as many were the antithesis of what should have been shown - adverts for Thai brides by post sites on a site listing baby names is putting carts before horses!

- 0%: the percentage of advertisements I was tempted to click on.

And then the language of the adverts:

- 49.6% were in Dutch
- 44% were in the language of the website
- 4.7% were in English (on non-English language sites)
- 1.6% were in the language of the country where the site was hosted.

If nobody clicks on the adverts then no payment need be made, so companies and criminals don't feel bound to think of the consequences of the current scatter gun approach. Short-term thinking. It does, though, have negative consequences. In the hope of getting enough click throughs to get a decent return, many webmasters are placing so much advertising on their sites that it has become difficult to locate the content. And we, the users, are becoming less and less inclined to look at, let alone click, on the adverts.

Facebook

So much for the web. But how about social communities where people have provided information and where even the simplest of data mining techniques could improve the relevance of advertising? How about that doyenne of the Internet: Facebook?

I am very careful and specific in my use of Facebook, which makes my profile a good test of their targetted advertising. They know my age, gender, the university I went to, that I speak English, Dutch and French (in that order) and that I and 95% of my "friends" play squash, and that almost every communication I have on that site is about squash.

That being the case I would expect Facebook to show me adverts in English, possible for sports equipment or similar. What I actually get:

- 71.4% of adverts in Dutch - they're looking again at my location, not my profile
- 26.8% in English
- 1.8% in German (eh? Did I say I speak German?? I don't.)
- 0% in French

- 0% relevant advertisements
- 100% irrelevant advertisements.

It is no surprise that Facebook ad performance is abysmal.

How many billions was Facebook deemed to be worth?

Are they mad?

I see bubbles.

Localization is evil!

2011-02-04T12:55:00.003+01:00

OK, now I've got your attention let me modify that statement: over-localization is undesirable.

Localisation (with the spelling localised to my own culture) is the process of adapting or modifying a product, service or website for a given language, culture or region. However, in almost everybody's mind, localisation is synonymous with translation, and any other modifications, such as making web forms suitable for local address structures, are quickly overlooked. Localisation is intimately bound up with the concept of locales, which give a country/region and language combination which can be used in the localisation process.

The whole problem with this system is that PLACES DO NOT SPEAK LANGUAGES - PEOPLE DO.

There are many indigenous people who do not speak a nationally recognised language in their country, and in our mobile world many of us do not speak the language of the place where we live, or have a preference for another. Yet this fact is overlooked in almost every attempt at localization made.

A case in point. This blog site is owned by Google, and Google win award after award for their localisation, presumably based on a count of the number of languages their interfaces and programs are available in rather than any intelligent application of the idea. Yet Google assume, like most of the rest, that places speak languages rather than people. Though I state clearly in my preferences that I wish to use this site in English, Google sees where my IP address is situated and changes the interface into Dutch. When I go to Prague and attempt to log in I am expected to master Czech. When I reach Athens I will be asked for my details in Greek.

HP.com is similar. Attempt to get into hp.com from outside the USA and you will be taken to a local page (or local to somewhere - I get to the UK page for some unknown reason). This might be regarded as a service by some, but it has consequences. A user in Bulgaria, for example, searching for information about an HP product may click on the link in the search engine referring to the hp.com site. The site then switches the user back to a local Bulgarian site, where that information page is not available, and the user is presented with a 404 page not found error.

Look, I am absolutely in favour of translating information and I regard myself as a reasonable polyglot. But a stop must be put to translating on the basis of place. Users must be given the ability to override locale settings so that they can use their own languages and not those attached to a place.

IAIDQ Blog carnival for Information/Data Quality, December 2010

2011-01-26T11:32:00.003+01:00

It's my pleasure and privilege to again be hosting the IAIDQ blog carnival, this time for posts made in December 2010. I'm glad to say that the festivities didn't reduce the flow of top quality posts, some of which I have highlighted here.

If you work in business and you're attempting to get a data quality initiative off the ground, William Sharp, over at the Data Quality Chronicle, has outlined a data quality program's 10 best practices. A useful checklist whatever your data quality background.

Jim Harris' steady stream of wisdom over at the Obsessive-Compulsive Data Quality Blog hardly let up in December, with a look back at the best data quality posts of 2010 (which I have to mention, because Jim kindly included not one but two of mine in the list!).

I've been thinking (probably rather more than is healthy) quite a lot recently about data silos, and Jim casts his usual sensible eye over just that subject when discussing The Good Data.

I don't know where Jim finds the time to read, but he's read "A Confederacy of Dunces" by John Kennedy Toole and he discusses our propensity to delete data in A Confederacy of Data Defects.

And finally, whilst Jim was looking back over 2011, Steve Sarsfield, in his data governance and data quality insider blog, was looking forward to 2011. The brave Steve has made six data management predictions for 2011. I'm wondering if that's a misprint - won't be be waiting until 2051 for most of those things to happen? ;-) What do you think?

The myth of deliverability.

2011-01-13T13:59:00.003+01:00

I am often asked whether an address is "deliverable", and not everybody is happy with my usual response that it depends on the mood of the postal staff on duty that day.

The point is that "deliverability" is unmeasurable, unscientific and has little basis in reality. Address validation software will often give a figure for the number of deliverable addresses within a file, such as 80%, but don't be fooled - this does not mean that, if you send a letter to each address within that file, 80% will get there and 20% will not. These numbers have been created to give some sort of feedback to the user, and to make files comparable with each other when run through the same software processes.

So how come there's no way to measure deliverability?

You could look at a country like The Netherlands, with its neat address system, and boldly state that a deliverable address is one where the postal code and the building number are present. Mind you, if either contain a typo, the mail may be deliverable, but not to the desired recipient. Equally, whilst TNT Post is happy with that much information, because it will get them to a letter box, to make a mailpiece deliverable to a person, more information is required - a sub-building indicator (as many addresses may share the same house number/postal code), a name, department and so on.

So, I've got all that. So the address is deliverable. Right?

A mailpiece containing my correct postal code and house number took 6 months to reach me. Not because the information was wrong, but because a second, stray, postal code had wormed its way into the address block, sending the mailpiece around the system ad infinitum. It was only when a particularly awake postal worker saw and crossed through the stray code that the mailpiece could get sent on its (correct) way.

But not having a full address, or even any address, does not make a mailpiece undeliverable! I remember the TV program That's Life! on the BBC successfully receiving mail sent with just a drawing of a prominent set of teeth on the front - an allusion to Esther Rantzen's somewhat toothy smile.

This Christmas a card was sent with this address on it:

Mr & Mrs T Burlingham?
Near the golf course in Thetford,
Norfolk.
Trevor is a photographer. This might help

It did help. This undeliverable address took just 2 days to arrive. But this isn't just a British thing. How about one of my favourites:

Translation:

Vukasin 6 years and Jelisaveta 3 years
I don't know the family name
PRUSKA GORA
SERBIA
Their father is big, he drives a Citroën Belingo.
He works on the little trains for tourists.
Postman: Please find them!

And he did!

Praise for a form

2011-01-04T14:39:00.006+01:00

You know that I have a thing about forms, and I spend a lot of time criticising them. But I did always said that if I found an example of a good form, I'd sing its praises on high.

So, this is me singing.

The form is not an international form - it's for the Dutch market and is in Dutch - but is shows some good practices that I'd like to highlight. The form is from ShopPartners .

Dutch addresses are of the few in the world where a whole address can be derived from very little information - in our case a postal code and a house number - and the form starts by asking for that information to save the customer from typing their whole address - and the less you make your customer type, the happier they are. They show very clearly the number of steps you'll have to go through to order, and the section to the right gives clear examples of what they want should there be any doubt in the mind of the customer.

The second step expands the form with the address autocompleted. Nothing earth shattering there.

The form asks whether the customer is a private individual or a business - a great way of preventing customers from having to fill in or skip over fields which are not relevant to them, such as VAT numbers.

But what did make me smile - no, it made me get up and jig around my office - was that this is the first Dutch form I have come across which does not ask your form of address (Mr or Mrs - no other choices are given) and use it to assume your gender. Somebody has thought about this, and apart from Mr and Mrs the customer can also specify a department name or the name of a family. Still no chance for me to add Dr, Professor, Lord, Sir, Dipl. Ing., Mag. or any other one of the hundreds of forms of address that exist; but I do get the change to choose NONE (my own preference) - an escape route for customers not covered by the options provided - and this is rarely allowed on forms.

And again, descriptive examples of what is required are provided to the right of the fields.

Now, plenty of usability experts will pull me up and mention element placement, colours and numerous other "problems" with this form, but for me it's a huge step forward! Well done ShopPartners!

Want to know more about collecting good data from your web form? Download my free e-book here. And no form to fill in either!

Robert the Carrot

2010-10-18T12:31:00.003+02:00

A friend told me an anecdote from the time he was working in a Chinese restaurant. A customer, called Robert, wanted to get a tattoo of his name in Chinese and so asked one of the Chinese staff to write down his nick name - Robbo - as a pictogram.

There's no Western "r" sound in Chinese, and the staff member subtly altered the pronunciation of Robbo to something closer to Lobbo - which means "carrot" in Chinese. The pictogram was duly drawn, and Robbo happily went off to get himself tattooed large as being "Carrot".

This resonated with my data quality genes in two ways. The first is what I call the Chinese Whisper effect. You know that game - one person whispers a word or phrase into the next's ear, and so on down the line, and what comes out at the end is often completely at odds with how it started. Data quality is like that - at every interface between information and data and between data systems, the quality of data goes a little more awry.

The second has to do with ignorance. Most organisations think their data is great simply because they don't understand it, and that's especially the case with names and addresses. If you can't see the problem, or, indeed, see that there is a problem, you can't correct it.

Robbo lives in ignorance about his tattoo and is probably still mighty proud about it. He may get a few sniggers if he ever goes to China, but that's about it. Unfortunately, data quality issues arising from processes which work like this can be much more dangerous.

Today we welcome ....

2010-10-10T10:35:00.002+02:00

.... five new countries and territories, and we say goodbye to one. The net number of countries and territories in the world without a postal code system increases by four.

Did you know? Had you noticed? When I asked two days ago at a speech at Post*Expo in Copenhagen, none of the 50 or so participants admitted to having any clue about it.

Today we're waving goodbye to The Netherlands Antilles. We're welcoming Curaçao and Sint Maarten as largely independent territories within the Kingdom of The Netherlands (as Aruba now is); and Bonaire, Sint Eustatius and Saba as special municipalities within The Netherlands.

How many weeks, months, years or decades will it take before organisations reflect these changes within their databases, processes and customer-facing systems? After all, Saint Martin and Saint Barthélemy (2007) are still largely unknown and unused; Serbia and Montenegro are still far too often lumped together, though they split in 2006; and some organisations still have Yugoslavia as a country, though that died a death in 2003.

Manage your own country list

Twice in the past couple of weeks, when pointing at errors in country lists, organisations have let me know that they will be "looking for a new source" for that list. Far too many organisations use incomplete and unsuitable lists provided by organisations such as the World Bank, United Nations or the ISO. These organisations have their own reasons and imperatives for creating and maintaining lists, which will not be the same as yours, and they must adjust lists to political pressures which rarely reflect reality on the ground.

If you need to keep your country list up to date, and you do, then manage your own. Any country or territory which has a de facto existence needs to be on your list. Though Guadeloupe is part of France, it's geographical location means that it needs to be listed separately to ensure correct address management. Saba may become part of The Netherlands but it won't use the same postal code system or, indeed the same currency. Kosovo must be on your lists because of the linguistic, cultural and addressing differences, regardless of how you stand on its relationship to Serbia.

If you can't rely on the list you're using now, use your own. I'm looking forward to Sint Eustatius and its new neighbours appearing on your website form dropdown very soon.

Avis - not trying hard enough

2010-09-28T11:13:00.003+02:00

Before you clock off and look elsewhere, this is not going to be a rant about how bad Avis customer service is - it's an illustration of how broken processes (in this case of information exchange and actions related to that information) can affect a company's bottom line.

I was in Croatia in September and booked a car from Avis for a trip into the mountains. I chose Avis because their location was close to where I was staying. Come the day of the collection, we went to the address given (at the Sheraton Hotel) but couldn't find Avis. At the reception we asked and they informed us that Avis had moved to a location a few kilometres away on 1st July. They were turning away "tens" of customers every day, who were all coming to search for Avis.

It took us a while to get to the new location, and when I vented my frustration to the staff there they just shrugged their shoulders. "We've e-mailed head office - what more can we do?". They weren't bothered about the "tens" of customers they were losing.

Back in The Netherlands I searched the global Avis website for a complaint form. No, they don't have that - they are clearly confident of the quality of their services - so I sent "feedback". After some days the reply came back that complaints had to be dealt with in the country of my residence, The Netherlands, even though they have no responsibility for the website. After some weeks I called Avis in The Netherlands (on a premium rate number - this is me trying to help this company to get back its customers by spending my own money) and found that the complaint was being processed - by sending it on to the office in Croatia. Even if Croatia did nothing (and what could they do except send another e-mail to head office?) the Dutch office would then pass this information on to me, and nothing would have actually happened except that the time of a lot of employees and a customer had been wasted. When I explained the problem to the customer service representative at the Dutch office, she admitted it took many months to get any changes made to the website at all.

In the meantime the website is still sending people to the wrong place. If we reckon on Avis losing 10 customers per day because of this I'm reckoning that, as of today, that's 900 customers.

Avis seems to think it's big enough and profitable enough to carry this sort of loss. A slight tweak in one of its processes (which would cost nothing - in fact, it would save money because staff wouldn't have to field complaints like mine) would have a huge effect on its bottom line.

Time to try harder, Avis?

The dangers of obscenity tagging ...

2010-09-10T10:22:00.002+02:00

I was pulled up sharp by this report about an XBox user whose account was suspended because he used "offensive language" in his details, the offence being that he lived in a city in the USA called Fort Gay.

I'm continually astonished by the stilted sensibilities of some Anglo-Saxon communities (I have never come across policies banning "offensive language" in systems for any other language) - like most of my fellow Europeans I can't imagine why the word "gay" should be deemed offensive. But this incident highlights the problematic use of obscenity lists which cause problems all the time for large numbers of innocent Internet users.

Gay, for example, is a common given and family name in the English-speaking world, and there are many streets and places named after those people. There are many people with the surnames Duck or Mouse, and some are called Donald or Michael; and they have a constant battle to achieve anything online. The good burgers of Scunthorpe in England have a very trying time with web sites and spam filters (think about it); and imagine the issues that the inhabitants of Dildo in Newfoundland, Condom in France and Fucking in Austria have.

If you do feel the need to check for obscenities, it needs to be done in a more knowledgeable, culturally-aware way than is currently the case.

How long is YOUR street name field?

2010-09-08T12:48:00.003+02:00

I was walking down the main street of Bihać, Bosnia, as one does, and I noted the street name:

You'd think this was long enough for a street name, but the sign actually contains abbreviations. To write this street name in full would take 89 characters. It could be correctly abbreviated to 20 characters, but would you know how? In how many ways might this street name be written? I may be more than a little weird, but I love this kind of diversity in addressing!

Definition drift

2010-07-16T10:47:00.008+02:00

A number of posts recently have drawn my attention again to the persistent problem of definitions and of definition drift. We are rarely able to agree a definition of any word or phrase in the data quality world before a new buzz term comes along to grab our attention. A great deal of this is due to fashion and marketing. Software and solution providers are constantly searching for new terms to launch in order to "persuade" (I'm being nice here) executives that they need to upgrade their current installations. Try as I might, for example, I cannot find a definition for Master Data Management that does not tally with what I understand to be good old plain data management, something we've been doing for years. Mark Goloboy noticed how some people are trying to replace the term "data quality" with "information quality". Though it is often not obvious to information workers, there is a huge difference between information and data (as us data workers know).

Altering terms in this way because of marketing, fashion or internal political needs is pernicious and does little to help data quality or, ultimately, your customers, for whom you should be working to improve your management of their data. The definition saga led me to start to build my own glossary of terms and to place that online for all to enjoy, in the hope that we can arrest some of the worst excesses before they take off. A recent post by Jim Harris exposed again how definitions can affect our working practices, and I felt the need to expand on his post and try to clarify my own thinking on data quality, what it is, what contributes to it, and how it affects information quality.

Data quality tends to have three main definitions: fitness for purpose (we'll come onto that later); data accurately representing the real world entity to which is refers (my own preference); and data being complete, current, consistent and accurate (or relevant, up to date and accurate; or complete, valid, consistent and timely; or accurate, correct, timely, complete, and relevant; or any of a number of similar properties...).

Without wishing to write a three-volume novel about this, let's have a look and see how some of these parameters do affect data quality, starting with continuing the discussion from Jim's post:

Validity versus Accuracy

Validity is that a piece of data satisfies a rule relating to the data itself. Accuracy indicates that the valid data applies to the entity for/about which the data is being gathered and stored. For example, US, CA, DE, and RU are all valid ISO 3166-2 country codes. XX is not. None of these are valid country codes for the country in which I live - in that case, only NL would be an accurate code.

1st January 1833 is a valid (Gregorian) date. It is a valid date for the date of birth of a human being. It is (currently) not a valid date as a date of birth for a human being still alive. 1st January 1961 is a valid data of birth for a human being still alive, but it is not an accurate data of birth if it were to be applied to me.

For these reasons, dashboards and data profiling tools need to be used with caution. They can check that every country code or date within a data file is valid, but they cannot check their accuracy.

Currency versus Timeliness versus Up to Date versus Of Its Time

Yes, even these terms vary in their definitions. For the most part Timeliness is regarded as a processing aspect, where data is made available to the worker at the time it is required - not a data quality issue, in my opinion. Currency is often regarded as synonymous with up-to-date, but data which is up to date (i.e. valid now) is not necessarily fit for a purpose. If your purpose is to know what I bought from your online shop in 2003, you'll need data from that time and of that time, rather than data from this year. This is also a definition of currency, but it does need to be mentioned. For me, this is all part of the accuracy and completeness of data - if I move then the address you may have for me is valid (the building still exists), the data is valid of its time (I did live there when you added it to your database), but I don't live there now, so it's not current and it's not accurate because the house it there but the entity for/about which you're collecting the data (me) isn't there any more.

Consistency and fitness for purpose

Consistency is NOT a pre-condition for data quality. If your database contains the information that I live in The Netherlands in a variety of forms (NL, NLD, The Netherlands, Nederland, Holland, Pays Bas, Niederlande) then the data is accurate, though represented in any number of ways. The data has quality but it is difficult to work with and process, and is therefore not fit for purpose. Counts to find the number of customers in The Netherlands will produce poor information leading to poor business decisions. Data which is made consistent can be used, and is therefore fit, for any purpose. If I know all records for entities within The Netherlands use the code NL, then I can print "The Netherlands" onto an envelope being sent from the USA, or "Niederlande" if it's from Germany, and that data need not be stored anywhere - it is derived from consistent data. I reject the definition of data quality being fitness for purpose - fitness for purpose is a consequence of data quality, not a definition of it.

OK, so I've tossed this off on a Friday morning, and may have missed some logical connections. What do you think? Please join the debate - I'll update this entry with any nuggets that are suggested.

Prevention or cure?

2010-07-06T15:09:00.004+02:00

I was looking through a pile of my old and dusty university essays a few weeks ago, nicely typed on a 40-year old manual typewriter (at that time the university's only computer was in a huge, well guarded room and the only way we were allowed to interact with it was with punch cards ...) and I found an essay with this title:

"Regional Water Authorities in Britain are Dominated by Engineers Trained Largely to Solve Water Supply Problems by Constructing New Facilities Rather Than by Minimising the Need for Them" (D.J. Parker and E.C. Penning-Rowsell). Discuss."

I had discussed, as directed, and agreed: instead of tempering our profligate and ever-increasing use of water, we just kept tapping into new resources to increase supply.

It wasn't just water that had this issue. And little has changed over the past 30 years.

Looking around, you see this pattern almost everywhere. Health services, for example, spend a little on prevention and a huge amount on curing. Police try to solve crimes after they occur but rarely attempt to prevent the crimes from occurring (and in many countries they are not allowed so to do). In fact, our whole society is based on the use/consume/experience now and the resolve/cure/clean up later paradigm.

So it's hardly surprising that businesses work the same way when it comes to data quality. Like the water authorities, they are dominated by people who are trained (and indoctrinated) to resolve problems as they arise rather than to prevent the problems from arising; and when they see the problems they envisage only technical solutions without any consideration for process or business structure changes. The bigger, more expensive and flashier the product, the more likely it is to be bought, regardless of its effectiveness at reducing the problem.

Shifting spending to prevention will reduce spending on the cure, and we'd be healthier for it - stopping us from taking up smoking will always be better than treating us for lung cancer. There'll always be a need to cures - like our bodies, data decays and we have to work on it to keep it healthy - but prevention works better. And is cheaper.

Vodafone, Samsung and a company in Almere ...

2010-06-03T15:46:00.003+02:00

Vodafone

Oh lovely - an e-mail from Vodafone offering me lots of discount if I extend my current mobile contract with them. "Just click here to see the goodies".

Right on time - my current 'phone had just thrown a wobbly. So I click on the link, log in, and ... nothing. "You cannot extend your contract. Call this number". Like the good automaton that I am, I call the number. "Oh yes, you can extend your contract, no problem. If you'd like to see the 'phones, pop down to your local Vodafone shop"

So I pop down and choose a 'phone just a little shinier and more sparkly than the previous one. And the nice gentleman says "But your current contract only runs out in November. You're not allowed to renew yet".

So I explain about the e-mail, he calls, somebody tweaks the system, and I renew.

And no criticism of the flunkies at Vodafone - "service" is not something the Dutch understand well - they don't even have a word for it (they use the English "service"), and they tend to equate "service" with "servitude", which doesn't sit well in this egalitarian society) But these flunkies were fine and resolved the issue without any fuss. What I do wonder about, though, is why Vodafone's data systems aren't talking to each other. The e-mail system and the contact centre system think one thing, the website and the shop system think something else entirely.

A company in Almere

So I get my nice, new, shiny and sparkly 'phone home, switch it on and (you'll like this Daragh) find that it may be shiny, but it's not new. 40 contact names, e-mails (about taxi expenses), voice messages, notes ... even pictures of the baby.

Looks like the 'phone had been tested and returned,but returned without any attempt being made to delete the data on it. I could even contact the company's e-mail server and check out the rest of their mails. I wasn't tempted (I may have been put off by the baby pictures, truth be told), but a message about data not leaving the company hasn't seeped through to all staff yet.

Samsung

So I take it back and get a new 'phone. Only this one is missing it's little pen ...

OK, this cost me not a lot (a few telephone calls and some cycling around Amsterdam, which is good for my condition). It costs Vodafone more, I reckon. And as for that company in Almere ...

I have a dream ...

2010-05-12T13:27:00.005+02:00

I have a dream. My dream is that one day those overpaid nincompoops who run many of our companies and organisations wake up to the importance of data, and start working with it accordingly.

If you're not persuaded of the importance of data, try imagining your organisation functioning without data (or its cousin, information, which is usually rooted in data) and see how far you get. No e-mails, no internet, no customer orders, no invoices. No telephone calls, no meetings, no discussions with colleagues, not even to discuss the weather, unless you're one of the very few organisations which is not affected by the weather (really, you'd be surprised).

How long would that situation be able to last? Minutes?

Why can't people understand the importance of data and its quality? Why don't we treat it in the same way that we treat other parts of our business? The very idea of an airline only maintaining its fleet when something went wrong with it would horrify all of us, but that's what we do with data. Few of us do not realise how preventing tooth decay not only saves us costly treatment and potentially a great deal of pain, but leaves us with far better teeth than any dentists ministrations could produce on badly maintained teeth. (Read Jim Harris' blog post on that topic here.)

So why do we wait until the CEO is told that $ 1 billion PROFIT was made instead of the actual $1 billion LOSS, with the resultant chaos, before we take data seriously? Clearly, unmaintained airlines falling from the sky make a greater immediate impact than data quality wrecks, but the results can be equally pernicious. Why must so many people waste so many hours trying to prove return on investment (ROI), when ANY and ALL data quality improvements are beneficial - I am yet to be persuaded that there is no return on any investment (in one form or another) on every improvement of data quality. Sadly, most businesses make money DESPITE their data quality, not because of it. (See Henrik Liliendahl Sørensen's post showing how simple it can be to show ROI here).

I have a dream of a revolution in data quality, where resources and focus are built into the prevention of data quality problems, rather than on trying to resolve them only when their detrimental effect becomes obvious; where as much control is put into data as is put into production, maintenance, finance, human resources and other aspects of organisations.

I have a dream. How long must it remain a dream?

IAIDQ Blog Carnival, April 2010

2010-05-04T17:50:00.016+02:00

I've dusted down the blog today to host April's IAIDQ blog carnival for information/data quality bloggers, a look at some of the month's best blog posts.

In keeping with this blog's focus, I've decided concentrate on posts about data quality (as a data issue) rather than on business or other practices, or on personnel issues; so I've largely bypassed posts about persuading executives to invest in improved data quality, data quality tools within businesses, return on investment and the like, though this is no reflection on the quality of those posts.

We can start with Daragh O Brien, a man who rarely utters a word I don't agree with, and he utters them always with great aplomb. His recounting of the difficulties of matching and moving his contact data from 'phone to 'phone in his post Do we have an App for that? shows well how real people have to grapple with data quality issues on a daily basis.

Also on the theme of data everywhere in our environment, and certainly not just within businesses, is the good Jim Harris' post Data, Data Everywhere, But Where Is Data Quality?. Jim, an obsessive compulsive blogger and independent consultant, speaker and writer looks at the avalanche of data we contend with daily, why its quality matters, and how we need to manage it.

I can't let the carnival go on without a mention of Dylan Jones, editor of Data Quality Pro and a prime mover in getting the importance of data quality recognised. Dylan's post is an Expert Interview with Jill Wanless (author of the Data Quality from the Ground Up blog). So I get to mention two data quality scions in a single paragraph.

I won't generally eat anything more exotic than a chicken and mushroom pie, and you'd have to tie me up and use a cattle prod to get me anywhere near IKEA, but Henrik Liliendahl Sørensen has continued his posts about data diversity with Data Quality and World Food. Henrik's work with, amongst others, Omikron, has given him a good understanding of the importance of understanding global diversity, and he blogs about it regularly.

Finally I'd like to make an honorable mention of Julian Schwarzenbach's final entry in his series The Data Zoo - How data personalities interact. Though I'm breaking my rule here of avoiding blog entries which revolve around data quality within businesses rather than as something generic, Julian's work in sifting and identifying the personalities involved in data quality work is a remarkable series, though I'm stuck with the feeling that I actually belong in each one of the nine categories identified, which is a trifle worrying ...

Apologies again to the writers of the excellent blog entries I had to exclude from this carnival, and I'm looking forward to next month's batch already.

Data Quality is a DATA issue

2010-03-23T10:36:00.004+01:00

In the blogosphere, on Twitter and at conferences I often come across mentions and discussions of whether data quality is a business issue or whether it is a technical/technological issue. It's normally an either/or question, with no other options considered.

And when I come across such discussions and statements I stick my fingers in my ears, sing la la la and try to think happy thoughts whilst I click on to something more to my liking. But sometimes I feel an overwhelming need to comment ...

Not that data quality CAN'T be a business issue - it can ... in businesses; and it can certainly be a technical issue. What gets me is the tunnel vision that surrounds this discussion. One would think that the only place that data exists is in businesses (and large ones at that), and that no data quality professionals existed outside them; and that data only exists on computers.

We are surrounded by data (and their cousins information and intelligence). It is in businesses but it is also to be found and used in huge quantities outside them in government, public utilities, health services. It is found in large businesses but is also to be found in huge quantities in small businesses where there is no talk of warehousing, OLAP, management buy-in or any other expression we can think of.

When a patient goes into surgery, for example, the purpose of data quality is not to make money but firstly to prevent a death (for example by transfusing blood of the wrong type) and secondly to achieve a health improvement to the patient.

Obviously, for those wrestling with data quality and company politics within large corporations every day data quality can appear to be a business issue. But we need to have a much more generic outlook with data quality.

Maybe we just need to be more careful in our use of language. "Poor data quality has an effect on the economics of a business": absolutely!

Data quality CAN be a business issue.
Data quality CAN be a technical issue.
Data quality CAN be a customer issue.
Data quality CAN be a health issue.

Data quality is ALWAYS a DATA issue.

And that's how I think we should regard it.

A true story of how data quality issues can cripple a business.

2010-01-19T16:35:00.003+01:00

I've just been told about a data quality issue which may not have cost millions to resolve, but illustrates very well the effect poor data quality (and lack of information quality) can have at every level.

Call 1: A café reports to its head office that its bank card payment terminal has stopped working. Head office assumes (assumption 1) that it is a technical issue and instructs the café (call 2) to call in the technical support from the equipment provider (call 3).

The equipment provider sends a technician (visit 1), finds no technical faults and assumes (assumption 2) that the problem lies with the telecommunications provider. They leave behind a hefty bill for the visit. Café calls (call 4) the telecommunications company, who check their systems and find no fault, and therefore assume (assumption 3) that the equipment manufacturer is at fault. Call 5 to the equipment manufacturer, who point the finger back to the telecommunications provider; call 6 to the telecommunications provider who .... well, you're getting the picture.

For 6 months this situation remained unresolved. Calls were made, technicians sent, invoices sent. More importantly, the café was losing customers that didn't have the cash to pay and wanted to use their bank cards. Trying to resolve the situation was costing their staff time and money and their good humour.

A new round of calls and eventually somebody at head office actually checked - and found that the bill had not been paid. The equipment was working again within 24 hours, but the costs of this situation were crippling for the café concerned.

Not only do we see how dangerous it is to make assumptions, but here also there's a clear information quality problem in the support department of the equipment manufacturers. When their technical service was called they could check on technical aspects of the equipment installed, but there was no link with the financial system, so nobody could tell the café that the switch had been thrown because the bill went unpaid.

Sometimes it doesn't matter how good your data is - you have to actually go and look at it and make sure you use it properly.

Dear Mr Other ...

2009-12-31T13:51:00.003+01:00

The year is ending and two calendars for 2010 arrive from "data quality" companies, each with strange ideas about my postal address.

The first is from a UK data quality exhibition organiser, who addresses me as:

Mr Other Graham Rhind

I can't imagine what is supposed to have been in the field which output as "Other", and I celebrate my individuality, but not in this particular way.

The address block (for my Dutch address) ends with a British postal code and "GB", struck out by the postal services to allow the mailing to reach me - eventually. I recognise the postal code as that I use when I am provided with a web form which does not allow me to add my Dutch postal code. This company is happy to invite foreigners to its exhibitions, it just won't allow them to register without providing false data.

The second is from a Spanish company, who puts my name below the final address line, guaranteed to delay mail because it's where the sorting machines expect to find the country name or the postal code. I am addressed a "D. Graham Rhind" - that's D for Don - and the postal code line reads:

52000 1018 VV Amsterdam

I guess that this company uses a Spanish CRM system that only allows a Spanish postal code to be entered. To allow my (not at all Spanish) details to be entered they have added the least used Spanish code in the postal code field (for Melilla) and then put my Dutch code into the place name field.

Remember, these are both data quality companies. I can see that I still have plenty of work to do to bring the message of our cultural diversity to all in 2010.

Informatica - a step forward in web form quality

2009-12-02T10:46:00.004+01:00

Yet another e-mail (from Informatica Netherlands this time) with news of a new white paper. Wearily (why wearily? check this blog entry to find out), I click to check the web form ...

Hey! Hang on a minute. Informatica have actually had the nous (British slang approximately meaning common sense, intelligence ...) to have pre-filled the form with my data (which they indeed already have in their system). And what's this? No state field? And yes, there's Montenegro in the country list, back in all its glory.

Could this be a result in my crusade for better Internet data collection? I gingerly change the country to Canada and yes! The province field appears! Not in a sensible place, unfortunately (if you're going to change fields, don't change them where the customer has already been in the form - the country should be asked beforehand).

But what can I say? RESULT!!

B-Eye Network - another web form of shame

2009-11-24T20:44:00.007+01:00

Another invitation to download a white paper, this time from the B-Eye Network, another organisation with data quality at its heart - except when it comes to its web form.

What hit me first with this form is that there is no indication of which fields are required, though you can be sure that they are there. In fact, to find out which fields I should fill in (according to B-Eye) I have to fill in the fields I can complete, hit the send button, and then hope for the best (or pray, depending on your personal preference). Only at that point will the form tell you what is required. And what is required shows a grave lack of understanding of the world out there.

I don't have to fill in a state (or a province), and I am very grateful for small mercies. But I do have to fill in a "last" name (that's a culturally loaded field label, by the way), even if I don't have one, and fill in a Zip/Postal Code, again, even if I live in one of the 60 or so countries or territories without one.

And then we look at the country drop down, always very instructive. I'll gloss over the "Falkland Islands (Islas Malvinas)" ... touchy subject ... and land instead on Serbia and Montenegro, a country that hasn't existed since 2006.

I await eagerly the next invitation to download a white paper. Who dares?

Chapeau Talend!

2009-11-23T10:47:00.001+01:00

Talend have corrected their web forms! Thanks to them for listening and reacting! Chapeau!

Broken business processes

2009-11-21T14:26:00.003+01:00

On the day that I posted about the poor web form design at Informatica I received an e-mail from Talend (another data quality company) inviting me to download a white paper. Inevitably, this wasn't free - I had to provide them with information. And on their drop down for country name they forgot Faeroe Islands but, more damagingly, included "Yugoslavia". Yugoslavia hasn't existed since 2003, and our memories shouldn't be so short that we forget the bloodshed which accompanied its disintegration.

Look, I know how difficult it is for companies to ensure that everything is known to all people in all departments. Information doesn't flow well - there are barriers everywhere, and though there are people at Talend and Informatica who know better than to make mistakes like this, they can't be everywhere checking everything before it gets posted.

But what really gets me down about these examples is that in both cases the companies concerned contacted me and let it be known that they saw the problems and would correct them. And yet in both cases the forms are still online and are still unchanged.

So how broken do your company processes have to be to allow such obvious embarrassments (people, you purport to be DATA QUALITY companies!) to remain online? What is standing in the way of actually correcting these forms? How many dissatisfied customers do you have to lose before anything changes? How bruised does my forehead have to become from bashing my head against these brick walls?

Somebody took the time to point out the errors. Do yourselves a favour Informatica and Talend - correct your forms!

Informatica and form rage

2009-10-30T10:00:00.006+01:00

Dear reader,

you'll know by now that the best way to achieve data quality is not by cleansing data after it has been collected - that's just an expensive way of mitigating the effects of poor data quality. You'll know that preventing data quality issues at source is a more effective and ultimately more cost effective way to manage data quality.

There are hundreds of thousands of companies who continue to attempt reactive data quality cleansing rather than instigating preventative data quality, and that won't change in the short (or even middle) term; but when you're a company working within the data quality sphere you have to be very careful about how you manage your own data quality, because if you don't ne'er do wells such as myself will be quick to pick up on it.

Informatica, a data quality company, posted a white paper - about data quality - here, and included with it a web form guaranteed to collect the worst quality data imaginable. (To me, a white paper is not free if I am expected to provide my information (which has value) in exchange for it - but that's another blog post ...). Now, don't get me wrong - I have nothing again Informatica as a company - they just seem (on this evidence) to have reached the size and structure which has stopped them being able to concentrate on data quality in all parts of their company, and with too many employees not understanding, or being part of, the data quality focus.

A quick look at the form and we can see some of the issues. Though I am allowed to enter data from my address in The Netherlands, I am forced to add an American state (or a Canadian province or territory) with which to pollute Informatica's data. The field labels suggest that my name is written in the same way as that of most Americans, that is given name first, family name last, and if I don't have a family name I'll have to make one up, because I can't leave that empty. I must add a postal code, even if my country doesn't have one, and though they have managed correctly to lose "Serbia and Montenegro" from their country list, they have lost Montenegro in the process.

When I pointed these errors out to Informatica they promised to recreate their forms, and they may be so doing; but it shouldn't take 15 days to stop a web form "State" field being a required field, one of the most obvious and widespread errors any web form can make, and the cause of more form rage than anything else. I hope they manage to get it sorted before next Tuesday, when I am presenting about web forms to the DDMA in Amsterdam - I'd like to be able to show a success story.

So, dear reader, do yourself a favour. Prevent your data becoming polluted at source. Look at your web forms. I'll make it easy for you - download my free e-book "Better data quality from your web form - Effective international name and address Internet data collection" and learn how to avoid those common errors. And I don't even ask you to fill in a form to get it ...

Data quality definitions: fit for purpose?

2009-10-02T10:40:00.002+02:00

As data quality professionals, some of us spend far too much time philosophising, particularly about how to define the term "Data quality".

Some professionals, particularly those in a business environment, define data quality as data which is fit for purpose. To me, far from clarifying, this definition throws up far too many new questions. Fit in what way, and for which purpose? Fit for my purpose or for his? Or both? Fit for the purpose I have now or those I may have in the future?

I don't like chain definitions, phrases that becomed defined by new phrases which themselves have to be defined - for example data quality = information quality = fit for purpose = ... This simply obfuscates the issues and moves us away from their core.

I also think we should avoid trying to attempt to bring definitions under umbrella terms when we are blessed with thousands of languages, each containing thousands of words, which can be used to define each issue. Instead of

"This data has no quality, because it doesn't help me do what I want to do"

wouldn't it be great if we said:

"The way this data has been provided to me is not fit for the purpose of calling all our customers as the telephone area code is not shown on the interface/printout"

without feeling we needed to park this under one or other defining phrase or buzz word?

After a couple of decades of intensive work with data, I firmly believe that data quality is an inherent property of the data itself and is not definable by what can be achieved with that data. But while I juggle with this issue in my head, I am open to other input. For me, data has quality if it is a true representation of the real world constructs to which it refers; being accurate, relevant, complete and up-to-date.

To me, if your data fulfils those criteria, there's nothing that can't be done with it and it could, if used properly, be fit for each and every purpose. In all my years working with data I've not found a case when this was not true.

Do YOU know of a case, real or imaginary, in which data that is accurate, relevant, complete and up-to-date would not be fit for each and every purpose? Note: we're talking about the data here, not information. If the data is not represented on your screen with the telephone area code, that's an information quality problem; but if the data is complete, relevant and up-to-date, the data will include the telephone area code which could therefore be used to make the information fit for purpose.

I'd love to hear of any examples! Leave a comment!

Data quality tools

2009-09-01T11:27:00.002+02:00

I've noticed an increasing tendency, especially in blogs, to use the term "data quality tool" to refer only to a piece of software.

We are doing the cause of data quality no favours at all by continuing the myth that data quality can only be improved through the purchase and use of a piece of software. Almost everything you do and touch within a company can affect data quality, and therefore a tool for improving data quality can range from a pencil and a piece of paper all the way to that Cray supercomputer over there in the corner. In fact, by far the most useful and effective data quality tool is the human brain. If we'd use it a bit more effectively, a great deal of data quality improvement would result!

Whois needs address validation

2009-07-11T12:17:00.005+02:00

A blog post here about a scam where people are sent to a site selling false anti-virus software demonstrated to me how introducing address validation to the Internet Corporation for Assigned Names and Numbers (ICANN) Whois database could help in the fight again such scammers.

The blog documents the websites through which the user is sent by the scam, and shows the whois entry for each of the sites.

Here's the first:

I don't think there is a Booth Street in Edmonton, but the real giveaway is the 6-digit postal code - not even close to a Canadian format.

The second:

Again, not a valid Brazilian postal code format.

The third:

An English-language address, in Melbourne, but in Russia? And with a 5-digit postal code? Shouldn't there be alarm bells going off somewhere about this obvious fakery??

Wouldn't we be better served if ICANN introduced validation, ranging from simple postal code format validation, through "can this address be in that country" validation right down to address-level validation, into their web registration database? I'm not fool enough to think that having to add a real address would stop scammers, but it would slow them down, as they'd have to find somebodys else's real address to add. Furthermore, having a database of only real addresses (i.e. of high data quality) instead of the current hodge-podge based on trust, would enable analysis of the data to improve identification of potential criminal activity. The current whois database may be fit for ICANN's purposes, but data quality it ain't.

Citroen and their Balkan confusion

2009-07-05T14:18:00.004+02:00

I was sent the link here, which shows a scan from a publication which appears to come from the car manufacturers Citroen and to date from 2009.

All looks fine until one looks more closely at the Balkans on this map.

Slovenia has expanded southwards, absorbing Croatia proper (including the capital, Zagreb) and Istria. Croatia has become but a rump of its former self, but has absorbed about half of Bosnia-Hercegovina, which is but a memory, as it's nowhere to be found on this map.

Citroen have succeeded where Slobodan Milosovic failed, and created a greater Serbia, including the Croatian area of Slavonia (and thus giving it a common border with Slovenia), Montenegro, Kosovo and a big chunk of Bosnia. In fact, only Macedonia is shown correctly, and even then the border has been drawn very approximately.

In fact, it looks as though somebody has taken a pre-1991 map showing Yugoslavia, and drawn in some borders where they think they might be, clearly more in hope than in expectation. Is a reminder needed of the number of people who died or were pushed out of their homes to fix the current borders of those countries? Should Citroen be thoroughly ashamed of publishing such a map?

Should road users everywhere be worried that the level of quality control during the manufacture of their cars matches that applied to their publications?

I do wonder how such errors get through the net - are people too lazy to reach for an atlas?

There goes another one

2009-06-04T18:00:00.003+02:00

My initial reaction to hearing that AddressDoctor was being taken over by Informatica was a sinking feeling - another one bites the dust.

Acquisitions of companies specialising in international name and address data are hardly new: Postalsoft (as First Logic) went to Business Objects before that was subsumed into SAP, who also snapped up Fuzzy! Informatik. QAS went to Experian and Global Address to Harte-Hanks.

For me, the best outcome of a takeover is that the acquired company is allowed to continue to run its business as independently of its owner as possible. Small companies tend to be more dynamic and react more quickly to change than their larger rivals. Global name and address data quality is a specialised business, and it requires the freedom to concentrate upon it without worries about the needs of a mother company.

The worst outcome is the absorption of the company into the mother company, so that it becomes a cog within a much larger corporate wheel. In my experience, when this happens the focus is lost, and, I have to say, the quality of the solution suffers. The needs of the larger company and its customers take precedence. I weep when a mature product disappears in this way. Even when it becomes part of the mother company's data quality suite, access to it is lost from the vast majority of small- and medium-sized companies who have the need for data quality but not the budget.

The vacuum created by the disappearance of a good global data quality company is not quickly or easily filled. Global address processing is an extraordinarily complex business. It takes years of knowledge collection, development of algorithms, trial and error before a product even approaches acceptable levels of quality in international data. There are quicker and easier ways to make money in the data world. It is eternally frustrating to witness this cycle of re-inventing the wheel as companies form, develop, are taken over and are absorbed into other companies, effectively disappearing.

I sincerely hope that the independent global data quality company, producing good and affordable solutions, don't become an endangered species.

What about the rest of the data?

2009-05-31T13:20:00.002+02:00

I'm currently reading a most worthy book about data quality, and, like most other books about data quality I've come across, its gaze is fixed completely on data held in large corporate entities. Large companies where data is amassed across myriad systems, current and legacy; where there is a separate IT department; where there are enough staff to create a data quality working group; where money is no object when it comes to tools to assess, process and cleanse that data.

But is that really where most of the world's data is held? Obviously it's where we most come in contact with it - when a large utility messes up its invoicing procedure, we know about it very quickly - but I would guess that more data is held in small spreadsheets and databases and documents in small- and medium-sized companies than is held in large corporations. I'm obviously not typical, but my own databases hold around 40 million records. In these small companies, there is unlikely to be distinct marketing or IT departments, no budget for data tools, not enough staff to create DQ teams.

Has anybody ever estimated how much data is held outside large corporations, or written about how they will go about improving their data quality? It must be the sun, as this is playing on my mind ...

How Microsoft Access can damage your data quality

2009-05-21T11:13:00.004+02:00

OK, so I've bitten the bullet. Much against my better judgement I'm making a concerted effort to learn Microsoft Access 2007, to allow me to produce some data tables with Unicode characters.

Plugging away, I found an interesting aspect of field input masks which is guaranteed to produce data quality issues. When adding a field mask to Access 2007, it "helpfully" provides a number of ready-made options:

Having a UK edition of the software, Microsoft have helpfully provided a telephone number mask and a postal code mask that it thinks covers the UK. Looking at the postal code mask itself:

and we see how it is made up. '0' indicates a required digit, 'L' a required letter, and '>' forces upper case.

Now, in the UK, AA99 9AA is indeed a valid postal code format. It is, however, only one of seven valid formats:

A9 9AA
A99 9AA
A9A 9AA
AA9 9AA
AA99 9AA
AA9A 9AA
AAA 9AA

Thus, whilst this mask can take OX19 6RY, it can't take OX9 6RY or SW1A 4WW or S1 1AA or N45 1AP or .... well, most of them.

Microsoft may feel that they are being helpful by adding this sample mask, but we all know that programmers, like most of us, will take any route that make their life easy, and are unlikely to make any attempt to alter this input mask to make it valid for the UK, let alone valid for postal codes in every other country. And we know that this happens - my God, don't we all suffer regularly at the hands of forms designed like this? Many programmers would not even be aware that the mask is not valid for most UK postal codes - they trust that the software provider has done their homework.

Back to the drawing board, please, Microsoft. This is not helping in the fight for better data quality.

Eurotunnelvision

2009-05-15T14:06:00.002+02:00

A slight move off topic, if I may. It's that time of the year again when the Eurovision Song Contest takes place, accompanied, as always, by the sounds of weeping, wailing and gnashing of teeth from the losers, who always try to find anything to blame apart from their own performances or the quality of their songs.

One of the most common complaints is the idea of block-voting, accompanied by a rather embarrassing lack of geographical knowledge. This year's Dutch song is doubtless very suitable for, and popular in, the local pubs of The Jordaan in Amsterdam, but is hardly material that will be popular elsewhere in Europe. Inevitably it failed to get through the semi-final. René Froger, one of the Dutch singers, immediately complained that this was due to block voting: "it's so strange that all the Balkan countries are through to the final", he wailed. And worst, no report I've seen pulled him up on it, seeding and strengthening the myth that everybody hates us/loves them.

Grabbing a pencil and a scrap of people, it took me no time at all to work out that six Balkan countries indeed made it to the final. And five did not. Not quite "all", and certainly no evidence of the great conspiracy so many profess to see. He'd have done better to point to the Nordic block, or the Caucasus block, or the Mediterranean block. If anybody should complain, it's the Central Europeans, none of whom got through.

Back to school, René!

For a map of those through, check Wikipedia here

Flying share

2009-05-14T13:08:00.003+02:00

Another great example of how ignorance affects data quality from web forms (and can lose customers and money!) has come my way, this time courtesy of the site Flying Share. Flying Share are offering their users in the US, UK, Canada and Australia, via this form, a free USB drive.

All well and good until you get to the "ZIP" field (ZIP? Not outside the USA, good people of Flying Share - you probably mean "Postal code"). And there you find that the field will accept five characters and no more. So anybody in the UK or Canada wanting a free drive must either give up at this point, or provide a truncated code.

As drives are being sent out postally, there will clearly be a huge problem of undeliverable and returned drives. I know that this error has been pointed out to the company concerned, but no action has yet been taken, though correcting it would take seconds.

I do wonder at what point anybody at Flying Share will feel the need to act on this. Perhaps when the costs of returns reaches astronomical proportions? How many (potential) customers might they have lost before then?

2009-05-05T11:23:00.003+02:00

I made a blog entry in March about Vietnam, where a postal code exists but without the populace knowing about it.

Two more examples caught my eye this week, in both cases where a country announced the introduction of a postal code system: Nigeria and Dominican Republic.

In both countries a postal code exists - in Nigeria's case since 2000 - and in both cases this information has been placed at one time or another on the national postal authority's website. It is very interesting that in all of the comments placed in reaction to these news items up to this point, only one persons suggested that they thought that a postal code system was already in place.

Regardless of the reasons why the existing postal code systems in either country had not been fully publicised before now, this is another indicator of how careful form designers need to be in their use of required fields. Nigeria may have a postal code system, but if nobody knows about it or uses it, requiring that field on a form will only lead to customer loss and data quality reduction.

For more information about web forms for an international audience, download the free e-book for here.

Free web form e-book

2009-04-24T10:45:00.000+02:00

I released today my new book: “Better data quality from your web form - Effective international name and address Internet data collection”. The good news is that it is completely free to download and use, so go to http://www.grcdi.nl/book4.htm to download it. Spread the word!

Web forms are a source of immense and continuous frustration for many people as the forms almost universally fail to take any account of variations in personal name and addressing conventions used throughout the world, so that customers have to struggle to clear the hurdles placed in their way by the forms. Reports suggest that almost 9 out of 10 customers have problems when attempting to carry out online transactions, and the result is a huge loss of income from lost custom paired with the collection of very poor quality data.

This e-book (in Acrobat pdf format), which is free to download and use, attempts to fill the gap left by most works about usability by concentrating on the experience that your international customers have with your form, and how it affects them, their relationship with you and your data quality.

This has been done using many examples and without going into too much depth about the idiosyncrasies of international personal names and addresses, and avoids technical discussions – links to get more information are provided where appropriate in the text.

Any feedback is most welcome.

Enjoy!

New product releases at GRC Database Information

2009-04-02T10:52:00.003+02:00

We've been busy in March preparing updates for most of our products. These include:

- an update to the Global Sourcebook for Address Data Management

- a new release for many of our data tables: place name/postal codes, address elements, job titles and others

- a new version of our address parsing, standardisation and formatting software GRCTools.

More information on our website.

Now you have your postal code system, how about telling us about it?

2009-03-07T08:44:00.004+01:00

My attention was caught today by an online discussion here hosted by the English-language Vietnam News, about introducing a postal code system to Vietnam to resolve the problem of duplicate thoroughfare names within cities.

Of interest to me is that Vietnam has had a postal code system for years, updating it from a 5-digit code to a 6-digit code in 2004. Clearly, though the system has been designed, it has not been implemented to the extent of informing the residents.

This is not as unusual as you might think. Some countries - Bahrain and Nicaragua spring to mind - are coy about their postal code systems, even to the extent of requesting people not to use them because sorting continues to be done manually. I've had people from Costa Rica swear blind to me that their country has no postal code system, even after I've pointed them towards the postal website describing it. Clearly, whilst some countries are happy to design postal code systems, the expense of implementing them - mechanised sorting systems, information dissemination and so on, tends to put a brake on following the projects through.

For the love of web forms!

2009-02-17T12:13:00.002+01:00

It may seem to you, faithful reader, that I have become a little obsessed with web data entry forms lately. Apart from being the front end of company databases that we crash into most regularly, it is also a reflection of the fact that I am currently working on a book about .... web data entry forms!

But I am not alone in this obsession. My colleagues across at the Data Value Talk blog have decided to try to think positively and to collect some examples of GOOD web data entry screens. If they get enough, they'll open up a competition for the best examples.

So, if you come across any well-designed (or badly designed!) web data entry forms, take a screen dump and scoot along to the Data Value Talk blog - I, for one, look forward to seeing them.

Stop. Step back. Think.

2009-02-10T12:15:00.004+01:00

Has creating web input forms become automatic? Does anybody ever test the forms they put online? Does anybody ever stop, take a step back, and think things through?

Yesterday, faced with yet another clone form expecting of me a typical United States' address, I came to the inevitable drop down requesting (no, demanding!) my state. The company gave no option to clear this field, nor to choose a non-US variant, so, as they had kindly defaulted to Alabama, I moved on.

For the next field, country, I chose "Netherlands". And look what happened:

Those who know me and my work will know that I have hammered on for years about making web forms dynamic, and that input fields should change on the basis of country and language. But why (oh why?), when a company follows this mantra, are the fields that cause this dynamic change ALWAYS added AFTER the fields that they change? Is it a conspiracy to annoy the customer? To make the customer work at their very hardest in order to buy the products?

Or is somebody just not thinking things through?

Data Quality: Perception versus Reality

2009-01-07T09:00:00.002+01:00

A new white paper, "Data Quality: Perception versus Reality", by Graham Rhind and sponsored by Capscan Ltd, reporting on the results of a data quality survey, is available for download here

BBC resurrects Serbia and Montenegro

2008-12-29T11:42:00.005+01:00

For a country that lasted only existed for a few years, Serbia & Montenegro has real staying power! In an online news report on the BBC News website from 26th December 2008, the BBC used a map of Serbia & Montenegro in a report instead of just Serbia.

To be fair, the image was corrected a few hours later:

but it does leave me wondering how long it will be before these errors are no longer being made. Perhaps a bloody war is required to force change into people's consciousness?

The world according to Barclaycard

2008-11-06T13:17:00.004+01:00

A letter in the Guardian pointed me to a competition being run by Barclaycard (a UK credit card company) to win an Aston Martin car, coinciding with the release of the newest James Bond film.

The terms and conditions take the term "geographically challenged" to new heights.

The United Kingdom, for the record, is the island of Great Britain with its islands, and Northern Ireland. If Barclaycard had wished to exclude Northern Ireland they could have used the term "Great Britain". "ROI" (Republic of Ireland) has been an independent nation for getting on for a century, and The Channel Islands are not now, and never have been, part of the United Kingdom.

One wonders, though, what the inhabitants of the Isle of Wight have done to upset Barclaycard!

GRC Database Information, welcome to the club!

2008-11-06T13:13:00.003+01:00

Yes, inevitably.

After creating these posts about companies taking years to realize that, for example,the country Serbia and Montenegro no longer exists, I thought it was perhaps best that I checked my online shop to be sure that there were no surprises there.

What did I find? Yep, Serbia & Montenegro, nestling in the countries list.

I hasten to add that this list is built in to the e-commerce software and can't be changed by the user, and the developers have since apologised and promised to correct it. But this does show how hard it can be to keep up with all real world changes in all parts of a company.

DHL, welcome to the club

2008-10-29T12:58:00.003+01:00

It didn't take long after writing the entry below about Barnes & Noble to find a further example of a company failing to effectively make its data quality initiative(s) continuous.

DHL has a list of country codes and postal code formats here.

Apart from missing a number of countries which have recently (and not so recently) created postal code systems, the list still contains Serbia and Montenegro, a country that hasn't existed since 2006;
it has the pre-1999 postal code format for Argentina, and the pre 2006/2007 formats for Armenia and Malta.

Postal code and addressing systems change at a surprisingly speedy rate, and information resources such as this one from DHL decay over time if they are not maintained. I don't know whether DHL's systems expect the same formats as are listed in this document (which I acknowledge has been posted on the Singaporean DHL site and may simply have been overlooked), but it would be, to say the least, commercially damaging if they did.

Barnes, Noble and hoops ...

2008-10-22T11:19:00.006+02:00

I recently came across this page on the Barnes & Noble online shop:

Entering international postal codes

explaining how they expected their customers to enter international postal codes so that their systems can validate the order.

One of the most obvious, but most overlooked, aspects of data quality is that it is a process and, importantly, that it is a continuous process. I'm all in favour of providing information, and lots of it, so that customers know what you expect, but posting a list like this and then forgetting about it demonstrates that the continuous aspect of data quality is being forgotten. The world has moved on, Barnes & Noble have not.

Like this, for example:

Yes, Argentina had a numeric, 4-digit postal code ... until 1999. Now their alphanumeric 7-digit code apparently is disallowed by Barnes & Noble. Or this:

Australia has never has a 7-digit postal code. I would give Barnes & Noble the benefit of the doubt an presume that they had mixed up the entries for Argentina and Australia, were it not for the insistence on a numerical code, whereas Argentinian codes contain 4 letters.

The other aspect of this page is the expectation that customers must jump through hoops to keep Barnes & Noble happy, to fit to B & N's technical requirements rather than B & N ensuring that their systems are correct. Take this example:

Grecian postal codes contain a space between the third and fouth digit. How long would a customer from Greece have to struggle to make their order? How much effort are they being put to to alter their (correct!) data to fit somebody else's system? Is this quality? Is this customer service.

No.

Personal names at Beijing: two steps forward, one step backwards

2008-08-13T11:42:00.002+02:00

Holding the Olympics Games in East Asia has had one small effect on spreading an understanding of global differences in personal name patterns. As East Asians write their name in a different order to us Anglo-Saxons, participant names are being written, for some countries, in the correct order, and with the family name shown in upper case:

WU Minxia
Stephanie RICE

It seems however, that this recognition is fairly limited: not all countries have had their personal name patterns recognised. And, with all that money spent, not a sign of a diacritical mark anywhere. A wasted opportunity to be truly global.

Does your airline know where it's going?

2008-07-10T14:19:00.008+02:00

I reckon that airlines have a very limited geographical understanding, limited to airports and without any understanding of the geographical context in which they are situated. I first began to suspect this when a data cleansing company (fortunately no longer extant) claimed American Airlines amongst its customers, and also claimed to be able to cleanse addresses for "more than 300 countries".

To save you having to run to an atlas to start counting, you'd be hard pressed, regardless of your definition of a country, to find many more than 240 on this planet.

My suspicions grew when I saw this SWISS map on the incomparable Strangemaps blog:

Now, the errors may not seem immediately obvious, but a great many cities have migrated significant distances away from their real locations. Pittsburgh is looking particularly moist, slap bang in the middle of Lake Eric, and Santiago in Chile has lost its name.

Then I was sure, after the same blog published a map from Aer Lingus:

I'm not sure how pleased the Americans would be to find so many of their cities placed in the north of Canada, but I'm pretty sure I know how Canadians would react.

I get the feeling that these errors have more to do with trying to get the maps looking nice and uncrowded than not knowing where places are; but this shows quite a lot of disrespect for the intelligence of their passengers. I'm looking forward to the next example, possibly showing London somewhere in the vicinity of Rome? I'm taking bets as to which airline it shall be!

Facebook

2008-07-10T14:13:00.002+02:00

One of the rules of an effective data collection web form is that drop downs should only be used when the number of possible answers is limited, and that all of those answers are given in the drop down.

The Guardian reported on 7th July that Facebook has blundered in this respect as their marital status question gives plenty of options - even including "it's complicated", but not including (same sex) civil partnerships (as available in the UK) or registered partnerships (as they are known in some other countries, such as The Netherlands).

It may not be possible for Facebook to research and know all the possible civil status possibilities around the world, but at least they have now provided an option to clarify the answer given.

Not that this may be much of a salve for those involved.

2008-06-16T13:31:00.002+02:00

How is your data?

Is everything working at the moment?

No weak links? No danger areas?

What happens when something causes a wave?

Will everything still be working? And how will your company's health be?

Credit card fraud

2008-06-13T15:04:00.002+02:00

Processing addresses is a challenge for all of us, credit card companies included. It was, therefore, no surprise to me to learn how cursory the address check on credit cards for online purchases actually is.

For many countries there is no address check.

For the UK the check is only on building number and the numbers of a postal code, so the check for 10 Downing Street, SW1A 2AA would be 1012. As a very large number of addresses in the UK will share this same code, it's a check that is very easy to get around.

Considering the amount credit card companies lose to fraudsters, and the stress involved for the legitimate credit card holder, isn't it time that these companies put some time and resources into learning about addresses?

Swiss Post

2008-06-01T15:05:00.005+02:00

Along with my order for the International AddressGuide (2007/2008 edition) from Swiss Post (more about that later) I received a "glow in the dark" postal code map of Europe.

Almost every inch of my walls is covered by postal code maps, and it looks very much like they take their data from the same source, as they share many idiosyncratic errors. Postal code area 59 in Italy, for example, is always missing; Leicester is shown as LG instead of LE on some of them, and so on.

I am, though, rather more concerned about this particular map being used as a marketing tool by Swiss Post to intimate their extensive coverage and knowledge. For the map shows Serbia & Montenegro as a single country (dissolved in 2006) and has Lithuania's pre-2004 postal code system.

Perhaps nobody will notice - after all, it glows in the dark! - and we all make mistakes - but I think somebody in Swiss Post's marketing department should have looked a bit harder before sending it out. It reflects badly on them when their international knowledge should be better than that.

Considering context

2008-04-21T13:21:00.002+02:00

One of my golden rules for improving data quality is to consider context. Any piece of data which is not considered in relation to other relevant data is highly likely to be misinterpreted. This is especially true of postal addresses.

This awareness of context may be the reason why I have a distinctive dislike of satellite navigation systems. They provide drivers with route information without providing any information about the context of that route, as would be available using a map. It gives the impression that drivers can switch off their brains and that their satellite navigation systems (inevitably using imperfect data) will lead them to precisely where they need to be. Road signs and other clues that they are going wrong can be ignored - the robotic voice can be followed without question.

Reports of satellite navigation disasters abound, and a new one has been reported this week. The village of Holdenhurst, near Bournemouth, UK, are having to change the name of their main street from Holdenhurst Road to Holdenhurst Village Road to try to reduce the number of lorry drivers being sent their by their satellite navigation systems instead of to Holdenhurst Road in nearby Bournemouth.

I, for one, will not be forsaking my paper maps.

Happy birthday ...

2008-02-29T15:00:00.002+01:00

... to all those born on 29th February. May I wish you much strength and patience when you're using those internet sites which won't accept that you were born on this date.

They still exist?

Inevitably.

Well meant, but ...

2008-02-28T14:02:00.002+01:00

Those of us who work with international data (and that's most companies on the planet these days) have to be very sensitive to the cultural norms and requirements of each country, nation, ethnic group, religion, language-speaker and so on. It is all too easy to make a mistake or a choice which will cause offence somewhere.

This was illustrated recently by a local hotel's attempt to foster better relations and understanding amongst their very varied workforce. A competition was held whereby members of staff who identified, and filled-in on a form, the nationalities of the greatest number of their colleagues, won a small prize. A very worthy cause which, however, caused great offence to my Croatian friend who works there as the list contained not "Croatia" but the defunct "Yugoslavia", and it used not the Yugoslavian flag, or even the Serbian national flag, but the Serbian state flag.

We all make mistakes, and they can be rectified. They can also be compounded, as this one was, as the hotel decided that it was unimportant and made no alteration to the paper concerned.

Hardly the way to demonstrate cultural understanding.

A country is not a nationality is not a language is not a flag ...

2008-01-28T15:02:00.000+01:00

It should be self-evident that a country name is not the same as a nationality, does not equate to a language and so on. That said, a surprising number of websites confuse particularly countries and nationalities. Often, a question asking for nationality is accompanied by a listing of countries.

On packaging, country names and flags are often used to indicate language, and this can backfire. On a package of candyfloss (British name) the information and ingredients listed under the heading UK: referred to the product as Cotton Candy (US and Canadian name) and used American spellings such as coloring instead of the British colouring.

Fit for purpose

2007-11-27T14:28:00.000+01:00

People rarely build a database system just for fun: they have a goal in mind, usually related to the data for which the database will form the receptacle. They will, therefore, need to consider always their objective(s) to ensure that the system they are building is fit for purpose.

It seems, however, that human nature often makes people concentrate on the process rather than the objective, and in the data world this directly affects data quality. I often come across companies and individuals who, when deciding between software A, which is slower but produces good quality data, and software B, which is faster but produces poor quality data, plump for software B because they have lost sight of the objective and are considering the process.

I often use the example of the Brussels underground system. The trains kept a strict timetable, which meant that each one would only stop at each station for a fixed number of seconds. The number of seconds did not appear to differ according to the volume of passengers at that station. As a result, the doors would close and trains would move off before all the passengers had had a chance to get on and off. Regular passengers knew about this, so when stepping onto the train would then simply stop where they were because they had to be in a position to get off again quickly, blocking the entrance and exit. At busy stations a scrum ensued as everybody tried to get on and off at the same time. Many a morning I had to walk to work from the stop after the one I had wanted because I hadn't managed to fight my way out in time.

Clearly, the transport company was concentrating on its process - sticking to its timetable - rather than its objective of moving passengers to where they needed to be.

I was reminded of this when I recently read of Amsterdam trams playing the same tricks. Some were whizzing past stops because they needed to stick to a timetable. Compounding this was the fact that the city of Amsterdam fine the tram company for each late running tram. Clearly nobody had thought this through properly - the city was basing its assessment of the tram company on the process rather than the objective. The tram company would be better off running all its trams without passengers and therefore avoiding fines, but that's hardly fit for purpose.

http://www.grcdi.nl

(c) 2007 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

The Black Swan and Data Quality

2007-11-12T14:14:00.001+01:00

First published online 19th October 2007

Nassim Nicholas Taleb, in his book The Black Swan, makes an interesting point which helped me to clarify in my mind why companies are too often willing to spend millions on attempting to cleanse and correct data after it has been collected rather than spending a fraction of that amount in increasing data quality through prevention, for example through validating data on input, a process that will always result in better quality data than any cleansing process employed later in the data process can produce.

What it boils down to is that reaction to problems can be perceived, measured and rewarded. Prevention results only in a lack of problems, which cannot be measured or rewarded. Most employees and managers feel more comfortable with the latter than the former.

To paraphrase Taleb, let us imagine that a legislator before 11th September 2001 introduced a law obliging airlines to fit armoured doors to the cabins of their planes, which were to be kept locked during flights. This would be unpopular with the airlines due to the extra costs involved, for example. Airlines might have to increase fares to cover costs, or reduce staffing levels, and that legislator would be unpopular. As this preventative measure would probably have ensured that the airlines used to attack New York and Washington on 11th September 2001 would not have been highjacked and the attacks would not have taken place, there would be a general perception that his measure was pointless and it would be impossible to measure what its effect had been because what it might be preventing is not taking place to measure.

My own favourite example of this is the millennium computer bug issue. Untold amounts of money were spent to prevent systems failing when the year changed to 2000, and domesday scenarios were described if the changes weren't made. As the clocks stuck midnight on 31st December 1999 no planes fell from skies and no nuclear power plants went into meltdown. Was there cheering and jubilation, back slapping and congratulations for the workers who had prevented this? On the contrary, they were labelled as a bunch of panic merchants who had exaggerated the problem to make a quick buck.

And so it is with data quality in almost all companies. An employee who can analyse badly collected data can show his bosses that there are, for example, errors in 50% of all records, and that after purchasing cleansing programs it could be reduced to 15%, is rewarded for producing measurable and verifiable results.

The better employee, who persuades his bosses to install systems to prevent data pollution earlier in the process can show, for example, errors in only 10% of the records, but as there is no worse start point with which to compare this figure, the employee could either be criticised that there are so many errors, despite the amount spent on prevention; or, more likely, be overlooked because nothing is being perceived as happening in their part of the company to draw managers' attentions.

I have seen this situation time and again in many different contexts. It is a rare manager indeed who can overcome the pressure to show figures in this way to justify their positions, and this reflects on data quality.

http://www.grcdi.nl

(c) 2007 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

Road to nowhere

2007-11-12T14:13:00.001+01:00

First published online 15th December 2006

I have been reading with interest the common reports of problems that motorists have been having with satellite navigation systems.

Having been able to inspect the data which these systems use, it comes as no surprise to me that there have been problems. Even had I not seen the data, we all know that even the smallest databases contain errors - databases of the size and complexity of those used in GIS systems, which attempt to represent an ever changing world, will inevitably contain many inaccuracies.

Many of the problems reported seem to revolve around motorists adopting a blind trust in their satellite navigation systems - something that almost everybody using data, including those doing it professionally, also tend to do. Show a person a piece of data, such as the string "London", and that person will make immediate assumptions about that data. However, one of my rules for good data management is to take data in context - that string may be a city in one context, but may refer to a name in another, for example.

Furthermore, the databases used by these satellite navigations systems do not contain data required for some drivers for choosing routes - height and width restrictions on roads, for example.

If users of satellite navigation systems were to take the data they are given in context - i.e. using that data along with information provided by road signs, maps and the evidence of their eyes, we would find fewer of them wedged in bends, unable to get up steep hills, and having to cross fields to get to roads they thought they were on already.

Other data users and managers would be well advised to consider if this metaphor applies to them ...

http://www.grcdi.nl

(c) 2006 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

The politics of personal names

2007-11-12T14:12:00.001+01:00

First published online 19th May 2006

The case of Ayaan Hirsi Ali, or Ayaan Hirsi Magan, raises interesting points about ignorance about, and the politics of, personal names. Ayaan Hirsi Ali was an outspoken member of the Dutch parliament. Originating in Somalia, she was naturalised as a Dutch citizen. Though it had been known for a long while that she has been born Ayaan Hirsi Magan, a sudden move by one of her colleagues resulted in the removal of her Dutch passport because she had lied about certain things on her application, including about her name.

Leaving aside the creeping structuralisation of racism within Dutch politics, this brings out some interesting points about personal names. Ayaan Hirsi Ali's name is Islamic. Moslim names have a greater flexibility that Western names, and they can change and flux as a person changes their personal situation - if they change jobs, for example, or have children. This is clearly not reflected in the decision taken against Ayaan Hirsi Ali.

Her case has also highlighted a number of others. In one case, for example, a Kurdish Iraqi by birth had his Dutch passport removed because he gave his Kurdish name whereas the Kurdophobe regime of Saddam (or Saddam Hussein, or Saddam Hussein al Tikriti or ..... - another flexible Moslim name) has registered the man with an Arabised first name.

Is this ignorance? Time to understand how personal names are created differently in different cultures.

http://www.grcdi.nl

(c) 2006 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

Reducing the need for scrap and rework with web data collection

2007-11-12T14:10:00.000+01:00

First published online 8th February 2006

When collecting data on the web, companies must allow diverse visitors to record their information in a way that is familiar and comfortable to them.

Though the Internet was once heralded as a solution for enabling cheap and effective data collection, experience has shown that this data is often too polluted to be useful in any business intelligence sense. This is not due to the medium, but to the poor understanding that most companies have of how to achieve quality data collection on the web without expensive scrap and rework.

The path normally followed by companies when choosing to collect data on the web is defined by the company decision-making structure and general ignorance of global diversity; and it dictates that scrap and rework will be a necessity. The path normally looks like this:

When a decision is made to collect customer information on the web, the short-term view of how to achieve this is usually chosen. A web data collection form can be up and running within a few hours. It does not usually require any special budgetary measures and answers the pressures from other company departments to get the data as quickly and cheaply as possible. Normal company structures militate against budget being made available to research and implement good data collection practices at the start of the process.

Little or no thought is given to this data collection page. The employees concerned stick with what they are familiar. They use the same fields, the same field labels and the same screen layout that they know from their own country. It is overlooked that a web page can be viewed from any place in the world, and that people from outside the company’s home country are likely to want to enter their details too. Similarly, it is overlooked that these visitors have personal details that do not coincide to the local norms. Ambiguous or country or language biased field labels will mean different things to different site visitors, causing them to provide different information based upon their interpretation. For example, a field entitled "Title"; may be filled in by one visitor with a form of address, by another with a job title and by yet a third with an academic title. In other countries, not only do people's name and address details consist of different components, they are also written in different ways. Their information may be too long to fit into the given fields; and required fields, for state or postal code, for example, will require them to enter nonsense information if their addresses do not contain such details. They may have more information than they can fit comfortably in the company’s web form. Because of this, they are required to shoehorn their data into the available space.

Data is collected, but it arrives in the database confused, concatenated, abbreviated, mis-fielded and completely useless. As the quantity of the data increases, it becomes clear that a major cleansing program is necessary to effectively use the data. Expensive software is acquired. This costs much more than it would have to create a good data entry system at the beginning. A better data structure must be identified, though this would have been better tackled before any data gathering began. In fact, the best way of getting top quality information from a customer is interacting with him or her at the time of data collection. This is true regardless of how expensive the software is, how many hours of labour are put into the process and how many processes are run.

The data is processed and a certain percentage is improved, but the stream of poor data from the data collection point continues. Thus, the data is assessed, scrapped and reworked as a continual process.

Companies do not often reach the end point of this path. Data remains of poor quality, with the resultant business process failures when the data or information from that data is used. Although these results are clearly not effective, this cycle has been followed by almost all companies. Not only does this result in bad data and its consequences (like poor customer image), but also an image and morale problem within the company. The data is not regarded as accurate, and is therefore underutilized. Budget is difficult to pin down to correct the problem because people are not confident about the outcome, and expensive processes do not show enough improvement to increase confidence. As people consider what has been spent already, they are reluctant to spend more.

The budgeting for web data collection should be moved to the beginning of the process, which is the design and execution of data collection processes and applications. With a small amount of investment and research, higher quality data can be collected from website visitors. Data collection pages, which dynamically alter form structure, order and language to the country and language of the visitor, allow visitors to record their information in a way that is familiar and comfortable to them. Field labels and lengths can be adjusted; and validation, both full postal and individual component validation, such as postal code length, can be implemented to reduce data pollution as much as possible. This is the only way that data can be collected on the web accurately enough to be fully used for business intelligence, without expensive scrap and rework.

http://www.grcdi.nl

(c) 2006 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

Integrated systems hide data

2007-11-12T14:08:00.000+01:00

First published online 29th March 2005

As data sets get larger and software systems more complex and powerful, data owners are being forced further and further away from their data.

Whether you hold data as an information resource, for marketing, CRM (Customer Relationship Marketing), database marketing, direct marketing or any other purposes, the success of your projects and your ability to comply with the Sarbanes-Oxley corporate governance act depend on the quality of your data.

Integrated data systems and software layers on top of your data often hide both the data and that data's quality problems, and often do not resolve the data quality problems. Technical aspects of large data projects, such as the use of SQL, conspire to isolate data from the user. Very few people now have the luxury of being able to browse through their data to allow their brains, the best data quality tool that exists, to work on that data.

This is bad news for data quality. My own analyses of international address data, even after cleaning, shows that sometimes more than 50% of data can be incorrect. If a company is analysing its sales, for example, on that basis, they could be making disastous business decisions.

Data and information quality cannot be assured simply by adding new system or software layers. The distance between the data owner and their data needs to be reduced again. A few moments of their time with their data can save enormous amounts of time and money in data quality requirements, and show immediately the data quality issues extant within that data.

(c) 2005 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

http://www.grcdi.nl

No shoehorns - asking the right questions

2007-11-12T13:56:00.000+01:00

First published online 12th November 2004

I recently gave a short seminar in London about global name and address management. Much of the time was taken up in emphasizing how culturally different names are addresses are around the world, and how systems must be adapted to match these differences, rather than trying to match real-world information to an inadequate system.

At the end, one of the questions was: "OK, I accept everything you say, but, if I'm stuck with a system which has been designed to take only American addresses, how can I shoehorn international addresses into that format?". The questioner was using Siebel. SAP users at the seminar all nodded in understanding. They all felt that they had to try to change the world because their systems demanded it.

The answer to the question, by the way, is that you can't. You can try, but you will damage your data, pollute your database and face huge bills to clean data up after the event. What is mysterious to me is why the users, large corporations who can vote with their wallets, don't turn this problem around and confront Siebel, SAP and cohorts, and demand that, given what they are charging for their systems, they make them more valid for storing global data. I think a few choice words and hinted threats to move to other systems from a small number of large users would make the suppliers move fast to do this. I haven't heard this being done, and I think it's time somebody started the ball rolling.

The secret is to ask the right question.

I recently read an article about parsing international personal names. The question asked by the author was "How would you split these names in order to put the proper elements in the surname field and given name field?". He then listed 6 personal names with different cultural origins. What I noted immediately was that one name was Islamic, where family names are not used; one name contained a generational name; one a patronym; one a preposition and so on. The idea that these names could have been shoehorned into an Anglo-Saxon given name/family name structure is not acceptable. In my eyes, the question being asked was wrong. Why try to force information into a system which is blatantly not suited to it? My question would have been: "How can I adjust my database structure to hold these personal name forms?"

Let's call time on showhorning and start asking the right questions!

(c) 2004 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

http://www.grcdi.nl

Postal authorities - who are their customers?

2007-11-12T13:48:00.000+01:00

First posted online 31st May 2004

The days of thinking of only the person who pays you for your service as your customer should be over. Following this theory, the supermarket's customer is the consumer, because he/she buys a product. The producer's customer is the supermarket, who buys the product from them. It is clear, however, that producers regard consumers as much as their customers as the supermarket. Whilst my local supermarket treats consumers like irritations (a supermarket chain heading for its biggest loss ever - maybe somebody should point out a potential link in this), the producers do their utmost to attract the consumer through advertising, packaging, shelf placement and so on. There are no chains of customers, there are networks of them.

In October 2003 I attended the European Mail Users' Forum in Brussels. Speakers came mainly either from large mailers or postal authorities. Of great interest to me was the way that both parties viewed their industry. There were some interesting examples of tunnel vision concerning particularly their customer chain. With a single honourable exception, postal authority speakers, for example, when speaking of their customers, were always referring to the mailers. In one way this is understandable - the mailers, after all, are the ones who pay for the delivery. This view, however, has no place in an industry where mail volumes are declining and the need for creative thinking is paramount. Only one speaker, that I heard, was forward thinking enough to consider that in postal matters there are two customers - for the postal authorities the customer is the one who sends the mail. For the mailer, the customer is the one who receives the mail. This being the case, the postal authorities need to take as much care in their attitude to the mailers' customer as to their own - without them, they have no business.

It is clear to me every day that the mail recipient is the poor man in the whole process. Though I was born and brought up in Britain, I live in Amsterdam. I still read a British newspaper, and follow the stories about the Royal Mail with interest. Though the British amongst you may not believe it, the service that the loss-making Royal Mail provides to the mail recipient is vastly superior to that provided by some postal companies which actually make a profit. When I lived in England I received mail before I left the house in the mornings. Here I cannot expect any mail before 11 am (2 pm on a Monday). If I want to get my mail before 9 am, I need to rent a postbox (kerching! $$$ Extra income for TPG). If collecting the mail from the postbox itself is a nuisance (and the number of post offices here is declining, just as it is in the UK), then I can pay TPG to deliver my mail from my postbox to my door. Kerching! $$$ TPG get money from the mailer and TWICE from the recipient to provide a service which those of us used to something better would regard as normal. Any wonder that they make a profit? The situation was not much different in Belgium when I lived there, except that you could not expect the mail before 2 pm any day.

As a mail receiver, I would much prefer the service offered by Royal Mail than the one I get from TPG.

When the post does arrive, it often arrives at the wrong place. In the Netherlands several different habitations often share the same house number. In the case of my house number it is five habitations - four flats and a houseboat. In each case, the delivery point is shown in the address by a suffix to the house number (or, in the case of the house boat opposite my front door, a prefix). Postmen and women often have a cavalier attitude to this information and throw the whole bundle in the nearest letterbox. If you stand at the end of the street and watch as the postman or women makes his/her rounds, you see behind them a sort of Benny Hill sketch developing - people coming out of their houses, milling around delivering the mail to the correct location. I may have to wait several days to get my mail, if my neighbour is not at home.

Apart from adopting a far more relevant attitude to the mail recipients, mail authorities would benefit immensely from improving their communication skills. This is almost universally applicable to postal authorities. I have spent a great deal of the last 14 years collecting information about postal systems and how they work - address formats, postal codes systems and so on. Many of my customers are very surprised that only a tiny amount of this information actually originated from postal authorities. There is a good reason for this. Postal authorities have been universally appalling at providing even basic information for their customers. Of all the letters, faxes and e-mails that I have sent in those 14 years requesting some basic information, I have received a reply to a single one (thumbs up Hong Kong Post, though I bruised myself when I fell off my chair in surprise).

I know that some of my colleagues are better at getting replies than I am, but it would be a shame if users had to learn a specific technique to get replies to their communications. Even the Internet sites of postal authorities show how communication needs to be given a much higher priority in their business plans. In order to use a postal system effectively, users need information. They need to know how to address a letter, whether a postal code is required, what the postal code is and so on; and this information needs to be provided in a language that they understand. If more users follow the rules laid down by the postal authorities for their addressing, logically it will cost the authorities less to process the mail piece and will lead to better use of their automation investment and greater profit (or less loss). However, dozens of postal authorities have no website. Many of those which do have sites fail to provide information which is essential, such as postal code information. Still others fail to understand that their postal system will be used by cross-border mailers as well as domestic mailers, and post information only in their local language. It is very interesting (and quite depressing) in many cases to compare the local language site of a postal authority with the English language version, and to see how much the postal authority don't think you need to know if you don't speak that local language. It's hardly surprising that companies are forced to come to me or one of my industry colleagues when they need this information.

Of course, there are exceptions, but any move towards a better understanding of the needs of the mail recipient and a greater communication towards all users, both mailers and recipients, would be both welcome and profitable.

(c) 2004 Graham Rhind. Reproduction only allowed with permission. Comment and dialogue welcome.

http://www.grcdi.nl