Monday, November 23, 2009
Chapeau Talend!
Talend have corrected their web forms! Thanks to them for listening and reacting! Chapeau!
Saturday, November 21, 2009
Broken business processes
On the day that I posted about the poor web form design at Informatica I received an e-mail from Talend (another data quality company) inviting me to download a white paper. Inevitably, this wasn't free - I had to provide them with information. And on their drop down for country name they forgot Faeroe Islands but, more damagingly, included "Yugoslavia". Yugoslavia hasn't existed since 2003, and our memories shouldn't be so short that we forget the bloodshed which accompanied its disintegration.

Look, I know how difficult it is for companies to ensure that everything is known to all people in all departments. Information doesn't flow well - there are barriers everywhere, and though there are people at Talend and Informatica who know better than to make mistakes like this, they can't be everywhere checking everything before it gets posted.
But what really gets me down about these examples is that in both cases the companies concerned contacted me and let it be known that they saw the problems and would correct them. And yet in both cases the forms are still online and are still unchanged.
So how broken do your company processes have to be to allow such obvious embarrassments (people, you purport to be DATA QUALITY companies!) to remain online? What is standing in the way of actually correcting these forms? How many dissatisfied customers do you have to lose before anything changes? How bruised does my forehead have to become from bashing my head against these brick walls?
Somebody took the time to point out the errors. Do yourselves a favour Informatica and Talend - correct your forms!

Look, I know how difficult it is for companies to ensure that everything is known to all people in all departments. Information doesn't flow well - there are barriers everywhere, and though there are people at Talend and Informatica who know better than to make mistakes like this, they can't be everywhere checking everything before it gets posted.
But what really gets me down about these examples is that in both cases the companies concerned contacted me and let it be known that they saw the problems and would correct them. And yet in both cases the forms are still online and are still unchanged.
So how broken do your company processes have to be to allow such obvious embarrassments (people, you purport to be DATA QUALITY companies!) to remain online? What is standing in the way of actually correcting these forms? How many dissatisfied customers do you have to lose before anything changes? How bruised does my forehead have to become from bashing my head against these brick walls?
Somebody took the time to point out the errors. Do yourselves a favour Informatica and Talend - correct your forms!
Friday, October 30, 2009
Informatica and form rage
Dear reader,
you'll know by now that the best way to achieve data quality is not by cleansing data after it has been collected - that's just an expensive way of mitigating the effects of poor data quality. You'll know that preventing data quality issues at source is a more effective and ultimately more cost effective way to manage data quality.
There are hundreds of thousands of companies who continue to attempt reactive data quality cleansing rather than instigating preventative data quality, and that won't change in the short (or even middle) term; but when you're a company working within the data quality sphere you have to be very careful about how you manage your own data quality, because if you don't ne'er do wells such as myself will be quick to pick up on it.
Informatica, a data quality company, posted a white paper - about data quality - here, and included with it a web form guaranteed to collect the worst quality data imaginable. (To me, a white paper is not free if I am expected to provide my information (which has value) in exchange for it - but that's another blog post ...). Now, don't get me wrong - I have nothing again Informatica as a company - they just seem (on this evidence) to have reached the size and structure which has stopped them being able to concentrate on data quality in all parts of their company, and with too many employees not understanding, or being part of, the data quality focus.

A quick look at the form and we can see some of the issues. Though I am allowed to enter data from my address in The Netherlands, I am forced to add an American state (or a Canadian province or territory) with which to pollute Informatica's data. The field labels suggest that my name is written in the same way as that of most Americans, that is given name first, family name last, and if I don't have a family name I'll have to make one up, because I can't leave that empty. I must add a postal code, even if my country doesn't have one, and though they have managed correctly to lose "Serbia and Montenegro" from their country list, they have lost Montenegro in the process.

When I pointed these errors out to Informatica they promised to recreate their forms, and they may be so doing; but it shouldn't take 15 days to stop a web form "State" field being a required field, one of the most obvious and widespread errors any web form can make, and the cause of more form rage than anything else. I hope they manage to get it sorted before next Tuesday, when I am presenting about web forms to the DDMA in Amsterdam - I'd like to be able to show a success story.
So, dear reader, do yourself a favour. Prevent your data becoming polluted at source. Look at your web forms. I'll make it easy for you - download my free e-book "Better data quality from your web form - Effective international name and address Internet data collection" and learn how to avoid those common errors. And I don't even ask you to fill in a form to get it ...
you'll know by now that the best way to achieve data quality is not by cleansing data after it has been collected - that's just an expensive way of mitigating the effects of poor data quality. You'll know that preventing data quality issues at source is a more effective and ultimately more cost effective way to manage data quality.
There are hundreds of thousands of companies who continue to attempt reactive data quality cleansing rather than instigating preventative data quality, and that won't change in the short (or even middle) term; but when you're a company working within the data quality sphere you have to be very careful about how you manage your own data quality, because if you don't ne'er do wells such as myself will be quick to pick up on it.
Informatica, a data quality company, posted a white paper - about data quality - here, and included with it a web form guaranteed to collect the worst quality data imaginable. (To me, a white paper is not free if I am expected to provide my information (which has value) in exchange for it - but that's another blog post ...). Now, don't get me wrong - I have nothing again Informatica as a company - they just seem (on this evidence) to have reached the size and structure which has stopped them being able to concentrate on data quality in all parts of their company, and with too many employees not understanding, or being part of, the data quality focus.

A quick look at the form and we can see some of the issues. Though I am allowed to enter data from my address in The Netherlands, I am forced to add an American state (or a Canadian province or territory) with which to pollute Informatica's data. The field labels suggest that my name is written in the same way as that of most Americans, that is given name first, family name last, and if I don't have a family name I'll have to make one up, because I can't leave that empty. I must add a postal code, even if my country doesn't have one, and though they have managed correctly to lose "Serbia and Montenegro" from their country list, they have lost Montenegro in the process.

When I pointed these errors out to Informatica they promised to recreate their forms, and they may be so doing; but it shouldn't take 15 days to stop a web form "State" field being a required field, one of the most obvious and widespread errors any web form can make, and the cause of more form rage than anything else. I hope they manage to get it sorted before next Tuesday, when I am presenting about web forms to the DDMA in Amsterdam - I'd like to be able to show a success story.
So, dear reader, do yourself a favour. Prevent your data becoming polluted at source. Look at your web forms. I'll make it easy for you - download my free e-book "Better data quality from your web form - Effective international name and address Internet data collection" and learn how to avoid those common errors. And I don't even ask you to fill in a form to get it ...
Friday, October 2, 2009
Data quality definitions: fit for purpose?
As data quality professionals, some of us spend far too much time philosophising, particularly about how to define the term "Data quality".
Some professionals, particularly those in a business environment, define data quality as data which is fit for purpose. To me, far from clarifying, this definition throws up far too many new questions. Fit in what way, and for which purpose? Fit for my purpose or for his? Or both? Fit for the purpose I have now or those I may have in the future?
I don't like chain definitions, phrases that becomed defined by new phrases which themselves have to be defined - for example data quality = information quality = fit for purpose = ... This simply obfuscates the issues and moves us away from their core.
I also think we should avoid trying to attempt to bring definitions under umbrella terms when we are blessed with thousands of languages, each containing thousands of words, which can be used to define each issue. Instead of
"This data has no quality, because it doesn't help me do what I want to do"
wouldn't it be great if we said:
"The way this data has been provided to me is not fit for the purpose of calling all our customers as the telephone area code is not shown on the interface/printout"
without feeling we needed to park this under one or other defining phrase or buzz word?
After a couple of decades of intensive work with data, I firmly believe that data quality is an inherent property of the data itself and is not definable by what can be achieved with that data. But while I juggle with this issue in my head, I am open to other input. For me, data has quality if it is a true representation of the real world constructs to which it refers; being accurate, relevant, complete and up-to-date.
To me, if your data fulfils those criteria, there's nothing that can't be done with it and it could, if used properly, be fit for each and every purpose. In all my years working with data I've not found a case when this was not true.
Do YOU know of a case, real or imaginary, in which data that is accurate, relevant, complete and up-to-date would not be fit for each and every purpose? Note: we're talking about the data here, not information. If the data is not represented on your screen with the telephone area code, that's an information quality problem; but if the data is complete, relevant and up-to-date, the data will include the telephone area code which could therefore be used to make the information fit for purpose.
I'd love to hear of any examples! Leave a comment!
Some professionals, particularly those in a business environment, define data quality as data which is fit for purpose. To me, far from clarifying, this definition throws up far too many new questions. Fit in what way, and for which purpose? Fit for my purpose or for his? Or both? Fit for the purpose I have now or those I may have in the future?
I don't like chain definitions, phrases that becomed defined by new phrases which themselves have to be defined - for example data quality = information quality = fit for purpose = ... This simply obfuscates the issues and moves us away from their core.
I also think we should avoid trying to attempt to bring definitions under umbrella terms when we are blessed with thousands of languages, each containing thousands of words, which can be used to define each issue. Instead of
"This data has no quality, because it doesn't help me do what I want to do"
wouldn't it be great if we said:
"The way this data has been provided to me is not fit for the purpose of calling all our customers as the telephone area code is not shown on the interface/printout"
without feeling we needed to park this under one or other defining phrase or buzz word?
After a couple of decades of intensive work with data, I firmly believe that data quality is an inherent property of the data itself and is not definable by what can be achieved with that data. But while I juggle with this issue in my head, I am open to other input. For me, data has quality if it is a true representation of the real world constructs to which it refers; being accurate, relevant, complete and up-to-date.
To me, if your data fulfils those criteria, there's nothing that can't be done with it and it could, if used properly, be fit for each and every purpose. In all my years working with data I've not found a case when this was not true.
Do YOU know of a case, real or imaginary, in which data that is accurate, relevant, complete and up-to-date would not be fit for each and every purpose? Note: we're talking about the data here, not information. If the data is not represented on your screen with the telephone area code, that's an information quality problem; but if the data is complete, relevant and up-to-date, the data will include the telephone area code which could therefore be used to make the information fit for purpose.
I'd love to hear of any examples! Leave a comment!
Tuesday, September 1, 2009
Data quality tools
I've noticed an increasing tendency, especially in blogs, to use the term "data quality tool" to refer only to a piece of software.
We are doing the cause of data quality no favours at all by continuing the myth that data quality can only be improved through the purchase and use of a piece of software. Almost everything you do and touch within a company can affect data quality, and therefore a tool for improving data quality can range from a pencil and a piece of paper all the way to that Cray supercomputer over there in the corner. In fact, by far the most useful and effective data quality tool is the human brain. If we'd use it a bit more effectively, a great deal of data quality improvement would result!
We are doing the cause of data quality no favours at all by continuing the myth that data quality can only be improved through the purchase and use of a piece of software. Almost everything you do and touch within a company can affect data quality, and therefore a tool for improving data quality can range from a pencil and a piece of paper all the way to that Cray supercomputer over there in the corner. In fact, by far the most useful and effective data quality tool is the human brain. If we'd use it a bit more effectively, a great deal of data quality improvement would result!
Saturday, July 11, 2009
Whois needs address validation
A blog post here about a scam where people are sent to a site selling false anti-virus software demonstrated to me how introducing address validation to the Internet Corporation for Assigned Names and Numbers (ICANN) Whois database could help in the fight again such scammers.
The blog documents the websites through which the user is sent by the scam, and shows the whois entry for each of the sites.
Here's the first:

I don't think there is a Booth Street in Edmonton, but the real giveaway is the 6-digit postal code - not even close to a Canadian format.
The second:

Again, not a valid Brazilian postal code format.
The third:

An English-language address, in Melbourne, but in Russia? And with a 5-digit postal code? Shouldn't there be alarm bells going off somewhere about this obvious fakery??
Wouldn't we be better served if ICANN introduced validation, ranging from simple postal code format validation, through "can this address be in that country" validation right down to address-level validation, into their web registration database? I'm not fool enough to think that having to add a real address would stop scammers, but it would slow them down, as they'd have to find somebodys else's real address to add. Furthermore, having a database of only real addresses (i.e. of high data quality) instead of the current hodge-podge based on trust, would enable analysis of the data to improve identification of potential criminal activity. The current whois database may be fit for ICANN's purposes, but data quality it ain't.
The blog documents the websites through which the user is sent by the scam, and shows the whois entry for each of the sites.
Here's the first:

I don't think there is a Booth Street in Edmonton, but the real giveaway is the 6-digit postal code - not even close to a Canadian format.
The second:

Again, not a valid Brazilian postal code format.
The third:

An English-language address, in Melbourne, but in Russia? And with a 5-digit postal code? Shouldn't there be alarm bells going off somewhere about this obvious fakery??
Wouldn't we be better served if ICANN introduced validation, ranging from simple postal code format validation, through "can this address be in that country" validation right down to address-level validation, into their web registration database? I'm not fool enough to think that having to add a real address would stop scammers, but it would slow them down, as they'd have to find somebodys else's real address to add. Furthermore, having a database of only real addresses (i.e. of high data quality) instead of the current hodge-podge based on trust, would enable analysis of the data to improve identification of potential criminal activity. The current whois database may be fit for ICANN's purposes, but data quality it ain't.
Sunday, July 5, 2009
Citroen and their Balkan confusion
I was sent the link here, which shows a scan from a publication which appears to come from the car manufacturers Citroen and to date from 2009.
All looks fine until one looks more closely at the Balkans on this map.

Slovenia has expanded southwards, absorbing Croatia proper (including the capital, Zagreb) and Istria. Croatia has become but a rump of its former self, but has absorbed about half of Bosnia-Hercegovina, which is but a memory, as it's nowhere to be found on this map.
Citroen have succeeded where Slobodan Milosovic failed, and created a greater Serbia, including the Croatian area of Slavonia (and thus giving it a common border with Slovenia), Montenegro, Kosovo and a big chunk of Bosnia. In fact, only Macedonia is shown correctly, and even then the border has been drawn very approximately.
In fact, it looks as though somebody has taken a pre-1991 map showing Yugoslavia, and drawn in some borders where they think they might be, clearly more in hope than in expectation. Is a reminder needed of the number of people who died or were pushed out of their homes to fix the current borders of those countries? Should Citroen be thoroughly ashamed of publishing such a map?
Should road users everywhere be worried that the level of quality control during the manufacture of their cars matches that applied to their publications?
I do wonder how such errors get through the net - are people too lazy to reach for an atlas?
All looks fine until one looks more closely at the Balkans on this map.

Slovenia has expanded southwards, absorbing Croatia proper (including the capital, Zagreb) and Istria. Croatia has become but a rump of its former self, but has absorbed about half of Bosnia-Hercegovina, which is but a memory, as it's nowhere to be found on this map.
Citroen have succeeded where Slobodan Milosovic failed, and created a greater Serbia, including the Croatian area of Slavonia (and thus giving it a common border with Slovenia), Montenegro, Kosovo and a big chunk of Bosnia. In fact, only Macedonia is shown correctly, and even then the border has been drawn very approximately.
In fact, it looks as though somebody has taken a pre-1991 map showing Yugoslavia, and drawn in some borders where they think they might be, clearly more in hope than in expectation. Is a reminder needed of the number of people who died or were pushed out of their homes to fix the current borders of those countries? Should Citroen be thoroughly ashamed of publishing such a map?
Should road users everywhere be worried that the level of quality control during the manufacture of their cars matches that applied to their publications?
I do wonder how such errors get through the net - are people too lazy to reach for an atlas?
Subscribe to:
Posts (Atom)