Dissertation Series - Resistances to Data Mining - Legislation concerns

On a similar note to the privacy concerns, some organisations may be concerned about making use of data mining because of the possible legal implications of doing so.  In the United Kingdom data legislation is primarily covered by the data protection act, section 12 specifically mentioning “rights in relation to automated decision-taking” (Information Commissioners Office B, 2012).

Dissertation Series - Resistances to Data Mining - Awareness

Before an organisation commits resources to any development there must have been a catalyst to drive the company down that development route.  Technical staff either have to make managers aware of new technologies that could be applied or senior managers need to be aware of these technologies and seek to employ them in their organisation. 

This rational can be applied to data mining, if there isn’t awareness in the organisation of data mining then it is unlikely it would be pursued as a future development.

Dissertation Series - Resistances to Data Mining - Accuracy

Data mining is in many cases used to forecast/predict an outcome, it does this with a degree of accuracy although it is important to note that is a forecast/prediction and not an actual.  For this reason results taken from data mining exercises should be acted on with this in mind.  Any the action taken as a result of data mining should be with the consideration as to what the impact would be if it were applied where the prediction incorrect.

These incorrect predictions are referred to as false positives, which is where something is being flagged as something it is not, where as a false negative is where something is not grouped as it should (Thrasingham, 1999 p.93).

Thuraisingham (1999 p.93) identifies the possible implications of acting on these “false positives”.

“if an agency finds incorrectly that its employee has carried out fraudulent acts and then  starts to investigate his behaviour, and if this is known to the employee, then it could damage him”

Conversely the same logic applies to false negatives “we do not want the data miner to return a result that the employee was well behaved when he is a fraud” (Thrasingham, 1999 p.93).

An area in which many consumers would have been exposed to the false positives of data mining is credit/debit card companies, as part of their fraud prevention systems banks look at consumer transaction patterns and place temporary blocks on cards that exhibit those that match patterns of stolen cards (AAAI, 2012).  A temporary block on a card requires the customer to contact their bank to unblock the card, again these false positives can be frustrating or even embarrassing for the individual concerned.

 “just because an individual makes a series of credit card purchase that are similar to those often made when a card is stolen does not mean that the card is stolen or that the individual is a criminal” (Dunham, 2003 p.16)

Dissertation Series - Resistances to Data Mining - Privacy

The introduction of new technologies or adaptation of existing technologies within an organisation can bring with it resistance from the different layers of the organisation; from the management resisting introduction to the operational staff resisting the use/uptake.  There are many reasons for management/operational staff resisting changes in an organisation and approaches to mitigating them (Davidson, 2009), so the focus will be those specific to data mining.

Privacy

Data mining against individuals inevitably makes use of large amounts of personal data (Busovky, 2011), with this brings concerns of data privacy and the high profile data breaches reported in the media (BBC, 2009).

Wilder and Soat (Wilder et al, 2001) cite an example of N2H2 a Seattle based company that provides internet filtering content software to schools, using that data they planned to sell the anonymised aggregated data.

“N2H2 began marketing the data, called Class Clicks, that it’s filtering tools collected on the website usage trends of elementary and high school students.  The data contained no names or personal information and complied with the new deferral Children’s Online Privacy Protection Act.  Yet N2H2’s new line of business brought such loud howls of protest from online privacy advocates that the company scrapped the effort”

A fictitious example is given by Wang and Liu (Wang et al, 2011) to illustrate the real privacy concerns that could exist when mining a medical database.

“released mining output can also be leveraged to uncover some combinations of symptoms that are so special that only rare people match them” “which qualifies as a severe threat to individuals privacy”

Many countries have legislation in place to protect individuals and ensure organisations put in place safe guards and controls to protect personal data, the main act in the United Kingdom being the Data Protection Act 1998, which covers many areas of data protection.  Specific to privacy the seventh principle of the act applies;

“Appropriate technical and organisational measures shall be taken against unauthorised or unlawful processing of personal data and against accidental loss or destruction of, or damage to, personal data” (Information Commissioners Office A, 2012).

There are techniques to prevent unauthorised disclosure of personal data through data mining:

Anonymisation

Privacy can be ensured through anonymising data, however simply removing customer reference number/names is not in itself always sufficient as discussed by Vaidya et al (2005 p.8) “just because the individual is not identifiable in the data is not sufficient; joining the data with other sources must not enable identification”.

An established approach to ensure that data is truly anonymised is “k-anonymity” which is a process that involves the grouping of individuals together within the data (Vaidya et al, 2005 p.8).

Suppression can also be introduced to hide groups/data that consist of small and easily identified sample sizes; this requires footnotes and an accompanying narrative to explain that this has been done; to prevent a misunderstanding of any summarised data (Vaidya et al, 2005 p.8).

Clearly defined use of data

Another method to control concerns about privacy is to clearly outline to the data subjects at point of data collection what the data will be used for and the associated benefits to them. 

This is evidenced by the success of the Tesco clubcard scheme and its changed perception amongst it customers and it’s the separation of its mailings from previously “dumb” junk mail.

“research consistency suggest that customers perceive the quarterly mailing from Tesco clubcard not as ‘junk mail’, but as personal mail” (Humby et al, 2004 p.116).

An example of poor understanding between the data subject and the organisaiton carrying out the data mining process is the case of pharmacies in the US that were selling data gathered from prescriptions to pharmaceutical companies to be data mined.  The pharmaceutical companies were then using that data to target marketing/sales towards specific doctors, based on the prescriptions they had written (Silverman, 2008).  The data subjects in this case (the doctors) represented by the American college of Physicians have opposed the use of this data for marketing (Walker, 2011).

However the example also speaks about the use of the data for other purposes;

“direct safety messages to doctors, to track disease progression, to aid law enforcement, to implement risk-mitigation programs, and to do post-marketing surveillance required by the FDA” (Walker, 2011)

It is where there is a benefit and consent between the data subject, the organisation and its use of the data mining, that there is less likelihood of resistance to data being mined.

SSRS Series - How to format addresses in SSRS

·         How to format an address block in SSRS, ideal for SSRS generated letters, invoices, bills and orders. 

·         Removes any blank address lines that the record has in your database.

·         Adds carriage returns after each populated line to give the distinctive address block appearance

There are two ways to approach msaaging/formatting fields into an address block in SSRS, you can either carry out your formatting in the SQL that populates the dataset or you can format the fields from the dataset in an SSRS expression. 

I’ve found that the easiest method is to format the address through an SSRS expression contained in the report, this keeps the SQL that populates the dataset as clear as possible and any issues associated with the expression are easily identifiable.

Example Data

·         Stored in dataset titled “DATASET”

·         The exact number/naming of fields will differ depending on your database.

MAILING_NAME	Mr J Davies
ADD1	The Whitehouse
ADD2
ADD3	Reindeer Road
ADD4
ADD5
TOWN	Reading
PCODE	NA13 6DF
COUNTRY	United Kingdom

Example expression

·         Create a new text box and add the following expression (change the name of the dataset/fields to reflect your data).

=First(Fields!MAILING_NAME.Value,”DATASET”)+IIF(First(Fields!MAILING_NAME.Value,”DATASET”) ="","",", "+VbCrLf)

+First(Fields!ADD1.Value,”DATASET”)+IIF(First(Fields!ADD1.Value,”DATASET”) ="","",", "+VbCrLf)

+First(Fields!ADD2.Value,”DATASET”)+IIF(First(Fields!ADD2.Value,”DATASET”) ="","",", "+VbCrLf)

+First(Fields!ADD3.Value,”DATASET”)+IIF(First(Fields!ADD3.Value,”DATASET”) ="","",", "+VbCrLf)

+First(Fields!ADD4.Value,”DATASET”)+IIF(First(Fields!ADD4.Value,”DATASET”) ="","",", "+VbCrLf)

+First(Fields!ADD5.Value,”DATASET”)+IIF(First(Fields!ADD5.Value,”DATASET”) ="","",", "+VbCrLf)

+First(Fields!TOWN.Value,”DATASET”)+IIF(First(Fields!TOWN.Value,”DATASET”) ="","",", "+VbCrLf)

+First(Fields!PCODE.Value,”DATASET”)+IIF(First(Fields!PCODE.Value,”DATASET”) ="","",""+VbCrLf)

+First(Fields!COUNTRY.Value,”DATASET”)

Example report view

Mr J Davies,

The Whitehouse,

Reindeer Road,

Reading,

NA13 6DF,

United Kingdom

Dissertation Series - Data Mining Modelling Techniques

Within data mining there are various modelling techniques that can be undertaken, the type used will depend largely on the situation, data available and type of problem the organisation is trying to be solved/addressed.

Description

Description is the identification of trends/patterns within data, an example of such a trend would be that learners with poor attendance don’t achieve as highly as learners that have good attendance (Daniel, 2004 p.11).

Estimation

Estimation uses complete records to estimate an outcome.  An example of this would be to estimate this years income based on previous years income (Daniel, 2004 p.11).

Prediction

Prediction is similar to estimation although uses incomplete records to predict an outcome (Veerman et al, 2009 p.13).  An example of this would be the prediction of fourth quarter sales, based on the sales in the previous three quarters of the year (Daniel, 2004 p.11)

Classification

A classification modelling technique forecasts and groups outcome into a category/classification, for example weather being sunny or a learner going on to study at university level (Daniel, 2004 p.11).

Clustering

The technique of clustering involves grouping together those with similar characteristics; an example of clustering would be the grouping together of customers who go on to buy a certain type of product.  New customers that also appear within that cluster could be targeted for that certain type of product that other customers in that cluster purchased (Daniel, 2004 p.11).

Association

The technique of association is sometimes referred to as basket analysis and is the method used to forecast additional items that a customer may wish to buy based on items they purchased previously (Aberer, 2007)

Dissertation series - Data mining overview

My masters dissertation specialised in the topic of data mining and I will be publishing some of the content that I assembled as part of my research for others to use.

Data mining is a discipline within IT that involves the manipulation of data through algorithms to extract undiscovered patterns/correlations (Dunham, 2003 p.3), often in large and diverse databases (Thuraisingham, 1999 p.2).  “Data mining uses classification algorithms and learns from the related or linked data” (Thiruvadi and Patel, 2011 p.711).

Gouda and Hassaan (2011 p.179) give a straight forward example of data mining;

“Let’s say the database records the books bought by each customer over a period of time.  The discovered patterns are the sequences of books most frequently bought by the customers.  An example could be that, “70% of the people who buy introduction to visual basic and introduction to C++ also buy introduction to Perl within a month.”  Stores can use these patterns for promotions, shelf placement etc.”

Data mining is distinctly different to traditional data manipulation (i.e. the use of SQL, a data analyst and reporting tools) where data analysts have to undertake a large amount of manual work (Daniel, 2004 p. xi) and usually have something specific that they are looking for (Dunham, 2003 p. 3) which would fail to uncover certain patterns.  Instead data mining makes use of a more automated and open minded approach to the identification of patterns (Hanna, 2004 p. 132).

Data mining can also be referred to as Knowledge Discovery in Databases (Baker, 2011), machine learning (Alpaydin, 2004 p.2) and data archaeology (Thuraisingham, 1999 p.2).

The application of data mining can be seen as an additional way of gaining further value from data collected and held “enhance the value of existing information resources” (Thearling, 2010) a common theme in area of data mining is the plethora of data stored by organisations “data rich” (Al-atta, 2011) and the inability to do anything with it (Kushima et al, 2011 p.215).