Dissertation Series - CRISP DM Step Six Deployment

Plan deployment

Deploying new software/solutions in an organisation requires a plan in order to avoid any issues or pitfalls; this could include raising awareness/allaying concerns amongst effected staff (Clark, 2012).  A deployment could also be carried out in a number of ways;-

Parallel adoption

The new system is run alongside existing systems, this does however mean that in some cases the effort is duplicated, although conversely if any issues are encountered or even if the new system completely fails then the old system is still in place (Weaver, 2004 p.232).

Phased adoption

This is where functionality of the new system is slowly phased in and teething problems emerge gradually rather than in one massive raft of changes (Weaver, 2004 p.232).

Pilot adoption

The pilot approach involves selecting a number of staff or a specific area of the business and introducing the system there, with the aim of gaining feedback and experience with the new systems to apply it when rolling out the new system to the rest of the organisation (Weaver, 2004 p.232).

Big bang adoption

The big bang approach is where a new system is introduced and replaces an existing system immediately with no crossover or where a new system (where an existing system is not in place) is introduced to the entire organisation in one phase.  Where existing systems are replaced using the big bang approach there can be issues encountered where the new system fails and there is no system in place to support the business activity (Weaver, 2004 p.232).

Plan monitoring and maintenance

The next sub-step involves putting controls in place so that any changes likely to affect the model are considered and documented (The Modelling Agency, 2000 p.33), an example of this would be changing the ways in which data is recorded.

Produce final report

A report is produced that documents the outcomes and products of the project (The Modelling Agency, 2000 p.33).

Review project

As with any development project a final review allows for lessons learnt during the project to be discussed and documented (OGC, 2005 p.333), this can mean that future projects take these into account and avoid making the same mistakes twice (The Modelling Agency, 2000 p.33).

Oracle SQL - Formatting dates into a more usable form (TO_CHAR)

Dates in oracle are stored in a format that doesn’t always make them friendly from a reporting point of view or dealing with data in Excel point of view, the TO_CHAR function can reformat your datetime fields so that they are more user friendly.

Examples

The statement is written with two parameters the exact name of the date field and the format (see second table).

TO_CHAR(DATEFIELD,’FORMAT’)

Statement	Result
TO_CHAR(DATEFIELD,’dd mm year’)	23 05 twenty thirteen
TO_CHAR(DATEFIELD,’dd/mm/yy’)	23/05/2013
TO_CHAR(DATEFIELD,’Day Month Year’)	23 May Twenty Thirteen
TO_CHAR(DATEFIELD,’dd/mm/yyyy hh24:mi’)	23/05/2013 18:38

Example formats

Any combination of the following formats can be used, although obviously certain combinations may not make any sense to your end users.

Format	Description	Example
Year/year	Year spelt out in text (with and without an upper case first digit	Twenty Thirteen / twenty thirteen
yyyy	Year number in full	2008
yy	Last two digits of year number	08
q	Quarter of the year	1 (February)
mm	Month number within year**	12 (December)
Mon / mon	Abbreviated month name (with and without an uppercase first digit)	Oct/oct
Month / month	Month name (with and without an upper case first digit)	October / october
w	Week number* (within month)	1 (01/05/2013)
ww	Week number* (within year) **	18 (01/05/2013)
d	Day of the week	1 (Monday)
dd	Day number within month**
Dy/dy	Abbreviated day name (with and without an uppercase first digit)	Mon / mon
Day / day	Day name (with and without an upper case first digit)	Monday / Monday
hh24	Hour of the day in 24 hour format**	16 (4pm)
hh	Hour of the day in 12 hour format**	04 (4pm)
mi	Minute of the hour**	52 (16:52)

*Important note about week numbers

Week numbers in oracle can be confusing as they start on the first day of the year and count seven days and then change, for example in 2013 the week number would increase each Wednesday.

**Suppression of zeros

This format can be prefixed with FM to suppress zeros, for example mm would display January as “01” where as if FMmm was used it would show “1”.

How to test you date formatting

The easiest method for testing the reformatting of your dates is to test them using the dual function, run the following SQL statement and substitute the area highlighted in yellow with the date format you are trying to use.

select to_char(to_date('01/05/2013 16:30','dd/mm/yyyy hh24:mi'),'Year') from dual

Dissertation Series - Educational Data Mining (EDM)

There are many sectors in which data mining can be applied, such as identifying buying habits in retail (BCS, 2002 p.30) and detecting fraud in the finance sector (Thiruvadi and Patel, 2011 p.710).  Educational data mining is an area of data mining that specifically focuses on the development of data mining with the unique data that comes from education (Baker, 2011).

Describing a model that looks at learner results Ayesha et al (2010 p.26) identifies the action that would be taken.

“the proposed model identifies the weak students before the final exam in order to save them from serious harm.  Teachers can take appropriate steps at right time to improve the performance of the student in final exam”

McGee (2008) talks about an implementation of an EDM model and its success.

“it used the trajectory analysis to identify 60 students at risk of failing state standardized test, and teachers developed plans to address their needs.  Only 10 ended up doing poorly” 

Dissertation Series - Barriers to successful Data Mining - Inconsistent data recording

Databases are capable of storing data from many sources and as such data can be entered/recorded by different individuals/organisations, which can introduce inconsistencies into the data.

For example data mined from clothing manufacturers could compare sizes (large medium etc) although the definition of that size could range between manufacturers.  This would have to be kept in mind and factored in when understanding the underlying data. 

Dissertation Series - Barriers to successful Data Mining - Missing data

“as more data is collected, the higher the likelihood of missing data” (Brown et al, 2003 p.611) and as such the treatment of the missing data must be taken into consideration and the effect that such treatment will have on the end result “missing data may be replaced with estimates.  This and other approaches to handling missing data can lead to invalid results in the data mining step” (Dunham, 2003 p.15).

Missing data may be as a result of individuals refusing to provide certain data, Brown and Kros (2003 p.612) refer to an example of a medical environment where “respondents may find certain survey questions offensive or they may be personally sensitive to certain questions”.

Dissertation Series - Barriers to succesful Data Mining - Data suitability

As with all technologies there are certain scenarios/situations where use of the technology will result in benefits not realised or even a negative outcome.  The reasons for this within data mining are as follows.

Data suitability

In some cases some or all of the data held by an organisation may be unsuitable for data mining and this is one of the reasons that some regard the simplicity of modern data mining with caution (Bramer, 1999 p.xii).

The principles of data warehousing can be applied in identifying suitability of data for data mining:-

• Subject orientated

It is important that there data is available that is related to the subject concerned, non subject related data that is mined will clearly result in false positives (Khan, 2005 p.151).

• Time variant

Any data that is considered for mining should be taken from an appropriate time period (Khan, 2005 p.152), data that is mined and that is particularly old may result in patterns being highlighted that no longer affect the business.

• Non volatile

Data that is to be mined should also be non volatile or static, with any amendments being on a periodic basis, this contrasts to databases which are subject to frequent change as transactions etc are processed (Khan, 2005 p.152).

• Integrated

Data that is to be used in data mining should be integrated which means pulling together the data from the various tables within the database and also includes where appropriate bringing data from other databases (Khan, 2005 p.151).

It is therefore not always possible to mine the data held by an organisation, this can be frustrating to staff who wish data mining to be applied in the organisation.

Dissertation Series - Resistances to Data Mining - Technological Requirements

Data mining requires technical resources in terms of hardware/software on which the processing will take place, as well as staff to develop and implement it.  Tesco in 1995 had collected a huge amount of data as a result of their clubcard program, but the technology to process it was available at the time (Humby et al, 2004 p.96).  In fact Tesco didn’t even have the staff/resources to process even a small percentage of the data (Humby et al, 2004 p.96), so had to outsource to Dunnhumby a data analysis company (Dunnhumby, 2012).  Dunnhumby took the collected Tesco data and performed analysis on 10% of the weekly transaction data (again because of technical limitations of the time) (Humby et al, 2004 p.97). 

As forecasted by Moores Law (Intel, 2012), computing processing power has dramatically increased since the Tesco example discussed from 1995.  However in some cases organisations might not have the equipment/staff to undertake the task in-house.  In which case the work could be outsourced or additional resources/staff brought into the organisation, in both cases this brings cost a resistance to change in itself.