|Teaching and Learning Forum 2005 [ Refereed papers ]|
Claudia Amonini, Nicholas Letch and Cherilyn Randolph
The University of Western Australia
Following a Senate decision in 2002 that the University community should embrace the concept of outcomes based education, there is pressure to develop outcome statements at degree, major and unit levels. A project to develop outcomes statements for a major in the Business School, UWA, provides an opportunity to reflect on the curriculum.
Three stakeholders are involved - students, lecturers and employers - from whom it is appropriate to elicit data to support the writing of outcomes statements. Each group has its own view of what a student should be able to perform as a result of studying this major and each view is valid. The varied nature of the three groups means that different techniques are needed. The methodology presented here is generally applicable to any major in any faculty.
Essentially, there are other stakeholders, students being the most obvious. It could be argued that most students do not have the life experience needed to make a significant input to defining outcome statements. However, because they will be working with them, their aspirations, language, and viewpoint must be incorporated. Universities are increasingly vocationally oriented and students are awake to the need to satisfy the requirements of their future employers. The third group of stakeholders therefore are employers.
Direct inquiry of any of these stakeholders is likely to result in more time spent debating outcomes based education than data collection. We opted to use other techniques, each chosen to be most appropriate for the type of stakeholder being targeted. This paper describes the techniques we have used. Not all of the work is complete but we have some results to present.
The Business School at The University of Western Australia awards two undergraduate degrees, Bachelor of Commerce and Bachelor of Economics. Students of both degrees must satisfy the requirements for at least one major stream of study and most students graduate with two or three majors. A major in Information Management (IM) is popular amongst students with an interest in computer technology (they typically pair it with another Business major) but is also chosen by students of accounting, finance, management and economics who see it as a valuable complement to their primary interest.
The name, Information Management, was chosen to reflect the broad range of topics that could be chosen by students, from the use of statistics to support decision making, through to systems development and social implications of technology adoption. However, Information Management is not a widely used term and does not easily translate into employment prospects as represented by job advertisements in newspapers. Although the academic staff (particularly those involved in teaching the units that make up the major) believe that this is a critically important subject to study, it is believed that this view is not held by prospective students. In essence, there is anecdotal evidence that suggests students do not recognise how their CVs will be improved by undertaking a major in Information Management.
The Academic Council resolution was recognised as an opportunity to review what is taught in the Information Management major and to document this in terms of the desired outcomes - sometimes referred to as 'exit outcomes' (Spady, 1988). The intention was to use the IM major outcome statement as a vehicle to establish outcome statements and competency scales for each of the units. Outcome statements at both the major and unit level, would provide both staff and students with a much clearer idea of what graduates will be able to do as a result of completing an IM major.
In order to include the perspectives of major stakeholders in the major we approached the identification of exit outcomes from several directions. Unit lecturers have perhaps the most tangible stake in the development of teaching and learning outcomes. Curriculum should be developed from outcomes not vice versa, but with units and teaching in place it would be naïve to adopt a green fields approach. The first line of attack, therefore, was to explore the content and objectives for each unit. We started by asking the lecturers in charge what their teaching objectives were.
The second stakeholder group that is likely to have a strong influence on outcomes are the potential employers of our graduates. Research has previously shown that there is frequently a gap between employer perceptions of graduate capabilities and the actual competencies of graduates (Trauth, Farwell, & Lee, 1993; Tye, Poon, & Burn, 1995). While we do not intend to replicate large scale studies of employer perceptions, we do feel that it is necessary to develop outcomes in line with what industry and businesses are looking for.
It could be argued that students are not sufficiently informed with respect to which outcomes are of value to them for their views to be a significant factor. In contrast, we believe that to be effective, the outcome statements we produce must reflect the criteria that undergraduates use to choose their degree and majors. At the very least, the expression of the outcome statements must be easily accessible to young people, particularly in language and emphasis. We therefore also intend to ask current students about their attitudes towards, and expectations from the IM major.
While the units within the major are not specifically driven by curricula recommended by professional and teaching bodies (eg ACM, AIS, ACS, etc), a review of recommendations provides a cross check and confirmation on what is appropriate (ACM, AIS & AITP, 2002; ACS, 1997; IRMA & DAMA, 1997).
This project is designed to stop at the specification of unit outcomes. It is expected that each lecturer in charge of a unit will take responsibility for devising appropriate assessment scales to match their unit outcomes. This is consistent with the 'designing back' principle advocated by Dalziell and Gourvenec (2003), also within The University of Western Australia.
Outcome Based Education means clearly focusing and organising everything in an educational system around what is essential for all students to be able to do successfully at the end of their learning experiences. This means starting with a clear picture of what is important for students to be able to do, then organising the curriculum, instruction, and assessment to make sure this learning ultimately happens (Spady, 1994, p1, in Killen, 2000).The sentiments in this quote are both laudable and achievable but require a different mind view of teaching and learning in higher education. Listing objectives for units is standard practice at this university. Relating those objectives to what students can do after graduation, as a result of taking the unit (outcomes) is the next step. Ensuring that the outcomes from each unit are compatible and complementary with those of the other units in the major is further progression towards a coherent and effective educational system. Applying outcomes to instruction and assessment is outside the scope of our project.
What outcomes based education means at our university can be assessed by reading what is available on the website created by the staff development group (CATL, undated).
|Objective||Data collection technique|
|Lecturers||Identify and explore perceptions of what is currently taught and how that contributes to the major.||Semi-structured interviews with lecturers using RepGrid technique.|
|Employers||Identify the outcomes expected by prospective employers|
|Students||Identify and explore student expectations of outcomes and their reasons for choosing/not choosing the major.||Focus groups with current undergraduate students
The RepGrid methodology is described by Kelly (1961) who developed the technique as part of his Personal Construct Theory, which is based around the concept that individuals have a personal view of the world that the clinical psychologist is attempting to reveal [Bannister, 1968 #525]. It is necessary to abstract and generalise personal constructs from what the target does, says, achieves. One useful technique is to explore what the target can be perceived to be and also by what it is not, which is the basis of the RepGrid test [Bannister, 1968 #525]. The requirements on such a test that is to be used in a clinical setting such as the need for 'useable terms', '... reveal those resources ... which might otherwise be overlooked ...' and '... may subsequently be tested and put to use' (Kelly, 1955, in Bannister & Mair, 1968, p.39) particularly resonate with our project.
There are similarities between the psychologist's appraisal of personal constructs of an individual and trying to characterise the Information Management major. Our lecturers need to abstract from the literal content of their unit to recognise the unit constructs, which can then form the basis of outcome statements. To give an example, a unit that includes lectures, tutorials and assessment on the use of presentation skills and PowerPoint is contributing to building communication skills. The relevant outcome is the ability to communicate in large groups. Generic skills such as this are the easiest to articulate. Abstracting a unit that teaches students to draw data flow diagrams is less obvious because the outcome from learning these techniques may be communication skills, software manipulation, work process analysis or all three and there is a flow-on effect to the evaluation of corporate based information systems taught in another unit.
Briefly, the grid in the name refers to a table with the constructs used by individuals to characterise the 'elements' of interest along one axis and the rating made by each individual about that element on the other. In our case the elements of interest are the units in the major, the individuals are the lecturers and the constructs are what we are trying to elicit in the first instance.
RepGrid has been widely used in a number of disciplines outside psychology including Information Systems (Hunter & Beck, 2000; Tan & Hunter, 2002), but its use in the development of outcome statements for majors in tertiary education is new. We applied the technique in two rounds of structured interviews. In the first we were focusing on extracting the constructs in structured interviews. We used the recommended approach of asking the interviewee to consider the units in groups of three and to identify aspects of commonality between two of units that is not shared by the third - the method of triads (Bannister & Mair, 1968; Fransella & Bannister, 1977). Analysis of these interviews reveals the constructs and these will form the basis of the outcome statements.
In the second round of interviews we will attempt to score each unit against the outcome statements as a way of confirming the constructs and reviewing the amount of overlap or complementary material in the units. This should also provide a first step towards building assessment scales.
Secondly, many UWA graduates take their first job outside the state. The project budget did not run to interviewing employers interstate or overseas.
An additional difficulty is that, in material aimed at new graduates, (job ads, flyers in the University careers office), employers do not articulate their needs beyond task specific (eg knowledge of tax laws if the job is tax accountant) and the more obvious generic skills like team work.
We adopted two approaches to overcome these difficulties. In order to gain an initial appreciation of the types of skills that are associated with job titles, we sampled job advertisements over a two week period and analysed the stated requirements. To expand on our understanding of employer requirements, we also conducted in depth interviews with human resource consultants and executive recruitment agents. These interviews confirmed that as intermediaries between universities and employers, the employment agents had a wider view of what skills, knowledge and experiences are valued both by employers of new graduates as well as those required for later career moves. They also had some influence on employers to help them to recognise the value of less obviously relevant knowledge.
Job ads review
Research into the skill requirements of IT professionals in Australia has previously used job advertisements as an indicator (Young, 1996). In our study, we wanted to examine the titles and skill requirements of published job advertisements as an initial assessment of whether the types of capabilities that we believed our graduates possess are actually in demand in the workplace.
The student cohort in the Information Management major is typically a mix of Western Australian school leavers and international fee paying students (primarily from Singapore). To reflect this mix, job advertisements from the local newspaper (The West Australian), a national newspaper (The Australian) and a Singaporean daily newspaper (The Straits Times) were analysed. In addition, the job advertisements posted on Seek.com were also examined over the same period.The analysis involved searching for job titles that appeared to represent the softer, less technical skills that are associated with IT professionals (eg. information systems analyst, business analyst, IT support manager).
In depth interviews
Some disciplines have an effective association that provides a conduit between employers and universities with respect to the content of undergraduate courses (eg CPA for Accountants). Although there is an Australian Computer Society (ACS), it does not have the influence of a compulsory professional body as membership is not a requirement for employment in related industries. As an alternative we chose to explore the university graduate careers service, and the Government online Gradlink site. We also approached recruitment agents and employers directly through in depth interviews.
Questions for the interviews included:
Essentially, this study required exploratory qualitative research to gain an understanding of the attitudes and perceptions of the target population prior to any statistically testing with a survey.
Focus group discussions with undergraduate students were used to explore salient attributes when selecting a major as well as the specific outcome expectations from undertaking an IM major. The focus group is a valuable alternative to an in depth interview as it can often generate more information due to the additional benefits from the interaction and dynamics of the group (Malhotra, 1993). The free flow discussion can trigger thoughts and feelings that are shared with the group. Focus groups are widely used in research to gather a range of opinions and experiences (Malhotra, 1993; Cavana et al 2004). Focus group participants were recruited via email. The participants were randomly selected from the following populations:
A moderator's outline was developed for use during the group discussions, and was divided into the following sections:
Clearly this early draft needed verifying. The University has a set of outcome statements embedded in its Academic Structures Plan (Haskell, 2001) . We took these and tabulated them against the outcome statements for the major and the units. (Outcome statements for the two degrees are not yet available.) Now we were in a position to present the framework to the rest of the teaching group. They had each already given time for the interviews so it was important to organise a meeting where exchange of views would be effective.
|IT Administrator/IT Manager||8|
|Project Support Officer/ Project Manager||7|
|Business Development Manager||3|
For most positions, analytical capabilities were deemed essential and several positions required specific technical expertise. All advertised positions required previous experience making them unsuitable for most new graduates. Some results from in-depth interviews include:
ACS (1997). The ACS Core Body of Knowledge for Information Technology Professionals. Sydney: Australian Computer Society.
Bannister, D. & Mair, J. M. M. (1968). The Evaluation of Personal Constructs. London: Academic Press.
CATL (Centre for the Advancement of Teaching and Learning) (undated). Implementing Outcomes Based Education @ UWA. http://www.catl.osds.uwa.edu.au/obe
Dalziell, T. & Gourvenec, S. (2003). Partnerships in outcomes based education. In Partners in Learning. Proceedings of the 12th Annual Teaching Learning Forum, 11-12 February. Perth: Edith Cowan University. http://www.ecu.edu.au/conferences/tlf/2003/pub/pdf/23_Dalziell_TanyaGourvenec_Susan.pdf
Fransella, F. & Bannister, D. (1977). A Manual for Repertory Grid Technique. New York: Academic Press.
Haskell, D. (2001). Academic Structures Plan. [retrieved 14 Oct 2004] http://www.admin.uwa.edu.au/reg/sec/acadstructures/ASWP-Rept-June.pdf
Hunter, M. G. & Beck, J. E. (2000). Using repertory grids to conduct cross cultural information systems research. Information Systems Research, 11(1), 93-101.
IRMA & DAMA (1997). IRM Curriculum Model (Draft). Hershey, PA: Information Resource Management Association.
Joseph, M. & Joseph, B. (1998). Identifying needs of potential students in tertiary education for strategy development. Quality Assurance in Education, 6(2), p90-.
Kelly, G. A. (1955). The Psychology of Personal Constructs (Vol. 1). New York: W.W. Norton & Company.
Killen, R. (2000). Outcomes-based education: Principles and possibilities. [retrieved 17 Jan 2003 from http://www.schools.nt.edu.au/curricbr/cf/outcomefocus/killen_paper.pdf; verified 24 Jan 2005 at http://www.acel.org.au/affiliates/nsw/conference01/ts_1.html
Spady, W. (1988). Organizing for results: The basis of authentic restructuring and reform. Educational Leadership, 46(2), 4-8.
Spady, W. (1994). Outcome-based education: Critical issues and answers. Arlington, V.A: American Association of School Administrators.
Tan, F. B. & Hunter, M. G. (2002). The Repertory Grid technique: A method for the study of cognition in information systems. MIS Quarterly, 26(1), 39-57.
Trauth, E. M., Farwell, D. W. & Lee, D. (1993). The IS Expectation Gap: Industry Expectations Versus Academic Preparation. MIS Quarterly, 1993(Sep), 293-303.
Tye, E. M. W., Poon, R. S. K. & Burn, J. M. (1995). Information systems skills: Achieving alignment between the curriculum and the needs of IS professionals in the future. Data Base Advances, 26(4), 47-61.
Young, J. (1996). The scope of qualifications, experience and skills Australian organisations require of system analysts and programmers. University of Tasmania, Hobart. Paper presented at the 7th Australasian Conference on Information Systems, University of Tasmania, Hobart.
|Contact author: Mrs Cherilyn Randolph|
School of Economics and Commerce
The University of Western Australia
M261, 35 Stirling Highway, Crawley, Western Australia 6009
Phone: +618 6488 1829 Fax: +618 6488 1055 Email: email@example.com
Please cite as: Amonini, C., Letch, N. and Randolph, C. (2005). Techniques for developing outcome statements. In The Reflective Practitioner. Proceedings of the 14th Annual Teaching Learning Forum, 3-4 February 2005. Perth: Murdoch University. http://lsn.curtin.edu.au/tlf/tlf2005/refereed/amonini.html
Copyright 2005 Claudia Amonini, Nicholas Letch and Cherilyn Randolph. The authors assign to the TL Forum and not for profit educational institutions a non-exclusive licence to reproduce this article for personal use or for institutional teaching and learning purposes, in any format (including website mirrors), provided that the article is used and cited in accordance with the usual academic conventions.