UTS site search

Dr Raymond Lister

Biography

2001 - Present
Senior Lecturer, Faculty of Engineering and Information Technology, University of Technology, Sydney.

1998 - 2000
Lecturer, School of Computing and Information Technology, University of Western Sydney, Nepean.

1994 - 1998
Lecturer, School of Computing Science, Queensland University of Technology.

1992 - 1993
Postdoctoral Research Fellow, Department of Electrical and Computer Engineering, University of Queensland.

1986 - 1992
Ph.D. candidate, Department of Computer Science, University of Sydney. My thesis was in the area of neural networks.

1985 - 1986
Programmer and Knowledge Engineer, Sydney Expert Systems Group, University of Technology, Sydney. I was the primary knowledge engineer in the development, with the CSIRO, of an expert system for interpreting the infra-red spectra of mineral bearing rocks.

1983 - 1985
Research Scientist, GEC Hirst Research Centre, London. I was a member of the Expert Systems Group, which undertook research work and acted as consultative group for other companies under the GEC umbrella; advising on the feasibility and implementation problems of expert system projects.

Professional

Since 2003, I have written a regular column, on Computer Education Research, for the "Bulletin" of the ACM's Special Interest Group on Computer Science Education (SIGCSE). The Bulletin is received by all members of SIGCSE world-wide.

I was co-Chair, of the 4th and 5th International Computing Education Research Workshops (ICER), the premier international conference devoted to computing education research. The 4th ICER was held in Sydney in 2008 and the 5th ICER was held at the University of California, Berkeley, in 2009.

I have edited two special issues of the "A" ranked journal, "Computer Science Education". The first was a special issue on Australasian Computing Education Research, (September 2007). The second was a special issue on Debugging by Novice Programmers (June 2008).

In the three years 2006–2008, I chaired the committee that wrote the Higher School Certificate (HSC) exam paper for the subject "Information Processes and Technology" (IPT). The IPT exam is taken by approximately 5,000 students.

Founding Chair, in 2008, of the Australasian Chapter of the SIGCSE. The SIGCSE is the ACM's Special Interest Group in Computer Science Education (SIGCSE).

Programme Chair, 7th (2007) "Koli Calling", an international conference on computing education research.

Programme Co-chair, Fifth (2003) and Sixth (2004) Australasian Computer Education Conferences: This is the only Australasian conference devoted to university computing education.

Working group Coordinator, for the 13th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), held in Madrid, Spain, in July 2008. The ITiCSE working groups allow computing educators, from different countries, to come together and collaborate on a topic of common interest. I co-coordinated the seven working groups at ITiCSE 2008, comprising a total of 57 members. I edited the combined working group reports, comprising a total of 98 pages, which were published in the SIGCSE's "Bulletin".

Image of Raymond Lister
Associate Professor, School of Software
Core Member, Centre for Human Centred Technology Design
BSc (Hons) (Syd), PhD (Syd)
Member, Association for Computing Machinery
 
Phone
+61 2 9514 1850
Room
CB11.07.117

Research Interests

My primary research interest is in the mental development of the novice computer programmer.

From 2004, I have been one of three conveners of the BRACElet project, which uses an action research model to study the performance of students on programming exams, with the aim of better understanding the problems students have with understanding computer programs. In 2007, I was joint winner (with Professor Jenny Edwards) of an $80K Fellowship from the Australian Learning and Teaching Council, which funded work on this project until 2010. In that funding period, 26 project participants, from 14 different educational institutions across seven different countries co-authored 16 research papers. I have pursued this research on the mental development of the novice programmers since 2004, when I led an international ITiCSE working group, comprised of eleven other members from six countries: USA, UK, Sweden, Finland, Denmark, and New Zealand. That research was published in Lister et al. (2004) A Multi-National Study of Reading and Tracing Skills in Novice Programmers.

Thus, unlike most academics, my research is located at the nexus with my teaching. Most academics lead a double life, bringing different modes of thinking to their teaching and their research. In their research lives, academics see themselves as part of a community that reaches beyond their own university. They read literature, attend conferences, publish, and participate in a public research which cycle repeats, with community members building upon each other's work. But in our teaching lives most academics lead a private life, rarely discussing their teaching beyond their own university, and rarely reading teaching-related literature.

I try to approach all teaching problems as research issues. Consequently, I am very active in the international computer science education research community. I have given five keynote/invited presentations at various national and international conferences. I have published over 70 papers on aspects of computing education.

Reflecting my research interest in the development of novice programmers, my primary teaching interest, and a great deal of my teaching experience, lies with teaching novice programmers.

Since 2005, I have taught introductory database courses.

Books

Lister, R.F. & Edwards, J. 2010, Teaching Novice Computer Programmers: bringing the scholarly approach to Australia A report on the BRACElet project, Australian Learning and Teaching Council, Sydney. Australia.

Chapters

Box, I. & Lister, R.F. 2005, 'Variation in students' conceptions of object-oriented information system development' in Vasilecas, O., Caplinskas, A., Wojtkowski, W., Wojtkowski, W.G., Zupancic, J. & Wrycza, S. (eds), Information systems development, Advances in theory, practice and education, Springer, New York, USA, pp. 439-451.
View/Download from: Publisher's site
Solomon, A.I., Sutcliffe, P.J. & Lister, R.F. 2003, 'Sorting Circular Permutations by Reversal' in Dehne, F., Sack, J.R. & Smid, M. (eds), Algorithms and Data Structures, Springer, Berlin, pp. 319-329.
View/Download from: Publisher's site

Conferences

Teague, D. & Lister, R.F. 2014, 'Longitudinal Think Aloud Study of a Novice Programmer', Computing Education 2014, Australian Computer Society Inc., Sydney, Australia, pp. 41-50.
Abstract: Recent research from within a neo-Piagetian perspective proposes that novice programmers pass through the sensorimotor and preoperational stages before being able to reason at the concrete operational stage. However, academics traditionally teach and assess introductory programming as if students commence at the concrete operational stage. In this paper, we present results from a series of think aloud sessions with a single student, known by the pseudonym Donald. We conducted the sessions mainly over one semester, with an additional session three semesters later. Donald first manifested predominately sensorimotor reasoning, followed by preoperational reasoning, and finally concrete operational reasoning. This longitudinal think aloud study of Donald is the first direct observational evidence of a novice programmer progressing through the neo-Piagetian stages.
Teague, D. & Lister, R.F. 2014, 'Manifestations of Preoperational Reasoning on Similar Programming Tasks', Computing Education 2014, Australian Computer Society Inc., Sydney, Australia, pp. 65-74.
Abstract: In this research paper, we study a simple programming problem that only requires knowledge of variables and assignment statements, and yet we found that some early novice programmers had difficulty solving the problem. We also present data from think aloud studies which demonstrate the nature of those difficulties. We interpret our data within a neo-Piagetian framework which describes cognitive developmental stages through which students pass as they learn to program. We describe in detail think aloud sessions with novices who reason at the neo-Piagetian preoperational level. Those students exhibit two problems. First, they focus on very small parts of the code and lose sight of the "big picture". Second, they are prone to focus on superficial aspects of the task that are not functionally central to the solution. It is not until the transition into the concrete operational stage that decentration of focus occurs, and they have the cognitive ability to reason about abstract quantities that are conserved, and are equipped to adapt skills to closely related tasks. Our results, and the neo-Piagetian framework on which they are based, suggest that changes are necessary in teaching practice to better support novices who have not reached the concrete operational stage.
Corney, M., Fitzgerald, S., Hanks, B., Lister, R.F., McCauley, R. & Murphy, L. 2014, 'Explain in Plain English Questions Revisited: Data Structures Problems', SIGCSE 14: Proceedings of the 45th ACM technical symposium on Computer science education, Association of Computing Machinery, New York, USA, pp. 591-596.
View/Download from: Publisher's site
Abstract: Recent studies have linked the ability of novice (CS1) programmers to read and explain code with their ability to write code. This study extends earlier work by asking CS2 students to explain object-oriented data structures problems that involve recursion. Results show a strong correlation between ability to explain code at an abstract level and performance on code writing and code reading test problems for these object-oriented data structures problems. The authors postulate that there is a common set of skills concerned with reasoning about programs that explains the correlation between writing code and explaining code. The authors suggest that an overly exclusive emphasis on code writing may be detrimental to learning to program. Non-code writing learning activities (e.g., reading and explaining code) are likely to improve student ability to reason about code and, by extension, improve student ability to write code. A judicious mix of code-writing and code-reading activities is recommended.
Teague, D., Ahadi, A. & Lister, R.F. 2014, 'Programming: Reading, Writing and Reversing', 19th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE14), ACM, New York, USA.
Abstract:
Teague, D., Ahadi, A. & Lister, R.F. 2014, 'Programming: Reading, Writing and Reversing', ITICSE14: Proceedings of the 2014 Innovation & Technology in Computer Science Education Conference, The Association for Computing Machinery, New York, New York, USA, pp. 285-290.
View/Download from: Publisher's site
Abstract: In this paper, we look at the concept of reversibility, that is, negating opposites, counterbalances, and actions that can be reversed. Piaget identified reversibility as an indicator of the ability to reason at a concrete operational level. We investigate to what degree novice programmers manifest the ability to work with this concept of reversibility by providing them with a small piece of code and then asking them to write code that undoes the effect of that code. On testing entire cohorts of students in their first year of learning to program, we found an overwhelming majority of them could not cope with such a concept. We then conducted think aloud studies of novices where we observed them working on this task and analyzed their contrasting abilities to deal with it. The results of this study demonstrate the need for better understanding our students' reasoning abilities, and a teaching model aimed at that level of reality.
Porter, L., Zingaro, D. & Lister, R.F. 2014, 'Predicting Student Success using Fine Grain Clicker Data', ICER14: Proceedings of the Tenth Annual International Conference on International Computing Education Research, The Association for Computing Machinery, New York, New York, USA, pp. 51-58.
View/Download from: Publisher's site
Abstract: Recent research suggests that the rst weeks of a CS1 course have a strong in influence on end-of-course student performance. The present work aims to rene the understanding of this phenomenon by using in-class clicker questions as a source of student performance. Clicker questions generate per-lecture and per-question data with which to assess student under-standing. This work demonstrates that clicker question performance early in the term predicts student outcomes at the end of the term. The predictive nature of these questions applies to code-writing questions, multiple choice questions, and the final exam as a whole. The most predictive clicker questions are identified and the relationships between these questions and final exam performance are examined.
Gluga, R., Kay, J., Lister, R.F., simon, S., Charleston, M., Harland, J. & Teague, D. 2013, 'A conceptual model for reflecting on expected learning vs. demonstrated student performance', Computing Education 2013, Australian Computer Society Inc, Sydney, Australia, pp. 77-86.
Abstract: Educators are faced with many challenging questions in designing an effective curriculum. What prerequisite knowledge do students have before commencing a new subject? At what level of mastery? What is the spread of capabilities between bare-passing students vs. the top-performing group? How does the intended learning specication compare to student performance at the end of a subject? In this paper we present a conceptual model that helps in answering some of these questions. It has the following main capabilities: capturing the learning specification in terms of syllabus topics and outcomes; capturing mastery levels to model progression; capturing the minimal vs. aspirational learning design; capturing condence and reliability metrics for each of these mappings; and annally, comparing and reflecting on the learning speciffcation against actual student performance. We present a web-based implementation of the model, and validate it by mapping the final exams from four programming subjects against the ACM/IEEE CS2013 topics and outcomes, using Bloom's Taxonomy as the mastery scale. We then import the itemised exam grades from 632 students across the four subjects and compare the demonstrated student performance against the expected learning for each of these. Key contributions of this work are the validated conceptual model for capturing and comparing expected learning vs. demonstrated performance, and a web-based implementation of this model, which is made freely available online as a community resource.
Teague, D., Corney, M., Ahadi, A. & Lister, R.F. 2013, 'A Qualitative Think Aloud Study of the Early Neo-Piagetian Stages of Reasoning in Novice Programmers', Volume 136 - Fifteenth Australasian Computing Education Conference, Australian Computer Society Inc, Sydney, Australia, pp. 87-96.
Abstract: Recent research indicates that some of the difficulties faced by novice programmers are manifested very early in their learning. In this paper, we present data from think aloud studies that demonstrate the nature of those difficulties. In the think alouds, novices were required to complete short programming tasks which involved either hand executing ("tracing") a short piece of code, or writing a single sentence describing the purpose of the code. We interpret our think aloud data within a neo-Piagetian framework, demonstrating that some novices reason at the sensorimotor and preoperational stages, not at the higher concrete operational stage at which most instruction is implicitly targeted.
Fidge, C.F., Hogan, J. & Lister, R.F. 2013, 'What vs. How: Comparing Students Testing and Coding Skills', Volume 136 - Fifteenth Australasian Computing Education Conference, Australian Computer Society Inc, Sydney, Australia, pp. 97-106.
Abstract: The well-known difficulties students exhibit when learning to program are often characterised as either difficulties in understanding the problem to be solved or difficulties in devising and coding a computational solution. It would therefore be helpful to understand which of these gives students the greatest trouble. Unit testing is a mainstay of large-scale software development and maintenance. A unit test suite serves not only for acceptance testing, but is also a form of requirements specification, as exemplified by agile programming methodologies in which the tests are developed before the corresponding program code. In order to better understand students conceptual difficulties with programming, we conducted a series of experiments in which students were required to write both unit tests and program code for non-trivial problems. Their code and tests were then assessed separately for correctness and coverage, respectively. The results allowed us to directly compare students abilities to characterise a computational problem, as a unit test suite, and develop a corresponding solution, as executable code. Since understanding a problem is a pre-requisite to solving it, we expected students unit testing skills to be a strong predictor of their ability to successfully implement the corresponding program. Instead, however, we found that students testing abilities lag well behind their coding skills.
Fitzgerald, S., Hanks, B., Lister, R.F., McCauley, R. & Murphy, L. 2013, 'What Are We Thinking When We Grade Programs?', SIGCSE '13: 44th ACM Technical Symposium on Computer Science Education, ACM, New York, NY, USA, pp. 471-476.
View/Download from: Publisher's site
Abstract: This paper reports on a mixed methods study which examines how four experienced instructors approached the grading of a programming problem. Two instructors used a detailed, analytic approach and two instructors employed a holistic approach. One instructor exhibited elements of a primary trait approach. Even though the four instructors used different grading scales and philosophies, their raw scores were highly correlated (Spearmans rho of .81) supporting the conclusion that experienced instructors usually agree on whether a program is very good or very bad. Clearly there is no single right way to grade programs. Further discourse should be encouraged for the benefit of both educators and students.
Ahadi, A. & Lister, R.F. 2013, 'Geek Genes, Prior Knowledge, Stumbling Points and Learning Edge Momentum: Parts of the One Elephant?', Proceedings of the 2013 ACM Conference on International Computing Education Research ICER, ACM, New York, NY, USA, pp. 123-128.
View/Download from: Publisher's site
ABSTRACT: Computing academics report bimodal grade distributions in their CS1 classes. Some academics believe that such a distribution is due to their being an innate talent for programming, a geek gene, which some students have, while other students do not have it. Robins introduced the concept of learning edge momentum, which offers an alternative explanation for the purported bimodal grade distribution. In this paper, we analyze empirical data from a real introductory programming class, looking for evidence of geek genes, learning edge momentum and other possible factors.
Sanders, K., Ahmadzadeh, M., Clear, T., Edwards, S., goldweber, M., Johnson, C., Lister, R.F., McCartney, R., Patitsas, E. & Spacco, J. 2013, 'The Canterbury QuestionBank: building a repository of multiple-choice CS1 and CS2 questions', Proceedings of the ITiCSE working group reports conference on Innovation and technology in computer science education-working group reports, ACM, New York, NY, USA, pp. 33-51.
View/Download from: Publisher's site
Abstract: In this paper, we report on an ITiCSE-13 Working Group that developed a set of 654 multiple-choice questions on CS1 and CS2 topics, the Canterbury QuestionBank. We describe the questions, the metadata we investigated, and some preliminary investigations of possible research uses of the QuestionBank. The QuestionBank is publicly available as a repository for computing education instructors and researchers.
Gluga, R., Kay, J., Lister, R.F., Kleitman, S. & Lever, T. 2012, 'Over-Confidence and confusion in using Bloom for programming fundamentals assessment', Proceedings of the 43rd ACM technical symposium on Computer science education, Association of Computing Machinery, New York, USA, pp. 147-152.
Abstract: This paper describes a web-based interactive tutorial that enables computer science tutors and lecturers to practice applying the Bloom Taxonomy in classifying programming exam questions. The structure, design and content of the tutorial are described in detail. The results of an evaluation with ten participants highlight important problem areas in the application of Bloom to programming assessments. The key contributions are the content and design of this tutorial and the insights derived from its evaluation. These are important results in continued work on methods of measuring learning progression in programming fundamentals.
Lister, R.F., Corney, M., Curran, J., D'Souza, D., Fidge, C.F., Gluga, R., Hamilton, M., Harland, J., Hogan, J., Kay, J., Murphy, T., Roggenkamp, M., Sheard, J., Simon, S. & Teague, D. 2012, 'Toward a Shared Understanding of Competency in Programming: An Invitation to the BABELnot Project', Fourteenth Australasian Computing Education Conference (ACE2012), Australian Computer Society Inc, Melbourne, Australia, pp. 53-60.
Abstract: The ICT degrees in most Australian universities have a sequence of up to three programming subjects, or units. BABELnot is an ALTC-funded project that will document the academic standards associated with those three subjects in the six participating universities and, if possible, at other universities. This will necessitate the development of a rich framework for describing the learning goals associated with programming. It will also be necessary to benchmark exam questions that are mapped onto this framework. As part of the project, workshops are planned for ACE 2012, ICER 2012 and ACE 2013, to elicit feedback from the broader Australasian computing education community, and to disseminate the projects findings. The purpose of this paper is to introduce the project to that broader Australasian computing education community and to invite their active participation.
Simon, S., Sheard, J., Carbone, A., Chinn, D., Laakso, M., Clear, T., de Raadt, M., D'Souza, D., Lister, R.F., Philpott, A., Skene, J. & Warburton, G. 2012, 'Introductory programming: examining the exams', Fourteenth Australasian Computing Education Conference (ACE2012), Australian Computer Society Inc, Melbourne, Australia, pp. 61-70.
Abstract: This paper describes a classification scheme that can be used to investigate the characteristics of introductory programming examinations. The scheme itself is described and its categories explained. We describe in detail the process of determining the level of agreement among classifiers, that is, the inter-rater reliability of the scheme, and we report the results of applying the classification scheme to 20 introductory programming examinations. We find that introductory programming examinations vary greatly in the coverage of topics, question styles, skill required to answer questions and the level of difficulty of questions. This study is part of a project that aims to investigate the nature and composition of formal examination instruments used in the summative assessment of introductory programming students, and the pedagogical intentions of the educators who construct these instruments.
Gluga, R., Kay, J., Lister, R.F., Kleitman, S. & Lever, T. 2012, 'Coming to terms with Bloom: an online tutorial for teachers of programming fundamentals', Fourteenth Australasian Computing Education Conference (ACE2012), Australian Computer Society Inc, Melbourne, Australia, pp. 147-156.
Abstract: This paper describes a web-based interactive tutorial that enables computer science tutors and lecturers to practice applying the Bloom Taxonomy in classifying programming exam questions. The structure, design and content of the tutorial are described in detail. The results of an evaluation with ten participants highlight important problem areas in the application of Bloom to programming assessments. The key contributions are the content and design of this tutorial and the insights derived from its evaluation. These are important results in continued work on methods of measuring learning progression in programming fundamentals.
Corney, M., Teague, D., Ahadi, A. & Lister, R.F. 2012, 'Some Empirical Results for Neo-Piagetian Reasoning in Novice Programmers and the Relationship to Code Explanation Questions', Fourteenth Australasian Computing Education Conference (ACE2012), Australian Computer Society Inc, Melbourne, Australia, pp. 77-86.
Abstract: Recent research on novice programmers has suggested that they pass through neo-Piagetian stages: sensorimotor, preoperational, and concrete operational stages, before eventually reaching programming competence at the formal operational stage. This paper presents empirical results in support of this neo-Piagetian perspective. The major novel contributions of this paper are empirical results for some exam questions aimed at testing novices for the concrete operational abilities to reason with quantities that are conserved, processes that are reversible, and properties that hold under transitive inference. While the questions we used had been proposed earlier by Lister, he did not present any data for how students performed on these questions. Our empirical results demonstrate that many students struggle to answer these problems, despite the apparent simplicity of these problems. We then compare student performance on these questions with their performance on six explain in plain English questions.
Teague, D., Corney, M., Ahadi, A. & Lister, R.F. 2012, 'Swapping as the Hello World of Relational Reasoning: Replications, Reflections and Extensions', Fourteenth Australasian Computing Education Conference (ACE2012), Australian Computer Society Inc, Melbourne, Australia, pp. 87-94.
Abstract: At the previous conference in this series, Corney, Lister and Teague presented research results showing relationships between code writing, code tracing and code explaining, from as early as week 3 of semester. We concluded that the problems some students face in learning to program start very early in the semester. In this paper we report on our replication of that experiment, at two institutions, where one is the same as the original institution. In some cases, we did not find the same relationship between explaining code and writing code, but we believe this was because our teachers discussed the code in lectures between the two tests. Apart from that exception, our replication results at both institutions are consistent with our original study.
Teague, D., Corney, M., Fidge, C.F., Roggenkamp, M., Ahadi, A. & Lister, R.F. 2012, 'Using Neo-Piagetian Theory, Formative In-Class Tests and Think Alouds to Better Understand Student Thinking: A Preliminary Report on Computer Programming', Proceedings of the 23rd Annual Conference for the Australasian Association for Engineering Education - The Profession of Engineering Education: Advancing Teaching, Research and Careers, Swinburne University of Technology, Australia, pp. 1-9.
BACKGROUND Around the world, and for many years, students have struggled to learn to program computers. The reasons for this are poorly understood by their lecturers. PURPOSE When the intuitions of many skilled lecturers have failed to solve a pedagogical problem, then a systematic research programme is needed. We have implemented a research programme based on three elements: (1) a theory that provides an organising conceptual framework, (2) representative data on how the class performs on formative assessment tasks, and (3) microgenetic data from one-on-one think aloud sessions, to establish why students struggle with some of the formative tasks. DESIGN / METHOD We have adopted neo-Piagetian theory as our organising framework. We collect data by two methods. The first method is a series of small tests that we have students complete during lectures, at roughly two week intervals. These tests did not count toward the students final grade, which affords us the opportunity to ask unusual questions that probe at the boundaries of student understanding. Think aloud sessions are the second data collection method, in which a small number of selected, volunteer students attempt problems similar to the problems in the in-class tests. RESULTS The results in this paper serve to illustrate our research programme rather than answer a single, tight research question. These illustrative results focus upon one very simple type of programming question that was put to students, very early in their first programming subject. That simple question required students to write code to swap the values in two variables (e.g., temp = a; a = b; b = temp). The common intuition among programming lecturers is that students should be able to easily solve such a problem by, say, week 4 of semester. On the contrary, we found that 40% of students in a class at one of the participating institutions answered this question incorrectly in week 4 of semester. CONCLUSIONS What is emerging from this research programme is evidence for three different ways in which students reason about programming, which correspond to the first three neo-Piagetian stages (Lister, 2011). In the lowest and least sophisticated stage, known as the sensorimotor stage, novices exhibit two types of problems: (1) misconceptions that are already well known in the literature on novice programmers (e.g., Du Boulay, 1989),and/or (2) an approach to manually executing (tracing?) code that is poorly organized and thus error prone. Novices at the next stage, known as the preoperational stage, can correctly trace code, but they cannot reliably reason about a program in terms of abstractions of the code (e.g., diagrams). It is only at the third stage, the concrete operational stage, where students begin to exhibit some capacity to reason about code abstractions. However, traditional approaches to teaching programming implicitly assume that students begin at the concrete operational stage.
Gluga, R., Kay, J., Lister, R.F. & Teague, D. 2012, 'On the reliability of classifying programming tasks using a neo-piagetian theory of cognitive development', International computing education research, Association of Computing Machinery (ACM), New York, USA, pp. 31-38.
View/Download from: Publisher's site
Abstract: Recent research has proposed Neo-Piagetian theory as a useful way of describing the cognitive development of novice programmers. Neo-Piagetian theory may also be a useful way to classify materials used in learning and assessment. If Neo-Piagetian coding of learning resources is to be useful then it is important that practitioners can learn it and apply it reliably. We describe the design of an interactive web-based tutorial for Neo-Piagetian categorization of assessment tasks. We also report an evaluation of the tutorial's effectiveness, in which twenty computer science educators participated. The average classification accuracy of the participants on each of the three Neo-Piagetian stages were 85%, 71% and 78%. Participants also rated their agreement with the expert classifications, and indicated high agreement (91%, 83% and 91% across the three Neo-Piagetian stages). Self-rated confidence in applying Neo-Piagetian theory to classifying programming questions before and after the tutorial were 29% and 75% respectively. Our key contribution is the demonstration of the feasibility of the Neo-Piagetian approach to classifying assessment materials, by demonstrating that it is learnable and can be applied reliably by a group of educators. Our tutorial is freely available as a community resource.
Gluga, R., Kay, J., Lister, R.F. & Lever, T. 2012, 'A unified model for embedding learning standards into university curricula for effective accreditation and quality assurance', Proceedings of the 23rd Annual Conference of the Australasian Association for Engineering Education, The Engineering & Science Education Research (ESER) group, Swinburne University of Technology, Melbourne, Australia, pp. 1-9.
Murphy, L., Fitzgerald, S., Lister, R.F. & McCauley, R. 2012, 'Ability to 'explain in plain english' linked to proficiency in computer-based programming.', International conference on computing education research, Association of Computing Machinery (ACM), New York, USA, pp. 111-118.
View/Download from: Publisher's site
Abstract: This study investigates the relationship between novice programmers' ability to explain code segments and their ability to write code. Results show a strong correlation between ability to correctly answer 'explain in plain English' (EiPE) questions and ability to write code indicating that there are aspects of reasoning about code that are common to both writing code and explaining code. Student explanations were categorized using the Structure of the Observed Learning Outcome (SOLO) taxonomy. The better programmers were more likely to articulate relational aspects of the algorithms. While earlier work also found such a link, the code writing in those earlier studies was done on paper. This is the first such result where the writing component was done with 'hands on' a computer. Our results add further evidence for the existence of an aspect of reasoning about code that is common to both explaining code and writing code, which in turn suggests that a judicious mix of teaching both code skills and code explaining skills may lead to a more effective process by which novices learn to reason about code.
Gluga, R., Kay, J. & Lister, R.F. 2012, 'Progoss: Mastering the Curriculum', Australian Conference on Science and Mathematics Education (ACSME2012), UniServe Science, Sydney, Australia, pp. 92-98.
Abstract: In education, we need to design effective degree programs of study that meet authoritative curricula guidelines. This is challenging because of the size of the curriculum and complexity of degree program structures. When dealing with data of this size and complexity, traditional spreadsheets are a clumsy way of storing the data. A database is a better option, especially when the database is accessible over the web. We created ProGoSs to effectively tackle this complexity. ProGoSs is a web-based system that maps curricula learning goals and mastery levels to individual assessment tasks across entire degree programs. ProGoSs enables academics to answer important questions such as: Does our degree teach the essential core defined in a recommended curriculum? Where in our degree are particular parts of the recommended curriculum taught? Does our degree ensure a solid progression in building skills? Where and how do we assess the learning achieved by bare-pass students on particular parts of the recommended curriculum? We present the design and implementation of ProGoSs and report on its evaluation by mapping multiple programming subjects from multiple universities to the ACM/IEEE Computer Science 2013 topics and learning objectives. This includes a mapping to various levels of Blooms Taxonomy to capture mastery.
Sheard, J., Simon, S., Carbone, A., Chinn, D., Laakso, M., Clear, T., de Raadt, M., DSouza, D., Harland, J., Lister, R.F., Philpott, A. & Warburton, G. 2011, 'Exploring programming assessment instruments: a classification scheme for examination questions', Proceedings of the ACM SIGCSE 2011 International Computing Education Research Workshop, Association of Computing Machinery, USA, pp. 33-38.
View/Download from: Publisher's site
This paper describes the development of a classification scheme that can be used to investigate the characteristics of introductory programming examinations. We describe the process of developing the scheme, explain its categories, and present a taste of the results of a pilot analysis of a set of CS1 exam papers. This study is part of a project that aims to investigate the nature and composition of formal examination instruments used in the summative assessment of introductory programming students, and the pedagogical intentions of the educators who construct these instruments.
Lister, R.F. 2011, 'Concrete and Other Neo-Piagetian Forms of Reasoning in the Novice Programmer', Thirteenth Australasian Computing Education Conference, Australian Computer Society, Australia, pp. 9-18.
Abstract This paper brings together a number of empirical research results on novice programmers, using a neo-Piagetian theoretical framework. While there already exists literature connecting programming with classical Piagetian theory, in this paper we apply neo-Piagetian theory. Using that neo-Piagetian perspective, we offer an explanation as to why attempts to predict ability for programming via classical Piagetian tests have yielded mixed results. We offer a neo-Piagetian explanation of some of the previously puzzling observations about novice programmers, such as why many of them make little use of diagrams, and why they often manifest a non-systematic approach to writing programs. We also develop the relatively unexplored relationship between concrete operational reasoning and programming, by exploring concepts such as conservation and reversibility.
Gluga, R., Kay, J., Lister, R.F. & Lever, T. 2011, 'An architecture for systematic tracking of skills and competence level progression in computer science', Proceedings of the 2nd Annual International Conferences on Computer Science Education: Innovation and Technology (CSEIT 2011) And Software Engineering & Applications (SEA 2011), Global Science & Technology Forum (GSTF), Singapore, pp. 65-69.
A typical Computer Science degree is three to five years long, consists of four to six subjects per semester, and two semesters per year. A student enrolled in such a degree is expected to learn both discipline-specific skills and transferable generic skills. These skills are to be taught in a progressive sequence through the duration of the degree. As the student progresses through the subjects and semesters of a degree, his skill portfolio and competence level for each skill is expected to grow. Effectively modeling these curriculum skills, mapping them to assessment tasks across subjects of a degree, and measuring the progression in learner competence level is, largely, still an unsolved problem. Previous work at this scale is limited. This systematic tracking of skills and competence is crucial for effective quality control and optimization of degree structures. Our main contribution is an architecture for a curriculum information management system to facilitate this systematic tracking of skill and competence level progression in a Computer Science context.
Corney, M., Lister, R.F. & Teague, D. 2011, 'Early Relational Reasoning and the Novice Programmer: Swapping as the Hello World of Relational Reasoning', Thirteenth Australasian Computing Education Conference, Australian Computer Society, Australia, pp. 95-104.
Abstract We report on a longitudinal research study of the development of novice programmers in their first semester of programming. In the third week, almost half of our sample of students could not answer an explain-in-plain-English question, for code consisting of just three assignment statements, which swapped the values in two variables. We regard code that swaps the values of two variables as the simplest case of where a programming student can manifest a SOLO relational response. Our results demonstrate that the problems many students face with understanding code can begin very early, on relatively trivial code. However, using traditional programming exercises, these problems often go undetected until late in the semester. New approaches are required to detect and fix these problems earlier.
Berglund, A. & Lister, R.F. 2010, 'Introductory Programming and the Didactic Triangle', Twelfth Australasian Computing Education Conference, Australian Computer Society, Australia, pp. 35-44.
In this paper, we use Kansanens didactic triangle to structure and analyse research on the teaching and learning of programming. Students, teachers and course content are the three entities that form the corners of the didactic triangle. The edges of the triangle represent the relationships between these three entities. We argue that many computing educators and computing education researchers operate from within narrow views of the didactic triangle. For example, computing educators often teach programming based on how they relate to the computer, and not how the students relate to the computer. We conclude that, while research that focuses on the corners of the didactic triangle is sometimes appropriate, there needs to be more research that focuses on the edges of the triangle, and more research that studies the entire didactic triangle.
Whalley, J. & Lister, R.F. 2009, 'The BRACElet 2009.1 (Wellington) Specification', Eleventh Australasian Computing Education Conference, Australian Computer Society Inc., Australia, pp. 9-18.
Abstract: BRACElet is a multi-institutional computer education research study of novice programmers. The project is open to new members. The purpose of this paper is to: (1) provide potential new members with an overview of BRACElet, and (2) specify the common core for the next data collection cycle. In this paper, BRACElet is taking the unusual step of making its study design public before data is collected. We invite anyone to run their own study using our study design, and publish their findings, irrespective of whether they formally join BRACElet. We look forward to reading their paper.
Hitchens, M. & Lister, R.F. 2009, 'A Focus Group Study of Student Attitudes to Lectures', Eleventh Australasian Computing Education Conference, Australian Computer Society Inc., Australia, pp. 93-100.
Abstract: This paper reports on the findings from focus groups, conducted at Macquarie University, on the attitudes of computing students to lectures. Students felt that two things were vital for a good lecture: (1) that the lecturer goes beyond what is written in the lecture notes; (2) that the lecture is interactive, by which students meant that the lecturer asks if students understand concepts and adjusts the delivery accordingly, and also the lecturer answers the students questions. The students in the focus groups also discussed what makes for a bad lectures: (1) lecturers reading straight from slides; (2) lecturers who blame the students, by saying that students dont work hard enough and are too lazy to turn up to lectures; and (3) lecturers who cover the material too slowly or too quickly. The most prominent reason given for not attending lectures was the timetabling of lectures in such a way that students had too few classes in one day to make the sojourn to university worthwhile. Any university seeking to improve attendance at lectures should perhaps look as much to improving its timetabling practices as it does to improving the practices of its individual lecturers.
Lister, R.F. & Box, I. 2009, 'A Citation Analysis of the ICER 2005-07 Proceedings', Eleventh Australasian Computing Education Conference Vol 31, No 5, Australian Computer Society Inc., Australia, pp. 119-128.
Abstract: This paper identifies the most commonly cited conferences, journals and books of the 43 papers within the first three ICER proceedings. A large array of conferences, journals, and books were cited. However, only a small set of journals and conferences were cited frequently, and the majority were only cited within a single paper, which is consistent with a power law distribution, as predicted by Zipf's Law. The most commonly cited books are concerned with education in general (29%) or psychology (20%), while 17% of books are concerned with computer science education and 12% with computing content. The citation results for ICER are contrasted with earlier published citation analyses of SIGCSE 2007 and ACE2005-07.
Lister, R.F. & Box, I. 2009, 'A Citation Analysis of the ACSC 2006-2008 Proceedings, with Reference to the CORE Conference and Journal Rankings', 32nd Australasian Computer Science Conference: Conferences in Research and Practice in Information Technology, Australian Computer Society Inc., Australia, pp. 7-16.
Abstract: This paper compares the CORE rankings of computing conferences and journals to the frequency of citation of those journals and conferences in the Australasian Computer Science Conference (ACSC) 2006, 2007 and 2008 proceedings. The assumption underlying this study is that there should be a positive relationship between citation rates and the CORE rankings. Our analysis shows that the CORE rankings broadly reflect the ACSC citations, but with some anomalies. While these anomalies might be minor in the larger scheme of things, anomalies need to be addressed, as the careers of individual academics may depend upon it. Rankings are probably here to stay, and this paper ends with some suggestions on how the rankings process should now evolve, so that it becomes more transparent.
Lister, R.F., Fidge, C.F. & Teague, D. 2009, 'Further Evidence of a Relationship between Explaining, Tracing and Writing Skills in Introductory Programming', 14th annual ACM SIGCSE conference on Innovation and technology in computer science education, Association for Computing Machinery, New York, NY, USA, pp. 161-165.
View/Download from: Publisher's site
ABSTRACT: This paper reports on a replication of earlier studies into a possible hierarchy of programming skills. In this study, the students from whom data was collected were at a university that had not provided data for earlier studies. Also, the students were taught the programming language "Python", which had not been used in earlier studies. Thus this study serves as a test of whether the findings in the earlier studies were specific to certain institutions, student cohorts, and programming languages. Also, we used a non-parametric approach to the analysis, rather than the linear approach of earlier studies. Our results are consistent with the earlier studies. We found that students who cannot trace code usually cannot explain code, and also that students who tend to perform reasonably well at code writing tasks have also usually acquired the ability to both trace code and explain code.
Venables, A., Tan, G. & Lister, R.F. 2009, 'A Closer Look at Tracing, Explaining and Code Writing Skills in the Novice Programmer', Fifth International Computing Education Research Workshop, Association for Computing Machinery, New York, NY, USA, pp. 117-128.
View/Download from: Publisher's site
Abstract: The way in which novice programmers learn to write code is of considerable interest to computing education researchers. One research approach to understanding how beginners acquire their programming abilities has been to look at student performance in exams. Lopez et al. (2008) analyzed student responses to an end-of-first-semester exam. They found two types of questions accounted for 46% of the variance on the code writing portion of the same exam. One of those types of question required students to trace iterative code, while the other type required students to explain what a piece of code did. In this paper, we investigate whether the results by Lopez et al. may be generally indicative of something about novice programmers, or whether their results are just an artifact of their particular exam. We studied student responses to our own exam and our results are broadly consistent with Lopez et al. However, we did find that some aspects of their model are sensitive to the particular exam questions used. Specifically, we found that student performance on explaining code was hard to characterize, and the strength of the relationship between explaining and code writing is particularly sensitive to the specific questions asked. Additionally, we found Lopez et al.'s use of a Rasch model to be unnecessary, which will make it far easier for others to conduct similar research.
Clancy, M., Caspersen, M.E. & Lister, R.F. 2009, 'Proceeding of the fifth International workshop on Computing education research', Fifth International Computing Education Research Workshop, Association for Computing Machinery, New York, NY, USA, pp. 1-165.
The fifth International workshop on Computing education research workshop attracted 24 submissions. All papers were double-blind peer-reviewed by members of the international program committee. After the reviewing, 13 papers (54%) were accepted for inclusion in the conference, written by authors across nine countries: Australia, Finland, Germany, Israel, New Zealand, Sweden, the Philippines, the United Kingdom, and the United States of America. The papers spanned a wide variety of topics, including tools and tool use; conceptions, preconceptions, and misconceptions; attitudes; collaborative learning; research categorization; teacher adaptation to new paradigms; and broad-scale adoption of CS innovation.
Clear, T., Edwards, J., Lister, R.F., Simon, B., Thompson, E. & Whalley, J. 2008, 'The Teaching of Novice Computer Programmers:Bringing the Scholarly-Research Approach to Australia', Computing Education 2008. Proc. Tenth Australasian Computing Education Conference (ACE 2008). CRPIT. 78, Australian Computer Society, Adelaide, pp. 63-68.
BRACElet is a multi-institutional multi-national research study of how novice programmers comprehend and write computer programs. This paper reviews the first action research cycle of the BRACElet project and, in the process, charts a path for the upcoming second cycle. The project remains close to educational practice, with much of the data being either data collected directly from exams sat by novices, or data from think-out-Ioud protocols where the task undertaken by a novice or an expert is modelled on an exam question. The first action research cycle analysed data in terms of the SOLO taxonomy. From think-aloud responses, the authors found that educators tended to manifest a SOLO relational response on small reading problems, whereas students tended to manifest a multistructural response. Furthennore, those students who manifested a relational response tended to do better overall in the exam than students who manifested a multistructural response. The second action research cycle will explore the relationship between the ability to read code and the ability to write code. Apart from reporting on the BRACElet project itself, this paper serves as an invitation for institutions and individuals to join the second action research cycle of the BRACElet project.
Berglund, A., Box, I., Eckerdal, A., Lister, R.F. & Pears, A. 2008, 'Learning educational research methods through collaborative research: the PhICER initiative', Tenth Australasian Computing Education Conference, Australian Computer Society, Australia, pp. 35-42.
Abstract: To accommodate an increasing interest in phenomenography among the computing education research community, the authors of this paper have organised two workshops, or intensive courses, in phenomenography, intended for researchers in computing education at a university level. Although the workshop programs contained lectures and smaller exercises, the emphasis was on preparing one joint publication from each of the workshops. The publication from the first workshop discussed the experience of being a teacher in computing, while the second focused on teachers experiences of the problems their students encounter when learning computing. The workshops and their impact on the community are discussed in this paper.
Lee, S.A. & Lister, R.F. 2008, 'Experiments in the Dynamics of Phase Coupled Oscillators When Applied to Graph Colouring', 31st Australasian Computer Science Conference (ACSC 2008), Australian Computer Society, Inc, Darlinghurst, Australia, pp. 83-89.
Abstract This paper examines the capacity of networks of phase coupled oscillators to coordinate activity in a parallel, distributed fashion. To benchmark these networks of oscillators, we present empirical results from a study of the capacity of such networks to colour graphs. We generalise the update equation of Aihara et al. (2006) to an equation that can be applied to graphs requiring multiple colours. We find that our simple multi-phase model can colour some types of graphs, especially complete graphs and complete k-partite graphs with equal or a near equal number of vertices in each partition. A surprising empirical result is that the effectiveness of the approach appears to be more dependent upon the topology of the graph than the size of the graph.
Berglund, A. & Lister, R.F. 2008, 'Debating the OO debate: where is the problem?', Conferences in Research and Practice in Information Technology, Vol 68: Koli Calling Proceedings of the Seventh Baltic Sea Conference on Computing Education Research, Australian Computer Society, Australia, pp. 171-174.
Abstract: In this paper we discuss problems related to the teaching of object-oriented programming (OOP). We argue that more research on how the computer science teacher understands OOP would be beneficial. Our argument takes its point of departure in three sets of studies: (1) an ongoing study on how computer science teachers understand core concepts of OOP, (2) a study of how the teaching of OOP is discussed within the CS community, and (3) a set of studies that discuss the different ways in which CS teachers experience their teaching. This paper reports on an ongoing study of the different ways in which computing science teachers understand object- oriented programming, and what they mean when use the term objects first.. The phenomenographic research approach has been applied to the analysis of a discussion that occurred in the SIGCSE-members mailing list. Two understandings of objects first have been identified: (1) as an extension of imperative programming, and (2) as conceptually different from imperative programming. These two understandings are illustrated via the differing ways in which computing science teachers use the term polymorphism.
Lister, R.F. & Simon, S. 2008, 'Koli Calling 2007', Proceedings of the 7th Baltic Sea Conference on Computing Education Research, Australian Computer Society, Australia, pp. 1-231.
The Seventh Baltic Sea Conference on Computing Education Research, Koli Calling 2007, was held in Finlands beautiful Koli National Park on 15-18 November 2007. Contributions to Koli Calling can take one of four forms. Research papers present unpublished original research. System papers describe tools for learning, instruction, or assessment in computing education, motivated by the didactic needs of computing. Discussion papers are shorter papers used to present novel ideas, proposals, prototypes, or work in progress. Posters are very short presentations describing novel approaches or work in progress. All papers and posters were double-blind peer reviewed by members of the international programme committee and additional reviewers. There were 30 long papers submitted (i.e. research or system papers). The acceptance rate for long papers was 43% (12 research papers and one system paper). The remaining long papers, and those papers submitted speci?cally as discussion papers, comprised a total of 28 papers, of which 15 (54%) were accepted as short papers. The authors of all these accepted papers are from 12 dierent countries.
Sheard, J., Carbone, A., Lister, R.F., Simon, B., Thompson, E. & Whalley, J. 2008, 'Going SOLO to Assess Novice Programmers', Proceedings of the 13th Annual SIGCSE Conference on innovation and Technology in Computer Science Education, Association of Computing Machinery, New York, NY, USA, pp. 209-213.
View/Download from: Publisher's site
ABSTRACT: This paper explores the programming knowledge of novices using Biggs' SOLO taxonomy. It builds on previous work of Lister et al. (2006) and addresses some of the criticisms of that work. The research was conducted by studying the exam scripts for 120 introductory programming students, in which three specific questions were analyzed using the SOLO taxonomy. The study reports the following four findings: when the instruction to students used by Lister et al. - "In plain English, explain what the following segment of Java code does" - is replaced with a less ambiguous instruction, many students still provide multistructural responses; students are relatively consistent in the SOLO level of their answers; student responses on SOLO reading tasks correlate positively with performance on writing tasks; postgraduates students manifest a higher level of thinking than undergraduates.
Lister, R.F., Caspersen, M. & Clancy, M. 2008, 'Proceeding of the fourth International workshop on Computing education research', Proceeding of the fourth International workshop on Computing education research, Association of Computing Machinery, New York, NY, USA, pp. 1-184.
The call for papers attracted 46 submissions. All papers were double-blind peer-reviewed by members of the international program committee. After the reviewing, 16 papers (35%) were accepted for inclusion in the conference, written by authors across 9 countries: Australia, Canada, Denmark, Finland, Germany, New Zealand, Sweden, the United Kingdom, and the United States of America
Lister, R.F. 2008, 'After the gold rush: toward sustainable scholarship in computing', Tenth Australasian Computing Education Conference, Australian Computer Society, Australia, pp. 3-18.
Abstract: In just thirty years, we have gone from punched cards to Second Life. But, as the American National Science Foundation (NSF) recently noted, undergraduate computing education today often looks much as it did several decades ago? (NSF, 2006). Consequently, todays Nintendo Generation? have voted with their feet. We bore them. The contrast between the changes wrought via computer research over the last 30 years, and the failure of computing education to adapt to those changes, is because computing academics lead a double life. In our research lives we see ourselves as part of a community that reaches beyond our own university. We read literature, we attend conferences, we publish, and the cycle repeats, with community members building upon each others work. But in our teaching lives we rarely discuss teaching beyond our own university, we are not guided by any teaching literature; instead we simply follow our instincts. Academics in computing, or in any other discipline, can approach their teaching as research into how novices become experts. Several recent multi-institutional research collaborations have studied the development of novice programmers. This paper describes some of the results from those collaborations. The separation of our teaching and research lives diminishes not just our teaching but also our research. The modern practice of stripping away all distractions to maximize research output is like the practice of stripping away rainforest to grow beef both practices appear to work, for a little while, but not indefinitely. Twenty-first century academia needs to bring teaching and research together, to form a scholarship of computing that is an integrated, sustainable, ecological whole.
Lister, R.F. & Box, I. 2008, 'A citation analysis of the ACE2005 - 2007 proceedings, with reference to the June 2007 CORE conference and journal rankings', Tenth Australasian Computing Education Conference, Australian Computer Society, Australia, pp. 93-102.
Abstract: This paper compares the CORE rankings of computing education conferences and journals to the frequency of citation of those journals and conferences in the ACE2005, 2006 and 2007 proceedings. The assumption underlying this study is that citation rates are a measure of esteem, and so there should be a positive relationship between citation rates and rankings. The CORE conference rankings appear to broadly reflect the ACE citations, but there are some inconsistencies between citation rates and the journal rankings. The paper also identifies the most commonly cited books in these ACE proceedings. Finally, in the spirit of Quis custodiet ipsos custodes?? the paper discusses some ways in which the CORE rankings process itself might in future be made more transparent and open to scholarly discourse.
Simon, S., Sheard, J., Carbone, A., de Raadt, M., Hamilton, M., Lister, R.F. & Thompson, E. 2008, 'Eight years of computing education papers at NACCQ', Proceedings of the 21st Annual Conference of the National Advisory Committee on Computing Qualifications, National Advisory Committee on Computing Qualifications, New Zealand, pp. 101-107.
Abstract: The 157 computing education papers from the past eight NACCQ conferences are categorised and summarised by a group of researchers from multiple institutions, with steps taken to measure and improve the consistency of classification. The papers are set predominantly in programming subjects, hardware/architecture/systems/ network subjects, and capstone projects. The bulk of the papers are about teaching/learning techniques, assessment techniques, teaching/learning tools, curriculum, and educational technology. Most of the papers are set within single subjects, a few in multiple subjects within a single program or department, and fewer still in a range of subjects across the whole institution or multiple institutions. Nearly a quarter of the papers either expound a position or outline a proposal; a large but diminishing proportion report on something such as a change of curriculum or approach; and a large and increasing proportion are clearly research papers, focusing on the analysis of data to answer an explicit research question.
Lister, R.F. & Box, I. 2008, 'A citation analysis of the SIGCSE 2007 proceedings', Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, Association of Computing Machinery, New York, NY, USA, pp. 476-480.
View/Download from: Publisher's site
Abstract: This paper identifies the most commonly cited conferences, journals and books among the 1398 citations made in the 122 publications of the SIGCSE 2007 proceedings. The SIGCSE 2007 authors cited a very large array of conferences, journals and books, but the majority are only cited within a single paper. There are only a very small set of journals and conferences cited frequently. Most books cited are concerned with technical information or are textbooks. Only 2% of books are concerned with computer science education and 23% with education in general. The picture that emerges from this citation analysis is that the SIGCSE community does not have a substantial core set of educational literature. Also, the epistemology of the SIGCSE community is primarily objectivist, with a focus on content, rather than a constructivist, student-centered focus on learning.
Lopez, M., Whalley, J., Robbins, P. & Lister, R.F. 2008, 'Relationships between reading, tracing and writing skills in introductory programming', Fourth International Workshop on Computing Education Research, Association of Computing Machinery, New York, USA, pp. 101-112.
View/Download from: Publisher's site
ABSTRACT: This study analyzed student responses to an examination, after the students had completed one semester of instruction in programming. The performance of students on code tracing tasks correlated with their performance on code writing tasks. A correlation was also found between performance on "explain in plain English" tasks and code writing. A stepwise regression, with performance on code writing as the dependent variable, was used to construct a path diagram. The diagram suggests the possibility of a hierarchy of programming related tasks. Knowledge of programming constructs forms the bottom of the hierarchy, with "explain in English", Parson's puzzles, and the tracing of iterative code forming one or more intermediate levels in the hierarchy.
Simon, S., Carbone, A., de Raadt, M., Lister, R.F., Hamilton, M. & Sheard, J. 2008, 'Classifying computing education papers: process and results', Fourth international Workshop on Computing Education Research, Association of Computing Machinery, New York, USA, pp. 161-172.
View/Download from: Publisher's site
ABSTRACT: We have applied Simon's system for classifying computing education publications to all three years of papers from ICER. We describe the process of assessing the inter-rater reliability of the system and fine-tuning it along the way. Our analysis of the ICER papers confirms that ICER is a research-intensive conference. It also indicates that the research is quite narrowly focused, with the majority of the papers set in the context of programming courses. In addition we find that ICER has a high proportion of papers involving more than one institution, and high proportions of papers on the themes of ability/aptitude and theories and models of teaching and learning.
Clear, T., Whalley, J., Lister, R.F., Carbone, A., Hu, M., Sheard, J., Simon, B. & Thompson, E. 2008, 'Reliably Classifying Novice Programmer Exam Responses using the SOLO Taxonomy', 21st Annual Conference of the National Advisory Committee on Computing Qualifications, National Advisory Committee on Computing Qualifications, New Zealand, pp. 23-30.
Abstract: Past papers of the BRACElet project have described an approach to teaching and assessing students where the students are presented with short pieces of code, and are instructed to explain, in plain English, what the code does. The student responses to these types of questions can be analysed according to the SOLO taxonomy. Some students display an understanding of the code as a single, functional whole, while other students cannot see the forest for the trees?. However, classifying student responses into the taxonomy is not always straightforward. This paper analyses the reliability of the SOLO taxonomy as a means of categorising student responses. The paper derives an augmented set of SOLO categories for application to the programming domain, and proposes a set of guidelines for researchers to use.
Doyle, B.J. & Lister, R.F. 2007, 'Why Teach Unix?', Australasian Conference on Computer Science Education, Australian Computer Society, Australia, pp. 19-26.
This paper examines computing academics' conceptions of the Unix operating system, and the purpose of teaching Unix. Interview transcripts from nine academics were analysed phenomenographically. A small number of qualitatively different conceptions of Unix were identified, within two broad categories. The first broad category manifested a technical approach to Unix. Within this broad category, the conceptions of Unix were, from the least to most sophisticated ! (1) Unix as a set of unrelated commands; (2) Unix as a command line interface superior to GUIs; and (3) Unix as a problem solving tool. The second broad category was a non technical conception of Unix, in which Unix was seen as a resource that is cheap, secure and robust. With regard to teaching Unix, two broad categories of reasons were identified ! practical and pedagogical. These results for teachers are broadly consistent with an earlier phenomenographic study of student conceptions of Unix.
Lister, R.F., Berglund, A., Box, I., Cope, C., Pears, A., Avram, C., Bower, M., Carbone, A., Davey, B., de, R.M., Doyle, B.J., Fitzgerald, S., Mannila, L., Kutay, C., peltomaki, P., Sheard, J., Sutton, K., Traynor, D., Tutty, J. & Venables, A. 2007, 'Differing Ways that Computing Academics Understand Teaching.', Australian Computer Science Communications - Proceedings of the ninth Australasian conference on Computing education, Australian Computer Society, Australia, pp. 97-106.
This paper presents first results from a wide-ranging phenomenographic study of computing academics' understanding of teaching. These first results focus upon four areas: the role of lab practical sessions, the experience of teaching success, conceptions of motivating and engaging students, and the granularity of the teacher's focus. The findings are comparable with prior work on the understandings of academics in other disciplines. This study was started as part of a workshop on phenomenography. Most participants at the workshop received their first training in phenomenography. This paper summarises the structure of the workshop.
Lister, R.F. 2007, 'The Neglected Middle Novice Programmer: Reading and Writing without Abstracting', 20th Annual Conference of the National Advisory Committee on Computing Qualifications, National Advisory Committee on Computing Qualifications, New Zealand, pp. 133-140.
Many teachers of novice programmers have lamented that students either seem to have a natural gift for programming, or have no gift for it at all. In this paper, we discuss a third group of students, the middle novice programmer. At the completion of their first semester of programming, these students can manifest a strong concrete grasp of the semantics of basic programming language constructs, by hand executing code, but they cannot reason about code at a higher goal/plan level. The research evidence presented in this paper for the existence of these middle novice programmers is from the analysis of twelve multiple choice questions, which students attempted as part of an end-of-first-semester exam.
Berglund, A. & Lister, R.F. 2007, 'How do students understand computer network protocols?', 20th Annual Conference of the National Advisory Committee on Computing Qualifications, National Advisory Committee on Computing Qualifications, New Zealand, pp. 15-20.
When computing education research is performed in a rigorous and theoretically sound way, it can inform and improve teaching. Having conducted a phenomenographic study of students understandings of computer network protocols, we present our findings and explain how they can inform the teaching of these protocols. We go on to explore what our results can tell us about the discipline of computer science as a whole.
Lister, R.F. 2006, 'Computer Science Teachers as Amateurs, Students and Reseachers', Koli Calling 2005- Proceeding of the Fifth Koli Calling Conference on Computer Science Education, Turku Centre for Computer Science, Turku, Finland, pp. 3-12.
Lister, R.F. 2006, 'Driving learning via criterion-referenced assessment using Bloom's taxonomy', UniServe Science 2006, UniServe Science, Sydney, Australia, pp. 80-88.
Thompson, E., Whalley, J., Lister, R.F. & Simon, B. 2006, 'Code classification as a learning and asssessment exercise for novice programmers', Proceeding of the 19th annual conference of the ational advisory committee on computing qualifications, National advisory comittee on computing qualifications, Hamilton, New Zealand, pp. 291-298.
Doyle, B.J. & Lister, R.F. 2006, 'Apreliminary phenomenographic study concerning student experiences of UNIX', Proceedings of the 19th annual conference of the national advisory comittee on computing qualifications, National advisory comittee on computing qualifications, Hamilton, New Zealand, pp. 73-78.
Solomon, A.I., Santamaria, D. & Lister, R.F. 2006, 'Automated testing of Unix command-line and scripting skills', Proceedings of the 7th international conference on information technology based higher education, UTS, Sydney, Australia, pp. 120-160.
Fincher, S., Lister, R.F., Clear, T., Robins, A., Tenenberg, J. & Petre, M. 2005, 'Multi-institutional, multi-national studies in CSEd research: some design considerations and trade-offs', Proceedings of the 2005 international workshops on computing education research, Association of computing machinery press, New York, USA, pp. 111-121.
Lister, R.F. 2005, 'Methods for evaluating the approproiateness and effectiveness of summative assessment via multiple choice examinations for technology-focused disciplines', Making a Difference: 2005 Evaluations and Assessment Conference, UTS, Sydney, Aust, pp. 75-84.
Lister, R.F. 2004, 'Teaching Java First: Experiments with Pigs-Early Pedagogy', Proceedings of the 6th Australasian Computing Education Conference, Australian Computer Society Inc, Sydney, Australia, pp. 177-183.
Lister, R.F., Box, I., Morrison, B., Westbrook, S. & Tenenberg, J. 2004, 'The Dimensions of Variation in the Teaching of Data Structures', SIGCSE Bulletin: Inroads ITiCSE 2004 Proceedings, Vol 36 #3 Sept 2004, Association for Computing Machinery, New York, USA, pp. 92-96.
View/Download from: Publisher's site
The current debate about the teaching of data structures is hampered because, as a community, we usually debate specifics about data structure implementations and libraries, when the real level of disagreement remains implicit -- the intent behind our teaching. This paper presents a phenomenographic study of the intent of CS educators for teaching data structures in CS2. Based on interviews with Computer Science educators and analysis of CS literature, we identified five categories of intent: developing transferable thinking, improving students' programming skills, knowing "what's under the hood", knowledge of software libraries, and component thinking. The CS community needs to first debate at the level of these categories before moving to more specific issues. This study also serves as an example of how phenomenographic analysis can be used to inform debate on syllabus design in general.
Prior, J.R. & Lister, R.F. 2004, 'The Backwash Effect on SQL Skills Grading', ITiCSE 2004 Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ACM Press, New York,USA, pp. 32-36.
View/Download from: Publisher's site
This paper examines the effect of grading approaches for SQL query formulation on students' learning strategies. The way that students are graded in a subject has a significant impact on their learning approach, and it is crucial that graded tasks are carefully designed and implemented to inculcate a deep learning experience. An online examination system is described and evaluated.
Collins, W.J., Tenenberg, J., Lister, R.F. & Westbrook, S. 2003, 'The Role of Framework Libraries in CS2', Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education, Association for Computing Machinery, New York, USA, pp. 403-404.
View/Download from: Publisher's site
The recent emergence of object-oriented framework libraries of classic data structures and algorithms such as the Standard Template Library and Java's Collection classes provides a set of general and efficient data structure components for use by practicing software developers. A number of textbook writers are beginning to incorporate the use of these frameworks into "CS2", the traditional first course in data structures at the university level.There has scarcely been a discussion of how these frameworks should best be incorporated into CS2, if they should be used at all. The proposed panel will examine the role of standardized framework libraries in the first data structures course at the university level. Panelists will focus on the following questions. What are the fundamental objectives of CS2, and what role might frameworks have in meeting these objectives? How does an instructor balance student needs for additional instruction in programming basics (e.g. arrays and pointers) versus practice in larger scale design and code reuse? What would be given up to incorporate frameworks into CS2? And to what extent is it important for students to construct elementary data structures from first principles? What assumptions about student cognition and learning does a pro- or con-frameworks approach imply.By trying to articulate answers to some of the above questions, we hope to raise the level of discussion concerning the evolution of the introductory computer science curriculum. This panel will thus try to make explicit, and hence available for critical examination, some of the arguments and assumptions that inform this debate.
Solomon, A.I., Sutcliffe, P.J. & Lister, R.F. 2003, 'Sorting Circular Permutations by Reversal', Algorithms and Data Structures Symposium (was Workshop on Algorithms and Data Structures), Springer-Verlag Berlin Heidelberg, Germany, pp. 319-328.
Lister, R.F. & Leaney, J.R. 2003, 'Bad Theory Versus Bad Teachers: Towards a Pragmatic Synthesis of Constructivism and Objectivism', Learning for an Unknown Future - Proceedings of the 2003 Annual International Conference of the Higher Educational Research and Development Society of Australasia Volume 26, Higher Educational Research and Development Society of Australasia, Milperra, NSW, Australia, pp. 429-436.
Lister, R.F. & Leaney, J.R. 2003, 'Introductory Programming, Criterion-Referencing and Bloom', Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education, The Association for Computing Machinery, New York, USA, pp. 143-147.
View/Download from: Publisher's site
In the traditional norm-referencing approach to grading, all students in a CS1 class attempt the same programming tasks, and those attempts are graded "to a curve". The danger is that such tasks are aimed at a hypothetical average student. Weaker students can do little of these tasks, and learn little. Meanwhile, these tasks do not stretch the stronger students, so they too are denied an opportunity to learn. Our solution is two-fold. First, we use a criterion-referenced approach, where fundamentally different tasks are set, according to the ability of the students. Second, the differences in the nature of the tasks reflect the differing levels of Bloom's taxonomy. Weaker CS1 students are simply required to demonstrate knowledge and comprehension; the ability to read and understand programs. Middling students attempt traditional tasks, while the stronger students are set open-ended tasks at the synthesis and evaluation levels.
Lister, R.F. & Leaney, J.R. 2003, 'First Year Programming: Let All the Flowers Bloom', Computing Education 2003. Fifth Australasian Computing Educational Conference Volume 20, Australian Computer Society Inc., Sydney, Australia, pp. 221-230.
Lister, R.F., Smith, I., Ray, M. & Hawson, G. 2001, 'Nave Bayesian Prediction of bleeding after heart by-pass surgery', Proceedings of ANZIIS 2001, Australian Research Centre for Medical Engineering, University of WA, Perth, WA, pp. 317-321.
Lister, R.F. & Jerram, P. 2001, 'Minimal Mark-Up of multiple choice exams using XML', Proceedings of ASCILITE 2001, Biomedical Multimedia Unit, University of Melbourne, Melbourne, pp. 101-104.
We describe a minimal XML mark-up for multiple choice exams. In our system, exams may be generated at anytime, by choosing a subset of questions from a pool. Furthermore, the system randomises the order of the choices within each chosen question. Thus a student may sit the exam many times. Our first version of this system has been described elsewhere. In this paper, we discuss the limitations of our first system that led to our current work, and give a description of the new system features, including XML tags for supporting collaborative authoring.
Lister, R.F. 2000, 'On blooming first year programming, and its blooming assessment.', Proceedings of the Australasian Conference on Computing Education, ACM Press, New York, pp. 158-162.
View/Download from: Publisher's site
The traditional approach to introductory programming has students writing complete programs, as early as possible. Also, the traditional emphasis is on the technology, not the explicit cognitive development of the student. This approach jumps to the fifth and sixth levels of Bloom's Taxonomy of Educational Objectives, when these last two levels depend upon competence in the first four levels. I describe my alternative approach that addresses the six stages in sequence, emphasising effective assessment processes for the first four levels.

Journal articles

Gluga, R., Kay, J., Lister, R.F., simon, S. & Kleitman, S. 2013, 'Mastering Cognitive Development Theory in Computer Science Education', Computer Science Education, vol. 23, no. 1, pp. 24-57.
View/Download from: Publisher's site
Abstract: To design an effective computer science curriculum, educators require a systematic method of classifying the difficulty level of learning activities and assessment tasks. This is important for curriculum design and implementation and for communication between educators. Different educators must be able to use the method consistently, so that classified activities and assessments are comparable across the subjects of a degree, and, ideally, comparable across institutions. One widespread approach to supporting this is to write learning objects in terms of Blooms Taxonomy. This, or other such classifications, is likely to be more effective if educators can use them consistently, in the way experts would use them. To this end, we present the design and evaluation of our online interactive web-based tutorial system, which can be configured and used to offer training in different classification schemes. We report on results from three evaluations. First, 17 computer science educators complete a tutorial on using Blooms Taxonomy to classify programming examination questions. Second, 20 computer science educators complete a Neo-Piagetian tutorial. Third evaluation was a comparison of inter-rater reliability scores of computer science educators classifying programming questions using Blooms Taxonomy, before and after taking our tutorial. Based on the results from these evaluations, we discuss the effectiveness of our tutorial system design for teaching computer science educators how to systematically and consistently classify programming examination questions. We also discuss the suitability of Blooms Taxonomy and Neo-Piagetian theory for achieving this goal. The Blooms and Neo-Piagetian tutorials are made available as a community resource. The contributions of this paper are the following: the tutorial system for learning classification schemes for the purpose of coding the difficulty of computing learning materials; its evaluation; new insights into the consistency that computing educators can achieve using Bloom; and first insights into the use of Neo-Piagetian theory by a group of classifiers.
Lister, R.F. 2012, 'A variation on Kvale's one thousand page question', ACM Inroads, vol. 3, no. 3, pp. 24-25.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2012, 'The CC2013 Strawman and Bloom's Taxonomy', ACM Inroads, vol. 3, no. 2, pp. 12-13.
This is a regular column that I write for this journal.
Lister, R.F. 2012, 'Teaching-Oriented Faculty and Computing Education Research', ACM Inroads, vol. 3, no. 1, pp. 22-23.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2012, 'Rare research: why is research uncommon in the computing education universe?', ACM Inroads, vol. 3, no. 4, pp. 16-17.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2011, 'COMPUTING EDUCATION RESEARCH: What If We Approached Teaching Like Software Engineering?', ACM Inroads, vol. 2, no. 1, pp. 17-18.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2011, 'Ten years after the McCracken Working Group', ACM Inroads, vol. 2, no. 4, pp. 18-19.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Clear, T., Whalley, J., Robbins, P., Philpott, A., Eckerdal, A., Laakso, M. & Lister, R.F. 2011, 'Report on the final BRACElet workshop: Auckland University of Technology, September 2010', Journal of Applied Computing and Information Technology, vol. 15, no. 1.
Abstract This paper reports on the thirteenth and final BRACElet workshop. In this paper we provide a brief retrospective review of the workshops and the findings that have resulted from this multi-institutional multinational investigation into the teaching and learning of novice programmers. Subsequently we report on the work undertaken during the final workshop and then discuss future avenues for research that have evolved as a result of the BRACElet project.
Lister, R.F. 2011, 'Programming, syntax and cognitive load', ACM Inroads, vol. 2, no. 2, pp. 21-22.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2011, 'Programming, Syntax and Cognitive Load (Part 2)', ACM Inroads, vol. 2, no. 3, pp. 16-17.
This is a regular invited column I write for this journal.
Lister, R.F. 2010, 'CS Research: Book Burning, Naturally Occurring Data, and the Stages of Pedagogic Grief', ACM SIGCSE Bulletin Inroads, vol. 41, no. 4, pp. 13-14.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2010, 'The naughties in CSEd research: a retrospective', SIGCSE Bulletin Inroads, vol. 1, no. 1 (March), pp. 22-24.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2010, 'Teaching the Super Profs to Fish', ACM Inroads, vol. 1, no. 2, pp. 16-17.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2010, 'Computing education research: geek genes and bimodal grades', ACM Inroads, vol. 1, no. 3, pp. 16-17.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2010, 'COMPUTING EDUCATION RESEARCH: The Closing of the CSEd Mind', ACM Inroads, vol. 1, no. 4, pp. 17-18.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2010, 'What If We Approached Teaching Like Software Engineering?', ACM Inroads, vol. 2, no. 1, pp. 17-18.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F., Clear, T., Simon, S., Bouvier, D.J., Carter, P., Eckerdal, A., Jackov, J., Lopez, M., McCartney, R., Robbins, P., Seppala, O. & Thompson, E. 2010, 'Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer', SIGCSE Bulletin Inroads, vol. 41, no. 4, pp. 156-173.
In New Zealand and Australia, the BRACElet project has been investigating students' acquisition of programming skills in introductory programming courses. The project has explored students' skills in basic syntax, tracing code, understanding code, and writing code, seeking to establish the relationships between these skills. This ITiCSE working group report presents the most recent step in the BRACElet project, which includes replication of earlier analysis using a far broader pool of naturally occurring data, refinement of the SOLO taxonomy in code-explaining questions, extension of the taxonomy to code-writing questions, extension of some earlier studies on students' 'doodling' while answering exam questions, and exploration of a further theoretical basis for work that until now has been primarily empirical.
Lister, R.F. 2009, 'CS research: rules for sustaining the discourse -- engage!', SIGCSE Bulletin Inroads, vol. 41, no. 2, pp. 72-74.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2008, 'The Originality Glut', SIGCSE Bulletin Inroads, vol. 40, no. 2, pp. 14-15.
View/Download from: Publisher's site
Lister, R.F. 2008, 'CS research: We are what we cite -- so where are we?', SIGCSE Bulletin Inroads, vol. 40, no. 4, pp. 16-18.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Whalley, J., Clear, T. & Lister, R.F. 2007, 'The Many Ways of the BRACElet Project', Bulletin of Applied Computing and Information Technology, vol. 5, no. 1, pp. 1-16.
This paper provides a retrospective snapshot of the first two years of a multi-institutional multi-national study (MIMN) in Computer Science Education called the BRACElet Project. This study has been inquiring into how novice programmers comprehend and write computer programs. The context for the study is outlined, together with details of how it has evolved and those who have participated. Some challenges encountered during the project are highlighted and pointers for the successful conduct of such a study are provided. The paper concludes by noting pitfalls to be avoided, some open research questions, and current plans for furthering the project.
Lister, R.F. 2007, 'Strangers, Enemies, Terms and Taxonomies', ACM SIGCSE Bulletin Inroads, vol. 39, no. 2, pp. 12-13.
View/Download from: Publisher's site
Lister, R.F. 2006, 'One room four meetings', ACM SIGCSE Bulletin, vol. 38, no. 4, pp. 11-13.
View/Download from: Publisher's site
Lister, R.F. 2006, 'Call me Ishmael: Charles Dickens meets Moby Book', ACM SIGCSE Bulletin, vol. 38, no. 2, pp. 11-13.
View/Download from: Publisher's site
simon, S., Fincher, S., Robbins, A., Box, I., Cutts, Q.I., de, R.M., Haden, P., Hamer, J., Hamilton, M., Lister, R.F., Petre, M., Sutton, K., Tolhurst, D. & Tutty, J. 2006, 'Predictors of success in a first programming course', Australian Computer Science Communications, vol. 28, no. 5, pp. 189-196.
This paper describes a multi-national, multiinstitutional study that investigated introductory programming courses. Student participants were drawn from eleven institutions, mainly in Australasia, during the academic year of 2004. A number of diagnostic tasks were used to explore cognitive, behavioural, and attitudinal factors such as spatial visualisation and reasoning, the ability to articulate strategies for commonplace search and design tasks, and attitudes to studying. The results indicate that a deep approach to learning was positively correlated with mark for the course, while a surface approach was negatively correlated; spatial visualisation skills are correlated with success; a progression of map drawing styles identified in the literature has a significant correlation with marks; and increasing measures of richness of articulation of a search strategy are also associated with higher marks. Finally, a qualitative analysis of short interviews identified the qualities that students themselves regarded as important to success in programming.
Bergin, J., Lister, R.F., Boucher Owens, B. & McNally, M. 2006, 'The first programming course: ideas to end the enrollment decline', ACM SIGCSE Bulletin, vol. 38, no. 3, pp. 301-302.
View/Download from: Publisher's site
Kutay, C. & Lister, R.F. 2006, 'Up close and pedagogical: Computing academics talk about teaching', Australian Computer Science Communications, vol. 28, no. 5, pp. 125-134.
This paper describes and enacts a process for bootstrapping a more systematic discussion of computing education within a school of computing at a research-intensive Australasian university. Thus far, the project has gone through three stages. In the first stage, some academics were interviewed about their approach to teaching. In the second stage, selected anonymous quotes from the interviews were presented and discussed by other interested members of the school at workshops. In the final stage, selected anonymous quotes from the interviews and workshops were placed on a web-based survey, to which interested members of the school responded. These forms of data will be used to drive further stages of debate within the school. The theoretical underpinnings of this project are Wenger's concept of a community of practice, phenomenography, and socially constructivism. The aim is not to instruct the academics in any "right way" to teach. Instead, the aim is to facilitate debate, where the teachers identify the problems, and in finding the solutions they construct their own "pedagogic reality". As facilitators of this process, the authors of this paper highlighted dialectically opposed views in quotes from the teachers, and then allow the teachers to synthesise those views into a more sophisticated view. Our ultimate project aim is to grow a teaching community that balances reified theories of teaching and learning with participation in a community of practice.
Fincher, S., Lister, R.F., Sheard, J., Tenenberg, J. & Young, A. 2006, 'Multi-institutional teaching communities in computer education', Australian computer sciences communications, vol. 28, no. 5, pp. 7-10.
simon, S., Cutts, Q.I., Fincher, S., Haden, P., Robins, A., Sutton, K., Baker, B., Box, I., de, R.M., Hamer, J., Hamilton, M., Lister, R.F., Petre, M., Tolhurst, D. & Tutty, J. 2006, 'The ability to articulate strategy as a predictor of programming skills', Australian computer science communications, vol. 28, no. 5, pp. 181-188.
A multi-national, multi-institutional study investigating introductory programming courses drew on student participants from eleven institutions, mainly in Australasia, during the academic year of 2004. A number of diagnostic tasks were used to explore cognitive, behavioural, and attitudinal factors such as spatial visualisation and reasoning, the ability to articulate strategies for commonplace search and design tasks, and attitudes to studying. This paper reports in detail on the task that required participants to articulate a commonplace search strategy. The results indicate that increasing measures of richness of articulation of a search strategy are associated with higher marks in the course.
goldweber, M., Bergin, J., Lister, R.F. & McNally, M. 2006, 'A comparison of different approaches to the introductory programming course', Australian computer science communications, vol. 28, no. 5, pp. 11-13.
Lister, R.F., Beglund, A., Clear, T., Bergin, J., Garvin-Doxas, K., Hanks, B., Hitchner, L., Luxton-Reilly, A., Sanders, K., Schulte, C. & Whalley, J. 2006, 'Research perspectives on the objects-early debate', ACM SIGCSE, vol. 38, no. 4, pp. 146-165.
Tolhurst, D., Baker, B., Hamer, J., Box, I., Lister, R.F., Cutts, Q.I., Petre, M., de, R.M., Robins, A., Fincher, S., simon, S., Haden, P., Sutton, K., Hamilton, M. & Tutty, J. 2006, 'Do map drawing styles of novice programming predict success in programming? A multi-national, multi institutional Study.', Australian Computer Science Communications, vol. 28, no. 5, pp. 213-222.
Lister, R.F., Simon, B. & Thompson, E. 2006, 'Not seeing the forest for the trees: Novice programmers and the SOLO taxonomy', ACM SIGCSE Bulletin, vol. 38, no. 3, pp. 118-122.
This paper reports on the authors use of the SOLO taxonomy to describe differences in the way students and educators solve small code reading exercises. SOLO is a general educational taxonomy, and has not previously been applied to the study of how novice programmers manifest their understanding of code. Data was collected in the form of written and think-aloud responses from students (novices) and educators (experts), using exam questions. During analysis, the responses were mapped to the different levels of the SOLO taxonomy. From think-aloud responses, the authors found that educators tended to manifest a SOLO relational response on small reading problems, whereas students tended to manifest a multistructural response. These results are consistent with the literature on the psychology of programming, but the work in this paper extends on these findings by analyzing the design of exam questions.
Whalley, J., Lister, R.F., Thompson, E., Clear, T., Robbins, P., Kumar, P. & Prasad, C. 2006, 'An Australasian study of reading and comprehension skills in novice programmers, using Bloom and SOLO taxonomies', Australin computer science communications, vol. 52, no. 5, pp. 243-252.
In this paper we report on a multiinstitutional investigation into the reading and comprehension skills of novice programmers. This work extends previous studies (Lister 2004, McCracken 2001) by developing a question set within two key pedagogical frameworks: the Bloom and SOLO taxonomies. From this framework of analysis some interesting emergent patterns relating the cognitive level of the questions to student performance have been identified.
Lister, R.F. 2005, 'Mixed methods: positivists are from Mars, constructivists are from Venus', ACM SIGCSE Bulletin Inroads, vol. 37, no. 4, pp. 18-19.
Lister, R.F. 2005, 'One Small Step Toward a Culture of Peer Review and Multi-Institutional Sharing of Educational Resources: A Multiple Choice Exam for First Semester Programming Students', Australin computer science communications, vol. 27, no. 5, pp. 155-164.
computer education research; programming; multiple choice; peer review; learning repository; multi-institutional
Hazzan, O., Impagliazzo, J., Lister, R.F. & Schocken, S. 2005, 'Using History of Computing to Address Problems and Opportunities in Computer Science Education', SIGCSE Bulletin Inroads, vol. 37, no. 1, pp. 126-127.
de Raadt, M., Hamilton, M., Lister, R.F., Tutty, J., Baker, B., Box, I., Cutts, Q.I., Fincher, S., Hamer, J., Haden, P., Petre, M., Robins, A., Sutton, K. & Tolhurst, D. 2005, 'Approaches to learning in computer programming students and their effect on success', Research and Development in Higher Education Series, vol. 28, pp. 407-414.
Lister, R.F. 2005, 'Grand Challenges', ACM SIGCSE Bulletin Inroads, vol. 37, no. 2, pp. 14-15.
Sanders, K., Fincher, S., Bouvier, D.J., Lewandowski, G., Morrison, B., Murphy, L., Petre, M., Richards, B., Tenenberg, J., Thomas, L., Anderson, R., Anderson, R., Fitzgerald, S., Gutschow, A., Haller, S., Lister, R.F., McCauley, R., McTaggart, J., Prasad, C., Scott, T., Shinners-Kennedy, D., Westbrook, S. & Zander, C. 2005, 'A multi-institutional, multinational study of programming concepts using card sort data', Expert Systems, vol. 22, no. 3, pp. 121-128.
View/Download from: Publisher's site
This paper presents a case study of the use of a repeated single-criterion card sort with an unusually large, diverse participant group. The study, whose goal was to elicit novice programmers' knowledge of programming concepts, involved over 20 researche
Lister, R.F., Adams, E., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Mostrom, J., Sanders, K., Seppala, O., Simon, B. & Thomas, L. 2004, 'A Multi-National Study of Reading and Tracing Skills in Novice Programmers', SIGCSE Bulletin Inroads, vol. 36, no. 4, pp. 119-150.
View/Download from: Publisher's site
Lister, R.F. 2004, 'Book Review: Computer Science Education Research', ACM SIGCSE Bulletin Inroads, vol. 36, no. 2, pp. 17-18.
Lister, R.F. 2004, 'A Clandestine Religious Meeting', ACM SIGCSE Bulletin Inroads, vol. 36, no. 4, pp. 17-18.
Lister, R.F. 2003, 'A Research Manifesto, and the Relevance of Phenomenography', ACM SIGCSE Bulletin Inroads, vol. 35, no. 2, pp. 15-16.
Lister, R.F. 2003, 'The Five Orders of Ignorance', ACM SIGCSE Bulletin Inroads, vol. 35, no. 4, pp. 16-17.