Dr Raymond Lister

Biography

2001 - Present
Senior Lecturer, Faculty of Engineering and Information Technology, University of Technology, Sydney.

1998 - 2000
Lecturer, School of Computing and Information Technology, University of Western Sydney, Nepean.

1994 - 1998
Lecturer, School of Computing Science, Queensland University of Technology.

1992 - 1993
Postdoctoral Research Fellow, Department of Electrical and Computer Engineering, University of Queensland.

1986 - 1992
Ph.D. candidate, Department of Computer Science, University of Sydney. My thesis was in the area of neural networks.

1985 - 1986
Programmer and Knowledge Engineer, Sydney Expert Systems Group, University of Technology, Sydney. I was the primary knowledge engineer in the development, with the CSIRO, of an expert system for interpreting the infra-red spectra of mineral bearing rocks.

1983 - 1985
Research Scientist, GEC Hirst Research Centre, London. I was a member of the Expert Systems Group, which undertook research work and acted as consultative group for other companies under the GEC umbrella; advising on the feasibility and implementation problems of expert system projects.

Professional

Since 2003, I have written a regular column, on Computer Education Research, for the "Bulletin" of the ACM's Special Interest Group on Computer Science Education (SIGCSE). The Bulletin is received by all members of SIGCSE world-wide.

I was co-Chair, of the 4th and 5th International Computing Education Research Workshops (ICER), the premier international conference devoted to computing education research. The 4th ICER was held in Sydney in 2008 and the 5th ICER was held at the University of California, Berkeley, in 2009.

I have edited two special issues of the "A" ranked journal, "Computer Science Education". The first was a special issue on Australasian Computing Education Research, (September 2007). The second was a special issue on Debugging by Novice Programmers (June 2008).

In the three years 2006–2008, I chaired the committee that wrote the Higher School Certificate (HSC) exam paper for the subject "Information Processes and Technology" (IPT). The IPT exam is taken by approximately 5,000 students.

Founding Chair, in 2008, of the Australasian Chapter of the SIGCSE. The SIGCSE is the ACM's Special Interest Group in Computer Science Education (SIGCSE).

Programme Chair, 7th (2007) "Koli Calling", an international conference on computing education research.

Programme Co-chair, Fifth (2003) and Sixth (2004) Australasian Computer Education Conferences: This is the only Australasian conference devoted to university computing education.

Working group Coordinator, for the 13th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), held in Madrid, Spain, in July 2008. The ITiCSE working groups allow computing educators, from different countries, to come together and collaborate on a topic of common interest. I co-coordinated the seven working groups at ITiCSE 2008, comprising a total of 57 members. I edited the combined working group reports, comprising a total of 98 pages, which were published in the SIGCSE's "Bulletin".

Image of Raymond Lister
Associate Professor, School of Software
Core Member, Centre for Human Centred Technology Design
BSc (Hons) (Syd), PhD (Syd)
Member, Association for Computing Machinery
 
Phone
+61 2 9514 1850
Fax
+61 2 9514 4535
Room
CB10.04.551

Research Interests

My primary research interest is in the mental development of the novice computer programmer.

From 2004, I have been one of three conveners of the BRACElet project, which uses an action research model to study the performance of students on programming exams, with the aim of better understanding the problems students have with understanding computer programs. In 2007, I was joint winner (with Professor Jenny Edwards) of an $80K Fellowship from the Australian Learning and Teaching Council, which funded work on this project until 2010. In that funding period, 26 project participants, from 14 different educational institutions across seven different countries co-authored 16 research papers. I have pursued this research on the mental development of the novice programmers since 2004, when I led an international ITiCSE working group, comprised of eleven other members from six countries: USA, UK, Sweden, Finland, Denmark, and New Zealand. That research was published in Lister et al. (2004) A Multi-National Study of Reading and Tracing Skills in Novice Programmers.

Thus, unlike most academics, my research is located at the nexus with my teaching. Most academics lead a double life, bringing different modes of thinking to their teaching and their research. In their research lives, academics see themselves as part of a community that reaches beyond their own university. They read literature, attend conferences, publish, and participate in a public research which cycle repeats, with community members building upon each other's work. But in our teaching lives most academics lead a private life, rarely discussing their teaching beyond their own university, and rarely reading teaching-related literature.

I try to approach all teaching problems as research issues. Consequently, I am very active in the international computer science education research community. I have given five keynote/invited presentations at various national and international conferences. I have published over 70 papers on aspects of computing education.

Reflecting my research interest in the development of novice programmers, my primary teaching interest, and a great deal of my teaching experience, lies with teaching novice programmers.

Since 2005, I have taught introductory database courses.

Book Chapters

Box, I. & Lister, R.F. 2005, 'Variation in students' conceptions of object-oriented information system development' in Vasilecas, O; Caplinskas, A; Wojtkowski, W; Wojtkowski, W,G; Zupancic, J; Wrycza, S; (eds), Information systems development, Advances in theory, practice and education, Springer, New York, USA, pp. 439-451.
View/Download from: UTSePress | Publisher's site

Conference Papers

Fitzgerald, S., Hanks, B., Lister, R.F., McCauley, R. & Murphy, L. 2013, 'What Are We Thinking When We Grade Programs?', SIGCSE '13:44th ACM Technical Symposium on Computer Science Education, Denver, Colorado, USA, March 2013 in SIGCSE '13: 44th ACM Technical Symposium on Computer Science Education, ed Camp, T., Tymann, P., Dougherty, J. D., Nagel, K., ACM, New York, NY, USA, pp. 471-476.
View/Download from: UTSePress | Publisher's site
Abstract: This paper reports on a mixed methods study which examines how four experienced instructors approached the grading of a programming problem. Two instructors used a detailed, analytic approach and two instructors employed a holistic approach. One instructor exhibited elements of a primary trait approach. Even though the four instructors used different grading scales and philosophies, their raw scores were highly correlated (Spearman++s rho of .81) supporting the conclusion that experienced instructors usually agree on whether a program is ++very good++ or ++very bad++. Clearly there is no single right way to grade programs. Further discourse should be encouraged for the benefit of both educators and students.
Ahadi, A. & Lister, R.F. 2013, 'Geek Genes, Prior Knowledge, Stumbling Points and Learning Edge Momentum: Parts of the One Elephant?', ACM Conference on International Computing Education Research, San Diego, CA, USA, August 2013 in Proceedings of the 2013 ACM Conference on International Computing Education Research ICER, ed Simon, B., Clear, A., and Cutts, Q., ACM, New York, NY, USA, pp. 123-128.
View/Download from: Publisher's site
ABSTRACT: Computing academics report bimodal grade distributions in their CS1 classes. Some academics believe that such a distribution is due to their being an innate talent for programming, a +geek gene+, which some students have, while other students do not have it. Robins introduced the concept of learning edge momentum, which offers an alternative explanation for the purported bimodal grade distribution. In this paper, we analyze empirical data from a real introductory programming class, looking for evidence of geek genes, learning edge momentum and other possible factors.
Gluga, R., Kay, J., Lister, R.F., Kleitman, S., Lever, T. 2012, 'Over-Confidence and confusion in using Bloom for programming fundamentals assessment', ACM technical symposium on Computer science education, Raleigh, North Carolina, USA, February 2012 in Proceedings of the 43rd ACM technical symposium on Computer science education, ed Laurie Smith King, L., Musicant, D., Camp, T., Tymann, P., Association of Computing Machinery, New York, USA, pp. 147-152.
View/Download from: UTSePress
Abstract: This paper describes a web-based interactive tutorial that enables computer science tutors and lecturers to practice applying the Bloom Taxonomy in classifying programming exam questions. The structure, design and content of the tutorial are described in detail. The results of an evaluation with ten participants highlight important problem areas in the application of Bloom to programming assessments. The key contributions are the content and design of this tutorial and the insights derived from its evaluation. These are important results in continued work on methods of measuring learning progression in programming fundamentals.
Lister, R.F., Corney, M., Curran, J., D'Souza, D., Fidge, C.F., Gluga, R., Hamilton, M., Harland, J., Hogan, J., Kay, J., Murphy, T., Roggenkamp, M., Sheard, J., Simon, S., Teague, D. 2012, 'Toward a Shared Understanding of Competency in Programming: An Invitation to the BABELnot Project', Australasian Computing Education Conference, Melbourne, Australia, January 2012 in Fourteenth Australasian Computing Education Conference (ACE2012), ed de Raadt, M. and Carbone, A., Australian Computer Society Inc, Melbourne, Australia, pp. 53-60.
View/Download from: UTSePress | Publisher's site
Abstract: The ICT degrees in most Australian universities have a sequence of up to three programming subjects, or units. BABELnot is an ALTC-funded project that will document the academic standards associated with those three subjects in the six participating universities and, if possible, at other universities. This will necessitate the development of a rich framework for describing the learning goals associated with programming. It will also be necessary to benchmark exam questions that are mapped onto this framework. As part of the project, workshops are planned for ACE 2012, ICER 2012 and ACE 2013, to elicit feedback from the broader Australasian computing education community, and to disseminate the project++s findings. The purpose of this paper is to introduce the project to that broader Australasian computing education community and to invite their active participation.
Simon, S., Sheard, J., Carbone, A., Chinn, D., Laakso, M., Clear, T., de Raadt, M., D'Souza, D., Lister, R.F., Philpott, A., Skene, J., Warburton, G. 2012, 'Introductory programming: examining the exams', Australasian Computing Education Conference, Melbourne, Australia, January 2012 in Fourteenth Australasian Computing Education Conference (ACE2012), ed de Raadt, M. and Carbone, A., Australian Computer Society Inc, Melbourne, Australia, pp. 61-70.
View/Download from: UTSePress | Publisher's site
Abstract: This paper describes a classification scheme that can be used to investigate the characteristics of introductory programming examinations. The scheme itself is described and its categories explained. We describe in detail the process of determining the level of agreement among classifiers, that is, the inter-rater reliability of the scheme, and we report the results of applying the classification scheme to 20 introductory programming examinations. We find that introductory programming examinations vary greatly in the coverage of topics, question styles, skill required to answer questions and the level of difficulty of questions. This study is part of a project that aims to investigate the nature and composition of formal examination instruments used in the summative assessment of introductory programming students, and the pedagogical intentions of the educators who construct these instruments.
Corney, M., Teague, D., Ahadi, A., Lister, R.F. 2012, 'Some Empirical Results for Neo-Piagetian Reasoning in Novice Programmers and the Relationship to Code Explanation Questions', Australasian Computing Education Conference, Melbourne, Australia, January 2012 in Fourteenth Australasian Computing Education Conference (ACE2012), ed de Raadt, M. and Carbone, A., Australian Computer Society Inc, Melbourne, Australia, pp. 77-86.
View/Download from: UTSePress | Publisher's site
Abstract: Recent research on novice programmers has suggested that they pass through neo-Piagetian stages: sensorimotor, preoperational, and concrete operational stages, before eventually reaching programming competence at the formal operational stage. This paper presents empirical results in support of this neo-Piagetian perspective. The major novel contributions of this paper are empirical results for some exam questions aimed at testing novices for the concrete operational abilities to reason with quantities that are conserved, processes that are reversible, and properties that hold under transitive inference. While the questions we used had been proposed earlier by Lister, he did not present any data for how students performed on these questions. Our empirical results demonstrate that many students struggle to answer these problems, despite the apparent simplicity of these problems. We then compare student performance on these questions with their performance on six explain in plain English questions.
Teague, D., Corney, M., Ahadi, A., Lister, R.F. 2012, 'Swapping as the Hello World of Relational Reasoning: Replications, Reflections and Extensions', Australasian Computing Education Conference, Melbourne, Australia, January 2012 in Fourteenth Australasian Computing Education Conference (ACE2012), ed de Raadt, M. and Carbone, A., Australian Computer Society Inc, Melbourne, Australia, pp. 87-94.
View/Download from: Publisher's site
Abstract: At the previous conference in this series, Corney, Lister and Teague presented research results showing relationships between code writing, code tracing and code explaining, from as early as week 3 of semester. We concluded that the problems some students face in learning to program start very early in the semester. In this paper we report on our replication of that experiment, at two institutions, where one is the same as the original institution. In some cases, we did not find the same relationship between explaining code and writing code, but we believe this was because our teachers discussed the code in lectures between the two tests. Apart from that exception, our replication results at both institutions are consistent with our original study.
Gluga, R., Kay, J., Lister, R.F., Kleitman, S., Lever, T. 2012, 'Coming to terms with Bloom: an online tutorial for teachers of programming fundamentals', Australasian Computing Education Conference, Melbourne, Australia, January 2012 in Fourteenth Australasian Computing Education Conference (ACE2012), ed de Raadt, M. and Carbone, A., Australian Computer Society Inc, Melbourne, Australia, pp. 147-156.
View/Download from: UTSePress | Publisher's site
Abstract: This paper describes a web-based interactive tutorial that enables computer science tutors and lecturers to practice applying the Bloom Taxonomy in classifying programming exam questions. The structure, design and content of the tutorial are described in detail. The results of an evaluation with ten participants highlight important problem areas in the application of Bloom to programming assessments. The key contributions are the content and design of this tutorial and the insights derived from its evaluation. These are important results in continued work on methods of measuring learning progression in programming fundamentals.
Teague, D., Corney, M., Fidge, C.F., Roggenkamp, M., Ahadi, A. & Lister, R.F. 2012, 'Using Neo-Piagetian Theory, Formative In-Class Tests and Think Alouds to Better Understand Student Thinking: A Preliminary Report on Computer Programming', 23rd Annual Conference for the Australasian Association for Engineering Education - The Profession of Engineering Education: Advancing Teaching, Research and Careers, Melbourne, Australia, December 2012 in Proceedings of the 23rd Annual Conference for the Australasian Association for Engineering Education - The Profession of Engineering Education: Advancing Teaching, Research and Careers, ed Llewellyn Mann & Scott Daniel, Swinburne University of Technology, Australia, pp. 1-9.
BACKGROUND Around the world, and for many years, students have struggled to learn to program computers. The reasons for this are poorly understood by their lecturers. PURPOSE When the intuitions of many skilled lecturers have failed to solve a pedagogical problem, then a systematic research programme is needed. We have implemented a research programme based on three elements: (1) a theory that provides an organising conceptual framework, (2) representative data on how the class performs on formative assessment tasks, and (3) microgenetic data from one-on-one think aloud sessions, to establish why students struggle with some of the formative tasks. DESIGN / METHOD We have adopted neo-Piagetian theory as our organising framework. We collect data by two methods. The first method is a series of small tests that we have students complete during lectures, at roughly two week intervals. These tests did not count toward the students++ final grade, which affords us the opportunity to ask unusual questions that probe at the boundaries of student understanding. Think aloud sessions are the second data collection method, in which a small number of selected, volunteer students attempt problems similar to the problems in the in-class tests. RESULTS The results in this paper serve to illustrate our research programme rather than answer a single, tight research question. These illustrative results focus upon one very simple type of programming question that was put to students, very early in their first programming subject. That simple question required students to write code to swap the values in two variables (e.g., temp = a; a = b; b = temp). The common intuition among programming lecturers is that students should be able to easily solve such a problem by, say, week 4 of semester. On the contrary, we found that 40% of students in a class at one of the participating institutions answered this question incorrectly in week 4 of semester. CONCLUSIONS What is emerging from this research programme is evidence for three different ways in which students reason about programming, which correspond to the first three neo-Piagetian stages (Lister, 2011). In the lowest and least sophisticated stage, known as the sensorimotor stage, novices exhibit two types of problems: (1) misconceptions that are already well known in the literature on novice programmers (e.g., Du Boulay, 1989),and/or (2) an approach to manually executing (++tracing+ ) code that is poorly organized and thus error prone. Novices at the next stage, known as the preoperational stage, can correctly trace code, but they cannot reliably reason about a program in terms of abstractions of the code (e.g., diagrams). It is only at the third stage, the concrete operational stage, where students begin to exhibit some capacity to reason about code abstractions. However, traditional approaches to teaching programming implicitly assume that students begin at the concrete operational stage.
Gluga, R., Kay, J., Lister, R.F., Teague, D. 2012, 'On the reliability of classifying programming tasks using a neo-piagetian theory of cognitive development', The ninth annual international conference on computing education research, Auckland, New Zealand, November 2012 in International computing education research, ed Clear, A., Sanders, K. and Simon, B., Association of Computing Machinery (ACM), New York, USA, pp. 31-38.
View/Download from: UTSePress | Publisher's site
Abstract: Recent research has proposed Neo-Piagetian theory as a useful way of describing the cognitive development of novice programmers. Neo-Piagetian theory may also be a useful way to classify materials used in learning and assessment. If Neo-Piagetian coding of learning resources is to be useful then it is important that practitioners can learn it and apply it reliably. We describe the design of an interactive web-based tutorial for Neo-Piagetian categorization of assessment tasks. We also report an evaluation of the tutorial's effectiveness, in which twenty computer science educators participated. The average classification accuracy of the participants on each of the three Neo-Piagetian stages were 85%, 71% and 78%. Participants also rated their agreement with the expert classifications, and indicated high agreement (91%, 83% and 91% across the three Neo-Piagetian stages). Self-rated confidence in applying Neo-Piagetian theory to classifying programming questions before and after the tutorial were 29% and 75% respectively. Our key contribution is the demonstration of the feasibility of the Neo-Piagetian approach to classifying assessment materials, by demonstrating that it is learnable and can be applied reliably by a group of educators. Our tutorial is freely available as a community resource.
Murphy, L., Fitzgerald, S., Lister, R.F., McCauley, R. 2012, 'Ability to 'explain in plain english' linked to proficiency in computer-based programming.', The ninth annual international conference on computing education research, Auckland, New Zealand, November 2012 in International conference on computing education research, ed Clear, A., Sanders, K. and Simon, B., Association of Computing Machinery (ACM), New York, USA, pp. 111-118.
View/Download from: UTSePress | Publisher's site
Abstract: This study investigates the relationship between novice programmers' ability to explain code segments and their ability to write code. Results show a strong correlation between ability to correctly answer 'explain in plain English' (EiPE) questions and ability to write code indicating that there are aspects of reasoning about code that are common to both writing code and explaining code. Student explanations were categorized using the Structure of the Observed Learning Outcome (SOLO) taxonomy. The better programmers were more likely to articulate relational aspects of the algorithms. While earlier work also found such a link, the code writing in those earlier studies was done on paper. This is the first such result where the writing component was done with 'hands on' a computer. Our results add further evidence for the existence of an aspect of reasoning about code that is common to both explaining code and writing code, which in turn suggests that a judicious mix of teaching both code skills and code explaining skills may lead to a more effective process by which novices learn to reason about code.
Gluga, R., Kay, J., Lister, R.F. 2012, 'Progoss: Mastering the Curriculum', Australian Conference on Science and Mathematics Education (ACSME2012), Sydney, Australia, September 2012 in Australian Conference on Science and Mathematics Education (ACSME2012), ed M. Sharma and A. Yeung, UniServe Science, Sydney, Australia, pp. 92-98.
View/Download from: UTSePress
Abstract: In education, we need to design effective degree programs of study that meet authoritative curricula guidelines. This is challenging because of the size of the curriculum and complexity of degree program structures. When dealing with data of this size and complexity, traditional spreadsheets are a clumsy way of storing the data. A database is a better option, especially when the database is accessible over the web. We created ProGoSs to effectively tackle this complexity. ProGoSs is a web-based system that maps curricula learning goals and mastery levels to individual assessment tasks across entire degree programs. ProGoSs enables academics to answer important questions such as: Does our degree teach the essential core defined in a recommended curriculum? Where in our degree are particular parts of the recommended curriculum taught? Does our degree ensure a solid progression in building skills? Where and how do we assess the learning achieved by bare-pass students on particular parts of the recommended curriculum? We present the design and implementation of ProGoSs and report on its evaluation by mapping multiple programming subjects from multiple universities to the ACM/IEEE Computer Science 2013 topics and learning objectives. This includes a mapping to various levels of Bloom++s Taxonomy to capture mastery.
Gluga, R., Kay, J., Lister, R.F., Lever, T. 2012, 'A unified model for embedding learning standards into university curricula for effective accreditation and quality assurance', Australasian Association for Engineering Education, Melbourne, Australia, December 2012 in Proceedings of the 23rd Annual Conference of the Australasian Association for Engineering Education, ed Llewellyn Mann & Scott Daniel, The Engineering & Science Education Research (ESER) group, Swinburne University of Technology, Melbourne, Australia, pp. 1-9.
View/Download from: UTSePress | Publisher's site
Lister, R.F. 2011, 'Concrete and Other Neo-Piagetian Forms of Reasoning in the Novice Programmer', Australasian Computing Education Conference, Perth, Australia, January 2011 in Thirteenth Australasian Computing Education Conference, ed Hamer, J. and de Raadt, M., Australian Computer Society, Australia, pp. 9-18.
View/Download from: UTSePress | Publisher's site
Abstract This paper brings together a number of empirical research results on novice programmers, using a neo-Piagetian theoretical framework. While there already exists literature connecting programming with classical Piagetian theory, in this paper we apply neo-Piagetian theory. Using that neo-Piagetian perspective, we offer an explanation as to why attempts to predict ability for programming via classical Piagetian tests have yielded mixed results. We offer a neo-Piagetian explanation of some of the previously puzzling observations about novice programmers, such as why many of them make little use of diagrams, and why they often manifest a non-systematic approach to writing programs. We also develop the relatively unexplored relationship between concrete operational reasoning and programming, by exploring concepts such as conservation and reversibility.
Corney, M., Lister, R.F. & Teague, D. 2011, 'Early Relational Reasoning and the Novice Programmer: Swapping as the Hello World of Relational Reasoning', Australasian Computing Education Conference, Perth, Australia, January 2011 in Thirteenth Australasian Computing Education Conference, ed Hamer, J. and de Raadt, M., Australian Computer Society, Australia, pp. 95-104.
View/Download from: UTSePress | Publisher's site
Abstract We report on a longitudinal research study of the development of novice programmers in their first semester of programming. In the third week, almost half of our sample of students could not answer an explain-in-plain-English question, for code consisting of just three assignment statements, which swapped the values in two variables. We regard code that swaps the values of two variables as the simplest case of where a programming student can manifest a SOLO relational response. Our results demonstrate that the problems many students face with understanding code can begin very early, on relatively trivial code. However, using traditional programming exercises, these problems often go undetected until late in the semester. New approaches are required to detect and fix these problems earlier.
Sheard, J., Simon, S., Carbone, A., Chinn, D., Laakso, M., Clear, T., de Raadt, M., D++Souza, D., Harland, J., Lister, R.F., Philpott, A. & Warburton, G. 2011, 'Exploring programming assessment instruments: a classification scheme for examination questions', INTERNATIONAL COMPUTING EDUCATION RESEARCH WORKSHOP, Providence, Rhode Island, USA, August 2011 in Proceedings of the ACM SIGCSE 2011 International Computing Education Research Workshop, ed Caspersen, M., Clear, A., Sanders, K., Association of Computing Machinery, USA, pp. 33-38.
View/Download from: UTSePress | Publisher's site
This paper describes the development of a classification scheme that can be used to investigate the characteristics of introductory programming examinations. We describe the process of developing the scheme, explain its categories, and present a taste of the results of a pilot analysis of a set of CS1 exam papers. This study is part of a project that aims to investigate the nature and composition of formal examination instruments used in the summative assessment of introductory programming students, and the pedagogical intentions of the educators who construct these instruments.
Gluga, R., Kay, J., Lister, R.F. & Lever, T. 2011, 'An architecture for systematic tracking of skills and competence level progression in computer science', Computer Science Education: Innovation and Technology, Singapore, December 2011 in Proceedings of the 2nd Annual International Conferences on Computer Science Education: Innovation and Technology (CSEIT 2011) And Software Engineering & Applications (SEA 2011), ed Varthini, B. P. & Westin, T., Global Science & Technology Forum (GSTF), Singapore, pp. 65-69.
View/Download from: UTSePress
A typical Computer Science degree is three to five years long, consists of four to six subjects per semester, and two semesters per year. A student enrolled in such a degree is expected to learn both discipline-specific skills and transferable generic skills. These skills are to be taught in a progressive sequence through the duration of the degree. As the student progresses through the subjects and semesters of a degree, his skill portfolio and competence level for each skill is expected to grow. Effectively modeling these curriculum skills, mapping them to assessment tasks across subjects of a degree, and measuring the progression in learner competence level is, largely, still an unsolved problem. Previous work at this scale is limited. This systematic tracking of skills and competence is crucial for effective quality control and optimization of degree structures. Our main contribution is an architecture for a curriculum information management system to facilitate this systematic tracking of skill and competence level progression in a Computer Science context.
Berglund, A. & Lister, R.F. 2010, 'Introductory Programming and the Didactic Triangle', Australasian Computing Education Conference, Brisbane, Australia, January 2010 in Twelfth Australasian Computing Education Conference, ed Clear, T. and Hamer, J., Australian Computer Society, Australia, pp. 35-44.
View/Download from: UTSePress | Publisher's site
In this paper, we use Kansanen++s didactic triangle to structure and analyse research on the teaching and learning of programming. Students, teachers and course content are the three entities that form the corners of the didactic triangle. The edges of the triangle represent the relationships between these three entities. We argue that many computing educators and computing education researchers operate from within narrow views of the didactic triangle. For example, computing educators often teach programming based on how they relate to the computer, and not how the students relate to the computer. We conclude that, while research that focuses on the corners of the didactic triangle is sometimes appropriate, there needs to be more research that focuses on the edges of the triangle, and more research that studies the entire didactic triangle.
Whalley, J. & Lister, R.F. 2009, 'The BRACElet 2009.1 (Wellington) Specification', Australasian Computing Education Conference, Wellington, New Zealand, January 2009 in Eleventh Australasian Computing Education Conference, ed Margaret Hamilton and Tony Clear, Australian Computer Society Inc., Australia, pp. 9-18.
View/Download from: UTSePress | Publisher's site
Abstract: BRACElet is a multi-institutional computer education research study of novice programmers. The project is open to new members. The purpose of this paper is to: (1) provide potential new members with an overview of BRACElet, and (2) specify the common core for the next data collection cycle. In this paper, BRACElet is taking the unusual step of making its study design public before data is collected. We invite anyone to run their own study using our study design, and publish their findings, irrespective of whether they formally join BRACElet. We look forward to reading their paper.
Hitchens, M. & Lister, R.F. 2009, 'A Focus Group Study of Student Attitudes to Lectures', Australasian Computing Education Conference, Wellington, New Zealand, January 2009 in Eleventh Australasian Computing Education Conference, ed Margaret Hamilton and Tony Clear, Australian Computer Society Inc., Australia, pp. 93-100.
View/Download from: UTSePress | Publisher's site
Abstract: This paper reports on the findings from focus groups, conducted at Macquarie University, on the attitudes of computing students to lectures. Students felt that two things were vital for a good lecture: (1) that the lecturer goes beyond what is written in the lecture notes; (2) that the lecture is interactive, by which students meant that the lecturer asks if students understand concepts and adjusts the delivery accordingly, and also the lecturer answers the students++ questions. The students in the focus groups also discussed what makes for a bad lectures: (1) lecturers reading straight from slides; (2) lecturers who ++blame the students++, by saying that students don++t work hard enough and are too lazy to turn up to lectures; and (3) lecturers who cover the material too slowly or too quickly. The most prominent reason given for not attending lectures was the timetabling of lectures in such a way that students had too few classes in one day to make the sojourn to university worthwhile. Any university seeking to improve attendance at lectures should perhaps look as much to improving its timetabling practices as it does to improving the practices of its individual lecturers.
Lister, R.F. & Box, I. 2009, 'A Citation Analysis of the ICER 2005-07 Proceedings', Australasian Computing Education Conference, Wellington, New Zealand, January 2009 in Eleventh Australasian Computing Education Conference Vol 31, No 5, ed Margaret Hamilton and Tony Clear, Australian Computer Society Inc., Australia, pp. 119-128.
View/Download from: UTSePress | Publisher's site
Abstract: This paper identifies the most commonly cited conferences, journals and books of the 43 papers within the first three ICER proceedings. A large array of conferences, journals, and books were cited. However, only a small set of journals and conferences were cited frequently, and the majority were only cited within a single paper, which is consistent with a power law distribution, as predicted by Zipf's Law. The most commonly cited books are concerned with education in general (29%) or psychology (20%), while 17% of books are concerned with computer science education and 12% with computing content. The citation results for ICER are contrasted with earlier published citation analyses of SIGCSE 2007 and ACE2005-07.
Lister, R.F. & Box, I. 2009, 'A Citation Analysis of the ACSC 2006-2008 Proceedings, with Reference to the CORE Conference and Journal Rankings', Australasian Computer Science Conference, Wellington, New Zealand, January 2009 in 32nd Australasian Computer Science Conference: Conferences in Research and Practice in Information Technology, ed Mans, B., Australian Computer Society Inc., Australia, pp. 7-16.
View/Download from: UTSePress | Publisher's site
Abstract: This paper compares the CORE rankings of computing conferences and journals to the frequency of citation of those journals and conferences in the Australasian Computer Science Conference (ACSC) 2006, 2007 and 2008 proceedings. The assumption underlying this study is that there should be a positive relationship between citation rates and the CORE rankings. Our analysis shows that the CORE rankings broadly reflect the ACSC citations, but with some anomalies. While these anomalies might be minor in the larger scheme of things, anomalies need to be addressed, as the careers of individual academics may depend upon it. Rankings are probably here to stay, and this paper ends with some suggestions on how the rankings process should now evolve, so that it becomes more transparent.
Lister, R.F., Fidge, C.F. & Teague, D. 2009, 'Further Evidence of a Relationship between Explaining, Tracing and Writing Skills in Introductory Programming', ACM Special Interest Group on Computer Science Education Conference, Paris, France, July 2009 in 14th annual ACM SIGCSE conference on Innovation and technology in computer science education, ed Brezillon, P., Russell, I., Marc Labat, J., Association for Computing Machinery, New York, NY, USA, pp. 161-165.
View/Download from: UTSePress | Publisher's site
ABSTRACT: This paper reports on a replication of earlier studies into a possible hierarchy of programming skills. In this study, the students from whom data was collected were at a university that had not provided data for earlier studies. Also, the students were taught the programming language "Python", which had not been used in earlier studies. Thus this study serves as a test of whether the findings in the earlier studies were specific to certain institutions, student cohorts, and programming languages. Also, we used a non-parametric approach to the analysis, rather than the linear approach of earlier studies. Our results are consistent with the earlier studies. We found that students who cannot trace code usually cannot explain code, and also that students who tend to perform reasonably well at code writing tasks have also usually acquired the ability to both trace code and explain code.
Venables, A., Tan, G. & Lister, R.F. 2009, 'A Closer Look at Tracing, Explaining and Code Writing Skills in the Novice Programmer', International Computing Education Research Workshop, Berkeley, CA, USA, August 2009 in Fifth International Computing Education Research Workshop, ed Clancy, M., Caspersen, M., Lister, R., Association for Computing Machinery, New York, NY, USA, pp. 117-128.
View/Download from: UTSePress | Publisher's site
Abstract: The way in which novice programmers learn to write code is of considerable interest to computing education researchers. One research approach to understanding how beginners acquire their programming abilities has been to look at student performance in exams. Lopez et al. (2008) analyzed student responses to an end-of-first-semester exam. They found two types of questions accounted for 46% of the variance on the code writing portion of the same exam. One of those types of question required students to trace iterative code, while the other type required students to explain what a piece of code did. In this paper, we investigate whether the results by Lopez et al. may be generally indicative of something about novice programmers, or whether their results are just an artifact of their particular exam. We studied student responses to our own exam and our results are broadly consistent with Lopez et al. However, we did find that some aspects of their model are sensitive to the particular exam questions used. Specifically, we found that student performance on explaining code was hard to characterize, and the strength of the relationship between explaining and code writing is particularly sensitive to the specific questions asked. Additionally, we found Lopez et al.'s use of a Rasch model to be unnecessary, which will make it far easier for others to conduct similar research.
Clear, T., Edwards, J.J., Lister, R.F., Simon, B., Thompson, E. & Whalley, J. 2008, 'The Teaching of Novice Computer Programmers:Bringing the Scholarly-Research Approach to Australia', Australasian Conference on Computer Science Education, Wollongong, January 2008 in Computing Education 2008. Proc. Tenth Australasian Computing Education Conference (ACE 2008). CRPIT. 78, ed Hamilton, S; Hamilton, M, Australian Computer Society, Adelaide, pp. 63-68.
View/Download from: UTSePress | Publisher's site
BRACElet is a multi-institutional multi-national research study of how novice programmers comprehend and write computer programs. This paper reviews the first action research cycle of the BRACElet project and, in the process, charts a path for the upcoming second cycle. The project remains close to educational practice, with much of the data being either data collected directly from exams sat by novices, or data from think-out-Ioud protocols where the task undertaken by a novice or an expert is modelled on an exam question. The first action research cycle analysed data in terms of the SOLO taxonomy. From think-aloud responses, the authors found that educators tended to manifest a SOLO relational response on small reading problems, whereas students tended to manifest a multistructural response. Furthennore, those students who manifested a relational response tended to do better overall in the exam than students who manifested a multistructural response. The second action research cycle will explore the relationship between the ability to read code and the ability to write code. Apart from reporting on the BRACElet project itself, this paper serves as an invitation for institutions and individuals to join the second action research cycle of the BRACElet project.
Lee, S.A. & Lister, R.F. 2008, 'Experiments in the Dynamics of Phase Coupled Oscillators When Applied to Graph Colouring', Australasian Computer Science Conference, Wollongong, Australia, January 2008 in 31st Australasian Computer Science Conference (ACSC 2008), ed Gillian Dobbie and Bernard Mans, Australian Computer Society, Inc, Darlinghurst, Australia, pp. 83-89.
View/Download from: UTSePress
Abstract This paper examines the capacity of networks of phase coupled oscillators to coordinate activity in a parallel, distributed fashion. To benchmark these networks of oscillators, we present empirical results from a study of the capacity of such networks to colour graphs. We generalise the update equation of Aihara et al. (2006) to an equation that can be applied to graphs requiring multiple colours. We find that our simple multi-phase model can colour some types of graphs, especially complete graphs and complete k-partite graphs with equal or a near equal number of vertices in each partition. A surprising empirical result is that the effectiveness of the approach appears to be more dependent upon the topology of the graph than the size of the graph.
Berglund, A. & Lister, R.F. 2008, 'Debating the OO debate: where is the problem?', Baltic Sea Conference on Computing Education Research, Koli National Park, Finland, November 2007 in Conferences in Research and Practice in Information Technology, Vol 68: Koli Calling Proceedings of the Seventh Baltic Sea Conference on Computing Education Research, ed Lister, R. and Simon, Australian Computer Society, Australia, pp. 171-174.
View/Download from: UTSePress | Publisher's site
Abstract: In this paper we discuss problems related to the teaching of object-oriented programming (OOP). We argue that more research on how the computer science teacher understands OOP would be beneficial. Our argument takes its point of departure in three sets of studies: (1) an ongoing study on how computer science teachers understand core concepts of OOP, (2) a study of how the teaching of OOP is discussed within the CS community, and (3) a set of studies that discuss the different ways in which CS teachers experience their teaching. This paper reports on an ongoing study of the different ways in which computing science teachers understand object- oriented programming, and what they mean when use the term objects first.. The phenomenographic research approach has been applied to the analysis of a discussion that occurred in the SIGCSE-members mailing list. Two understandings of objects first have been identified: (1) as an extension of imperative programming, and (2) as conceptually different from imperative programming. These two understandings are illustrated via the differing ways in which computing science teachers use the term polymorphism.
Lister, R.F. & Box, I. 2008, 'A citation analysis of the SIGCSE 2007 proceedings', ACM Special Interest Group on Computer Science Education Conference, Portland, OR, USA, March 2008 in Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, ed Susan Rodger, John Dougherty, Mark Guzdial, Sue Fitzgerald, Ellen Walker, Association of Computing Machinery, New York, NY, USA, pp. 476-480.
View/Download from: UTSePress | Publisher's site
Abstract: This paper identifies the most commonly cited conferences, journals and books among the 1398 citations made in the 122 publications of the SIGCSE 2007 proceedings. The SIGCSE 2007 authors cited a very large array of conferences, journals and books, but the majority are only cited within a single paper. There are only a very small set of journals and conferences cited frequently. Most books cited are concerned with technical information or are textbooks. Only 2% of books are concerned with computer science education and 23% with education in general. The picture that emerges from this citation analysis is that the SIGCSE community does not have a substantial core set of educational literature. Also, the epistemology of the SIGCSE community is primarily objectivist, with a focus on content, rather than a constructivist, student-centered focus on learning.
Sheard, J., Carbone, A., Lister, R.F., Simon, B., Thompson, E. & Whalley, J. 2008, 'Going SOLO to Assess Novice Programmers', ACM Special Interest Group on Computer Science Education Conference, Madrid, Spain, June 2008 in Proceedings of the 13th Annual SIGCSE Conference on innovation and Technology in Computer Science Education, ed June Amillo, Cary Laxer, Ernestina Menasalvas, Alison Young, Association of Computing Machinery, New York, NY, USA, pp. 209-213.
View/Download from: UTSePress | Publisher's site
ABSTRACT: This paper explores the programming knowledge of novices using Biggs' SOLO taxonomy. It builds on previous work of Lister et al. (2006) and addresses some of the criticisms of that work. The research was conducted by studying the exam scripts for 120 introductory programming students, in which three specific questions were analyzed using the SOLO taxonomy. The study reports the following four findings: when the instruction to students used by Lister et al. - "In plain English, explain what the following segment of Java code does" - is replaced with a less ambiguous instruction, many students still provide multistructural responses; students are relatively consistent in the SOLO level of their answers; student responses on SOLO reading tasks correlate positively with performance on writing tasks; postgraduates students manifest a higher level of thinking than undergraduates.
Lopez, M., Whalley, J., Robbins, P. & Lister, R.F. 2008, 'Relationships between reading, tracing and writing skills in introductory programming', International Workshop on Computing Education Research, Sydney, Australia, September 2008 in Fourth International Workshop on Computing Education Research, ed Raymond Lister, Michael E Caspersen and Mike Clancy, Association of Computing Machinery, New York, USA, pp. 101-112.
View/Download from: UTSePress | Publisher's site
ABSTRACT: This study analyzed student responses to an examination, after the students had completed one semester of instruction in programming. The performance of students on code tracing tasks correlated with their performance on code writing tasks. A correlation was also found between performance on "explain in plain English" tasks and code writing. A stepwise regression, with performance on code writing as the dependent variable, was used to construct a path diagram. The diagram suggests the possibility of a hierarchy of programming related tasks. Knowledge of programming constructs forms the bottom of the hierarchy, with "explain in English", Parson's puzzles, and the tracing of iterative code forming one or more intermediate levels in the hierarchy.
Simon, S., Carbone, A., de Raadt, M., Lister, R.F., Hamilton, M. & Sheard, J. 2008, 'Classifying computing education papers: process and results', International Workshop on Computing Education Research, Sydney, Australia, September 2008 in Fourth international Workshop on Computing Education Research, ed Raymond Lister, Michael E Caspersen and Mike Clancy, Association of Computing Machinery, New York, USA, pp. 161-172.
View/Download from: UTSePress | Publisher's site
ABSTRACT: We have applied Simon's system for classifying computing education publications to all three years of papers from ICER. We describe the process of assessing the inter-rater reliability of the system and fine-tuning it along the way. Our analysis of the ICER papers confirms that ICER is a research-intensive conference. It also indicates that the research is quite narrowly focused, with the majority of the papers set in the context of programming courses. In addition we find that ICER has a high proportion of papers involving more than one institution, and high proportions of papers on the themes of ability/aptitude and theories and models of teaching and learning.
Lister, R.F. 2008, 'After the gold rush: toward sustainable scholarship in computing', Australasian Computing Education Conference, Wollongong, Australia, January 2008 in Tenth Australasian Computing Education Conference, ed Simon and Margaret Hamilton, Australian Computer Society, Australia, pp. 3-18.
View/Download from: UTSePress | Publisher's site
Abstract: In just thirty years, we have gone from punched cards to Second Life. But, as the American National Science Foundation (NSF) recently noted, ++undergraduate computing education today often looks much as it did several decades ago+ (NSF, 2006). Consequently, today++s ++Nintendo Generation+ have voted with their feet. We bore them. The contrast between the changes wrought via computer research over the last 30 years, and the failure of computing education to adapt to those changes, is because computing academics lead a double life. In our research lives we see ourselves as part of a community that reaches beyond our own university. We read literature, we attend conferences, we publish, and the cycle repeats, with community members building upon each other++s work. But in our teaching lives we rarely discuss teaching beyond our own university, we are not guided by any teaching literature; instead we simply follow our instincts. Academics in computing, or in any other discipline, can approach their teaching as research into how novices become experts. Several recent multi-institutional research collaborations have studied the development of novice programmers. This paper describes some of the results from those collaborations. The separation of our teaching and research lives diminishes not just our teaching but also our research. The modern practice of stripping away all ++distractions++ to maximize research output is like the practice of stripping away rainforest to grow beef ++ both practices appear to work, for a little while, but not indefinitely. Twenty-first century academia needs to bring teaching and research together, to form a scholarship of computing that is an integrated, sustainable, ecological whole.
Berglund, A., Box, I., Eckerdal, A., Lister, R.F. & Pears, A. 2008, 'Learning educational research methods through collaborative research: the PhICER initiative', Australasian Computing Education Conference, Wollongong, Australia, January 2008 in Tenth Australasian Computing Education Conference, ed Simon and Margaret Hamilton, Australian Computer Society, Australia, pp. 35-42.
View/Download from: UTSePress | Publisher's site
Abstract: To accommodate an increasing interest in phenomenography among the computing education research community, the authors of this paper have organised two workshops, or intensive courses, in phenomenography, intended for researchers in computing education at a university level. Although the workshop programs contained lectures and smaller exercises, the emphasis was on preparing one joint publication from each of the workshops. The publication from the first workshop discussed the experience of being a teacher in computing, while the second focused on teachers++ experiences of the problems their students encounter when learning computing. The workshops and their impact on the community are discussed in this paper.
Lister, R.F. & Box, I. 2008, 'A citation analysis of the ACE2005 - 2007 proceedings, with reference to the June 2007 CORE conference and journal rankings', Australasian Computing Education Conference, Wollongong, Australia, January 2008 in Tenth Australasian Computing Education Conference, ed Simon and Margaret Hamilton, Australian Computer Society, Australia, pp. 93-102.
View/Download from: UTSePress | Publisher's site
Abstract: This paper compares the CORE rankings of computing education conferences and journals to the frequency of citation of those journals and conferences in the ACE2005, 2006 and 2007 proceedings. The assumption underlying this study is that citation rates are a measure of esteem, and so there should be a positive relationship between citation rates and rankings. The CORE conference rankings appear to broadly reflect the ACE citations, but there are some inconsistencies between citation rates and the journal rankings. The paper also identifies the most commonly cited books in these ACE proceedings. Finally, in the spirit of ++Quis custodiet ipsos custodes?+ the paper discusses some ways in which the CORE rankings process itself might in future be made more transparent and open to scholarly discourse.
Simon, S., Sheard, J., Carbone, A., de Raadt, M., Hamilton, M., Lister, R.F. & Thompson, E. 2008, 'Eight years of computing education papers at NACCQ', National Advisory Committee on Computing Qualifications, Auckland, New Zealand, July 2008 in Proceedings of the 21st Annual Conference of the National Advisory Committee on Computing Qualifications, ed Samuel Mann and Mike Lopez, National Advisory Committee on Computing Qualifications, New Zealand, pp. 101-107.
View/Download from: UTSePress | Publisher's site
Abstract: The 157 computing education papers from the past eight NACCQ conferences are categorised and summarised by a group of researchers from multiple institutions, with steps taken to measure and improve the consistency of classification. The papers are set predominantly in programming subjects, hardware/architecture/systems/ network subjects, and capstone projects. The bulk of the papers are about teaching/learning techniques, assessment techniques, teaching/learning tools, curriculum, and educational technology. Most of the papers are set within single subjects, a few in multiple subjects within a single program or department, and fewer still in a range of subjects across the whole institution or multiple institutions. Nearly a quarter of the papers either expound a position or outline a proposal; a large but diminishing proportion report on something such as a change of curriculum or approach; and a large and increasing proportion are clearly research papers, focusing on the analysis of data to answer an explicit research question.
Clear, T., Whalley, J., Lister, R.F., Carbone, A., Hu, M., Sheard, J., Simon, B. & Thompson, E. 2008, 'Reliably Classifying Novice Programmer Exam Responses using the SOLO Taxonomy', National Advisory Committee on Computing Qualifications, Auckland, New Zealand, July 2008 in 21st Annual Conference of the National Advisory Committee on Computing Qualifications, ed Samuel Mann and Mike Lopez, National Advisory Committee on Computing Qualifications, New Zealand, pp. 23-30.
View/Download from: UTSePress | Publisher's site
Abstract: Past papers of the BRACElet project have described an approach to teaching and assessing students where the students are presented with short pieces of code, and are instructed to explain, in plain English, what the code does. The student responses to these types of questions can be analysed according to the SOLO taxonomy. Some students display an understanding of the code as a single, functional whole, while other students cannot ++see the forest for the trees+ . However, classifying student responses into the taxonomy is not always straightforward. This paper analyses the reliability of the SOLO taxonomy as a means of categorising student responses. The paper derives an augmented set of SOLO categories for application to the programming domain, and proposes a set of guidelines for researchers to use.
Doyle, B.J. & Lister, R.F. 2007, 'Why Teach Unix?', Australasian Conference on Computer Science Education, Ballarat, Australia, January 2007 in Australasian Conference on Computer Science Education, ed Mann.S; Simon, Australian Computer Society, Australia, pp. 19-26.
View/Download from: UTSePress
This paper examines computing academics' conceptions of the Unix operating system, and the purpose of teaching Unix. Interview transcripts from nine academics were analysed phenomenographically. A small number of qualitatively different conceptions of Unix were identified, within two broad categories. The first broad category manifested a technical approach to Unix. Within this broad category, the conceptions of Unix were, from the least to most sophisticated ! (1) Unix as a set of unrelated commands; (2) Unix as a command line interface superior to GUIs; and (3) Unix as a problem solving tool. The second broad category was a non technical conception of Unix, in which Unix was seen as a resource that is cheap, secure and robust. With regard to teaching Unix, two broad categories of reasons were identified ! practical and pedagogical. These results for teachers are broadly consistent with an earlier phenomenographic study of student conceptions of Unix.
Carbone, A., de Raadt, M., Kay, J., Lister, R.F., Litchfield, A.J., Raban, R., Roe, P., Santamaria, D., Sheard, J., Shepherd, J., Solomon, A.I. & Thomas, R. 2007, 'The Carrick Vision and Computing Education: Four Case Studies in Multi-institutional Collaboration', Australasian Conference on Computer Science Education, Ballarat, Australia, January 2007 in Australian Computer Science Communications, ed Mann.S; Simon, Australian Computer Society, Australia, pp. 3-8.
View/Download from: Publisher's site
The Carrick Institute is an initiative of the Australia federal government. It is aimed at generating strategic change in Australian University education, via grants and other awards to approximately $20 million annually. By previous Australian standards, the potential funding for projects is large. However, the Carrick Institute has a well focused vision, and grant applications need to be aligned with that vision. This paper first describes some key aspects of the Carrick vision, before describing four multi-institutional computing education projects that successfully attracted funding from the Carrick Institute in 2006. Three of the projects are funded under Carrick++s Priority Program, and are concerned with different aspects of automated assessment: (1) assessing Unix scripting skills, (2) self and peer assessment in groupwork, and (3) the assessment of novice programmers. The fourth project is funded under Carrick++s Disciplinary-Based Initiatives Scheme. Commonalities in the structure of these three projects are observed.
Lister, R.F., Berglund, A., Box, I., Cope, C., Pears, A., Avram, C., Bower, M., Carbone, A., Davey, B., de Raadt, M., Doyle, B.J., Fitzgerald, S., Mannila, L., Kutay, C., peltomaki, p., Sheard, J., Sutton, K., Traynor, D., Tutty, J. & Venables, A. 2007, 'Differing Ways that Computing Academics Understand Teaching.', Australasian Conference on Computer Science Education, Ballarat, Australia, January 2007 in Australian Computer Science Communications - Proceedings of the ninth Australasian conference on Computing education, ed Mann.S; Simon, Australian Computer Society, Australia, pp. 97-106.
View/Download from: UTSePress | Publisher's site
This paper presents first results from a wide-ranging phenomenographic study of computing academics' understanding of teaching. These first results focus upon four areas: the role of lab practical sessions, the experience of teaching success, conceptions of motivating and engaging students, and the granularity of the teacher's focus. The findings are comparable with prior work on the understandings of academics in other disciplines. This study was started as part of a workshop on phenomenography. Most participants at the workshop received their first training in phenomenography. This paper summarises the structure of the workshop.
Lister, R.F. 2007, 'The Neglected Middle Novice Programmer: Reading and Writing without Abstracting', National Advisory Committee on Computing Qualifications, Nelson, New Zealand, July 2007 in 20th Annual Conference of the National Advisory Committee on Computing Qualifications, ed Mann, S. and Bridgeman, N., National Advisory Committee on Computing Qualifications, New Zealand, pp. 133-140.
View/Download from: UTSePress
Many teachers of novice programmers have lamented that students either seem to have a natural gift for programming, or have no gift for it at all. In this paper, we discuss a third group of students, the middle novice programmer. At the completion of their first semester of programming, these students can manifest a strong concrete grasp of the semantics of basic programming language constructs, by hand executing code, but they cannot reason about code at a higher goal/plan level. The research evidence presented in this paper for the existence of these middle novice programmers is from the analysis of twelve multiple choice questions, which students attempted as part of an end-of-first-semester exam.
Berglund, A. & Lister, R.F. 2007, 'How do students understand computer network protocols?', National Advisory Committee on Computing Qualifications, Nelson, New Zealand, July 2007 in 20th Annual Conference of the National Advisory Committee on Computing Qualifications, ed Mann, S. and Bridgeman, N., National Advisory Committee on Computing Qualifications, New Zealand, pp. 15-20.
View/Download from: UTSePress
When computing education research is performed in a rigorous and theoretically sound way, it can inform and improve teaching. Having conducted a phenomenographic study of students++ understandings of computer network protocols, we present our findings and explain how they can inform the teaching of these protocols. We go on to explore what our results can tell us about the discipline of computer science as a whole.
Lister, R.F. 2006, 'Computer Science Teachers as Amateurs, Students and Reseachers', Baltic Sea Conference on Computing Education Research, Koli NAtional Park Eastern Finland, November 2005 in Koli Calling 2005- Proceeding of the Fifth Koli Calling Conference on Computer Science Education, ed Salakoski, T., Mantyla, T., Laakso, M, Turku Centre for Computer Science, Turku, Finland, pp. 3-12.
View/Download from: UTSePress
Simon, B., Lister, R.F. & Fincher, S. 2006, 'Multi-institutional computer science education research: A review of recent studies of novice understanding', FIE Frontiers in Education, San Deigo, USA, October 2006 in Frontiers in Education 2006, ed Lord, S & Hayhurst, D, IEEE, San Diego, CA, USA, pp. S4E-12-S4E-17.
View/Download from: UTSePress | Publisher's site
There has been a recent proliferation of multinational, multi-institutional computer science education research exploring issues surrounding novice computer science student understanding. The large numbers of students studied in these efforts and their breadth in terms of student background (both technically and culturally) has led to significant interest in the work. In this paper, we summarize each study, identifying the populations studied, the types of data collected, the analyses performed and review the primary results. We also report on current ongoing work related to and derived from these efforts
Lister, R.F. 2006, 'Driving learning via criterion-referenced assessment using Bloom's taxonomy', Uniserve Science Symposium, Sydney, Australia, September 2006 in UniServe Science 2006, ed N/A, UniServe Science, Sydney, Australia, pp. 80-88.
View/Download from: UTSePress
Thompson, E., Whalley, J., Lister, R.F. & Simon, B. 2006, 'Code classification as a learning and asssessment exercise for novice programmers', National Advisory Committee on Computing Qualifications, Wellington, New Zealand, July 2006 in Proceeding of the 19th annual conference of the ational advisory committee on computing qualifications, ed Mann, S; Bridgeman, N, National advisory comittee on computing qualifications, Hamilton, New Zealand, pp. 291-298.
View/Download from: UTSePress
Doyle, B.J. & Lister, R.F. 2006, 'Apreliminary phenomenographic study concerning student experiences of UNIX', National Advisory Committee on Computing Qualifications, Wellington, July 2006 in Proceedings of the 19th annual conference of the national advisory comittee on computing qualifications, ed Mann, S; Bridgeman, N., National advisory comittee on computing qualifications, Hamilton, New Zealand, pp. 73-78.
View/Download from: UTSePress
Solomon, A.I., Santamaria, D. & Lister, R.F. 2006, 'Automated testing of Unix command-line and scripting skills', international conference on information technology based higher education, Sydney, Australia, July 2006 in Proceedings of the 7th international conference on information technology based higher education, ed Housego, S, UTS, Sydney, Australia, pp. 120-160.
View/Download from: UTSePress
Fincher, S., Lister, R.F., Clear, T., Robins, A., Tenenberg, J. & Petre, M. 2005, 'Multi-institutional, multi-national studies in CSEd research: some design considerations and trade-offs', International workshops on computing education research, Seattle, USA, October 2005 in Proceedings of the 2005 international workshops on computing education research, ed Anderson, R; Fincher, S.A; Guzdial, M., Association of computing machinery press, New York, USA, pp. 111-121.
View/Download from: UTSePress
Lister, R.F. 2005, 'Methods for evaluating the approproiateness and effectiveness of summative assessment via multiple choice examinations for technology-focused disciplines', Evaluations and Assessment Conference, Sydney, Aust, November 2005 in Making a Difference: 2005 Evaluations and Assessment Conference, ed Kandlbinder,P;, UTS, Sydney, Aust, pp. 75-84.
View/Download from: UTSePress
Lister, R.F. 2004, 'Teaching Java First: Experiments with Pigs-Early Pedagogy', Australasian Computing Education Conference, Dunedin, New Zealand, January 2004 in Proceedings of the 6th Australasian Computing Education Conference, ed Lister,R; Young, A.L., Australian Computer Society Inc, Sydney, Australia, pp. 177-183.
View/Download from: UTSePress
Lister, R.F., Box, I., Morrison, B., Westbrook, S. & Tenenberg, J. 2004, 'The Dimensions of Variation in the Teaching of Data Structures', Annual Conference on Innovation and Technology in Computer Science Education, Leeds, UK, July 2004 in SIGCSE Bulletin: Inroads ITiCSE 2004 Proceedings, Vol 36 #3 Sept 2004, ed Impagliazzo,J., Association for Computing Machinery, New York, USA, pp. 92-96.
View/Download from: UTSePress | Publisher's site
The current debate about the teaching of data structures is hampered because, as a community, we usually debate specifics about data structure implementations and libraries, when the real level of disagreement remains implicit -- the intent behind our teaching. This paper presents a phenomenographic study of the intent of CS educators for teaching data structures in CS2. Based on interviews with Computer Science educators and analysis of CS literature, we identified five categories of intent: developing transferable thinking, improving students' programming skills, knowing "what's under the hood", knowledge of software libraries, and component thinking. The CS community needs to first debate at the level of these categories before moving to more specific issues. This study also serves as an example of how phenomenographic analysis can be used to inform debate on syllabus design in general.
Lister, R.F. 2004, 'Objectives and Object-Oriented Programming', National Advisory Committee on Computing Qualifications, Christchurch, New Zealand, July 2004 in Proceedings of the 17th Annual Conference of the National Advisory Committee on Computing Qualifications, ed Mann,S; Clear,T., National Advisory Committee on Computing Qualifications, Hamilton, New Zealand, pp. 13-19*.
View/Download from: UTSePress | Publisher's site
Prior, J.R. & Lister, R.F. 2004, 'The Backwash Effect on SQL Skills Grading', ACM Special Interest Group on Computer Science Education Conference, Leeds,UK, June 2004 in ITiCSE 2004 Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ed Boyle,R; Clark,M; Kuma,A., ACM Press, New York,USA, pp. 32-36.
View/Download from: UTSePress | Publisher's site
This paper examines the effect of grading approaches for SQL query formulation on students' learning strategies. The way that students are graded in a subject has a significant impact on their learning approach, and it is crucial that graded tasks are carefully designed and implemented to inculcate a deep learning experience. An online examination system is described and evaluated.
Collins, W.J., Tenenberg, J., Lister, R.F. & Westbrook, S. 2003, 'The Role of Framework Libraries in CS2', ACM Special Interest Group on Computer Science Education Conference, Reno, Nevada, USA, February 2003 in Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education, ed Unknown, Association for Computing Machinery, New York, USA, pp. 403-404.
View/Download from: Publisher's site
The recent emergence of object-oriented framework libraries of classic data structures and algorithms such as the Standard Template Library and Java's Collection classes provides a set of general and efficient data structure components for use by practicing software developers. A number of textbook writers are beginning to incorporate the use of these frameworks into "CS2", the traditional first course in data structures at the university level.There has scarcely been a discussion of how these frameworks should best be incorporated into CS2, if they should be used at all. The proposed panel will examine the role of standardized framework libraries in the first data structures course at the university level. Panelists will focus on the following questions. What are the fundamental objectives of CS2, and what role might frameworks have in meeting these objectives? How does an instructor balance student needs for additional instruction in programming basics (e.g. arrays and pointers) versus practice in larger scale design and code reuse? What would be given up to incorporate frameworks into CS2? And to what extent is it important for students to construct elementary data structures from first principles? What assumptions about student cognition and learning does a pro- or con-frameworks approach imply.By trying to articulate answers to some of the above questions, we hope to raise the level of discussion concerning the evolution of the introductory computer science curriculum. This panel will thus try to make explicit, and hence available for critical examination, some of the arguments and assumptions that inform this debate.
Solomon, A.I., Sutcliffe, P.J. & Lister, R.F. 2003, 'Sorting Circular Permutations by Reversal', Algorithms and Data Structures Symposium (was Workshop on Algorithms and Data Structures), Ottawa, Ontario, Canada, July 2003 in Algorithms and Data Structures Symposium (was Workshop on Algorithms and Data Structures), ed Dehne, F; Sack, J.R; Smid, M, Springer-Verlag Berlin Heidelberg, Germany, pp. 319-328.
View/Download from: UTSePress
Lister, R.F. & Leaney, J.R. 2003, 'Bad Theory Versus Bad Teachers: Towards a Pragmatic Synthesis of Constructivism and Objectivism', Higher Education Research and Development Society of Australasia Annual Conference, Christchurch, New Zealand, July 2003 in Learning for an Unknown Future - Proceedings of the 2003 Annual International Conference of the Higher Educational Research and Development Society of Australasia Volume 26, ed Bond, C; Bright, P, Higher Educational Research and Development Society of Australasia, Milperra, NSW, Australia, pp. 429-436.
View/Download from: UTSePress
Lister, R.F. & Leaney, J.R. 2003, 'Introductory Programming, Criterion-Referencing and Bloom', ACM Special Interest Group on Computer Science Education Conference, Reno, Nevada, USA, February 2003 in Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education, ed Unknown, The Association for Computing Machinery, New York, USA, pp. 143-147.
View/Download from: Publisher's site
In the traditional norm-referencing approach to grading, all students in a CS1 class attempt the same programming tasks, and those attempts are graded "to a curve". The danger is that such tasks are aimed at a hypothetical average student. Weaker students can do little of these tasks, and learn little. Meanwhile, these tasks do not stretch the stronger students, so they too are denied an opportunity to learn. Our solution is two-fold. First, we use a criterion-referenced approach, where fundamentally different tasks are set, according to the ability of the students. Second, the differences in the nature of the tasks reflect the differing levels of Bloom's taxonomy. Weaker CS1 students are simply required to demonstrate knowledge and comprehension; the ability to read and understand programs. Middling students attempt traditional tasks, while the stronger students are set open-ended tasks at the synthesis and evaluation levels.
Lister, R.F. & Leaney, J.R. 2003, 'First Year Programming: Let All the Flowers Bloom', Australasian Computing Education Conference, Adelaide, Australia, February 2003 in Computing Education 2003. Fifth Australasian Computing Educational Conference Volume 20, ed Greening, T; Lister, R, Australian Computer Society Inc., Sydney, Australia, pp. 221-230.
Lister, R.F. & Jerram, P. 2001, 'Minimal Mark-Up of multiple choice exams using XML', Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Melbourne, Victoria, December 2001 in Proceedings of ASCILITE 2001, ed Kennedy G, Keppell M, McNaught C, Petrovic T, Biomedical Multimedia Unit, University of Melbourne, Melbourne, pp. 101-104.
View/Download from: UTSePress | Publisher's site
We describe a minimal XML mark-up for multiple choice exams. In our system, exams may be generated at anytime, by choosing a subset of questions from a pool. Furthermore, the system randomises the order of the choices within each chosen question. Thus a student may sit the exam many times. Our first version of this system has been described elsewhere. In this paper, we discuss the limitations of our first system that led to our current work, and give a description of the new system features, including XML tags for supporting collaborative authoring.
Lister, R.F., Smith, I., Ray, M. & Hawson, G. 2001, 'Nave Bayesian Prediction of bleeding after heart by-pass surgery', ANZIIS 2001, Perth, WA, November 2001 in Proceedings of ANZIIS 2001, ed Linggard, Australian Research Centre for Medical Engineering, University of WA, Perth, WA, pp. 317-321.
View/Download from: UTSePress

Journal Articles

Gluga, R., Kay, J., Lister, R.F., simon, s. & Kleitman, S. 2013, 'Mastering Cognitive Development Theory in Computer Science Education', Computer Science Education, vol. 23, no. 1, pp. 24-57.
View/Download from: UTSePress | Publisher's site
Abstract: To design an effective computer science curriculum, educators require a systematic method of classifying the difficulty level of learning activities and assessment tasks. This is important for curriculum design and implementation and for communication between educators. Different educators must be able to use the method consistently, so that classified activities and assessments are comparable across the subjects of a degree, and, ideally, comparable across institutions. One widespread approach to supporting this is to write learning objects in terms of Bloom+s Taxonomy. This, or other such classifications, is likely to be more effective if educators can use them consistently, in the way experts would use them. To this end, we present the design and evaluation of our online interactive web-based tutorial system, which can be configured and used to offer training in different classification schemes. We report on results from three evaluations. First, 17 computer science educators complete a tutorial on using Bloom+s Taxonomy to classify programming examination questions. Second, 20 computer science educators complete a Neo-Piagetian tutorial. Third evaluation was a comparison of inter-rater reliability scores of computer science educators classifying programming questions using Bloom+s Taxonomy, before and after taking our tutorial. Based on the results from these evaluations, we discuss the effectiveness of our tutorial system design for teaching computer science educators how to systematically and consistently classify programming examination questions. We also discuss the suitability of Bloom+s Taxonomy and Neo-Piagetian theory for achieving this goal. The Bloom+s and Neo-Piagetian tutorials are made available as a community resource. The contributions of this paper are the following: the tutorial system for learning classification schemes for the purpose of coding the difficulty of computing learning materials; its evaluation; new insights into the consistency that computing educators can achieve using Bloom; and first insights into the use of Neo-Piagetian theory by a group of classifiers.
Lister, R.F. 2012, 'Teaching-Oriented Faculty and Computing Education Research', ACM Inroads, vol. 3, no. 1, pp. 22-23.
View/Download from: UTSePress | Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2012, 'The CC2013 Strawman and Bloom's Taxonomy', ACM Inroads, vol. 3, no. 2, pp. 12-13.
View/Download from: UTSePress | Publisher's site
This is a regular column that I write for this journal.
Lister, R.F. 2012, 'A variation on Kvale's one thousand page question', ACM Inroads, vol. 3, no. 3, pp. 24-25.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F. 2012, 'Rare research: why is research uncommon in the computing education universe?', ACM Inroads, vol. 3, no. 4, pp. 16-17.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Lister, R.F., Clear, T., Simon, S., Bouvier, D.J., Carter, P., Eckerdal, A., Jackov+, J., Lopez, M., McCartney, R., Robbins, P., Seppala, O. & Thompson, E. 2010, 'Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer', SIGCSE Bulletin Inroads, vol. 41, no. 4, pp. 156-173.
View/Download from: UTSePress
In New Zealand and Australia, the BRACElet project has been investigating students' acquisition of programming skills in introductory programming courses. The project has explored students' skills in basic syntax, tracing code, understanding code, and writing code, seeking to establish the relationships between these skills. This ITiCSE working group report presents the most recent step in the BRACElet project, which includes replication of earlier analysis using a far broader pool of naturally occurring data, refinement of the SOLO taxonomy in code-explaining questions, extension of the taxonomy to code-writing questions, extension of some earlier studies on students' 'doodling' while answering exam questions, and exploration of a further theoretical basis for work that until now has been primarily empirical.
Lister, R.F. 2008, 'CS research: We are what we cite -- so where are we?', SIGCSE Bulletin Inroads, vol. 40, no. 4, pp. 16-18.
View/Download from: Publisher's site
This is a regular invited column I write for this journal.
Whalley, J., Clear, T. & Lister, R.F. 2007, 'The Many Ways of the BRACElet Project', Bulletin of Applied Computing and Information Technology, vol. 5, no. 1, pp. 1-16.
View/Download from: UTSePress
This paper provides a retrospective snapshot of the first two years of a multi-institutional multi-national study (MIMN) in Computer Science Education called the BRACElet Project. This study has been inquiring into how novice programmers comprehend and write computer programs. The context for the study is outlined, together with details of how it has evolved and those who have participated. Some challenges encountered during the project are highlighted and pointers for the successful conduct of such a study are provided. The paper concludes by noting pitfalls to be avoided, some open research questions, and current plans for furthering the project.
Bergin, J., Lister, R.F., Boucher Owens, B. & McNally, M. 2006, 'The first programming course: ideas to end the enrollment decline', ACM SIGCSE Bulletin, vol. 38, no. 3, pp. 301-302.
View/Download from: Publisher's site
Lister, R.F. 2006, 'One room four meetings', ACM SIGCSE Bulletin, vol. 38, no. 4, pp. 11-13.
View/Download from: Publisher's site
Lister, R.F. 2006, 'Call me Ishmael: Charles Dickens meets Moby Book', ACM SIGCSE Bulletin, vol. 38, no. 2, pp. 11-13.
View/Download from: Publisher's site
simon, s., Fincher, S., Robbins, A., Box, I., Cutts, Q.I., de Raadt, M., Haden, P., Hamer, J., Hamilton, M., Lister, R.F., Petre, M., Sutton, K., Tolhurst, D. & Tutty, J. 2006, 'Predictors of success in a first programming course', Australian Computer Science Communications, vol. 28, no. 5, pp. 189-196.
View/Download from: UTSePress
This paper describes a multi-national, multiinstitutional study that investigated introductory programming courses. Student participants were drawn from eleven institutions, mainly in Australasia, during the academic year of 2004. A number of diagnostic tasks were used to explore cognitive, behavioural, and attitudinal factors such as spatial visualisation and reasoning, the ability to articulate strategies for commonplace search and design tasks, and attitudes to studying. The results indicate that a deep approach to learning was positively correlated with mark for the course, while a surface approach was negatively correlated; spatial visualisation skills are correlated with success; a progression of map drawing styles identified in the literature has a significant correlation with marks; and increasing measures of richness of articulation of a search strategy are also associated with higher marks. Finally, a qualitative analysis of short interviews identified the qualities that students themselves regarded as important to success in programming.
Kutay, C. & Lister, R.F. 2006, 'Up close and pedagogical: Computing academics talk about teaching', Australian Computer Science Communications, vol. 28, no. 5, pp. 125-134.
View/Download from: UTSePress
This paper describes and enacts a process for bootstrapping a more systematic discussion of computing education within a school of computing at a research-intensive Australasian university. Thus far, the project has gone through three stages. In the first stage, some academics were interviewed about their approach to teaching. In the second stage, selected anonymous quotes from the interviews were presented and discussed by other interested members of the school at workshops. In the final stage, selected anonymous quotes from the interviews and workshops were placed on a web-based survey, to which interested members of the school responded. These forms of data will be used to drive further stages of debate within the school. The theoretical underpinnings of this project are Wenger's concept of a community of practice, phenomenography, and socially constructivism. The aim is not to instruct the academics in any "right way" to teach. Instead, the aim is to facilitate debate, where the teachers identify the problems, and in finding the solutions they construct their own "pedagogic reality". As facilitators of this process, the authors of this paper highlighted dialectically opposed views in quotes from the teachers, and then allow the teachers to synthesise those views into a more sophisticated view. Our ultimate project aim is to grow a teaching community that balances reified theories of teaching and learning with participation in a community of practice.
Tolhurst, D., Baker, B., Hamer, J., Box, I., Lister, R.F., Cutts, Q.I., Petre, M., de Raadt, M., Robins, A., Fincher, S., simon, s., Haden, P., Sutton, K., Hamilton, M. & Tutty, J. 2006, 'Do map drawing styles of novice programming predict success in programming? A multi-national, multi institutional Study.', Australian Computer Science Communications, vol. 28, no. 5, pp. 213-222.
View/Download from: UTSePress | Publisher's site
Lister, R.F., Simon, B. & Thompson, E. 2006, 'Not seeing the forest for the trees: Novice programmers and the SOLO taxonomy', ACM SIGCSE Bulletin, vol. 38, no. 3, pp. 118-122.
View/Download from: UTSePress
This paper reports on the authors use of the SOLO taxonomy to describe differences in the way students and educators solve small code reading exercises. SOLO is a general educational taxonomy, and has not previously been applied to the study of how novice programmers manifest their understanding of code. Data was collected in the form of written and think-aloud responses from students (novices) and educators (experts), using exam questions. During analysis, the responses were mapped to the different levels of the SOLO taxonomy. From think-aloud responses, the authors found that educators tended to manifest a SOLO relational response on small reading problems, whereas students tended to manifest a multistructural response. These results are consistent with the literature on the psychology of programming, but the work in this paper extends on these findings by analyzing the design of exam questions.
Fincher, S., Lister, R.F., Sheard, J., Tenenberg, J. & Young, A. 2006, 'Multi-institutional teaching communities in computer education', Australian computer sciences communications, vol. 28, no. 5, pp. 7-10.
View/Download from: Publisher's site
simon, s., Cutts, Q.I., Fincher, S., Haden, P., Robins, A., Sutton, K., Baker, B., Box, I., de Raadt, M., Hamer, J., Hamilton, M., Lister, R.F., Petre, M., Tolhurst, D. & Tutty, J. 2006, 'The ability to articulate strategy as a predictor of programming skills', Australian computer science communications, vol. 28, no. 5, pp. 181-188.
View/Download from: UTSePress
A multi-national, multi-institutional study investigating introductory programming courses drew on student participants from eleven institutions, mainly in Australasia, during the academic year of 2004. A number of diagnostic tasks were used to explore cognitive, behavioural, and attitudinal factors such as spatial visualisation and reasoning, the ability to articulate strategies for commonplace search and design tasks, and attitudes to studying. This paper reports in detail on the task that required participants to articulate a commonplace search strategy. The results indicate that increasing measures of richness of articulation of a search strategy are associated with higher marks in the course.
goldweber, m., Bergin, J., Lister, R.F. & McNally, M. 2006, 'A comparison of different approaches to the introductory programming course', Australian computer science communications, vol. 28, no. 5, pp. 11-13.
View/Download from: Publisher's site
Whalley, J., Lister, R.F., Thompson, E., Clear, T., Robbins, P., Kumar, P. & Prasad, C. 2006, 'An Australasian study of reading and comprehension skills in novice programmers, using Bloom and SOLO taxonomies', Australin computer science communications, vol. 52, no. 5, pp. 243-252.
View/Download from: UTSePress
In this paper we report on a multiinstitutional investigation into the reading and comprehension skills of novice programmers. This work extends previous studies (Lister 2004, McCracken 2001) by developing a question set within two key pedagogical frameworks: the Bloom and SOLO taxonomies. From this framework of analysis some interesting emergent patterns relating the cognitive level of the questions to student performance have been identified.
Lister, R.F., Beglund, A., Clear, T., Bergin, J., Garvin-Doxas, K., Hanks, B., Hitchner, L., Luxton-Reilly, A., Sanders, K., Schulte, C. & Whalley, J. 2006, 'Research perspectives on the objects-early debate', ACM SIGCSE, vol. 38, no. 4, pp. 146-165.
View/Download from: UTSePress
Sanders, K., Fincher, S., Bouvier, D.J., Lewandowski, G., Morrison, B., Murphy, L., Petre, M., Richards, B., Tenenberg, J., Thomas, L., Anderson, R., Anderson, R., Fitzgerald, S., Gutschow, A., Haller, S., Lister, R.F., McCauley, R., McTaggart, J., Prasad, C., Scott, T., Shinners-Kennedy, D., Westbrook, S. & Zander, C. 2005, 'A multi-institutional, multinational study of programming concepts using card sort data', Expert Systems, vol. 22, no. 3, pp. 121-128.
View/Download from: UTSePress | Publisher's site
This paper presents a case study of the use of a repeated single-criterion card sort with an unusually large, diverse participant group. The study, whose goal was to elicit novice programmers' knowledge of programming concepts, involved over 20 researche
Lister, R.F. 2005, 'Grand Challenges', ACM SIGCSE Bulletin Inroads, vol. 37, no. 2, pp. 14-15.
Lister, R.F. 2005, 'Mixed methods: positivists are from Mars, constructivists are from Venus', ACM SIGCSE Bulletin Inroads, vol. 37, no. 4, pp. 18-19.
Lister, R.F. 2005, 'One Small Step Toward a Culture of Peer Review and Multi-Institutional Sharing of Educational Resources: A Multiple Choice Exam for First Semester Programming Students', Australin computer science communications, vol. 27, no. 5, pp. 155-164.
View/Download from: UTSePress
computer education research; programming; multiple choice; peer review; learning repository; multi-institutional
Hazzan, O., Impagliazzo, J., Lister, R.F. & Schocken, S. 2005, 'Using History of Computing to Address Problems and Opportunities in Computer Science Education', SIGCSE Bulletin Inroads, vol. 37, no. 1, pp. 126-127.
de Raadt, M., Hamilton, M., Lister, R.F., Tutty, J., Baker, B., Box, I., Cutts, Q.I., Fincher, S., Hamer, J., Haden, P., Petre, M., Robins, A., Sutton, K. & Tolhurst, D. 2005, 'Approaches to learning in computer programming students and their effect on success', Research and Development in Higher Education Series, vol. 28, pp. 407-414.
View/Download from: UTSePress
Lister, R.F., Adams, E., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Mostrom, J., Sanders, K., Seppala, O., Simon, B. & Thomas, L. 2004, 'A Multi-National Study of Reading and Tracing Skills in Novice Programmers', SIGCSE Bulletin Inroads, vol. 36, no. 4, pp. 119-150.
View/Download from: UTSePress | Publisher's site