National Centre for Research Methods

Comparative judgement methods are used by social scientists to construct reliable measurement scales that are difficult to construct using traditional methods. Such scales typically measure important but nebulous theoretical ideas such as ‘beauty’ or ‘writing quality’. Comparative judgement involves no coding schemes and no categorisation; instead judges make holistic decisions about which of two presented objects better represents the theoretical idea of interest. Comparative judgement methods have a long history but are only now gaining traction due to technological developments making them more accessible to researchers. The course will take attendees through the entire process of using comparative judgement to construct reliable measurement scales.

The course covers:

• The background and theory of comparative judgement
• How to choose suitable objects and judges
• How to design and run a comparative judgement study online
• How to process and interpret study outcomes
• How to evaluate reliability and validity

By the end of the course participants will:

• Know how to run a comparative judgement study using the free NoMoreMarking website
• Know how to process and interpret comparative judgement outcomes

IMPORTANT: Please note that this course includes computer workshops. Before registering please check that you will be able to access the software noted below. Please bear in mind minimum system requirements to run software and administration restrictions imposed by your institution or employer with may block the installation of software.

• Access to the website NoMoreMarking.com via a computer browser.
• I will demonstrate and provide code for processing data using R. Installing R prior to the online workshop is helpful but not essential.

Optional reading: examples of studies that used comparative judgement:

Jones, I., Bisson, M., Gilmore, C., & Inglis, M. (2019). Measuring conceptual understanding in randomised controlled trials: Can comparative judgement help? British Educational Research Journal, 45(3), 662–680. https://doi.org/10.1002/berj.3519

Rycroft-Smith, L., & Stylianides, A. J. (2022). What makes a good educational research summary? A comparative judgement study of mathematics teachers’ and mathematics education researchers’ views. Review of Education, 10(1), e3338.

Optional reading of an online how-to guide for comparative judgement:

https://www.notion.so/nmm/No-More-Marking-for-researchers-70cb4eec46d547cd91c65ff2066d415f

Course Times:

Day 1 (08.09.22): 10am to 1pm

Day 2 (09.09.22): 10am to 1pm

While ethical issues are faced by all social researchers the ethical issues posed by some digital platforms, digital spaces and mediated interactions can raise distinct ethical challenges (and opportunities).

This one-day course will consider what it means to ethically collect data and to work with potentially sensitive data when using digital research tools and when researching digital spaces and objects.  It will introduce researchers to principles underpinning ethical practice in digital contexts and will explore the complexities of managing issues of informed consent, anonymity and confidentiality in and through digitally mediated environments.

Course Timings: 11:00 – 16:00
This course is suitable for researchers at any career stage who wish to gain an understanding of ethics in digital research.

Many students in the social sciences, especially in political science, public policy and public administration who decide to undertake qualitative or interpretive research feel they are unqualified to do so. They express deep-seated confusion about the reliability and generalizability of data, results, and conclusions. They feel that interpretive approaches lack the type of specialised training that has become commonplace in quantitative political science. The aim of this course is to redress this gap. We will equip students with a toolkit that will enable them to both conceptualise and execute an interpretive project. 

The course covers:

• Situating the interpretive approach in relation to other ways of doing political science research by reference to the philosophical, epistemological, and methodological assumptions on which these approaches are based;
• The theoretical and analytical tools students need to design and conduct their research project;
• The toolkit of methods used by interpretive scholars to collect data, including ethnographic and interview-based methods;
• The standards that will both ensure results are reliable and maximise the impact of findings; and
• Guidance on the norms and principles used to analyse data in an interpretive project.
• An introduction to comparative interpretive research

By the end of the course participants will:

• Be able to describe the strengths and features of the interpretive approach   
• Be able to develop and justify a sophisticated design for interpretive research
• Have experience interpreting rich qualitative data

This one-day online, interactive course will provide a practical introduction to qualitative longitudinal enquiry. The morning session will explore key design features of this methodology, including how to build time into a study, how to sample through time, how to generate temporal data, the ethics of longitudinal enquiry, and the potential to create real-time impact in policy processes. The afternoon session will focus on the intricate nature of QL analysis. The course will comprise two lectures and two interactive workshops (see below for further details). The course will be delivered by Bren Neale, a specialist in QL research and the author of two books on this methodology.

This half day short course will introduce delegates to the opportunities for qualitative secondary analysis (QSA) offered by the Irish Qualitative Data Archive and the Digital Repository of Ireland.  It will focus on two themes:

• Opportunities for the creation of cross-national data assemblages for QSA
• Reflection on some challenges associated with sharing and re-using potentially traumatising data and Facebook data.

The course will spotlight the innovative Archiving Reproductive Health collection.  Funded by the Wellcome Trust, this uniquely broad collection focuses on the 2015 referendum to liberalise access to abortion in Ireland. The collection includes interviews with activists and doctors. Organisational data has been donated by some of the main groups campaigning for increased access to abortion, including a wide collection of pro-choice artists work. Other collections include a database of tweets from the referendum period, a collection of posts from the InHerShoes Facebook page and a collection of images of posters and streets in the time period leading up to the referendum. The addition of interviews conducted in twenty years leading up to the referendum and in the years after the referendum add to the longitudinal perspective of this social movement for change.

In this one day course we will explore how to use R to import, manage and process spatial data. We will also cover the process of making choropleth maps, as well as some basic spatial analysis. Finally, we will cover the use of loops to make multiple maps quickly and easily, one of the major benefits of using a scripting language to make maps, rather than traditional graphic point-and-click interface.

The course covers:

• Using R to import, manage and process spatial data
• Design and creation of choropleth maps
• Basic spatial analysis
• Working with loops in R to create multiple maps

 By the end of the course participants will:

• Use R to read in CSV data & spatial data
• Know how to plot spatial data using R
• Join spatial data to attribute data
• Customize colour and classification methods
• Understand how to use loops to make multiple maps
• Know how to reproject spatial data
• Be able to perform point in polygon operations
• Know how to write shapefiles

This course is ideal for anyone who wishes to use spatial data in their role. This includes government & other public sector researchers who have data with some spatial information (e.g. address, postcode, etc.) which they wish to show on a map. This course is also suitable for those who wish to have an overview of what spatial data can be used for. Although no previous experience of spatial data is required it would be beneficial (eg Google Maps).

Latent variable models relate a set of observed (or manifest) variables to a set of latent (or unmeasured) variables. In practice, this entails the use of models to summarise the relationship between a series of highly associated variables. It will be demonstrated that these models are specific examples of a wider family of measurement models.

This one-day course will showcase latent variable models through the application of these models to social science data. Particular emphasis will be given to estimation and interpretation of latent class models in Stata.
It is most suited to empirical social science researchers with knowledge of statistical data analysis methods.
Course will be delivered on-line from 10:00 – 16:00

Participants will need to use their own computers and have Stata installed. Familiarity with Stata is helpful but not essential.

Target audience: Postgraduate students and researchers who have a background in statistically-orientated social science data analysis.

•    Introduction to latent variable models
•    Comparison of factor analysis, latent trait analysis and latent class analysis
•    Estimation and interpretation of latent class models using Stata
•    Practical exercise: Estimation of latent variable models in Stata

Participants will:
•    Gain a clearer understanding of the concept of latent variable models
•    Be able to select an appropriate latent variable model
•    Estimate and interpret latent class models using Stata
•    Be aware of potential issues during model estimation




Across the social sciences there is widespread agreement that longitudinal data (e.g. studies that repeatedly contact individuals) provides powerful research resources to examine both social change and social stability. There is now a broad portfolio of longitudinal data available to social science researchers. Many social science research questions can be adequately answered without longitudinal data; however, most research projects will benefit from the addition of longitudinal data analysis, and some research questions can only feasibly be answered using longitudinal data.

This is a two day workshop on longitudinal data and research using statistical methods.

The workshop is specifically designed for social scientists, and social science data and examples will be showcased throughout the workshop. The workshop will focus on the research value of longitudinal data and explore sources of longitudinal data. Participants will be introduced to the analysis of repeated cross-sectional data, duration models and models for panel data. The emphasis will be on interpreting outputs (e.g. from data analysis software packages) and understanding results (e.g. in published papers).

The event is intended to be engaging and informative and there will be audience participation and practical demonstrations.

This is not a practical workshop and it does not provide training in the use of data analysis software. It will however provide a strong theoretical foundation for future engagement at practical workshops that are designed to provide hands-on training in data analysis.

A high level of mathematical ability is not required, but participants should ideally have undertaken an introductory statistics and data analysis course (e.g. a semester long module as part of a Masters degree) or have attended the one day workshop on Statistical Modelling (held the day before).

Researchers who are at any career-stage are welcome.

A high level of mathematical ability is not required, but participants should ideally have undertaken an introductory statistics and data analysis course (e.g. a semester long module as part of a Masters degree) or have attended the one day workshop on Statistical Modelling (held the day before).

Measuring behaviour is a non-trivial activity. The techniques that are required to measure behaviour accurately and reliably are frequently glossed over in research methods training courses.

This workshop will focus on the methods and wider considerations associated with systematically observing and recording behaviour. Many of the principles and techniques apply to non-human animals as well as humans.

The workshop is orientated to researchers working in fields such as psychology, anthropology and human sciences.

Course Timings: 11:00 – 16:00
Researchers at any career stage are welcome.

The workshop is orientated to researchers working in field such as psychology, anthropology and human sciences, but will showcase techniques that are also relevant to collecting non-human data.

This course will focus on the conceptual understanding of multilevel modelling and its relevance for robust analysis of evidence from randomised controlled trials, with case studies from educational and clinical trials. It will focus on ‘meaning’ and application of multilevel models instead of computations. The course will run for three days with the first day focusing on the transition from linear regression models to multilevel models. Practical examples with simple exercises will be used to motivate the need for a more robust approach than t-tests or linear regressions in randomised controlled trials. The different sources of variability will be discussed as well as their implications on effect size. The course will primarily be taught in R, but we would also be able to support individual exercises in STATA. This is an intermediate course that requires good understanding of the linear regression model as a prerequisite.

The course covers:

• Simple and multiple linear regression
• Overview of multisite and cluster randomised controlled trials
• Hierarchical and correlated data structures
• Random intercepts models
• Random site by intervention models
• Multilevel models for longitudinal data

By the end of the course participants will:

• Gain practical skills in converting data to long form
• Make a link between study design and analytical choice
• Gain practical skills in applying multilevel models and interpreting results
• Acquire necessary skills to check robustness of results from educational or clinical trials

This course is suitable for postgraduate students, researchers, trial statisticians and methodologists and participants should have a basic understanding of statistical methods including the linear regression model and analysis of variance. Participants require access to computer with R enabled software. It is recommended to use RStudio for coding and running R.

Uncertainty in research design and practice is having a continuing impact on researchers’ methodological considerations.

This half day course explores uncertainty, anticipation, ignorance and surprise, and offer practical support in relation to design and methodologies. 

PAR aims to create a space for researcher and participants to co-produce knowledge and where relevant, action for change. PAR is considered as a research paradigm in itself, that embodies a particular set of concepts under which researchers operate (Minkler and Wallerstein 2008). These include respect for diversity, community strengths, reflection of cultural identities, power-sharing, and co-learning (Minkler 2000). In this session we will explore these principles, the cyclical approach to PAR and what this means in practice. Participants will be given the opportunity to learn terminology, understand participation in community engaged research, explore how power and positionality can change health outcomes in PAR, and learn about a variety of participatory methods and how they have been applied in different contexts, globally and within the UK. Participants will also be provided with the space to explore challenges they are facing in designing or implementing community engaged collaborative research within a discussion clinic forum. 

Ethical research is better quality research. This online course (one day, taught over two mornings) is designed to raise your awareness of why and how you need to think and act ethically in practice throughout your research work. The current system of ethical review by committee can lead to the misleading sense of having ‘done ethics’. This course shows you how to conduct research which is truly ethical. It also provides the opportunity for discussion of your own ethical dilemmas, if you wish.

The course covers:

• Research ethics in context – ethical breaches past and present, ethics dumping, ethics activism
• Potential ethical pitfalls at each stage of the research process, from question setting to aftercare
• How to think and act ethically throughout research

By the end of the course participants will:

• Recognise the importance of context for ethical decision-making
• Understand why they need to think and act ethically throughout research work
• Be clearer about potential ethical pitfalls at different stages of the research process
• Know how to approach ethical thought and action at any point in their research

This course is aimed at Doctoral students, early career researchers (any discipline). Also practice-based/applied researchers. Possibly government researchers too, and independent researchers.

Randomised controlled trials (RCTs) are heralded as the gold standard of research design in the social sciences. RCT principles are used in research at all levels of complexity from evaluating national social policies to experimenting with the impact of website designs (there often known as A/B testing). This course is for social researchers who have a firm grasp of the foundations of quantitative research methods (e.g., linear regression and confidence intervals) and would like to learn how to design and analyse randomised controlled trials. The course incorporates a blend of presentations and participatory sessions, using examples from the social sciences.

Dr Andi Fugard (they/them) is a Research Director in NatCen Evaluation. They have experience designing, project managing, and analysing data from randomised controlled trials in mental health and education. Before joining NatCen, Andi was a Senior Lecturer in Social Science Research Methods at Birkbeck, University of London, where they directed postgraduate programmes in Social Research, and Lecturer in Educational Psychology Research Methods at University College London.

Ludwig Wittgenstein is widely considered one of the greatest and most influential philosophers of the 20th century who in his later work produced a radical and distinctive approach to philosophical analysis, which had far-reaching ramifications for research in the social and human sciences. What does Wittgenstein and Witgensteinian philosophy have to say to Social Scientists and Psychologists in the 21st century?

Wittgenstein’s impact on the social sciences can be divided into a number of phases, some of which (2 & 3) spawned distinctive research programmes:

1. Via reception of the work of the Wittgensteinian philosopher Peter Winch and his influential book The Idea of a Social Science and its Relation to Philosophy, published in the mid-20th Century.

2. The combining of Wittgensteinian insights with Ethnomethodology and Conversation Analysis (EMCA) beginning in the early 1970s and found in the work of the ‘Manchester School’ of Ethnomethodology and Conversation Analysis in the work of authors such as Wes Sharrock, Rod Watson and Bob Anderson in addition to associated figures such as Mike Lynch (Cornell) and Jeff Coulter (Boston).

3. The combining of Wittgensteinian insights with Michael Billig’s Rhetorical Analysis, Ethnomethodology and Conversation Analysis to form Discursive Psychology, initiated by Johnathan Potter, Derek Edwards and Margaret Wetherell, beginning in the 1980s at Loughborough University and being developed by others in the decades that have followed.

Run across two afternoons, this course will revisit the philosophical sources that inform a Wittgensteinian approach to questions in the social and human sciences and explore these alongside the approach to interaction found in Ethnomethodology.

The social world is complex and messy. Statistical models provide a formal approach to evaluate data, test ideas and investigate research questions.

This is a one-day Stata workshop on statistical models for social science data analysis.

The workshop will concentrate on models within the generalized linear modelling framework. It will cover linear regression, and models for binary, categorical, ordered categorical and count data. The focus of the workshop will be on social science applications, and social science data and research questions will be showcased throughout. The emphasis will be on interpreting outputs (e.g. from data analysis software packages) and understanding published results.

•    Be able to interpret statistical modelling results
•    Understand linear regression models
•    Understand models for binary, categorical, ordered categorical and count data
•    Understand the model building process (e.g. selecting variables, goodness of fit and model criticism)
•    Be aware of some statistical data analysis software solutions

The social world is complex and messy. Statistical models provide a formal approach to evaluate data, test ideas and investigate research questions.

This is a one day workshop on statistical models for social science data analysis. It will introduce the underlying concepts associated with multivariate analysis using statistical models. The workshop will concentrate on models within the generalized linear modelling framework. It will cover linear regression, and models for binary, categorical, ordered categorical and count data. The focus of the workshop will on social science applications, and social science data and research questions will be showcased throughout. The emphasis will be on interpreting outputs (e.g. from data analysis software packages) and understanding published results.

The event is intended to be engaging and informative and there will be audience participation and practical demonstrations.

This is not a practical workshop and it does not provide training in the use of data analysis software. It will however provide a strong theoretical foundation for future engagement at practical workshops that are designed to provide hands-on training in data analysis.

A high level of mathematical ability is not required, but participants should ideally have undertaken an introductory statistics and data analysis course (e.g. a semester long module as part of a Masters degree).

Researchers who are at any career-stage are welcome.

A high level of mathematical ability is not required, but participants should ideally have undertaken an introductory statistics and data analysis course (e.g. a semester long module as part of a Masters degree).

We are pleased to announce that registrations are now open for the Timescapes 10 Festival, a major celebration of progress and advance in qualitative longitudinal methods.

Jointly run through the Timescapes Archive and the National Centre for Research Methods, the Timescapes 10 Festival celebrates ten years since the conclusion of the original Timescapes Programme of research.  

A key aim of Timescapes was to facilitate advances in Qualitative Longitudinal (QL) research, archiving and the re-use of QL data. Through a mixture of international symposia, panel sessions, video provocations, sandpits, and demonstrator events, this online only festival showcases the huge amount of progress in these methodological fields, as well as international advances in: 

• Qualitative Longitudinal research 
• Qualitative Secondary Analysis 
• Qualitative archiving 
• Decolonising Archives 
• Qualitative data in panel surveys 
• Auto/biographical methods 
• Big Qual 
• Mixed Methods 
• Ethics 
• Data Integrity and Data Management 
• Time and temporality, temporal methodologies.  

Participants on this one-day training course will learn how to conduct research aimed at improving accountability for climate change.  The course will show participants how to generate data that is focussed on key actors in a particular geographic/industrial/economic context, and build a case study that they can use in their research project, or in a stand alone project.

The course will teach participants to navigate specialist secondary sources and apply investigative methods to produce unique analysis of responsible actors, the role they play in climate change and the benefits they gain from environmentally harmful practices.

Join us for an exciting and flexible online course that will develop your knowledge and skills for working with qualitative data at scale: ‘big qual’.  Discover cutting-edge analytic methods with applications across archival research, social media research, secondary analytic practice and other research arenas in which large amounts of diverse qualitative data, including longitudinal, are brought together and analysed.

This course amounts to around 7 hours study over a 3-day period.  It is tailored so that you can dip-in-and-out of resources and activities, self-directing your study to fit in with your work and life commitments.  Live online sessions are also scheduled once a day with the course tutors for tailored support.

Gain guided experience of the unique four-step Breadth and Depth Method to enable you to combine extensive coverage with intensive illumination, moving between the span of big qual analysis and the detail of qualitative engagement. 

In this course you will:

• gain hands-on experience of accessing, searching, obtaining and organising large amounts of qualitative data;
• practice identifying and exploring datasets using contextual metadata;
• learn how to undertake preliminary exploration across the breadth of big qual using basic text analysis software;
• acquire an understanding of using analytic software to identify points of interest and select cases to follow up;
• learn about and undertake in-depth analysis of selected cases to deal with context, complexity and detail; and
• understand the iterative relationship between extensive and intensive explorations enabled by the Breadth-and-Depth Method.

The course is suitable for early career as well as more experienced researchers.  The course is accessible for researchers with quantitative, as well as those with qualitative skills, but you will need to be familiar with the purpose and process of qualitative research.