Contextualising Youth Justice Interventions: Making the Case for Realist Synthesis

Abstract

This article examines the problematic reductionism and decontextualising nature of hegemonic youth justice intervention evaluation and offers a way ahead for a realistic, context-sensitive approach to intervention evaluation in the youth justice field. It opens by considering how the development of risk-based youth justice interventions in England and Wales flowed from and fed into the modernisation and resultant partiality of the ‘evidence-base’, which shaped youth justice practice. It then moves to a critical review of the emergence and continued influence of risk-based interventions and the ‘What Works’ intervention evaluation framework in youth justice. In the closing discussion, this article envisages the potential of taking a realist approach to the evaluation of youth justice interventions to mitigate the limitations of current approaches to intervention selection and the evaluation of their ‘effectiveness’.

1. Introduction

The ‘What Works’ framework for evaluating intervention ‘effectiveness’ has dominated youth justice prevention practice in both the US and UK since the 1990s. It is a framework synergistic with the evidence-based ‘Risk Factor Prevention Paradigm’, which prioritises interventions targeting criminogenic ‘risk factors’ through experimental methods. However, the risk-led, preventative interventions prioritised by the ‘What Works’ evaluation framework are increasingly the subject of critique on the grounds that they privilege a narrow range of ‘evidence’ generated through rigid, experimental measures of ‘quality’, often based on explicit randomisation into control and treatment groups. We explore these ideas in this paper, starting from the premise that such approaches fail to appreciate how context influences the way interventions operate and are, therefore, premised on a limited understanding of causality. These critiques have been particularly vociferous from those advocating realist approaches to analysis [1] and evaluation [2,3,4]. Furthermore, whilst the realist approach to evaluation has gained traction in other fields such as crime reduction [5] and healthcare [6], realist approaches have had limited impact on understanding the evidence base for preventative interventions in youth justice.

Accordingly, this article examines the reductionist and decontextualized understanding that emerges from the ‘What Works’ framework for evaluating the effectiveness of youth justice interventions and offers a way ahead for a realistic, context-sensitive approach to intervention evaluation, which would facilitate a more comprehensive evidence base to better support youth justice practice. We begin with an overview of the development of youth justice interventions in England and Wales, considering how these developments flowed from, and fed into, the modernisation and partiality of the ‘evidence-base’ shaping practice in the Youth Justice System (YJS) of England and Wales. The Youth Justice System was established in England and Wales following the 1998 Crime and Disorder Act, with the primary aim of preventing offending and reoffending amongst children and young people aged 10–17. The Act created Youth Offending Teams (YOTs) to work with children entering the YJS, multi-agency teams comprised of representatives from statutory agencies (police, health, probation, local authorities), along with staff from other relevant services in Local Authority areas. The practice of YOTs was monitored, managed and funded by the Youth Justice Board, a non-departmental public body created to advise government on youth justice policy to support YOTs with implementation [7]. What follows is a critical review of the emergence and continued influence of risk-based interventions alongside the ‘What Works’ intervention evaluation framework that have shaped evidence-based, ‘effective’ practice. Finally, we envisage the potential of taking a realist approach to the evaluation of youth justice interventions, which could mitigate the decontextualization embedded in current approaches to intervention selection and evaluation of their ‘effectiveness’.

2. The Evolution of Intervention in Youth Justice

The use of formal means of intervention with ‘delinquent’ children and those engaging in criminal activities became engrained in youth justice practice in the United Kingdom (UK) following the 1933 Children and Young Persons Act. Despite its welfarist undertones [7], the Act was undeniably both correctionalist and rehabilitative. This was evident in the strategic themes and foci for intervention that became extended in the 1969 Children and Young Person’s Act, which promoted the use of treatment, care proceedings and care orders as alternatives to custody and criminal proceedings [8]. Correctionalist and rehabilitative ideals continued to dominate wider discussions of penology and punishment up until the 1980s [9]. However, rapidly rising crime rates increased disillusionment with state efforts to try and rehabilitate offenders when such strategies began to fall out of favour primarily with the middle classes [10]. The genesis of this disillusionment came from the findings of a review of rehabilitative intervention programmes [11], which provided a summary of ‘treatments’ for offenders, drawn from all reports published in English from 1945–1967 [12]. Martinson’s review concluded that there was a lack of evidence demonstrating the effectiveness of rehabilitative interventions—leading to his infamous assertion that ‘nothing works’ in respect of achieving recidivism for adults, or children. However, Martinson’s assertion has been often misrepresented, because this assertion was posed as a question, rather than a statement:—‘Do all of the studies lead us irrevocably to the conclusion that nothing works?’ [11] p. 48. Notwithstanding this misrepresentation, the collapse of the rehabilitative ideal certainly reached its denouement during the 1970s, with Martinson’s article serving as a timely reflection of an already notable decline in confidence in rehabilitation:

‘virtually everyone came to forsake the rehabilitative ideal. Through an intersection of cataclysmic social events and strong intellectual currents, a social movement of sorts had risen up to purge systems of control of the insidious social welfare ideology that used the rhetoric of benevolence to mask the abuse of power. Martinson’s [9] study did not spark such thinking; its publication served only to confirm what we already knew’ [13], p. 302.

The decline of the rehabilitative ideal in criminal justice around the 1970s was matched by a similar period of despondency regarding the efficacy of interventions in youth justice [14]. Both the ‘hard’ punitive intervention approaches, popular with the Thatcher and Major Conservative governments in the UK, and the ‘soft’ needs-led welfarist approaches to intervention, were deemed to be ineffective in reducing recidivism amongst children who had offended; a feeling that was compounded by enduring concerns (often moral panics) about perceived increases in the extent and severity of youth crime and government impotence in addressing the problem [15,16]. The subsequent diversionary focus of UK youth justice policy and practice during the 1980s—concurrently a ‘decade of diversion’ and a ‘successful revolution’ in youth justice [17] —demonstrated a move away from formal intervention as the central means of addressing youth crime. Whilst this reduced the number of children being drawn into the formal system [9], the approach did little to resolve the perennial public concerns about rising youth crime, which were perpetuated by political rhetoric and agendas, as well as by media misrepresentations [16,18].

Rapidly changing socio-economic and socio-political contexts during the 1980s and early 1990s exerted particularly significant pressure for change, both in understandings of youth criminality and in formal youth justice responses to it. As globalisation precipitated sweeping socio-economic changes across westernised, anglophone societies, this created increasing uncertainty, anxiety and concerns about the associated ‘risk’ and ‘threats’ to the safety of populations [16,18,19]. The need to predict, control and manage these perceived risks and threats [20], which were readily and easily ascribed to younger, ostensibly criminal populations, became a key focus of societal responses to globalised pressures. This began to characterise the trajectory and evolution of youth justice; arguably seen as a devolution in terms of explanatory reductionism (as discussed in later sections). Neo-liberal ideas also began to influence and shape the social policy of westernised/anglophone societies from the late 1970s into the 1980s [21]. In respect of youth justice, neoliberalism incorporated both existing concerns about crime contexts and a contemporary focus on prescribing practice, which constrained freedom of choice and held individuals, families and communities responsible for the existence of crime [22,23]. Strategies of responsibilisation, managerialism and ‘evidence-based’ policy and practice emerged from the neoliberalist influence on youth justice.

Indeed, since around the last quarter of the twentieth century, a broad political consensus has shaped discussions of crime and criminal justice, particularly in states characterised by Pemberton [24] p. 65 as belonging to a ’neo-liberal harm reduction regime’ where punitive policies and tough rhetoric are the foci. Where criminal justice is concerned, countries such as the UK and US (amongst others) have pursued distinctly neo-liberal and thus less welfarist approaches to criminal justice. Neoliberal criminal justice policies are seen to be ‘heavily authoritarian in nature’ with punishment characterised by exclusionary methods, often including the reliance on high rates of imprisonment in privately owned facilities as a mechanism for incapacitation [24] p. 62. This neoliberal influence resulted in a waning focus on the social contexts of crime and an increased focus on individual responsibility and accountability in youth justice. In particular, there was a focus on the responsibilisation and management of individual children who had committed crime, or who were deemed at risk of committing crime, due to the measurement of psychosocial risk factors. In light of this focus, a more prescriptive approach to the “rehabilitation” of children emerged, which implied that crime-producing deficits should be corrected by the justice system to inhibit future criminal involvement [13].

It was in the 1990s, however, that ideas associated with neo-liberalism began to firmly shape social and criminal justice policy, which was concerned less with issues of crime prevention (crime rates had been falling for years) and concentrated instead on concerns with public disorder and the increasingly pervasive labelling of so-called ‘problematic’ populations, such as children [7]. In youth justice, neo-liberal strategies began to gain traction, marginalising the social contexts of crime and privileging explanations focused on assigning responsibility for crime to individuals, families and communities [23,25]—in other words, ‘responsibilising’ these groups for crime (i.e., holding them primarily responsible), regardless of age or capacity. Dubbed the ‘punitive turn’ in youth justice [19], these developments were oriented to the management of the risks and threats presented to the public by so-called problematic and dangerous children [16]. The responsibilising and punitive nature of neo-liberal strategies was manifested in the expansion of youth justice apparatus and infrastructure, notably: vertical expansion—burgeoning prison populations and use of incapacitation; and horizontal expansion—the proliferation and diversification of (cost-effective) techniques of management, regulation, control and surveillance [7,26]. The increasing application of punitive strategies in the YJS resulted in the creation of restrictive community sentences such as Intensive Supervision and Support Programme (ISSP) for ‘high risk’ children [27]. This encompassed ideas of ‘vertical expansion’ (e.g., increasing use of custody for children) and ‘horizontal expansion’ (e.g., the proliferation and diversification of technologies to regulate, control and scrutinise children) [26]. Accordingly, these globalised, neo-liberal and punitive pressures catalysed the modernisation of strategies and techniques of youth justice, along with the methods and practices employed for generating evidence to shape explanations of and responses to youth offending [28,29].

3. Modernising Youth Justice Intervention: Managing ‘Effectiveness’

When the New Labour Government came to power in 1997 in the UK, the prevention of youth crime became the primary aim for the Youth Justice System (YJS) created in the Crime and Disorder Act 1998. The Act determined that children aged 10–17 could become subject to assessment, monitoring and intervention activities if they were convicted of a crime or deemed to be at risk of involvement in crime.

A broad approach was initially taken in respect of intervention in the YJS, because at this point the YJB ‘decided at that point it wanted to let a thousand flowers bloom’ [30] p. 25, having had insufficient time to consider what they wanted from intervention projects and what evaluation arrangements might be appropriate. For example, there was an initial emphasis on restorative justice-based interventions, stemming from the recommendations of the Misspent Youth report [31] (which predated the neoliberalist influence) and the provisions of the Crime and Disorder Act, although these activities tended to vary in nature and quality. Therefore, whilst risk-led interventions were not the only focus of preventative activities in the early days of the YJS, they later became the consistent approach to intervention in youth justice practice in England and Wales [32], underpinned by their strong recommendations in the Audit Commission’s Misspent Youth report [31]. Their modernising, ‘evidence-based’, responsibilising/individualising nature aligned well with the emerging neo-liberal influences on youth justice.

The neo-liberal strategy of managerialism was adopted as the key mechanism through which this preventative aim was to be pursued. Managerial approaches entailed the control, prescription, restriction and regulation of children who entered the YJS and also the practice (e.g., assessment, supervision, planning, interventions) of the youth justice professionals, organisations and systems working with them. Structurally, the Youth Justice Board (YJB) for England and Wales was established to monitor and manage the newly created formalised YJS. Their predominant focus was the work of (similarly newly created) Youth Offending Teams (YOTs)—multi-agency teams in every local authority area, containing professionals from statutory agencies (i.e., police, probation, health, local authority—typically social work departments) and other organisations, where appropriate (e.g., third sector, charities). At that time, the YJB also monitored and managed the practice of the ‘Secure Estate’ of custodial institutions for children in England and Wales—secure children’s homes, secure training centres and secure young offender institutions. The Secure Estate is now managed by the Youth Custody Service, but practice continues to adhere to YJB guidance and standards. These YOTs were tasked with preventing offending and reoffending by children who had entered the YJS. The YJB managed and evaluated the practice of YOTs and their staff (against the prevention aim) through a series of technical mechanisms. This included routine data collection relating to ‘Key Performance Indicators’ (e.g., reductions in first-time entry into the YJS, reoffending, custody) and inspection against prescribed ‘national standards’ for practice, as well as ‘case management guidance’ for working with children at different stages of the YJS [33]. The work of youth justice professionals was underpinned by an adherence to ‘Key Elements of Effective Practice’ (KEEPs)—a series of guidance documents outlining ‘the essential elements of practice with all children at all stages of the YJS’ [34] p. 6. Crucially, for practice interventions (the central vehicle for preventing offending) to be prescribed and recommended as ‘effective’, they had to be grounded in ‘evidence-based practice’ (EBP). In this sense, EBP was a modernising criminal justice approach emphasising the need for all practice to be accountable, transparent and defensible through linkages to an established evidence base [35]. The application of EBP in youth justice constituted a guide for practice and resource allocation that signified a move away from the purportedly overly discretionary, less consistent, uncoordinated and expensive systemic responses of the past [20,30,34], as noted by Stephenson et al. [32] p. 10:

‘The conscientious, explicit and judicious use of current best evidence in making decisions regarding the prevention of offending by individual young people based on skills which allow the evaluation of both personal experience and external evidence in a systematic and objective manner’

Therefore, EBP was seen by some as a tool of neo-liberal managerialism by central government to control, regulate and prescribe the interpretation and implementation of national youth justice policy in localised (evidence-based) practice. This was facilitated by regular monitoring processes, data collection and non-negotiable adherence to (regularly evaluated) performance indicators, guidelines, checklists and prescribed procedures [36,37,38]. Government commitment to EBP as a vehicle for managing children (and staff) in England and Wales was demonstrated in their collection, dissemination and utilisation of social scientific research ‘evidence’ allegedly illustrating the ‘effectiveness’ of youth crime prevention interventions [39]. Managerialist strategies began to heavily influence the nature of the growing ‘evidence-base’ on which youth justice practice and preventative interventions were established and developed, privileging quasi-positivist evidence that focused on individualistic/psychosocial ‘causes’ of crime, which aligned well with prescriptions for managerialist practices [36], experimental intervention and experimental evaluation frameworks [29,35]. However, this emerging reductionism, illustrated by a psychosocial bias when explaining and responding to youth crime, diverted attention away from examining and targeting broader contextual influences on both youth offending (e.g., causal mechanisms) and the success/failure of preventative interventions, such as those that were potentially socio-structural (e.g., poverty, unemployment, inequalities) and relational (e.g., interactions, systemic processes), that had been widely researched in areas of criminology since the 1980s [40,41]. We return to this point later in the article.

4. Precautionary Prevention: Managing Risk, Not Tackling Causes

A defining feature of neo-liberal approaches to criminal justice and youth justice was the turn away from trying to solve problems by unpacking their underlying causes and their related contexts, towards a ‘precautionary turn’ in the management of risk (conceptualised as a predictor, rather than a cause) through intervention. Precautionary approaches began to underpin youth justice interventions with a fundamental ‘pre-emptive logic’ [42]. Pre-emptive logic catalysed moves towards preventive intervention with children ‘at risk’ of offending, ergo intervening before the event occurs and referred to as ‘pre-crime’ [43]. The precautionary turn in criminal and youth justice corresponded with developing ideas, such as the conditions of Beck’s [20] ’risk society’ and Giddens’ [44] ‘reflexive modernisation’, where behaviours and activities of members of society became less concerned with the accrual of ‘goods’ and more preoccupied with the avoidance of ‘bads’ [45]. In other words, behaviours become increasingly shaped by wanting to avoid victimisation, broadly conceived, whether that be the victim of adverse weather, nuclear fallout, terror attacks or other experiences of crime. Built into this is an entire industry, with media simultaneously heightening the sense of fear and directing us towards those we should fear [46], children at risk of offending, for example, or those already engaging in crime. Contemporary aspects of youth justice can be seen in this context as understandings of causation subverted into a focus on psychosocial ‘risk factors’ functioning as ‘predictors’ of the likelihood of future offending [47]. As a consequence, the design and nature of interventions in youth justice became rooted in the assessment of these ‘risk factors’ as a means to effectively, efficiently and economically manage and ameliorate individual children’s offending predispositions.

The use of ‘risk’ as the primary lens for viewing the children who offended (or who were on the cusp of offending) became redolent across the fabric of youth justice policy making and practice in England and Wales from the early 21st century onwards. Its use also justified the prescription of youth justice interventions, as the identification of groups of children assessed as ‘at risk’ or ‘high risk’ of offending as targets for intervention mobilised a preference for individualised, psychosocial and decontextualised interventions informed by children’s assessed levels of ‘risk’ of reoffending [48]. Notably, it was clear from their outset that the KEEP guidance would be heavily reductionist and skewed towards the risk factor evidence base [16] because of the implicit ‘hierarchy of evidence’ [49] p. 6. For instance, the YJB placed significant pressure on KEEP authors to conduct systematic reviews of evidence that privileged the ‘Campbell Collaboration’ guidelines [50,51,52] when assessing and promoting ‘effective’ assessment and intervention practice. These guidelines favoured the use of interventions highlighted by ‘What Works’ evaluations as preventing crime, which had an overriding emphasis on targeting decontextualised psychosocial risk factors [53,54,55]. Indeed, the predominance of the Risk Factor Prevention Paradigm (RFPP) evidence across the KEEP documents rendered ‘certain research questions… “unaskable” because they cannot be addressed using experimental methods’ [50] p. 219. This negated any potential for the production of inconvenient evidence [56] which was contrary to risk-based understandings of youth offending and the ‘effective’ preventative interventions designed to target the causes identified in these understandings, in accordance with the established ‘What Works’ framework.

5. The Hegemony of ‘What Works’ in Youth Justice Intervention

At the start of the 21st century, driven by government policies of responsiblisation, managerialism and early intervention (Stephenson), the emerging scientific, experimental ‘What Works’ framework provided an ideal vehicle for the risk-led ‘new youth justice’ [57], which had been prioritised in the YJS of England and Wales [31]. Most notably, it provided practice with a reliable and modernising body of demonstrably ‘effective’ evidence-based crime prevention interventions and established what constitutes ‘effective’ intervention. The popularity of the ‘What Works’ approach to evaluation began to gain momentum both in the US and UK, providing hope against the backdrop of the ‘nothing works’ era of rehabilitative despondency. This promised criminology, and in particular youth criminology, ‘robust’ explanations such as those achieved in the natural sciences, therefore, the associated experimental methods became dominant in these disciplines [58]. The ‘What Works’ promise of reducing recidivism through treatment programmes ‘rooted in criminological knowledge’ and evaluated using ‘rigorous science’ [59] p. 1 chimed with the neo-liberal and risk-based thinking, which was influencing the direction of youth justice policy during the late 1990s, most notably due to its genesis in a single, hugely influential study (Box 1):

Box 1. Preventing Crime: What Works, What Doesn’t, What’s Promising.

In 1996, US Congress had commissioned a comprehensive evaluation of the effec-tiveness of crime prevention programmes across the USA entitled ‘Preventing Crime: What Works, What Doesn’t, What’s Promising’ [53]. This evaluation was required to prioritise programmes using ‘rigorous and scientifically recognized standards and methodologies’ and to give special emphasis to ‘factors that relate to juvenile crime … including “risk factors” in the community, schools, and family’. Sherman et al. [53] utilised the ‘Scientific Methods Scale’ (SMS), influenced by Cook and Campbell [60], to assess the methodological quality of individual crime preven-tion evaluations according to three methodological criteria: Control over extraneous (external, unmeasured) variables that might influence the relationship between in-tervention and outcome, measurement error and statistical power to detect the ef-fects of a programme [61]. The SMS consists of five levels of analysis, escalating in rigour and detail, summarised by Farrington [47] p. 14 as:

• Correlation between a prevention program and a measure of crime at one point in time.
• Measures of crime before and after the program, with no comparable control condition.
• Measures of crime before and after the program in experimental and compara-ble control conditions.
• Measures of crime before and after the program in multiple experimental and control units, controlling for other variables that influence crime (e.g., by matching, prediction scores or statistical controls).
• Random assignment of units to program and control conditions.

Level five on this scale represents the ‘Randomised Controlled Trial’ (RCT), the so-called ‘gold standard’ of evaluation methodology, much favoured by the UK Government (Evident in guidance from the Cabinet Office What Works Centres https://www.gov.uk/guidance/what-works-network accessed on 25 May 2021) and YJS. According to Sherman et al. [53], a risk-focused intervention exemplifies ‘What Works’ in youth crime prevention if it is ‘reasonably certain’ that it prevents crime or reduces risk factors in the social context in which it is evaluated. The intervention must also have findings that are generalisable to similar settings in other places and times, in addition to at least two successful evaluations at level three on the SMS or above. Similarly, programmes were heralded as examples of ‘what’s promising’ if the level of certainty regarding reduction of crime or risk factors was too low to support generalisable conclusions, but there was some empirical basis for predicting that further research could support such conclusions, in addition to at least one successful evaluation at level three. Sherman et al. classified ‘what doesn’t work’ as programmes that they were reasonably certain failed to prevent crime or reduce risk factors (or rather, cannot be measured as reducing crime and reducing risk accord-ing to the SMS).

The ‘What Works’ framework for evaluating prevention activities was immediately attractive to politicians because these reviews presented quick, evidence-based, off-the-shelf solutions [62] that prescribed ‘effectiveness’ and facilitated the central premise of the RFPP. As the framework had already demonstrated its effectiveness in the field of health and medicine [32], the implications of the ‘What Works’ endeavour in youth justice appeared clear cut—the impact of different intervention dosage, with different children, could be successfully measured through experimentation, randomised control trials and meta-analysis [14]. The purported benefits of ‘What Works’ for evaluating youth justice intervention ‘evidence’ was their association with improving and informing practice through their examination, categorisation and synthesis of intervention research, but also through commissioning research to fill gaps in knowledge [63]. Furthermore, the ‘gold standard’ mark of quality attributed to the experimental “What Works” body of research was even more compelling in youth justice, because developmental or clinical psychologists often led this research [64] using scientific methods and statistical measurement [65]. The use of ‘strong evidence’ was comprised of using controlled experimental research with comparison groups to demonstrate whether an intervention worked or not [53] pp. 2–17. This prescriptive approach meant that if an intervention had not been evaluated via experimental methods, then it was unlikely to be recommended under the ‘What Works’ category in these reviews.

However, there have also been significant issues and limitations associated with this approach. Not surprisingly, these reviews tend to identify intervention programs that are empirically supported by the research traditions in these fields [64]—namely the RFPP. Indeed, Sherman et al. [53] acknowledged in his pioneering evaluation that congress had specifically required a comprehensive evaluation of effectiveness, which gave special emphasis to “(a) reductions in delinquency, juvenile crime, youth gang activity, youth substance abuse, and other high-risk factors; (b) reductions in the risk factors in the community, schools, and family environments that contribute to juvenile violence; and (c) increases in the protective factors that reduce the likelihood of delinquency and criminal behavior.” Therefore, the ‘What Works’ framework has perpetuated and privileged risk-based preventative interventions. As a consequence, risk-focussed prevention programmes gained widespread popularity with Youth Offending Teams and formed the basis of KEEP assessment and intervention guidance (as previously highlighted). This was the inevitable result of the dominant risk-based focus of the ‘What Works’ framework [53] alongside risk-based explanations of offending influencing preventative responses in youth justice.

6. ‘What Works’: Perpetuating Partiality

The limitations associated with the intervention research ‘evidence’ prioritised by the ‘What Works’ approach to evaluation in youth justice arise from two key issues: The partiality of explanations, which are intrinsically reductionist, and the suitability of methodological approaches taken in this research for use with children in contact with the YJS.

The ‘What Works’ framework is inherently predisposed to the selection of experimental research [48]. It is commonly aligned with risk-based approaches and, arguably, it therefore promotes the design, delivery and subsequent evaluation of reductionist interventions. Furthermore, it could be also seen as an agenda linked closely to modernising notions of ‘evidence-based’ and ‘effective’ practice in youth justice [32], which have become synonymous with the evaluation of largely risk-based methods of preventative intervention (e.g., with the prevention of offending as the main outcome measure). Sherman et al.’s pioneering ‘What Works’ review laments the lack of robust intervention evaluation in youth justice, suggesting tighter control of extraneous variables in future research. However, this recommendation could be seen to equate to the effect of the reduction of important explanatory contextual features due to privileging the minimisation of measurement errors and the use of ‘statistical power to detect meaningful differences’ [53] pp. 2–18. This narrow, restricted, decontextualised approach to the evaluation of youth justice interventions illustrates how psychosocial ‘causes’ have been elevated atop an evidential hierarchy largely because their developmental psychological origins align better with the scientific, experimental standards required by the hegemonic ‘What Work’s’ evaluation framework. These explanations (which are often single factor) fail to explore other potentially criminogenic influences such as criminalisation processes and relationship-related causes of offending. This absence of context in those youth crime prevention studies and interventions prioritised by ‘What Works’ evaluations provides the platform for critique over the ecological validity of experimental research, extending the argument that the ‘What Works’ approach leads to a partial evaluation and assessment of youth justice interventions. For example, small-scale qualitative research, such as practitioner case studies, tend to be overlooked because their methods lie at the bottom of the ‘hierarchy of evidence’, which informs scientific scales, resulting in this type of research scoring poorly in these evaluations. In contrast, priority status is given to research that is devoid of the important contextual elements, which are often only illustrated in qualitative research. Indeed, this is born out in Sherman et al.’s comments on their ‘What’s Unknown’ category:

‘This category includes major variations on program content, social setting, and other conditions which limit the generalisability of even of programs coded as working or not … programs with unknown effects should not be judged deficient … because a program has not been evaluated properly does not mean that it is failing to achieve its goals’ [53], pp. 2–21.

Subsequently, ‘What Works’ Centres, such as the What Works Centre for Crime Reduction, became popular in the UK as they were seen to be synonymous with improving and informing policy and practice across a range of policy areas through their examination, categorisation and synthesis of intervention research and also through commissioning research to fill gaps in knowledge [63]. These were developed because of the popularity of the approach and have come to accept that what they have traditionally deemed robust methods for evaluating the ‘effectiveness’ of youth crime prevention interventions should now be accompanied by a more systematic analysis of what is working, where and why [63]. This method is evident in a more recent effectiveness review entitled ‘What works in managing young people who offend?’ from the Ministry of Justice for England and Wales [66], which considered the management of children involved in crime and the effectiveness of interventions in reducing their reoffending. A systematic Rapid Evidence Assessment (REA) was conducted to evaluate international evidence using the Maryland Scientific Methods Scale, but this was accompanied by an adapted version of the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI) scale to assess the relevance and methodology of mixed-method and qualitative studies, which provide more context and explain how approaches to intervention can offer the best chance of success. However, the authors of the Ministry of Justice study also accept that this approach will lead to the exclusion of some effective programmes:

‘Use of two [Weight of Evidence] approaches was designed to encompass a diverse range of quantitative and qualitative methodologies. However, each essentially assesses the quality of evaluation, rather than the intervention. As such, potentially successful interventions may have been excluded from this review’ [66].

Attempts to avoid decontextualised understandings of intervention effectiveness is also reflected in the YJB’s Theory of Change guidance [67] p. 4, which introduces more of a focus on context alongside the use of ‘robust’ evidence to ‘confidently’ evaluate the effectiveness of an intervention:

‘what makes theory of change complicated or challenging are the social problems we are looking to address and the context in which you are working’

Unfortunately, the guidance tends to operationalise ’context’ as observable static ‘features’ of the situation or population [41], rather than the dynamic (changeable) and relational features that shape the mechanisms through which the intervention works [41]. Consequently, it is evident that context is now being acknowledged as an important consideration in effectively evaluating mechanisms/causality [67] and intervention success [66]; however, this approach is somewhat restricted, because the analytical frameworks for understanding causality and context remain largely reductionist.

Therefore, the influence of decontextualised and reductionist understandings of causation and intervention effectiveness continue to influence the design of preventative interventions by youth justice practitioners, which maintains the reliance on the identification and targeting of psychosocial risk factors [35]. This is seen to be because these interventions are often evaluated as ‘effective’ in ‘What Works’ effectiveness reviews, owing to the fact that they score more highly on the evaluation criteria because of their experimental and quantifiable evidence-base [35,68]. Therefore, it can be argued that (quasi) experimental, risk-based interventions are not necessarily more ‘effective’ than alternative forms of intervention in preventing youth crime—they are merely more attuned to the ranking requirements of the Maryland Scale that underpins the ‘What Works’ framework because of their experimental, ‘scientific’ designs [53,54,69]. This risk-based, experimental bias is evident from Sherman et al.’s [53] comments above about the ‘What’s Unknown’ category and in their acknowledgement of the limitations of the Maryland Scale, conceding the criticism from realists [70] that the scale was too restrictive and inflexible to adequately evaluate research on intervention programmes. On reflection, they concluded that they would have raised the cut off point for the scientific threshold and acknowledged that excluding ‘moderately rigorous studies would waste a great deal of information that could be useful for policymaking’ [53] pp. 2–20. Consequently, the decontextualised and reductionist nature of the ‘What Works’ evaluation framework raises critical concerns over its capacity to provide youth justice policymakers and practitioners with a clear, replicable and valid guide for populating and delivering ‘effective’ preventative interventions [62,63].

Consequently, the research methods prioritised in ‘What Works’ evaluations of preventative interventions in youth justice are a crucial limitation of the approach because it has perpetuated an overreliance on RCT/quasi-experimental methods (thus excluding other sources of evidence) enabling only partial understandings of intervention ‘effectiveness’—with ‘partial’ conceptualised here in the dual sense of being biased towards risk-based interventions, which target individualised, psychosocial influences on youth offending, and incomplete in the sense of neglecting to address broader contextual influences (e.g., socio-structural factors) [35]. The reductionism of the ‘What Works’ lens/framework results in an over-consideration of a limited range of potential causes (e.g., psychosocial bias) and a relative neglect of contextual considerations, both of which artificially reduce the breadth and depth of our understanding of intervention effectiveness in the real world. Individualised evidence, statistical measurement and experimental methods conspire to be reductionist, neglecting the depth and complexity of the contexts and mechanisms associated with the success or failure of preventative interventions [71]. In essence, by reducing and simplifying the approach to evidence generation and evaluation in the youth justice field, the ‘What Works’ approach actively overlooks ‘how context conditions the potential for rehabilitation’ [12] pp. 105–106, which is essential in gaining a comprehensive understanding of both the persistence and desistence of offending and how interventions may change behaviour. Furthermore, as the ‘What Works’ mode of analysis has its origins in positivist research, it seeks to develop empirically grounded theories by identifying patterns amongst variables. Positivists work within a successionist framework—observing an association between independent and dependent variables [72] is sufficient to demonstrate causation if these have been observed within a sufficiently ‘closed’ system such that alternative explanations for the observed associations can be ruled out. However, this model of causations fails to account for why or how the dependent variable(s) impacts on or causes a change in an independent variable. As the successionist model of causation rests only on what is observable, it fails to account for the unobservable generative mechanisms that underpin one event causing another. Ergo, that one event succeeds another is not an explanation for why this happens.

In addition, we question whether experimental methods are suitable in assessing the effectiveness of youth justice interventions. Whilst experimental methods in medical and clinical contexts (e.g., evaluating drug treatment) may be more successful in creating a closed system and thus are better able to isolate the differing effects of dosage between patients (though this claim is subject to debate [73]), the complexities and unpredictability of human behaviour in the real world are not so easily isolated; nor effectively monitored when reduced to arbitrary numbers and assessed by statistical means. The decontexualisation of the aspects involved in youth justice interventions glosses over the dynamicism of human action, particularly in respect of the unpredictability of children’s behaviours:

‘These methods ignore the emergent nature of the social world. People’s agency, the factors that could make a difference are ignored or, at best, turned from rich relational accounts into variables, things to be empirically recorded’ [4], p. 4.

Consequently, youth justice interventions (which are predominantly situated in non-experimental, community contexts) do not sit so well with these methods of evaluation [32,71], and their application can be severely restricted in terms of this ecological validity. Arguably, the ‘What Works’ framework, particularly categories 3–5 and most notably the RCT ‘gold standard’, are largely unsuitable for assessing the effectiveness of youth justice interventions in real-world contexts [68,74,75]. Whilst the ‘What Works’ approach to evaluation has been criticized for similar reasons in other research fields [5,6], the dominance of this approach in youth justice is largely unchallenged [68,76].

7. The Potential of Realist Youth Justice

Notwithstanding the limitations and criticisms of the ‘What Works’ framework, it remains the dominant approach for evaluating the ‘effectiveness’ of youth justice interventions in the industrialised Western world. Moreover, an overriding focus on psychosocial risk-based interventions [77] has sat hand in hand with the results of the ‘What Works’ research-base. The continued influence of the use of risk in youth justice practice assessment has meant that a deficit-focussed approach has been taken in relation to preventing offending amongst children by the YJS. Indicators of vulnerability (physical or emotional) and indicators of serious harm to others form the main body of assessment, resulting in a reduced understanding of a child’s offending [78]. As the RFPP quasi-positivistic understandings of offending and experimental methods have been privileged in youth justice (as previously discussed), simplistic assumptions of cause and effect have dominated understandings and approaches to rehabilitative interventions, whereby offending is seen to be the outcome of risk factors usually identified early in the life course [79]. In criminology (and other fields), increasing attention is being paid to alternative forms of explanation that question this understanding of causation. These developments can be seen under the umbrella of realist social science [1,12,80,81,82]. Broadly speaking, realism starts with a different understanding of causality that acts as a critique of ‘successionist’ versions associated with more positivist methods. For realists, that one event succeeds another is not an explanation for why this happens nor is this evidence that it will always happen (a point also noted by Popper in his famous White Swans analogy [83]). In criminal justice, and youth justice, this can be justified on the grounds that the way in which the dependent variable(s) impacts on or causes a change in an independent variable is not always (if ever) observable, but equally nor is it uniform.

The reductionist and decontextualising limitations of the ‘What Works’ approach could be addressed through the use of realist approaches to the evaluation of youth justice interventions, particularly in generating a broader evidence base for better understanding causal mechanisms and the role of context in youth offending and its prevention. For Wong and colleagues [84], realism starts with a view of reality that cannot be directly measured and, therefore, an alternative way of viewing cause is necessary, which moves beyond the traditional successionist model of causation associated with experimental designs [72]. As successionism examines the relationship between the dependent variable(‘s) impact on the independent variable, the unit of analysis—the variable—is the principle causal agent. Here, variables are the pivot around which research revolves and they are used to indicate the presence, or absence, of the phenomenon in question. A successionist view of causation allows that observing an association between variable X and Y and creating a closed system to rule out alternative explanations is sufficient to establish that X causes Y, thus forming the purest conditions for positivist analysis. Youth justice practice has become heavily influenced by a predisposition for these successionist causal explanations [12] because of the dominance of RFPP understandings of offending. Critics point out that these successionist causal explanations are nothing more than ‘constant conjunctions’, which are not necessarily indicative of how causal powers operate outside of the methods used to discover them [1]. The generative understanding of causality espoused by realists argues that there are underlying generative forces that we need to understand to explain causation, which could contribute to the evidence base used by youth justice practitioners so that they can properly consider the influence of context, thereby expanding understandings of why children offend and how we should respond. When successionist understandings are applied to youth justice, for example, we can see some emerging problems for understanding and responding to youth offending. Although a unified practice management model has promoted a more professional and accountable approach to the delivery of youth justice, evidence does not confirm the logic that preventing offending is linked to reducing measurable levels of risk [48,85]. Generative understandings can help to reveal the influence of other important contextual factors, observable features (space place people things) that trigger or block interventions and context as relational and dynamic features that can shape the mechanisms through which interventions work [41]. Indeed, there is an emerging literature that demonstrates that there is not a linear progression from exposure to risk and subsequent offending [86] and that risk factors alone do not successfully capture the trajectory of criminal careers or the different transitions into adulthood [87]. For example, MacDonald [86] p. 380 analysed the extended transition into adulthood of children in a deprived area of Northern England, and acknowledged the dynamic and cumulative effect of experiences of risk factors (poverty, ill-health, criminal victimisation, unemployment, poor schooling, offending, drug use and homelessness) on children’s lives, but concluded that:

‘transitions of whatever sort—whether they be ‘conventional’, ‘delinquent’ or somewhere in-between—do not roll on deterministically to foregone conclusions’.

Most children in this group had experienced a high number of risk factors, but most did not go on to criminal careers. Some grew out of crime when they became partners/parents, but at times, the processes of desistance were fragile and, therefore, often chronically relapsing [88]. There is, however, little account of these social and structural identity models of transition in the quasi-positivist risk factor literature. In addition, there remains a gap in the youth justice literature over the significance of contextual factors in understanding the mechanisms of offending and how these can lead to diverse outcomes [48]. In other words, as suggested by the transitioning into and out of criminal careers in non-uniform ways by the young people in the MacDonald et al. studies [86,87,89], the aim of more valid causal explanation in youth justice should be to produce theories of the mechanisms that explain both the presence and absence of ‘uniformity’. The attendant interplay between contexts, mechanisms and outcomes is the foundation of realist approaches to evaluation [2,12].

A key benefit of taking a realist approach to the evaluation of interventions is that it can consider the elements that underlie the mechanisms of change, in other words, ‘Context as the relational and dynamic features that shape the mechanisms through which the intervention works’ [41] p. 1. Realism is a philosophy situated between positivism (the belief that there is a real world out there that can be observed) and constructivism (the belief that reality is socially constructed and thus cannot be observed) [90], therefore it can enable broader and more in-depth analyses of causal mechanisms. Bhaskar [1] notes how reality can be stratified into three levels characterised by the real (causal powers or mechanisms of objects and agents), the actual (what happens if, and when, those mechanisms are activated) and the empirical (events which are experienced or observed) [91]. Events observable in the empirical domain are the product of unobservable activity in the real and actual domains of children’s lives. Thus, realists espouse a ‘generative’ understanding of causation, which would seek to understand the generative causal mechanisms that can initiate a child’s offending and also help them to desist. Mechanisms are not variables or attributes and thus not always directly measurable [92]. Instead, they are ‘processes describing the human actions that have led to the uniformity’ [72] p. 1. As they depend on this choice-making capacity of children and the groups they belong to, the emergence of social uniformities is always highly conditional:

‘According to this perspective it is not ‘programmes’ that work: rather it is the underlying reasons or resources that they offer subjects that generate change. Causation is also reckoned to be contingent. Whether the choices or capacities on offer in an initiative are acted upon depends on the nature of their subjects and the circumstances of the initiative. The vital ingredients of programme(s) … are … ‘generative mechanisms’ and… ‘contiguous context’ [93], p.342.

Consequently, Realist approaches move beyond the explanatory ‘black box’ of successionist models. This generative understanding of causation underpins the realist approach to synthesis. The promise for evaluation in youth justice is that, rather than summating evidence from experimental and quasi-experimental studies to isolate the average effect size of an intervention, realists seek to explore how contextual features shape the mechanisms through which interventions produce their outcomes by integrating both quantitative and qualitative evidence. This fuller, more in-depth understanding could provide policymakers and practitioners with a stronger evidence base, which can help them better support children to prevent offending and reoffending.

Some scholars have attempted to devise synthesis methods that build bridges between realist and positivist approaches [94] p. 21 as ‘a means of marshalling the best evidence available to meet policymakers’ and practitioners’ needs’. These methods can challenge evidence hierarchies, such as those used by the ‘What Works’ evaluation framework in youth justice and demonstrate how different kinds of evidence can speak to theories embedded in policy and practice to facilitate a better understanding of causal and contextual influences on intervention effectiveness. For example, Johnson et al. [95] p. 462 note, the ‘mediators’ and ‘moderators’ approach used by some meta-analysts has many of the same functions as realism:

‘Mediators describe the chains of events (or intermediate outcomes) that occur between a treatment and the ultimate outcomes produced…. Moderators … refer to variables that may explain variation in outcomes across different studies. They can include circumstances associated with differences in the efficacy of the intervention, such as the type of location [or] the study methods employed.’

It is important to note here that many realists, including Pawson, do not accept the idea that ‘mediators’ can be equated with mechanisms or that ‘moderators’ are synonymous with contextual features [96,97]. Side-stepping these debates in a pragmatic move, Johnson et al. introduced their own framework to guide evidence synthesis to enable policy makers with limited budgets to be able to grasp evidence that ‘speaks to how interventions work and where they might do most effectively’ [95] p. 463. Drawing on lessons from the moderators and mediators and from realist approaches, the EMMIE framework was designed as a system to analyse the quality and coverage of existing systematic reviews of (youth) crime prevention interventions but also as a framework to enhance the explanatory value of future systematic reviews [5]. Therefore, a large body of previously neglected in-depth youth justice literature could be used to enlighten understandings of intervention effectiveness, such as practitioner and qualitative research, which would not meet the scientific criteria required by the ‘What Works’ evaluation framework. EMMIE concentrates not only on the effects (E) but encourages reviewers to look for evidence on the moderators (M) mechanisms (M) the nature of implementation (I) and costs, or economics (E) of interventions. In other words, EMMIE provides a framework for reviewers to examine not just the effect of the intervention, but also the causal mechanisms through which the interventions work, the contextual features that shape the way interventions work and their impact and the practical implementation issues for policymakers.

The significance of approaches like EMMIE is that they are filtering into the ‘What Works’ networks and now provide the methodological foundation for the ‘What Works’ Centre for Crime Reduction, established in the UK in 2013 and hosted by the College of Policing (https://whatworks.college.police.uk/About/Pages/default.aspx accessed on 6 June 2021). There is also evidence of EMMIE reviews entering the crime prevention evidence base. Sidebottom et al.’s [97] review of gating alleyways as a situational crime prevention intervention drew on the EMMIE framework to consider not just whether providing lockable gates would have a crime reduction effect, but to understand the ways in which this might or might not happen. The importance of the review was that it started with theories of how gating alleyways might work, which most meta-analyses lack, because they contend that gating alleyways will either reduce crime or not. The review then harnessed evidence from both qualitative and quantitative studies to test these theories and their crime reduction impact. These theories broadly state that it increases the effort on behalf of the would-be offender. By applying the EMMIE framework, the authors were able to ascertain that this indeed was significant, but that this also places little emphasis ‘on the actions of the impacted residents’ [97].

Future youth justice policy, intervention programme design and its implementation could also be better informed through the use of realist methods as it allows consideration of important contextual features neglected by the ‘What Works’ approach. These include how the implementation of an intervention can shape its effectiveness and its impact on a child, as well as considering individual, family, social, economic and other features. The EMMIE framework encourages researchers to engage in further kinds of interrogation when evaluating (youth justice) interventions, which can extend understandings of the efficacy of different responses to youth crime by considering contextual differences and causal mechanisms. For example, Sidebottom et al.’s [97] review of situational crime prevention sought to ascertain what impacts might be yielded if residents actions were more concerted. For example, if residents decided to grow plants in the newly created private space, afforded by gating, or installed trellises and maintained cleanliness, emergent forms of internal social control were generated. This in turn creates new mechanisms: The fixing of broken windows—creating a sense of guardianship and increased surveillance of the space; or the removal of excuses—increasing the challenging of unfamiliar faces in the locale, both of which have crime reduction impacts [5]. The EMMIE review also showed how these mechanisms rely on a non-trivial amount of community cohesion and a mutual sense of purpose, which can heighten or lessen the impact of the mechanism, depending on its nature and extent. Therefore, the benefits of using this method to synthesise youth justice intervention research from a broad spectrum of methods, from a wide range of disciplines, offers the potential to consider not only ‘What Works’, but what works for which children, in what circumstances, in what respect and how. A more comprehensive approach to intervention evaluation can, consequently, lead to a better understanding of how to achieve improved outcomes for children involved in the YJS.

8. Conclusions

The evaluation of the effectiveness of youth crime prevention interventions in the Western world has been dominated by the quasi experimental ‘What Works’ framework, which has privileged risk-based, individualised, psychosocial and decontextualised explanations of youth offending and has thus prioritised risk-based responses. The predominance of risk-based approaches to youth crime prevention, in particular, is a function of the underlying quasi-positivist assumptions, often derived from developmental psychology, contending that causes of youth offending are simplistic and lie within the individual. A cause and consequence of this simplistic understanding has been that preventative interventions have been increasingly evaluated using frameworks that prioritise RCTs and associated experimental designs, which ignore wider contextual features that might shape the effectiveness of these interventions.

The arguments forwarded within this paper recommend the adoption of the Realist Synthesis evaluation methodology by the youth justice field to enable a more holistic understanding of how criminogenic influences and intervention programme theories have been conceptualised in youth justice. Compared to the hegemonic ‘What Works’ evaluation framework, Realist Synthesis offers the sector far more capacity and utility when seeking to identify contextualised explanations of youth offending. It examines both the causal mechanisms through which interventions work (or do not work) to produce their intended and unintended effects and it also interrogates the vital contextual influences on the mechanisms of change that underpin intervention effectiveness [41]. Adopting this approach, therefore, will enable stakeholders to identify where there are implementation issues in youth justice and/or fundamental flaws in the implicit assumptions about causality and how interventions work. Closer examination of the potential of Realist Synthesis strongly suggests that the field of youth justice would benefit greatly from this alternate framework, as crime prevention and healthcare fields have already long benefitted from its use. Realist evaluation could broaden the explanatory scope of the existing youth justice evidence base, so the understanding of what programmes work could be further expanded, and consideration could also be given to whom programmes work for, in what circumstances and why [12].

In summation, the use of realist approaches would facilitate more detailed reviews of the ‘effectiveness’ of preventative interventions in youth justice that are more sensitive to the complexities of children’s differing experiences, engagements and local circumstances. We believe that this offers a platform for synthesising the contextual elements of youth offending and youth justice intervention programmes that are often neglected in contemporary evidence-based practice.