My current PhD module is on the research ‘technology’ of systematic reviewing. This type of research study is a manifestation of the evidence-based practice movement driven by the desire to make sure that research informs practice and/or policy. Systematic reviewing arose in the world of medicine as a way of drawing together the findings of different ‘Randomised control trials’ in order to come up with a better answer to whether the intervention x leads to an outcome y. The method of systematic review was/is hailed as better than traditional literature reviews which were criticised for cherry-picking the studies that fit with what an author wants to say. My own view is that the traditional literature review actually has a different purpose – to scope out existing research in an area to highlight the ‘niche’ for a proposed piece of research and as Boell and Cecez-Kecmanovic (2014) eloquently argue can be undertaken just as rigorously.
Anyway, as I’ve gone through the module, I’ve begun to understand that the term systematic review now goes well beyond the original ‘what works’ review of the Cochrane collaboration. There are a multitude of different approaches to identifying and synthesising both quantitative and qualitative information held in research literature underpinned by a variety of study designs – like other forms of research they arise from different epistemological perspectives and therefore approach the task in different ways in order to answer different types of questions. There are now articles of systematic review methods leading to different typologies and a multitude of terms (see for example, Dixon-Woods et al, 2005; Gough et al, 2012; and, Grant and Booth, 2009) and more that focus on different ‘stages’ of the review process especially synthesis (see Barnett-Page and Thomas, 2009).
As I near the end of the module, I’ve started to wonder about the degree to which systematic reviewing can be undertaken systemically. The systems practitioner in me is rearing its head! As Ray Ison once said to me – “research is a practice too” – words which I directly hold responsible for me doing a PhD in the first place [depending on the day I am having that may be blame or gratitude!]
Could I in fact think about designing a systematic review through the framework of a systemic inquiry? Juggling the C-ball (for contextualising) by practising methodological pluralism – with awareness of course of the limitations – can this happen within one systematic review? Rather than get annoyed with the idea of needing to write a protocol up-front, can I think instead that it is my articulation of a design for learning? Can I do a ‘design turn‘ in my approach? (whether or not my tutors and the research community as a whole will recognise this as ‘acceptable’ is a different matter).
I turned to the C-ball chapter in Ray’s book (Ison, 2010) and found myself drawn to the table showing Checkland’s (1985) distinction between the ‘hard’ systematic tradition of systems thinking and ‘soft’ systemic tradition. So, I thought I’d take each of the characteristics and make a tentative ‘shot’ at applying them to the practice of systematic reviewing…
| The hard systematic thinking tradition (from Ison, 2010, p.158) |
How I perceive this being realised in approaches to systematic literature review |
| oriented to goal seeking | protocol that defines procedures that will get you to your goal (i.e. answer to a question) |
| assumes the world contains systems that can be engineered | looking for an effective intervention for a problem |
| assumes systems models to be models of the world (ontologies) | naive realism/realism |
| talks the language of ‘problem’ and ‘solutions’ | as above |
| allows the use of powerful techniques | the ultimate quest in the hierarchy of evidence is to do a meta-analysis |
| assumes that there is a ‘right’ answer | seeking ‘the’ answer to a question |
| may lose touch with aspects beyond the logic of the problem situation | already criticisms that systematic reviews ignore context |
The above highlights the ‘traditional’ domain of application for the systematic review – that of a ‘what works’ review. So now to Checkland’s characteristics of systemic thinking
| The soft systemic thinking tradition (from Ison, 2010, p.158 |
How I see the potential for this being realised in approaches to systematic literature review |
| oriented to learning | more of a process of ‘discovering’ the literature; finding out or exploring |
| assumes the world is problematical but can be explored by using systems models | the ‘protocol’ describes your ‘learning system’ – your purpose, the boundaries you will draw. But recognises that this is always subject to review |
| assumes system models to be intellectual constructs (epistemologies) | mmm – need to think some more about that |
| talks the language of ‘issues’ and ‘accommodations’ | research questions tend to be about phenomena or the body of research literature itself rather than an intervention |
| is available to all stakeholders including professional practitionerskeeps in touch with the human content of the situation | more of an emphasis on participation of stakeholders at all stages of the review process |
| does not produce the final answers and accepts the inquiry is never-ending | recognition more of the unfolding story-line of research and that only offering a tentative position which itself is socially and historically situated |
| remains aware that there are dimensions of the situation to which linear logic does not apply | some review questions speak more of ‘forces and factors’; influence and issues |
It seems to me that there are already some echoes of researchers taking elements of a ‘systemic’ approach in their systematic reviews (should I be calling them ‘systemic reviews’ or ‘systematic reviews practised systemically’?) but perhaps the systemic framing can be taken further.
My intention in this exercise isn’t to suggest that one is better than the other – systemic and systematic aren’t an either or – it’s just that in some cases it may be more appropriate to start off a journey into the confusing messy world of research literature in a systemic way. In other cases, where you have a clear question – like ‘does this medicine work for this clearly defined diagnostic category?’ – then systematic thinking comes to the fore. What’s more a systemic orientation to systematic reviewing practice doesn’t mean a lack of rigour, it’s just that you are not approaching the method in a formulaic, recipe-like manner regardless of what you are learning about the nature of the research in the area. A systemic approach means that you may craft together from different approaches and adapt them according to your review question – questions that act as compasses rather than an anchor (Eakin and Mykhalovskiy, 2003).
I’m glad I made this connection – I can cope with a systemic orientation to ‘systematic reviewing practice’ – even though the language trips you over a little.
References
Barnett-Page, E. and Thomas, J. (2009), Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology, 9, p.59.
Boell, S.K. and Cecez-Kecmanovic, D. (2014), On being ‘systematic’ in literature reviews in IS. Journal of Information Technology, Online Advance.
Checkland, P. (1985), From Optimizing to Learning: A Development of Systems Thinking for the 1990s. The Journal of the Operational Research Society, 36(9), pp.757–767.
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B. and Sutton, A. (2005), Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy, 10(1), pp.45–53B.
Eakin, J.M. and Mykhalovskiy, E. (2003), Reframing the evaluation of qualitative health research: reflections on a review of appraisal guidelines in the health sciences. Journal of Evaluation in Clinical Practice, 9(2), pp.187–194.
Gough, D., Thomas, J. and Oliver, S. (2012), Clarifying differences between review designs and methods. Systematic Reviews, 1(28).
Grant, M.J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26(2), pp.91–108.
Ison, R. (2010), Systems Practice: how to act in a climate-change world, Milton Keynes/London: The Open University/Springer Publications.
Interesting line of inquiry. The Cochrane reviews attempt to answer the question ‘what is the best answer given all the evidence that is available at this point of time?’. However it is assumed that evidence will grow and change and that answer may change. However health care interventions usually consist of a number of actions which need to come together to be effective. For example for an anti-hypertensive medication to be effective even at a patient level, an individual’s blood pressure needs to be properly assessed, the most appropriate medication prescribed, dispensed and taken regularly, and dosage may need to be modified based on ongoing monitoring. There are even more components if the aim is an effective population intervention. It is much harder to get clear answers about such questions from traditional systematic reviews. The Australian Prevention Partnership Centre (www.saxinstitute.org.au/our-work/preventing-chronic-disease/) was set up to look at systems perspectives on prevention of lifestyle related chronic conditions. Through our knowledge synthesis group we are grappling with similar issues to you and would be happy to share our thoughts. If interested I can put you in touch with the relevant research fellows.
Hi Andrew
thank you for taking the time to read my post and provide comments.
You’ve prompted me to consider another dimension of drawing together ‘systems thinking’ and ‘systematic reviews’ – that of the issue of context of practice in using review findings. I think that realist reviews take this stance – considering the issue of context as well as mechanism and outcome? This seems to be a good way of building systems models of ‘how the world works’ – the most (in)famous manifestation in public health being the Foresight Obesity Map. This seems to fit with Checkland’s characteristics of hard systematic thinking – if we can discover enough about the ‘system’ of obesity then we should be able to intervene to engineer it differently. Approaches like systems dynamics are good for this orientation.
I think my post above focussed more on the tensions I sometimes experience as someone who values the usefulness of being systemic (in all my practices including research practice), but feels constrained by the ‘norms’ and ‘technologies’ of research practice that I am currently learning about. It seems that every so often I have to make a radical ‘reframing’ of what it is I understand I am doing in order to keep a systemic outlook on my research practice.
Thanks for this opportunity to reflect some more.
Helen