Case studies describing the experience of working with decision-makers, at both a country and global level.
TB MAC Guidance Document:
These case studies were compiled in 2017, in companionship with the TB MAC/WHO Guidance for country-level TB modelling. Each case study highlights an instance in an individual country-level modelling application where a particular principle found in the Guidance document (shown below in bold) was felt to be especially pertinent. See the Guidance document for more details. Modelling teams and countries have been redacted to avoid sensitivities.
A modeller’s experience of ‘Relevance’
During the application of a model to multiple countries, it became clear that important differences existed in the extent to which the quality of care in the private sector was an issue. To make sure the model was relevant to each setting, the modelling team consulted with each of the country programmes, to ensure that assumed parameters were applicable to each setting.
A modeller’s experience of ‘Realism’
During a country visit, discussions between the modelling team and staff from the national TB programme highlighted that screening protocols did not reflect the reality of implementation, with substantial variation found. Through triangulation with data from other national modelling efforts, and agreement with local experts, model assumptions were made that reflected actual practice, including barriers to implementation and associated time-implications.
A modeller’s experience of ‘Appropriateness of model structure’
During a model application in countries in Asia and Africa, it became clear that the model needed appropriate structure to capture the complete screening population, including those who did not have TB and were at risk of being misdiagnosed (false positive) with TB. With this structure, the model was able to provide a more complete impression of resources needed (e.g. people screened) and potential negative consequences of proposed screening programmes.
A modeller’s experience of ‘Consideration of all evidence’
During a review of evidence to inform modelling related to the introduction of preventive therapy for contacts and active case finding (ACF), the modelling group identified important differences in the type of evidence available for these two interventions. Specifically, epidemiological evidence about the efficacy of ACF was available from cluster randomised trials, while evidence about the efficacy of preventive therapy was available from clinical studies at the level of individuals. The differences in the type of evidence available, and the limited evidence regarding the effectiveness of both interventions under programmatic conditions, were made clear in the modelling report.
A modeller’s experience of ‘Validation’
In a recent country modelling application, two types of validity checks were particularly valuable: (i) face validity and (ii) technical validity. In the case of face validity, country partners were engaged from the outset, to check that the model faithfully captured their epidemiological situation. Some estimates of HIV/TB burden were found to be too high, a discrepancy that was traced back to subnational data that needed updating. The subnational data (and the model) were then updated. For technical validity, model methods, scenarios and results were shared with two independent experts for peer review. Their critical comments helped to improve the model.
A modeller’s experience of ‘Informativeness’
During a multi-country modelling exercise to examine the impact of radically expanded TB programs, it became clear that the plausibility of modelled strategies could not be judge based only on epidemiological outcomes. To allow county program experts to judge the feasibility of a strategy, models were used to predict additional outcomes describing the changes in program performance (numbers needing to be screened, improvements in cure rates) required to achieve greater impact. In some cases this led to original strategy descriptions being judged implausible, and strategies were revised to be more realistic.
A modeller’s experience of ‘Transparency’
During the preparation of modelling results for use in a country application to the Global Fund, the structure of the model, and its strengths and limitations were discussed at length with in-country stakeholders. In addition, a smaller in-country modelling team tested the key assumptions and model functionality. Opening up the “black box” led to critical questions being raised, which helped to improve model structure and documentation, and also the calibration of the model to country data.
A modeller’s experience of ‘Timeliness’
A country application was timed to evaluate the impact of reforming a national TB programme. The model team captured the current TB epidemic and response as well as alternative response scenarios, using findings from a pilot project on patient-centred care which had recently become available. Results outlining an optimized TB response strategy were reviewed and released in time to inform a dialogue around national TB care. During this dialogue agreement was reached on key components of reform to the national TB programme.
A modeller’s experience of ‘Country ownership’
In a recent country modelling application undertaken in the context of a Global Fund grant, there was a strong emphasis on country leadership and engagement from the outset. A country modelling team was created with representatives from all major organizations working on TB in the country, including the NTP, the WHO country office and the principle recipient (PR) of the Global Fund. Following training to use the modelling tools, the country modelling team led planning, execution and communication of the modelling work, for both domestic and international stakeholders.
A modeller’s experience of ‘Iteration’
Estimates of disease burden (such as those from WHO’s Global TB Report) are released annually as new surveillance and survey data become available. Modelling groups have evaluated past modelling projections and results in light of new burden estimates or data, such as prevalence surveys or inventory studies. As part of preparations for a Global Fund application, one modelling group returned to previous projections and reviewed these in the context of new prevalence survey data.
Other published literature:
The South African TB Think Tank
National Tuberculosis Programmes (NTPs) require specialist input to support the development of policy and practice informed by evidence, typically against tight deadlines. A national TB think tank was established to advise the NTP in support of evidence-informed policy. Support was provided for activities, including meetings, modelling and regular telephone calls, with a wider network of unpaid expert advisers under an executive committee and working groups. Intervention evaluation used desktop analysis of documentary evidence, interviews and direct observation.
The TB Think Tank evolved over time to acquire three key roles: an ‘institution’, a ‘policy dialogue forum’ and an ‘interface’. Although enthusiasm was high, motivating participation among the NTP and external experts proved challenging. Motivation of working groups was most successful when aligned to a specific need for NTP decision making. Despite challenges, the TB Think Tank contributed to South Africa’s first ever TB and human immunodeficiency virus (HIV) investment case, and the decision to create South Africa’s first ever ring-fenced grant for TB. The TB Think Tank also assisted the NTP in formulating strategy to accelerate progress towards reaching World Health Organization targets.
With partners, the TB Think Tank achieved major successes in supporting evidence-informed decision making, and garnered increased funding for TB in South Africa. Identifying ways to increase the involvement of NTP staff and other experts, and keeping the scope of the Think Tank well defined, could facilitate greater impact. Think tank initiatives could be replicated in other settings to support evidence-informed policy making.
Link to full text here.
Coming in 2019…