USAID/REAL Resilience Measurement and Analysis: Frequently Asked Questions
Below you will find a variety of questions that we have gathered from a number of REAL learning events. If you have a question about resilience measurement and analysis that you do not see indicated below, please contact us here with your question.
Measuring Shocks, Stresses, and Resilience Capacities & Analyzing Resilience
What makes a research question a “resilience research question”?
Resilience research should be guided by complementary questions based on an analytical framework that includes the central pillars of resilience analysis: resilience capacities; identified shocks and stresses (and downstream effects); coping strategies; resilience responses; and well-being outcomes. While it is possible to analyze resilience capacities without a shock or stress, we cannot reach valid conclusions about whether these capacities, when combined with responses ultimately result in household and/or community resilience in the absence of identifiable shocks or stresses. Relevant examples of resilience research questions are provided in Guidance Note 5: Resilience Analysis.
Are there useful ways to conduct resilience analyses using cross-sectional data? Or do we need panel data?
Current analytical methods for resilience analysis of USAID-funded projects are often based on both cross-sectional data and panel data. Resilience analyses using cross-sectional data is collected from statistically representative samples through baseline studies, routine monitoring and endline evaluations. Resilience analysis based on cross-sectional data entails first applying “weights” (determined as part of the baseline analysis) to individual components of resilience capacity and then applying these weights in the empirical model used at the endline. This allows for assessing how the components that contribute to resilience change over time (t0 to t1) for a given outcome. Resilience analysis using panel data employs the same basic methodology but allows for a deeper understanding of the dynamics of change for the same households over time, particularly if collected at more than two points in time. The REAL Associate Award will soon issue guidance on Utilizing Panel Data for Resilience Measurement which illustrates that it provides stronger analytical power and precision for a given sample size than cross-sectional data. Subjective resilience can be used to measure resilience cross-sectionally; it can also be captured with panel data.
Cross-sectional analysis can be used to answer questions around population trends. For example, do sources of resilience (i.e., individual resilience capacities) change over time or context? Longitudinal analysis can be used to assess what proportion of a population is resilient and what factors are predictive generally. In other words, who is resilient and why?
How are resilience capacities defined? Are these locally defined, measured, and interpreted through participatory research?
Guidance Note 3 of the Resilience Measurement Practical Guidance Series shares the now widely accepted definitions of absorptive, adaptive and transformative resilience capacities. The exact components of these three types of resilience capacities are driven by the local context, specific shocks and stresses, and target populations.
How is women's empowerment associated with resilience measures?
Previous evaluations confirm that gender-based differences between men and women in their mobility, time use, and ownership and control of income, assets and other resources affect the capacities of individuals and households to respond effectively to a particular shock or stress. Additional detail on the relationships between gender and resilience can be found in the Concept Note on Integrating Gender into Resilience Analysis. As part of its Resilience in Action Series, REAL has also developed detailed guidance for promoting gender equity and social inclusion as a key element of strengthened resilience.
How can we measure resilience at impact level? What are some possible indicators to measure resilience?
First, we must reiterate the difference between measuring resilience and measuring a resilience capacity. Second, resilience itself is not an impact measure. The outcome indicators we are interested in remain the high level impact indicators used by typical development projects (e.g. food security, nutrition, socio-economic status, health status, etc.). Resilience measurement is focused on determining how wellbeing fluctuates over time in the face of shocks, and how investments intended to strengthen absorptive, adaptive and transformative capacities reduce this fluctuation. Because measuring resilience requires repeated measures of wellbeing over time in the face of shocks, identifying a single indicator that captures the temporal and nuanced nature of resilience is nearly impossible. Instead, it requires capturing several indicators that reflect various dimensions of resilience capacities over time and analyzing them together to create a comprehensive and coherent picture. Detailed guidance on specific indicators used in resilience analysis of USAID-funded programming is available in the Resilience and Resilience Capacities Measurement Options resources. Other interesting proxy indicators for resilience are also being developed as part of this growing body of research, including subjective dimensions of resilience recently published by Béné, Frankenberger, Griffin, Langworthy, Mueller, and Martin (2019).
How do we design resilience analysis when we are unable to have a counterfactual?
Resilience analysis does not require a counterfactual, per se, unless it is used as part of an impact evaluation. Resilience analysis is also used in performance evaluations involving baseline and endline comparisons with panel data, but no counterfactual. When no counterfactual is available, we can use recall techniques in both quantitative and qualitative studies to ask respondents about changes in their resilience capacities, the factors that contributed to those changes, and how changes in resilience capacities have affected wellbeing outcomes among individuals, households and communities. The caveat is that this approach has limitations in terms of people’s ability to accurately recall information, meaning that measures are not purely objective.
For resilience analysis, it is important to integrate shocks and stresses. How do we approach this when we are unable to measure these events?
Resilience cannot be measured without the presence of shocks or stresses. In instances where primary data collection does not include information on previous, current or anticipated shocks and stresses, secondary data sources can be effectively used to provide this information, including data on weather patterns that contribute to droughts and floods, the occurrence of conflict, market data on economic shocks, and other data on present disasters. In addition to empirical data on shocks and stresses is available, it is also important to gauge the perceptions of risk among diverse individuals and groups within a given context. Qualitative analysis can provide rich insight into at least two key dimensions of risk perception that influence whether and how populations draw on resilience capacities in response to shocks and stresses: 1) perceived risks associated with experiencing certain shocks and stressors; and 2) perceptions of potential responses to shocks and stresses once an adverse event has occurred.
How do you develop a composite indicator for resilience capacities? How do you assign weights for such indicators?
Because a “capacity” is both multi-faceted and unobservable, it is treated as a latent characteristic. To address this challenge, composite indicators of resilience capacities are built around a set of components that can be observed and measured. Factor analysis is then used to identify the relevant characteristics of community and household resilience, which fall under each capacity. The inter-correlations of these characteristics allow the creation of a single variable ("Index”) to measure that capacity. The current practice for developing resilience indices and assigning weights is presented in full in the Methodological Guide for the Resilience and Resilience Capacities Measurement Options
When we speak of the resilience of individuals and households rebounding from exposure to shocks and stresses, what is the ideal equilibrium point? If households are food insecure prior to the shock/stress, should we say they are resilient if they rebound back to this initial point?
Essentially yes — in the short-term individuals, households and communities are on a resilience pathway if they are able to manage a shock or stress in a way that does not negatively impact their food security or another measure of well-being. However, over the longer-term, resilience strategies should focus on sustainable solutions that help lift the most vulnerable (e.g. poor, food insecure) out of a state of heightened exposure to shocks and stresses. This underscores the idea that resilience is not an "end state" that is measurable by some threshold, but rather represents a pathway that leads people to maintain or improve their well-being even when they experience a shock or stress. The findings of Recurrent Monitoring Surveys of affected populations in Ethiopia provides clear evidence of this relationship between shocks and maintenance of wellbeing outcomes.
Risk and Resilience Assessment
Is the Strategic Resilience Assessment (STRESS) approach to risk and resilience assessment appropriate for use in both development and emergency contexts? If so, what is the most appropriate stage in the program cycle to conduct the risk resilience assessment?
Yes — STRESS, like other Risk and Resilience Assessments, can and should be adapted and applied in various contexts to better understand key factors related to risk and resilience at individual, household, community and systems levels. In both humanitarian crises and development contexts, findings from Risk and Resilience Assessments can inform program design, implementation, monitoring and impact evaluation.
Should a Risk and Resilience Assessment be done at the beginning (e.g. first year) of a program or before designing a program?
Ideally, Risk and Resilience Assessments should be done prior to designing an intervention or a portfolio of programs. At the same time, a Risk and Resilience Assessment can also provide valuable information for improving ongoing programs (e.g., during the first year) depending on the operating context, the purpose of the assessment (e.g. strategy development, program design, adaptive management, evaluation) and the knowledge gaps it seeks to address. Prior to implementation, a Risk and Resilience Assessment can be instrumental in identifying key constraints, pertinent shocks and stresses, system dynamics, vulnerable populations, target geographies, and opportunities for strengthening resilience capacities. Lacking this information, programs aimed at strengthening resilience may be ineffectively designed and targeted, insufficiently monitored, and in some cases, may even run counter to basic principles of ‘do no harm’.
When appropriately designed and implemented, Risk and Resilience Assessments are also very useful for supporting adaptive management of programs in various operating environments. At the portfolio level, Risk and Resilience Assessments can inform the development of cross-program results frameworks that link distinct interventions and facilitate effective sequencing, layering and integration of activities to reduce risk and strengthen absorptive, adaptive and transformative capacities.
Is the STRESS methodology used for devising country-level resilience strategies or at project level only? How is the methodology most effectively and efficiently applied given the time it takes to generate the information?
Mercy Corps has used the STRESS process to inform country strategies (e.g. Niger), as well as sub-national strategies (e.g. Northeast Nigeria, Karamoja, Uganda) and program strategies (e.g. Democratic Republic of the Congo). Guidance Note 1 in the Resilience Measurement Practical Guidance Series explains three potential levels of engagement for a Risk & Resilience Assessments — low, medium and high — depending on the resources available, the purpose of the assessment (process and product-wise), the geographic scope and the availability of pre-existing data. This general categorization helps inform decision-making on the most suitable (best) scope and scale of the assessment, weighing the desired outputs and outcomes in light of feasibility considerations (timeframe, resources, staff capacity, opportunities for stakeholder engagement).
Should sector-specific Risk and Resilience Assessments be done beforehand — prior to going to the field?
Generally, we advise a sector-neutral approach to Risk and Resilience Assessments to ensure that all constraints to (and enablers of) resilience, and the linkages between them are given adequate consideration. Sector-specific assessments that have already been done can (and should) inform the risk and resilience assessment process.
How can Risk and Resilience Assessments effectively integrate the perspectives of participating communities on resilience (from shocks inventory to well-being outcomes) in such a long, rigorous process?
Data collection and analysis in Risk and Resilience Assessments is an iterative process, becoming more refined as knowledge is gained and greater understanding reflected in new lines of inquiry. Keeping in mind that context matters, collecting, and where appropriate, interpreting field-level data with key stakeholders representing targeted geographies and vulnerable populations is a critical step of the Risk and Resilience Assessment process. When complemented with secondary data, this “ground-truthing” analysis of primary data can ensure that the assessment is asking the appropriate questions and that responses accurately capture the perspectives and experiences of diverse stakeholders. Guidance Note 1: Risk and Resilience Assessments shows how ensuring and encouraging the active participation of a diverse set of partners and teams is key to the assessment process. Therefore it is important that data collection be carried out in a participatory manner – incorporating focus group discussions, risk and resource mapping, key informant interviews, historical timelines – that empowers community members to share their unique insights.
Resilience Monitoring and Evaluation at the Activity Level
How do we measure resilience from a cross-thematic perspective, oftentimes we are challenged to mainstream resilience?
To answer this question first we must distinguish between measuring resilience and measuring a resilience capacity.
TANGO conceptualizes resilience according to the USAID and Resilience Measurement Technical Working Group (RMTWG) definitions. Drawing on these broader definitions, and focusing at the household level, its operational definition of resilience is:
Resilience is the ability of households to recover from shocks: to maintain stability in their well-being or return to their pre-shock well-being or better.
The measure of resilience it employs aligns very closely with this definition. It is the change in households’ food security from before to after a clearly-defined shock. This measure is termed “realized resilience” because it is a post-shock depiction of how households actually fared over the course of a shock. (D’Errico and Smith 2020).
While resilience itself is an ability to manage or recover, resilience capacities are determinants of resilience. The three dimensions of resilience capacity measured by TANGO are absorptive capacity, adaptive capacity, and transformative capacity. Guidance Note 3 provides in depth instruction into how to define, identify, measure and analyze resilience capacities.
Resilience is not equivalent to resilience capacity, and resilience is not equivalent to well-being itself. In regards to the latter, households with low well-being can have high resilience and households with high well-being can have low resilience, depending on how each group’s livelihoods are affected by a particular shock. Thus, measures of well-being do not always align strongly or even positively with measures of resilience.
For the purposes of this question, resilience capacities are the sources of resilience that enable appropriate and positive responses to protect or improve well-being in the context of shocks and stresses (see Bene, Frankenberger, and Nelson 2015 for more details). Capacities are multi-faceted, deeply contextual and nuanced in their function and role in relation to shocks and stress, requisite responses, and desired well-being outcomes. Resilience capacities are also multi-level, from individuals, to households, communities, and the systems they occupy. For these two reasons, measuring resilience capacities transcends sectoral-specific perspectives to measure important intermediate outcomes from a multi or cross-sectoral perspective.
Would you recommend developing a resilience results chain during a proposal design workshop or at a start-up workshop or both?
Guidance Note 5 recommends explicitly identifying distinct elements of the resilience results chains’ – well-being outcomes, intermediate outcomes, resilience capacities, resilience responses, interventions and outputs – during the design stage. Given the dynamic nature of shocks and stresses, a resilience results chain should be updated as needed as additional learning is available.
Does a resilience Theory of Change depend on the specific shock(s) or stress(es) being addressed?
Any given theory of change should be context-specific and explicitly reflect the particular objectives of the program. Guidance Note 3 explains how resilience Theories of Change should describe the ways in which activities strengthen the ability of participants to prevent, prepare for, mitigate and adapt to shocks and stresses. Given dynamic operating environments, Theories of Change (as well as results frameworks, monitoring and evaluation plans) should be reviewed periodically to ensure they are responsive to observed changes on the ground.
Would you give us examples of how you have used results frameworks to guide the design, implementation and monitoring of resilience programs?
During the Session 2 of the REAL Short Course on Resilience Measurement a case study was presented (starting at 34 minutes, 34 seconds) to provide an example from a livestock program in Mongolia where they developed resilience results chains, and established regular results chain review sessions to periodically revisit and revise the results chains throughout the life of the project. Two subsequent case studies presented in Guidance Note 5 of the Resilience Measurement Guidance Note Series provides further examples of how resilience results chains may be used to inform program design and implementation.
Can you share any tips for advocating for use of panel data collection for annual surveys (to enable resilience monitoring) given that annual surveys are often used to report on output indicators?
Having open communication lines with the donor about the purpose of routine resilience monitoring, and how it complements standard data collection and reporting requirements, can enable greater flexibility in data collection and reporting. A key point in advocating for use of panel data is that it enables the determination of the specific activities and/or combinations of interventions that strengthen the capacities of people to manage shocks. Regular collection of panel data through RMS is particularly useful for enabling program staff to make adaptive management decisions including refinement of the activity design or implementation strategy rather than waiting for this to be determined through a final impact evaluation. Guidance Note 6 offers a relevant example of how RMS data collection was coordinated with annual monitoring surveys to gain insight into the impact of specific interventions in Nepal.
We are evaluating the long-term impact of a program intended to strengthen resilience in East Africa and we are doing this four years aft er completion of the project. What are the similarities/differences between standard impact evaluation and post-project evaluation of a resilience program when it comes to design?
There is a substantial need for both “standard” impact evaluation (hereafter referred to as impact evaluation) and “post-project” impact evaluations of resilience investments, where the former takes place at the end of project activities but before the official close and the latter taking place over a longer time period after the official close of a project. Many of the similarities/differences between an impact evaluation and a long-term impact evaluations (LTIEs) are not unique to evaluating the impacts of resilience investments, and the ERIE (2018) Guide for Planning Long-Term Impact Evaluations offers useful insights for program managers. A primary similarity is that both should rely on a counterfactual by definition as an impact evaluation. However, as the time horizon for an LTIE grows, so do the challenges in establishing/maintaining an adequate counterfactual, e.g. due to challenges tracking respondents, contamination of the counterfactual group, limited availability of project staff and participants with substantive memory of the project implementations mechanisms, etc. One key difference between the two is often the types of outcomes evaluated, where an impact evaluation might have impact level indicators (e.g. food security, nutrition, etc.) it will also look at intermediate outcomes like various resilience capacities. Long-term impact evaluations are well placed to measure changes in wellbeing outcomes over time. Regardless of the similarities and differences between the two, they should be viewed as complementary and planned for at the outset of a new program that is theorized to have longer term impact on building resilience.
What happens when there is no clear, pervasive shock to trigger an RMS? Perhaps rainfall and food security are bad in some parts of the focus zone but not others?
The Recurrent Monitoring Survey (RMS) carried out in West Africa for Burkina and Niger fits that scenario where you don’t have widespread covariate shocks across a large area but rather a number of idiosyncratic that are affecting individual households. Findings from recurrent monitoring of the PAHAL program in Nepal provide another relevant example. As described in Guidance Note 6, an RMS still allows you to see how people are managing stresses over time where various seasonal and/or idiosyncratic circumstances prevent individual households from escaping chronic poverty and/or food insecurity. In this seasonal (non-covariate shock) model, the RMS can either follow-up with all households from the baseline/preliminary round or a smaller subset. However, since it might not be clear which households will experience different shocks and stresses during each round, caution should be exercised when reducing the sample size.