Evidence has been central to the development of Housing First. It is through the collection of good quality evidence that Housing First became influential in homelessness policy debates in North America and was able to attract and then sustain funding. In Europe, the emerging evidence base for Housing First has shown that it can work in a diverse range of countries, which have significant differences in their welfare systems, housing systems, culture and levels of economic prosperity. As is shown in the Appendix, Housing First evaluations are reporting successes in countries as diverse as Denmark, England, France, the Netherlands, Portugal, Scotland and Spain.

There are several benefits to evaluating Housing First services:

• Strong evidence has been fundamentally important in persuading governments, charities and homelessness service providers to consider using Housing First. Federal government in the USA regards Housing First as a service model of proven effectiveness((Tsemberis, S. (2010) Housing First: The Pathways Model to End Homelessness for People with Mental Illness and Addiction Minnesota: Hazelden.)) and systematic evaluations have led to Housing First becoming central to the Canadian and French homelessness strategies. Evaluation has been crucial to promoting the idea of Housing First, in demonstrating that Housing First works and in showing that Housing First can be cost-effective. However, evaluations of Housing First must be of good quality and should ideally contrast the Housing First approach with existing services, if the evidence is to be persuasive.
• Measuring outcomes systematically and carefully allows a Housing First service to assess how well it is performing. Good quality evaluation allows Housing First services to learn about any limitations in their support or housing provision, enabling improvements to be made.
• Evaluation showing good performance can help Housing First services ensure they have funding in place and help make the case for Housing First services to be expanded.
• Evaluating Housing First is the main means by which good practice and important lessons about providing Housing First can be learned. Conducting and sharing evaluations can be very useful for everyone involved in developing and providing Housing First services.

Evaluation presents risks as well as opportunities. Attention must be paid to how information on performance is collected, because an evaluation that is not well designed or properly conducted can undermine the case for an individual Housing First service and Housing First in general. Anyone undertaking an evaluation of Housing First needs to be clear that the evaluation, if it is properly conducted, will not report that a Housing First service is perfect. There will be at least some minor issues that need addressing and, while the rates at which Housing First will end homelessness are, on current evidence, usually very high compared to most other homelessness services, Housing First will not work well for absolutely everyone in all circumstances.

Evidence can certainly help support Housing First, indeed it can be crucial to ensuring that the idea is promoted and that Housing First services are sustainably funded. The use of good quality evidence has been fundamental to successfully promoting Housing First in North America. However, collecting evidence does present some risks because it can highlight limitations as well as successes. It is also important to note that while philanthropists, charities and governments will not expect Housing First to report perfect results, they may not always be persuaded by evidence, even if a Housing First service is very successful.

5.2.1. Process Evaluation

Process evaluation refers to exploring how a service works. This means understanding the philosophy of Housing First. Ensuring the design of a Housing First service is fully understood, i.e. how the Housing First service is supposed to work is a very important first step in evaluation. A key measure here is the level of fidelity (similarity) to the original Housing First model. Fidelity refers to the core principles of Housing First and the operational detail of the successful original model of Housing First.

5.2.2. Fidelity

Assessing fidelity is the starting point of an evaluation of Housing First. Fidelity refers to how closely a service follows the core principles of Housing First (see Chapter 2). If a service does not follow the core principles, it should not be regarded as Housing First and should not be evaluated as an example of Housing First.

Fidelity measurement can also be described as testing for paradigm or model drift (moving away from the original model), which is a fundamental principle of any service evaluation. This means making sure that the Housing First service being tested is close to the original service design, i.e. that a service has not drifted away from, or was never really close to, the core principles of Housing First. In evaluation, this is very important because it tells the evaluators and anyone hearing about the results of an evaluation whether or not a successful Housing First service, or a Housing First service with problems, had high or low fidelity with the core principles of Housing First. This is important because success or failure may both be heavily influenced by fidelity and it is crucial to understand whether, for example, poor results from a particular Housing First service could be explained by low fidelity. The evidence from Europe so far suggests that success in Housing First is linked to high fidelity with the core principles((Pleace, N. and Bretherton, J. (2013) The Case for Housing First in the European Union: A Critical Evaluation of Concerns about Effectiveness European Journal of Homelessness, 7(2), 21-41 http://housingfirstguide.eu/website/the-case-forhousing-first-in-the-european-union-a-critical-evaluation-of-concerns-about-effectiveness/)). Housing First services that follow the core principles, although they work in European countries with sometimes very different welfare, health, housing and homelessness systems, have all delivered good results in ending homelessness (see Chapter 1).

Fidelity tests exist in North America, are being developed for use in Europe and are also being developed and used in individual European countries. The operational details may vary, e.g. whether or not social housing is used, or whether a service employs an integrated multidisciplinary team, intensive case management or a combination of support (see Chapter 3 and Chapter 4). Operational details may also need to vary to allow for differences in context between European countries, e.g. differences in health, welfare and housing systems. However, adherence to the core principles of Housing First cannot vary if a service is to be viewed as high fidelity.

Examples of Housing First fidelity tests include:

• The Pathways to Housing First fidelity measure((http://www.housingfirsttoolkit.ca/sites/default/files/Revised_HF_Self-Assessment_Survey_12-23-13.pdf)).
• The Canadian At Home/Chez Soi programme fidelity measure((http://www.housingfirsttoolkit.ca/sites/default/files/AtHomeFidelityScale.pdf)).
• The Full Service Partnership (FSP) fidelity measure((http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4097835)).

5.2.3. Effectiveness Evaluation

The evaluation of effectiveness centres on what a Housing First service is achieving. This aspect of evaluation includes the progress that a Housing First service is making in terms of delivering the outcomes it is designed to deliver. It is also important for an evaluation to understand what the people using a Housing First service think about Housing First.

The evaluation of effectiveness starts by exploring the ways in which a Housing First service is delivered. Alongside understanding the structure of the Housing First service and observing how it works, this also involves mapping the range of partner agencies involved, how the service is funded and how the networks that a Housing First service relies on are structured and function. In order to understand the effectiveness of a Housing First service, it is very important to understand how the Housing First service is designed and how it operates. This involves understanding how a Housing First service is targeted, what it is designed to achieve and what the roles of the Housing First staff team are.

After assessing fidelity, an evaluation must explore the outcomes that a Housing First service is designed to achieve. This means testing whether or not a Housing First service is achieving what it is supposed to achieve, both in terms of outcomes and the views of the people using the Housing First service.

Exploring the effectiveness of a Housing First service has several dimensions:

• Promoting housing sustainment and a lasting exit from homelessness. This is sometimes also called housing retention.
• Enhancing the health and well-being of Housing First service users,
• Improving the social integration of Housing First service users.
• The cost-effectiveness of Housing First.

5.3.1. Use of Validated Measures

Validated measures are questions that have been repeatedly tested and found to produce consistent results. Validated measures can enhance the quality of an evaluation and mean that the results are more likely to be regarded as accurate. Validated measures can include:

• Validated questions on mental and physical health.
• Validated questions on quality of life.
• Validated questions on social integration and social support.

Some validated measures are widely used at national level, but there are also examples of measures that are used internationally. Some examples of validated measures include (note this list is illustrative only):

• The SF-12((The SF-12 health questionnaire is available at: https://www.hss.edu/physician-files/huang/SF12-RCH.pdf)) and SF-36((Available at: http://www.shcdenver.com/Portals/902/web-content/files/JamesGenuario/JG-health%20questionnaire. pdf)) measures of health and well-being.
• Lehman’s Quality of Life Interview((Pleace, N. with Wallace, A. (2011) Demonstrating the Effectiveness of Housing Support Services for People with Mental Health Problems: A Review London: National Housing Federation)) (QoLI).
• Quality-Adjusted Life Years (QALYs) used in Health Economics.
• The Self-Sufficiency Matrix((Available at: http://www.selfsufficiencymatrix.org/zrm-int.aspx)) (SSM) developed in the US and adapted for use in the Netherlands.
• The SAMSHA (Substance Abuse and Mental Health Services Administration) scale((Pleace, N. with Wallace, A. (2011) Demonstrating the Effectiveness of Housing Support Services for People with Mental Health Problems: A Review London: National Housing Federation. Vide supra)).

5.3.2. Key Questions for Evaluation

• Housing First has three sets of interrelated goals (see Chapter 2):
• Promoting housing sustainment and a lasting exit from homelessness (also known as housing retention).
• Enhancing the health and well-being of Housing First service users, including:
  • Mental health.
  • Physical health.
  • Limiting illness and disability.
  • Drug and alcohol use (where this has been an issue for someone using Housing First services).
• Improving the social integration of Housing First service users, including:
  • Gains in social support and self-esteem.
  • Engagement in community and civic life.
  • Structured and productive activity and economic integration.
  • Working on nuisance, criminal or anti-social behaviour (where this has been an issue for someone using Housing First services).
• The cost-effectiveness of Housing First, which has two dimensions:
  • The cost-effectiveness of Housing First compared to other homelessness services.
  • The cost offsets, i.e. savings, that Housing First can potentially generate for other types of service, e.g. Housing First can produce reductions in expenditure for health services because it changes the ways in which long-term homeless people use health services and their level of use of medical services falls.

Measurement of outcomes of Housing First centres on these three sets of goals. Successful outcomes for Housing First rest on achieving as much as possible for each individual service user. Housing sustainment is an achievement, but it is a limited achievement if someone is isolated, bored, feels stigmatised or is not experiencing improvements in their health and well-being. Equally, gains in social support are an achievement, but Housing First is not working well if a service user is not sustaining their housing as well.

Overall effectiveness for Housing First rests on achieving successes across a range of outcomes, related to housing, health, well-being and social integration. When successes are achieved, it is important to understand those successes in terms of the overall well-being and situation of each Housing First service user.

Evaluations of Housing First may also need to include an assessment of cost-effectiveness. This element of evaluation looks at the relative cost-effectiveness of Housing First compared to other models of homelessness services. Evaluating cost effectiveness can also include assessing whether Housing First generates wider savings in public spending.

5.3.3. Housing Sustainment

Housing sustainment can be measured in three main ways:

• Length of time a Housing First service user has lived in the same home. This approach has some advantages:
  • It is a simple measure that is instantly understandable. If a Housing First service user has been living in their home for a year, this is a clear indication of housing sustainment.
  • The measure gives an idea of housing stability, i.e. if Housing First service users are typically remaining in the first apartment they are housed in for a year or more, this indicates that housing is being very effectively sustained.
• Time spent in an apartment compared to time spent sleeping and living in other situations. This approach:
  • Provides a night-by-night measure of where Housing First service users are and allows relative changes to be recorded. For example, if someone were living rough (on the street) for three nights a week prior to using Housing First and living rough drops to one or two nights a month, there is a clear gain.
  • Can be hard to interpret unless very carefully recorded. It needs to be clear whether nights in an apartment are within the same apartment or not, or whether there was a reason for someone not to be in their apartment for a given number of nights.
• Individuals’ feelings about their homes. This approach:
• Enables assessment of how a Housing First service user feels about their home and how settled they are.
• Looks at the success of housing in a wider sense, including:
  • whether someone feels physically safe in their home;
  • whether their home is affordable;
  • whether their home has all the facilities they need;
  • whether their home is of an adequate standard (damp, poor repair or poor space standards);
  • views on the neighbourhood where their home is located;
  • how happy a Housing First service user is with their home.

5.3.4. Health and Well-Being

There are three ways to measure health and well-being:

• Use very basic measures based on people’s own judgement about how their health is and whether there are any changes in drug/alcohol use (where this is relevant).
  • Using basic measures of whether someone feels they are getting better or worse, in terms of their physical health, mental health and drug/alcohol use, is very simple.
  • Answers will be subjective, i.e. they will be influenced by an individual’s interpretation of their health and well-being, which may be more positive, or more negative, than the view a medical professional would take.
  • Answers cannot be compared systematically, because the information being collected is not consistent (Housing First service users will not all interpret their health and well-being in the same way as each other).
• Use validated measures of health and well-being. A validated measure is one which has been repeatedly tested and found to be accurate in recording health and well-being. An example is the SF-12 health questionnaire, which has been widely used in surveys and statistical research, which establishes basic information on physical and mental health. This approach:
  • Allows the collection of data that can be compared over time and across Housing First service users, because questions and responses take place within a clearly-defined and consistent framework.
  • Collects data that may carry more influence in the outside world, because they use recognised standards of measurement that have been tested.
  • Will be more complex and expensive to administer than just asking very simple questions about health.
• Employ external evaluation of health and well-being. Medical teams and psychiatrists could be used to test health and well-being among Housing First service users over time. This is feasible and is likely to generate evidence that is taken seriously by external agencies, but may be difficult to fund.

5.3.5. Social Integration

In some respects, social integration is the hardest of the various outcomes to measure:

• Social support, participation in community and civic life and the nature and extent of structured activity are very subjective. When two individuals receive the same levels of social support, one may report that they are isolated and bored, and the other may feel supported and happy.
• Social integration can be interpreted in different ways for different groups of people. In Europe, it is quite common to talk about the lack of ‘community’ in poor areas as a social problem, but not to view the lack of ‘community’ in rich areas as being a social problem. It is important not to impose an ideal of what a ‘citizen’ should be on people using Housing First, when most other citizens do not match that same ideal.
• Validated measures of social support are available, but this is an area where qualitative outcome measurement, i.e. talking to Housing First service users about their lives and level of social integration, may be the most effective way to collect information.
• Measurement of social integration must take into account the other needs, characteristics and experiences of Housing First service users. If many people using a Housing First service have ongoing, limiting illnesses, this will influence how much success can be achieved with economic integration.

Measurements of social integration might include the following:

• Social support
  • Is the user in contact with their family?
  • Is the user in contact with friends?
  • Do they have a partner?
  • Do they have esteem support, a sense they are valued by others, and what is their level of self-esteem?
  • Do they have access to instrumental (practical) support from friends, family and/or a partner?
  • Do they have sufficient social companionship?
  • Are there people they can ask for advice and/or talk to?
• Community and civic participation
  • Does a Housing First service user participate in community events?
  • What are their relationships with their neighbours like?
  • Do they socialise within their community?
  • Do they participate in social media focused on their community?
  • Do they vote?
  • Do they volunteer in their community?
• Structured activity and paid work
  • Does a Housing First service user participate in the creative or performing arts?
  • Are they in education or receiving training?
  • Are they volunteering (in any capacity)?
  • Are they participating in a work placement/work experience scheme?
  • Are they in paid work?

5.3.6. Cost-Effectiveness

The measurement of the cost-effectiveness of Housing First services is heavily reliant on access to good quality, detailed, data. It is possible to produce estimations of cost-effectiveness, but these are less influential than detailed information that clearly shows Housing First delivering effective services. It is important to note that cost-benefit analysis is a distinctive, highly detailed and complex form of economic evaluation which should not be confused with evaluation of cost-effectiveness. There are two basic tests of cost-effectiveness which can be used for Housing First or other homelessness services((Pleace, N., Benjaminsen, L., Baptista, I. and Busch-Geertsema (2013) The Costs of Homelessness in Europe: An Assessment of the Current Evidence Base Brussels: FEANTSA http://housingfirstguide.eu/website/the-costs-of-homelessnessin-europe-an-assessment-of-the-current-evidence-base)):

• Is Housing First achieving better results than existing homelessness services for the same level of spending and/ or for a lower level of spending?
• Is Housing First producing cost offsets((Ibid.)), i.e. reductions in expenditure, for other publicly funded services? For example, by ending long-term and repeated homelessness, Housing First may produce savings for emergency health services, mental health services, drug and alcohol services, the criminal justice system, welfare systems and other homelessness services. It is important to explore whether these savings are realisable, i.e. the reductions in long-term and repeated homelessness delivered by Housing First really do allow publicly-funded services to reduce spending.

Planning an Evaluation

When designing an evaluation, it is useful to look at how other Housing First services (or programmes or strategies using Housing First) have been evaluated and also to look at any criticism of those evaluations. The Internet is a good source of information and resources such as Google Scholar can provide information on the evaluations that have been conducted, with access to some free resources. Major evaluations of Housing First, which tend to be supported by large, publicly-funded organisations, often produce reports which are freely available on the Internet. Some guidance on evaluation is also available on the Canadian Housing First Toolkit((http://www.housingfirsttoolkit.ca/evaluate)).

Evaluation can be comparative, which can include experimental or randomised control trials, in which two exactly matched groups (a minimum of 100 people in each group is desirable) are monitored. One group uses Housing First and the other uses existing homelessness services. Over the course of a year or more, outcomes for those using Housing First are compared with those for homeless people using existing homelessness services. These comparisons are expensive to conduct, but produce high quality evidence if they are carefully designed and precise. Randomised control trials (RCTs) of this sort have been used to test the French and the Canadian Housing First programmes and have generally reported very positive results (see Chapter 1).

Housing First has also been evaluated using comparison-group, or quasi-experimental, research. Again, these evaluations compare one group using Housing First with another group using existing homelessness services, but the groups are not precisely matched and can be smaller. This kind of evaluation can still be influential, but is generally viewed as being less accurate.

Many evaluations of Housing First are observational, which means looking at the people using a Housing First service and assessing how effective the service is in addressing their homelessness, improving their health and well-being and promoting social integration (e.g. being part of a community, having social support from friends, family and a partner, see 5.3). While this approach to evaluation can produce useful and persuasive evidence, the lack of a direct comparison with other homelessness services can mean the results are seen as less convincing than evaluations using RCTs or quasi-experimental approaches.

It is very important to consider the resources and objectives of an evaluation carefully. This includes thinking through what the evaluation is testing, what arguments it may be used to support, how much time and money are available and the potential criticisms that might be made of the results. While RCTs are often described as the best possible form of evaluation, they can still be the subject of criticism and their results may be rejected, particularly if there is seen to be a problem with design or a lack of precision. An RCT cannot be done cheaply and will involve a lot of resources if it is going to be truly persuasive. Equally, a much cheaper way of evaluating, an observational approach, while it has limitations, can still be highly persuasive.

Another consideration is who will be responsible for an evaluation. An evaluation is less likely to be influential if it is produced by the organisation providing a Housing First service, than if an evaluation uses independent researchers. This is not to suggest that an in-house evaluation (an evaluation of Housing First services by the people providing the Housing First service) has no value. The evidence from a good quality in-house evaluation can still be influential. Nevertheless, the argument that an in-house evaluation will be less likely to record or report problems may be used to question the results of an in-house evaluation.

Evaluations should always include feedback from Housing First service users. Giving service users a clear voice should enable any deficiencies in Housing First services to be identified and corrected. Equally, when Housing First is performing well, service users will have a detailed understanding of good practice that can be learned from and shared. Ensuring that the people using Housing First have a voice in evaluation is useful for the following reasons:

• Homeless people are experts by experience; they understand their own needs and what support they require better than anyone else does. The views of service users on how well a Housing First service is working are a very important part of an evaluation. Both the strengths and any limitations of Housing First are best understood by talking to the people using the service.
• The direct experience of homeless people using Housing First, when Housing First is working well, is a powerful way of conveying the effectiveness of Housing First. Statistics can be used to make the case for Housing First, but that case can be made more powerfully when positive opinion from service users is combined with statistical evidence.

Using qualitative methods, i.e. talking to people using Housing First in an open way, which allows and encourages them to express their opinions, is the best way to learn from their experience. It is also possible to understand opinion through statistical surveys, but it is important that surveys are not designed solely by researchers without any consultation with the people using Housing First, who are likely to have useful views on the kinds of questions that should be asked.

How an evaluation is done depends on what the wider goals of Housing First are. For example, if Housing First is being tested for the first time in a particular country, region or municipality, it will make sense to use experimental (RCT) or comparative approaches to research. When it has not been used before, Housing First needs to be tested to see how well it performs when compared to existing homelessness services. Depending on the results of that evaluation, Housing First may then be used on a larger scale.

If the existing evidence is strong enough, either based on a local evaluation or the international evidence base, it may be decided that there is no need to comparatively evaluate Housing First services. Instead, evaluation can be mainly about outcome monitoring, to ensure that the Housing First service is performing as expected and to look for any problems.

Evaluation also needs to be proportionate. A relatively expensive evaluation, such as an RCT evaluation, is only really practical when looking at a large Housing First service or Housing First programme, not for testing a single, small Housing First service. This is because, to be robust, an RCT should involve at least 200 people (100 using Housing First and 100 using other services). It can still be very valuable to look at single Housing First services comparatively, but smaller-scale services with, for example, 20 service users can also be evaluated using quasi-experimental or observational approaches.