Skip to main content

Evaluation of headache service quality indicators: pilot implementation in two specialist-care centres

Abstract

Background

Evaluating quality of health care is increasingly recognized as an important contributor to the advancement of health-care delivery. We recently developed a set of quality indicators for headache care, intended to be applicable across countries, cultures and settings so that deficiencies in headache care worldwide might be recognized and rectified. These indicators themselves require evaluation and proof of fitness for purpose. This pilot study begins this process.

Methods

We tested the quality indicators in the tertiary headache centres of the University of Duisburg-Essen in Essen, Germany, and the Hospital da Luz in Lisbon, Portugal. Using seven previously-developed enquiry instruments, we interrogated health-care providers (HCPs), including doctors, nurses, psychologists and physiotherapists, as well as consecutive patients and their medical records.

Results

The questionnaires were easily understood by both HCPs and patients and were not unduly time-consuming. The results from the two headache centres were comparable despite their differences in structure, staffing and language. These findings met the purpose of the study.

Diagnoses were made according to ICHD criteria and critically evaluated during follow-up. However, diagnostic diaries and instruments assessing burden and response to treatment were not always in place or routinely utilised. Triage systems adjusted waiting times to urgency of need. Treatment plans included pathways to other specialities. Patients felt welcomed, reassured and educated, and were mostly satisfied. Discussion points arose over inclusion of psychological therapies in treatment plans; over recording of outcomes; over indicators of efficiency and equitability (protocols to limit wastage of resources, systems to measure input costs and means of ensuring equal access to the services); and over protocols for reporting serious adverse events.

Conclusion

This pilot study to assess feasibility of the methods and acceptability of the instruments of headache service quality evaluation was successful. The project is ready to be taken into its next stages.

Background

As a feature of health care, quality is obviously desirable. Evaluating quality of health care is increasingly recognized as a necessary link in the advancement of health-care delivery. Yet it is not always clear what “quality” is in this context, and how it is achieved or whether it has been. With regard to headache care, quality has not–until very recently–been defined; quality indicators for headache care that have been developed in the past, in the absence of any agreed meaning of “quality”, are limited to diagnosis and treatment in specific health-care settings, or to specific types of headache.

To address this, so that deficiencies in headache care worldwide might be recognized and rectified, the Global Campaign against Headache [1, 2] in collaboration with the European Headache Federation (EHF) brought together a service quality evaluation (SQE) group of health-services researchers and headache specialists. Their first completed task, after a literature review [3], was to develop a definition of quality, which they conceived as residing in nine separate domains (see Table 1):

Table 1 The nine domains of quality in a headache service (4)

“Good quality headache care achieves accurate diagnosis and individualized management, has appropriate referral pathways, educates patients about their headaches and their management, is convenient and comfortable, satisfies patients, is efficient and equitable, assesses outcomes and is safe” [4].

Quality evaluation would require application of sets of quality indicators for each of these nine domains separately, and formulation of these was the group’s second completed task [4]. They developed 30 in all, along with related assessment instruments, designed to be applicable across countries, cultures and settings.

Implementation requires the deployment of all of these, at various levels within health systems, most particularly in primary care, which is where most headache should be managed [5, 6]. But prior to this, a number of evaluation studies are needed–in multiple settings, countries and cultures, to demonstrate acceptability of the instruments, with low barriers to usage, and fitness for purpose. Evaluation will require interrogation of health-care providers (HCPs), including doctors, nurses, psychologists and physiotherapists, and of patients, which is not otherwise part of implementation.

Here we report the pilot evaluation study, undertaken in two specialist centres in Europe primarily to learn whether such an enquiry is feasible, and whether the questionnaires are easily understood by both HCPs and patients and not unduly time-consuming. The study continues the collaborative project (headache service quality evaluation) between EHF and Lifting The Burden (LTB) within the Global Campaign against Headache [1, 2], which is conducted by LTB in official relations with the World Health Organization.

Methods

The study was approved by the ethics committees of the University of Duisburg-Essen and Hospital da Luz. Informed consent was obtained from all study participants.

Study settings and subjects

The headache clinic of the University of Duisburg-Essen is a tertiary headache centre established within a major teaching hospital in Essen, Germany. Medical care is delivered as outpatient or day-clinic care by one senior physician, two residents, two psychologists and one physiotherapist supported by two nurses/secretaries. The centre is supported as necessary by other specialties within the hospital.

The headache clinic of the Hospital da Luz is a specialized centre within a private hospital in Lisbon, Portugal. Medical care includes a daily outpatient clinic, specialist support to the emergency department and inpatient care. One neurology consultant coordinates the centre, which works with physicians from multiple other specialities (physical and rehabilitation medicine, psychiatry, dentistry, gynaecology and obstetrics) and with other HCPs from the hospital, including psychologists, physiotherapists and nurses. The centre has one part-time secretary who coordinates care with the clinical secretaries of each speciality.

In each centre we interviewed the HCPs most involved in delivering the service, and studied consecutive patients and their records. Numbers of patients at each centre were determined by what were feasible.

Instruments

There were five instruments in total. Among these were three questionnaires: one each for doctors, other HCPs and patients. The last took the form of an exit questionnaire, which patients were asked to complete and return within 2 weeks. In addition, some items of information were extracted from the patients’ records and from central service records.

Table 2 provides an overview of the methods and instruments used. The indicators studied (column one) are grouped in their domains A to I as defined by the SQE group [4]. Column three (Means of enquiry) indicates the method of enquiry: e.g., by questionnaire or chart review. Column two (Measure) explains how each was assessed (many as “yes”/”no”, some quantitatively as continuous measures).

Table 2 Methods and outcomes of implementation of quality indicators in each centre

Questionnaires were translated from their English originals into German and Portuguese by three authors (ZK, CG and RG) according to Lifting The Burden’s translation protocol for hybrid documents [7], using forward and backward translation, with colleagues and patients not involved in the study providing review and assistance.

Data collection

HCPs and patients completed questionnaires anonymously. Data from patients’ records were extracted by a medical student (AS) in Essen and by a clinical research assistant in Lisbon. Information from central service records was supplied by the HCPs.

Analysis

Demographic and clinical data are provided as numerical values, percentages or mean values with standard deviations (SDs). No statistical tests were performed.

Results

In Essen we interviewed six HCPs: one senior physician, two residents, two nurses and one psychologist. We interviewed and reviewed the records of 89 patients, of whom 22 were male, 67 female, mean age 42.5 ± 12.7 years. Most (61) had migraine, 53 episodic and eight chronic; 13 had tension-type headache (TTH), 9 episodic and four chronic; 18 had medication-overuse headache (MOH), one had cluster headache (CH) and one had trigeminal neuralgia (TN). Five patients were given two diagnoses.

In Lisbon we interviewed 10 HCPs: one neurologist, one dental surgeon, two psychiatrists, two physiotherapists and four gynaecologists. We interviewed and reviewed the records of 50 patients, five male and 45 female, mean age 37.3 ± 11.2 years. Again, most patients (40) had migraine, seven had TTH, four had MOH and one each had new daily persistent headache, TN and orgasmic cephalalgia. Four patients were given two diagnoses.

On a general level, questionnaires proved easy to apply, were readily understood and accepted by both HCPs and patients, and not unduly time consuming. None of the specific enquiries caused or led to difficulties. Evaluation of each clinic according to the quality indicators is shown in Table 2. Despite the two different languages, German and Portuguese, the different settings and the very different structures of the two centres, the findings were very comparable. We summarise them by domain.

Domain A: accurate diagnosis

Diagnoses were made according to current ICHD criteria at both centres, documented after the first visit and reviewed after follow-up with the support of diagnostic diaries. Percentages of positive responses were ≥98 % in Essen and ≥92 % in Lisbon (Table 2).

Domain B: individualized management

A triage system to identify urgent cases existed in both clinics, but waiting-list times until first appointment were not documented. Mean time allocated to patients’ visits (according to patients’ reports) was 46 min for the first visit in Essen (with wide variation), but half that (and with much less variation) in Lisbon. Patient satisfaction (Essen 100 %, Lisbon 92 %) did not appear to be greatly influenced by this (Table 2). Treatment plans followed international guidelines. Lisbon made less use of psychological treatment. Disability measures existed in both clinics; disability was well documented in Lisbon (100 %), but not at all in Essen. In both clinics, only one third of patients were routinely followed to ascertain optimal outcomes (Table 2).

Domain C: appropriate referral pathways

Both clinics were well established with referral pathways for urgent and specialist consultations (Table 2).

Domain D: education and reassurance of patients

The great majority of patients expressed satisfaction (Essen ≥99 %; Lisbon ≥92 %) on both counts (Table 2).

Domain E: convenience and comfort

More patients (94–98 %) than HCPs (60–67 %) found the service environment clean and comfortable; most patients felt welcomed (Essen 100 %, Lisbon 94 %). Waiting times in the clinic varied quite widely around means of 20–23 min, and were unsatisfactory for a sizeable minority of patients (Essen 12 %, Lisbon 22 %) and 40 % of HCPs in Lisbon (Table 2).

Domain F: patient satisfaction

Overall satisfaction with their management was expressed by most patients (Essen 96 %, Lisbon 74 %) (Table 2).

Domain G: efficiency and equitability

Neither clinic had a protocol to avoid wastage of resources. Running costs were calculated, but the information was available to senior management only. Neither clinic was able to offer equal access to headache services for all who might need it.

Domain H: outcome assessment

Outcome measures were available in both clinics (in Essen only for symptom burden, in Lisbon also for disability), but they were used only in Lisbon. Neither clinic evaluated quality of life.

Domain I: safety

The service was safe in Lisbon, and probably also in Essen. Prescribed drugs were always well documented in both centres. In both, also, there were no recorded serious adverse events (SAEs), but the pilot identified absence in Essen of any formal protocol to ensure that SAEs were reported (Table 2).

Discussion

This was a pilot study to assess feasibility of the methods and acceptability of the instruments before committing resources to a large multicentre implementation study, which–still in the evaluation phase of the project–must be done next. The pilot was successful in this purpose: the questionnaires were found easy to apply and were readily understood and acceptable, and, importantly, the whole process proved not unduly time consuming. All this was so in both German and Portuguese languages, and in two centres with very different structures–one in a university teaching hospital and one in a private hospital.

Also in these two very different settings, findings with regard to quality were quite comparable. We can summarise these findings, while emphasising that here we were testing the concepts behind the quality evaluation project, not the centres, and we were certainly not comparing the centres. Triage systems were in place to adjust waiting times to first appointment to urgency of need. Waiting times to be seen were generally considered acceptable and most patients felt welcomed, reassured and satisfied. Diagnoses were made according to current ICHD criteria, and were critically evaluated during follow-up visits. Treatment plans included psychological therapies (variably utilised) and pathways to other specialities. Outcomes were assessed on symptom measures (such as headache intensity and frequency) and on disability burden in Lisbon. However, instruments to assess outcomes were not always in place, or not routinely utilised.

Both services would be considered to have failed on indicators of efficiency and equitability. Protocols for limiting wastage of resources did not exist; input costs were measured, but the personnel utilising resources were not informed of them; equal access to the services was not ensured.

The services appeared safe, but we say more about this below.

There are two lines of thought worth further discussion. One is to look at these findings and, where problems appeared, question whether these truly reflected issues of quality. The other is to identify the next steps in evaluation of the quality indicators.

Pursuing the first of these, we see no issues arising in domain A. In domain B, the centres differed markedly in their utilisation of psychological management approaches (100 % at Essen, 32 % at Lisbon), although both had access. They differed even more in recording disability in patients’ notes in order that management plans might reflect this (0 % at Essen, 100 % at Lisbon). We will come back to whether or not these are issues of quality. Both centres showed evidence of routinely following-up only one third of patients. In primary care, the expectation would be close to 100 %, whereas specialist-care practice entirely appropriately sends many patients back to primary care, with detailed advice for follow-up. Domains C and D raised no issues, interestingly in the latter case since it recorded patient satisfaction, which is notoriously fickle. That fickleness may be evident in domain F: Lisbon may be disappointed with an overall patient satisfaction rating of 74 % in the face of much higher scores elsewhere (although only 78 % for waiting times–domain E). Do patients really attach so much more importance to waiting time than to time spent with the doctor (92 % satisfied), information received (92 %) and reassurance (94 %), cleanliness and comfort (94 %) and a warm welcome (94 %)? When developing these indicators, the SQE group found it impossible to exclude patient satisfaction [4], but it can be difficult to understand what are its determinants. In fact, waiting times were clearly a problem at Lisbon, since the HCPs expressed even less satisfaction with them (60 %), but only to a limited extent are they within the control of clinicians. These quality indicators are equally for service managers, who have responsibility for resource allocation and for protocols avoiding wastage of resources, and who, perhaps, can do something to promote equitable access.

The SQE group did not incorporate clinical outcomes themselves into the quality indicators because they found no objective basis for stipulating what outcomes should be considered optimal in individual cases in particular settings [4]. Instead they called for evidence that outcomes were recorded against recognised outcome measures. Neither centre apparently attached great importance to this: Lisbon made some use of recognised outcome measures in two thirds (68 %) of cases, Essen not in any. Both centres did, nonetheless, record outcomes in terms of symptoms–in particular, headache frequency–in patients’ records. The quality issues related to this are threefold. First, accurate recording of outcomes serves patient follow-up and guides achievement of best outcomes. Second, it is part of good record-keeping, failure of which would be clear evidence of a quality deficiency. However, recognised outcome measures are not needed for either of these purposes: any system that has meaning to its users may be sufficient, including simple but careful recordings of symptoms in patients’ notes. But the third is this: how does a service know what its outcomes are; how can it assess itself against benchmarks; how can it improve (or even know whether it needs to improve)? This purpose is quite different, and requires that outcomes are formally recorded, in a standard manner, and it is for this purpose that recognised outcome measures are mandatory.

Since no outcome measures are yet universally accepted, it is part of the service quality evaluation project to suggest those that might be used and include them in the evaluations. The HURT questionnaire was developed as an outcome measure specifically for the purpose of achieving best outcomes [8]. While it was intended primarily for non-specialists, it has much broader utility [9], reflected in the fact that both centres had it available. Lisbon made use of it for the majority of their patients. For disability burden, the HALT questionnaire [10], like MIDAS from which it is derived [11], records lost productive time–which correlates strongly with disability. The two instruments have been widely validated, and HALT is available in multiple European translations. As for quality of life, while WHOQoL-8 [12] was suggested, there is no evidence that this is useful as an outcome measure in headache management. Neither is there good support for any alternative. Future studies may conclude that this indicator should not be retained.

Finally, on the matter of safety, the issue again is one of record-keeping. It may be that a protocol for reporting SAEs at Essen had not been required because none had occurred, but managers should have recognised the need to have one in place. It is not a minor point. Essen would have been considered to have failed on this indicator: it could not be known for certain whether there had been no SAEs at Essen or whether they were merely not recorded.

The second line of thought is to consider the next steps in developing the methodology and instrumentation of headache service quality evaluation. Following this pilot, the quality indicators will be implemented in many centres in a Europe-wide study supported by the European Headache Federation and Lifting The Burden, still in specialist care. The protocol will be the same, with the seven instruments used to evaluate performance. This extended study will serve two purposes. First it will confirm (or not) that the indicators are fit for purpose, with or without some degree of refinement. Second, by establishing what is majority practice, it will guide the setting of benchmarks against which quality may be judged. Uncertainties, such as the extent to which psychological approaches should be used in management, and whether and to what extent management plans should be based on disability assessments, on which Essen and Lisbon clearly differed, may not belong in indicators of quality. We shall look for consensus in this study. There will be both qualitative and quantitative benchmarks to be set. The danger, of course, is to assume that majority practice correctly sets the benchmark; but that is a matter for these future studies.

The stage after that will be to take the process into non-specialist care–primary care in particular. It is envisaged that the finally-agreed quality indicators should not themselves be varied when taken into primary care, but the benchmarks may be different.

Conclusion

This pilot study to assess feasibility of the methods and acceptability of the instruments of headache service quality evaluation was successful. The project is ready to be taken into its next stages.

Abbreviations

CH:

Cluster headache

EHF:

European Headache Federation

ICHD:

International classification of headache disorders

HCP:

Health-care practitioner

LTB:

Lifting The Burden

MOH:

Medication-overuse headache

SAE:

Serious adverse event

SD:

Standard deviation

SQE:

Service quality evaluation

TTH:

Tension-type headache

TN:

Trigeminal neuralgia

References

  1. Steiner TJ (2004) Lifting the burden: The global campaign against headache. Lancet Neurol 3:204–205

    Article  PubMed  Google Scholar 

  2. Steiner TJ (2005) Lifting The Burden: The global campaign to reduce the burden of headache worldwide. J Headache Pain 6:373–377

    Article  PubMed Central  PubMed  Google Scholar 

  3. Peters M, Perera S, Loder E, Jenkinson C, Gil Gouveia R, Jensen R, Katsarava Z, Steiner TJ (2012) Quality in the provision of headache care. 1: systematic review of the literature and commentary. J Headache Pain 13:437–447

    Article  PubMed Central  PubMed  Google Scholar 

  4. Peters M, Jenkinson C, Perera S, Loder E, Jensen R, Katsarava Z, Gil Gouveia R, Broner S, Steiner T (2012) Quality in the provision of headache care. 2: defining quality and its indicators. J Headache Pain 13:449–457

    Article  PubMed Central  PubMed  Google Scholar 

  5. Steiner TJ, Paemeleire K, Jensen R, Valade D, Savi L, Lainez MJA, Diener H-C, Martelletti P, Couturier EGM, on behalf of the European Headache Federation and Lifting The Burden: The Global Campaign to Reduce the Burden of Headache Worldwide (2007) European principles of management of common headache disorders in primary care. J Headache Pain 8(suppl 1):S3–S21

    PubMed  Google Scholar 

  6. World Health Organization, Lifting The Burden (2011) Atlas of headache disorders and resources in the world 2011. WHO, Geneva

    Google Scholar 

  7. Peters M, Bertolote JM, Houchin C, Kandoura T, Steiner TJ (2007) Translation protocol for hybrid documents. J Headache Pain 8(suppl 1):S45–S47

    Google Scholar 

  8. Buse DC, Sollars CM, Steiner TJ, Jensen RH, Al Jumah MA, Lipton RB (2012) Why HURT? A review of clinical instruments for headache management. Curr Pain Headache Rep 16:237–254

    Article  PubMed  Google Scholar 

  9. Westergaard MLS, Steiner TJ, MacGregor EA, Antonaci F, Tassorelli C, Buse DC, Lipton RB, Jensen RH (2013) The Headache Under-Response to Treatment (HURT) questionnaire: assessment of utility in headache specialist care. Cephalalgia 33:245–255

    Article  PubMed  Google Scholar 

  10. Steiner TJ (2007) The HALT and HART indices. J Headache Pain 8(suppl 1):S22–S25

    Google Scholar 

  11. Stewart WF, Lipton RB, Kolodner KB, Sawyer J, Lee C, Liberman JN (2000) Validity of the Migraine Disability Assessment (MIDAS) score in comparison to a diary-based measure in a population sample of migraine sufferers. Pain 88:41–52

    Article  CAS  PubMed  Google Scholar 

  12. Schmidt S, Mühlan H, Power M (2005) The EUROHIS-QOL 8-item index: psychometric results of a cross-cultural field study. Eur J Public Health 16:420–428

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This study was supported by the European Headache Federation and by Lifting The Burden, a UK-registered non-governmental organisation conducting the Global Campaign against Headache in official relations with the World Health Organization.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zaza Katsarava.

Additional information

Competing interests

ZK and RJ are directors and trustees of the European Headache Federation. ZK, RJ and TJS are directors and trustees of the Lifting The Burden.

No author has other conflicts of interest relevant to the subject matter of this manuscript.

Authors’ contributions

ZK and TJS conceived the study and developed the protocol with input from RGG and RJ. ZK, SS, RGG, RJ, CG and AS collected and analysed the data. ZK and TJS drafted the manuscript. All authors were involved in critical review of the first and subsequent drafts, and approved the final version.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0), which permits use, duplication, adaptation, distribution, and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Katsarava, Z., Gouveia, R.G., Jensen, R. et al. Evaluation of headache service quality indicators: pilot implementation in two specialist-care centres. J Headache Pain 16, 53 (2015). https://doi.org/10.1186/s10194-015-0537-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s10194-015-0537-1

Keywords