Scholarly article on topic 'Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT'

Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT Academic research paper on "Clinical medicine"

0
0
Share paper
Academic journal
BMJ Open
OECD Field of science
Keywords
{""}

Academic research paper on topic "Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT"

Open Access

Research

accessible medical research

Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT

Sebastian Voigt-Radloff,1 Maud Graff,2 Rainer Leonhart,3 Michael Hüll,4 Marcel Olde Rikkert,5 Myrra Vernooij-Dassen6

To cite: Voigt-Radloff S, Graff M, Leonhart R, et al. Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT. BMJ Open 2011;1:e000094. doi:10.1136/bmjopen-2011-000094

< Prepublication history for this paper is available online. To view these files please visit the journal online (http:// bmjopen.bmj.com).

Received 8 February 2011 Accepted 4 May 2011

This final article is available for use under the terms of the Creative Commons Attribution Non-Commercial 2.0 Licence; see http://bmjopen.bmj.com

For numbered affiliations see end of article.

Correspondence to

Sebastian Voigt-Radloff;

sebastian.voigt@

uniklinik-freiburg.de

ABSTRACT

Background: The positive effects of the Dutch Community Occupational Therapy in Dementia programme on patients' daily functioning were not found in a multicentre randomised controlled trial (RCT) in Germany.

Objectives: To evaluate possible effect modification on the primary outcome within the German RCT with regard to (1) participant characteristics, (2) treatment performance and (3) healthcare service utilisation; and (4) to compare the design and primary outcome between the German and the original Dutch study.

Methods: (1) The impact of participant baseline data on the primary outcome was analysed in exploratory ANCOVA and regression analyses. (2) Therapists completed questionnaires on context and performance problems. The main problems were identified by a qualitative content analysis and focus-group discussion. Associations of the primary outcome with scores of participant adherence and treatment performance were evaluated by regression analysis. (3) Utilisation rates of healthcare services were controlled for significant group differences. (4) Differences in the Dutch and German study design were identified, and the primary outcome was contrasted at the item level.

Results: (1) Participant characteristics could not explain more than 5% of outcome variance. (2) The treatment performance of some active intervention components was poor but not significantly associated with the primary outcome. (3) There were no significant group differences in the utilisation of healthcare resources. (4) In contrast to the Dutch waiting-control group, the active intervention in the German control group may have reduced group differences in the current RCT. The German patients demonstrated a higher independence at baseline and less improvement in instrumental activities of daily living.

ARTICLE SUMMARY

Article focus

■ Lessons to be learnt from a process evaluation alongside a pragmatic multicentre randomised controlled trial, specifically on the effectiveness of a Dutch community occupational therapy in Dementia programme within the German healthcare system, and more generally for cross-national transfer studies.

Key messages

■ Exploratory analyses of process data did not reveal a major bias within the German trial from the specific baseline characteristics of the participants, variances in the treatment performance or the utilisation of other healthcare resources during the treatment period.

■ Compared with the original Dutch study, the German sample showed more independence at baseline in items of instrumental activities of daily living and less improvement within an identical treatment period. The Dutch waiting-group design versus the German multicentre parallel group design with an active control arm, as well as the extensive training and expertise of the Dutch therapists might also help to explain the differences in the results from the Dutch and the German randomised controlled trial. To address these findings, future cross-national transfers of complex interventions should be prepared by small-scaled pilot studies assessing the applicability of the intervention, the appropriateness of inclusion criteria and the specific patient needs in the target country.

Conclusion: The differences in outcome may be explained by a more active control treatment, partially poor experimental treatment and less room for improvement in the German sample. Future cross-

Process evaluation in an RCT on a complex intervention in dementia

ARTICLE SUMMARY

Strengths and limitations of this study

■ The association of variance in treatment performance and primary outcomes was investigated alongside a German randomised multicentre trial. Data on the primary outcome were available at the item level from both German and Dutch original studies.

■ The exploratory statistical analysis of change to baseline in daily functioning in the German sample was based on data which showed variance comparable with the Dutch sample but were possibly limited by a restricted range at baseline. The quality of the treatment performance was measured only by the therapists' self-report.

national transfers should be prepared by pilot studies assessing the applicability of the intervention and patient needs specific to the target country.

Trial registration: International Clinical Trials Registry Platform, DRKS00000053.

INTRODUCTION

New guidance from the British Medical Research Council states that developing and evaluating complex interventions can be a lengthy process. All steps should be sufficiently addressed. These steps include (1) the development of the intervention, (2) a pilot study on feasibility, (3) a randomised controlled trial (RCT) on effectiveness and (4) an evaluation of implementation in healthcare practice.1 Cross-national transfer of complex intervention can speed up the uptake of innovative and effective programmes from one country to another. Time and resources might be saved when an intervention programme that has already been developed, piloted and evaluated on effectiveness in one country can be directly proven regarding effectiveness in the healthcare context of another country. We followed this approach by transferring the Dutch evidence-based Community Occupational Therapy in Dementia Programme (COTiD)2 to the German healthcare system and testing its effectiveness in a seven-centre RCT.3 However, the highly positive effects of the Dutch COTiD on patients' daily functioning could not be found. Process evaluation is recommended as being highly valuable in RCTs to provide an insight into unexpected intervention failure.4 Differences in participants as well as aspects of treatment performance and contextual factors should be assessed with regard to their associations with the primary outcome.5-7 Based on these recommendations, our process evaluation investigated four research questions. We evaluated both possible bias within the German study (question 1 e3) and differences between the Dutch and German RCT (question 4). 1. Did specific patient or carer characteristics influence a patient's outcome after the intervention?

2. What problems and variations in experimental treatment performance could be identified in the study context, and did they influence the daily functioning of patients?

3. What differences in the utilisation of further healthcare resources during the treatment period could be identified, and did they influence the daily functioning of patients?

4. What differences between the Dutch and the German study could be identified in terms of design and primary outcome?

METHODS

Specific participant characteristics of the German sample

The outcome of interest was daily functioning, indicated by two measurement instruments, the Interview for Deterioration in Daily Living Activities in Dementia (IDDD) and the Perceive, Recall, Plan and Perform System of Task Analysis (PRPP). The IDDD performance scale records the patients' need for assistance in 11 basic and instrumental activities of daily living.8 In the PRPP, the number of errors occurring during the performance of a self-chosen daily living task is measured.9 An ANCOVA was used to investigate the mean changes from baseline in the IDDD and PRPP between the COTiD and control group controlling for (1) the patient's age, gender, education and financial limitation; and the daily activities, mood and cognition at baseline; and (2) the carer's gender, education, relationship to the patient; and the sense of competence and mood at baseline. Data were collected with standardised measurement instruments as described in the study protocol and were in line with a recent health-technology assessment of risk or protective factors for Alzheimer's disease.10 11 The percentage variance explained in mean changes from baseline in the IDDD and PRPP was assessed using multiple regression.

COTiD performance in the German experimental group

Therapists completed semistructured questionnaires during and after the treatment period. (Questionnaires are available in German from the corresponding author.) During the treatment phase, they reported reasons for a problematic performance in 20 subpro-cesses for each experimental case. The subprocesses were defined according to the study protocol (table 1 and 4). After the treatment period, therapists described and rated their professional experience in the field and their valuation of introduction, pilot phase and supervision, as well as inhibiting and facilitating processes at the study site. A qualitative content analysis with inductive category development was used to identify the main performance problems from the comments given in the questionnaires.1216 A focus-group discussion served as a member check, in order to achieve consensus among the therapists about the main performance prob-lems.17 21

Furthermore, the therapists dichotomously scored the 20 treatment subprocesses as performed either with or

Process evaluation in an RCT on a complex intervention in dementia

Table 1 Statements by therapists stating main performance problems within the therapeutic subprocesses

Setting therapy goals

Educating patient in new skills

Adapting physical or social environment

Training of carer's competence

'Priorisation by the patient was difficult, because he was very uncritical.'

'The carer wants immediately to talk about problem solving. I again and again had to suggest

the procedure [of systematic shared goal setting].'

'Patient needs much guidance. Concentration and endurance [are] very limited. Assistance for simple tasks [is needed].'

'Activities agreed on could not be carried out twice due to apathy and depressive mood.'

'In addition, patient had dyspraxia, which made training difficult.'

'[There was a] lack of training due to the negative attitude of the carer.'

'It is difficult for the patient to accept the disease. Therefore a high degree of convincing is

needed in each session.'

'The carer is the house owner and refuses any adaptation.'

'[Adapting physical environment] does not succeed because the carer is ostensibly open for intervention, but in reflective talks reluctant and negative.'

'An adaptation [of the physical environment] seems not reasonable to the patient, although [it is] necessary.'

'[Adapting physical environment] is possible only step by step, because the patient reacts on it with reluctance.'

'The patient lives rather reclusively, wishes no changes [in the social environment].' 'The patient is very anxious and avoiding [change]'. 'The son strongly adheres to old patterns of interaction'.

'The family dynamic is very fixed. Both daughters seem to have difficulty in just letting the

mother [patient] simply do ... Changes take place, but very slowly. [It is] questionable,

whether there will be work on the goals after the intervention is finished.'

'In the community, there is no day care and no care centre for people with dementia.'

'The son [is] often not or only temporary present at the sessions.'

'[The carer is] many a time overstrained and tries to give away [the responsibility] to the

therapist.'

'It is difficult for the carer to get used to something new. He quickly falls back into old patterns [of behaviour] without being aware of it.'

'[The carer] seems to be very overstrained and burdened by the disease. He needs additional

professional support, for example, from a psychologist'.

'The carer has need for support, but refuses any offer of support for himself.'

'The carer mostly sees only his own problems. He cannot or only very rarely empathises with

the patient. Offers of support are refused.'

'There are difficulties in the interaction between the family and the patient. The patient plays off the caring family members against each other.'

without problems. These scores were used to oper-ationalise the quality of performance for each case. The best quality was indicated by 100% when all subprocesses were performed without any problems.

The therapists also rated patient adherence regarding the cooperation during the interview, the goal setting and the training; as well as regarding the patient's daily changing mental capacity, their collaboration with the carer and regarding the acceptance of innovations. Additionally, the carer adherence was assessed with regard to their cooperation during the scheduling, the interview, the goal setting and the training of supervision, as well as with regard to their encouragement of the patient, the acceptance of support service and the implementation of innovations. Therapists rated these indicators for adherence on a five-point-Likert scale ranging from 'very facilitating for the treatment performance' (=1) to 'very hindering for the treatment performance' (=5).

Correlations between the mean changes from baseline in the IDDD and PRPP and scores of the performance

quality and the participant adherence were calculated (Pearson coefficient). An exploratory regression analysis was deemed to be appropriate for smaller samples, and this was to evaluate whether such scores could explain variance in the mean changes from baseline in the IDDD and PRPP.22

Utilisation of healthcare resources in the German study

The Resource Utilisation in Dementia23 was applied to collect patient data during the treatment period on (1) the number of consultations with general practitioners, neurologists/psychiatrists and other medical specialists; (2) the time for individual therapy such as physio-, speech- or psychotherapy; (3) the time for group therapy such as cognitive stimulation or exercise groups; (4) times of receiving nursing or domestic home care; (5) the number of technical aids implemented within the patient's home; and (6) increasing, decreasing, constant or no intake of acetyl cholinesterase inhibitors. Furthermore, comorbidity indicating a possible need for further healthcare services was rated using the Cumulative Illness

Process evaluation in an RCT on a complex intervention in dementia

Rating Scale.24 These data were tested for the significance of group differences between the experimental and the control arm (non-parametric two-tailed Manne Whitney U test owing to the negative skewness of the data distribution).

Comparison between the Dutch and the German study

Regarding the patient characteristics and change from baseline to follow-up on the primary outcome, we compared the single IDDD items of the Dutch and the German sample within the identical measurement period of 5 weeks from baseline to the first follow-up measurement at week 6. This was in order to assess whether both samples had the same room for improvement in items which indicate the need for assistance in daily activities. Furthermore, we compared the expertise of the Dutch and German therapists in terms of pre-experience with the experimental intervention and intensity of treatment delivery (patients per therapist) and the study designs regarding the control-group intervention.2 3

RESULTS

Specific participant characteristics of the German sample

The mean changes to baseline in the IDDD and the PRPP were associated neither with carers' socio-demo-graphic or baseline assessment data nor with patients' socio-demographic data or baseline mini mental state (table 2). We found a minor correlation of mean changes to baseline in the IDDD with patients' mood at baseline (Cornell Scale for Depression in Dementia (CSDD)25; r=0.21; p=0.044). A stepwise regression analysis using patient and carer characteristics as listed in table 2 could not explain more than 5% of variance in

change over time in the patient's daily functioning. An adjusted ANCOVA using the patient's baseline values of the CSDD, the PRPP and the IDDD as independent variables did not yield any significant group differences in the dependent variables, which were the mean changes to baseline in the IDDD and the PRPP (results not shown). This indicated that after correction for baseline scores of mood and daily functioning, there were still no significant differences on the primary outcome in the German sample with moderate to good daily functioning at baseline.

COTiD performance in the German experimental group

Eleven therapists from seven study sites delivered the COTiD to 54 patients. The therapists' characteristics (table 3) varied in previous years in dementia care from 1 year part time to 11 years full time, in perceived facilitators from quite facilitating to slightly hindering and in the quality of treatment performance from 52 to 90% of optimal performance. The data did not provide stable patterns in the sense that many previous years in dementia care and high values of perceived facilitators did lead to a high quality of treatment performance or vice versa.

The quality of the subprocess performance (table 4) did also vary from receiving full medical information in 52 of 54 cases (96%) to successfully adapting physical environment in 24 cases (44%). Subprocesses relating to therapeutic active agents as identified by Graff et at5 were performed with no problems at only a low frequency with 76% for setting therapy goals, 46% for training of patient's skills, 44 and 46% for adapting physical and social environment, and 59 and 54% for

Table 2 Pearson correlation coefficient of specific participant characteristics and mean changes to baseline in the Interview

for Deterioration in Daily Living Activities in Dementia and the Perceive, Recall, Plan and Perform System of Task Analysis

(German completers of the Community Occupational Therapy in Dementia Programme and control group)

Perceive, Recall, Plan and Interview for Deterioration in

Perform System of Task Daily Living Activities in

N Analysis24 change to baseline Dementia23 change to baseline

Patient

Age 104 -0.02 0.11

Gender 104 -0.16 -0.11

Education 104 -0.13 -0.02

Financial limitation 93 0.07 0.14

Mood, Cornell Scale for Depression in 95 0.16 0.21*

Dementia25 baseline

Cognition, Mini-Mental State 104 0.02 0.10

Examination baseline

Gender 104 0.08 -0.11

Education 104 -0.12 0.11

Relationship to patient 104 0.15 0.09

Sense of competence, Sense of 103 -0.06 -0.02

Competence Questionnaire26 baseline

Mood, Center for Epidemiologic 103 0.09 0.09

Depression Scale27 baseline

*p<0.05; **p<0.0001 (two-tailed).

Process evaluation in an RCT on a complex intervention in dementia

Table 3 Characteristics of the 11 therapists who delivered Community Occupational Therapy in Dementia Programme to 54 patients with Alzheimer's (German completers of the experimental intervention)

Treatment

Basic data Perceived facilitators! performance

Years in Years in

occupational dementia Pre-existing Study Site Quality

Age Gender therapy care knowledge preparation support Total Cases (%)Z

27 Male 3 3 2.2 2.0 1.1 1.8 3 86

31 Female 8 5 1.6 2.0 1.2 1.6 3 89

45 Female 9 7 1.8 2.5 2.2 2.2 5 94

44 Female 7 1* 3.2 1.7 1.8 2.2 2 64

40 Male 13 13* 3.8 3.0 2.4 3.1 10 81

34 Female 5 3 3.8 2.5 2.8 3.0 10 90

54 Male 11 11 2.0 3.5 2.7 2.7 4 73

36 Female 11 9 2.3 2.7 2.6 2.5 1 52

39 Female 18 12* 2.4 2.5 2.9 2.6 2 74

40 Female 6 3* 2.4 3.3 2.9 2.9 2 59

32 Female 9 9* 4.2 3.3 3.4 3.4 12 84

*Part time.

tScored by therapists with 1=very much facilitating, 2=facilitating, 3=neutral, 4=hindering, 5=very much hindering. Z100%=all treatment subprocesses were performed without problems.

training of carer's competence in instruction and problem-solving. In the questionnaires, the therapists commented on the main performance problems in these subprocesses (table 4).

Association between COTiD performance and primary outcome

We found no significant associations between the scores of COTiD performance and changes to baseline in the IDDD and in the PRPP (detailed data not shown). Since there was a poor performance in those subprocesses which were related to active therapeutic agents (nos 14 to 20; table 4), we further analysed the association between the performance score of these subprocesses and the changes to baseline in the IDDD and in the PRPP. A minimal correlation was found, r=0.268 (p=0.05) only with the PRPP. No association was found between carer adherence and the changes to baseline in the IDDD and in the PRPP. The score of patient adherence and the change to baseline in the PRPP demonstrated a moderate correlation of r=—0.317 (p=0.02). The subsequent regression analysis revealed that patient adherence could explain 10% of the variance (p=0.02) in the PRPP change to baseline. The IDDD change to baseline could not be explained by patient or carer adherence, or by the quality of treatment performance.

Utilisation of healthcare resources in the German study

The COTiD group had a somewhat higher comorbidity index, slightly more visits to general practitioners and fewer hours for nursing or domestic home care. Negative skewness in the data distribution indicated that many participants had a low utilisation rate, and only a few participants had a high intensity of resource utilisation (table 5). However, we found no significant differences on group level within the German trial in any resource

utilisation or comorbidity. The subgroups of patients with decreasing or increasing acetylcholinesterase-inhibitor medication were too small to detect any significant group differences. However, the daily functioning in the COTiD group was not better than in the control group, although more COTiD patients received acetylcholinesterase inhibitors at a constant level (COTiD, 63% vs control, 52%).

Comparison between the Dutch and the German study

Differences in the room for improvement in the IDDD

Table 6 shows that the COTiD group in the Dutch sample did improve notably in household instrumental activities of daily living (IADL), only marginally in basic activities of daily living and not at all in handling finances. Graff et atr defined 20% improvement as being clinically relevant, which is indicated by a pre—post-treatment difference of 0.8 on item-level. The household IADL items demonstrated such differences and, therefore, a high responsiveness to the COTiD programme. Thus, the household IADL items can be presumed to be a therapeutic window basically providing room for improvement given a sufficient need for assistance in these items at baseline. Comparing the Dutch and German COTiD groups, the baseline values in these IADL items differed considerably more than in the other IDDD items.

The German patients showed much less need for assistance in this area. The limited room for improvement in the German sample is obvious when regarding the baseline differences between the Dutch and the German sample. Analysis of a German subsample matched to the Dutch sample with a comparable need for assistance in these household IADL items at baseline was not possible owing to the low number of German patients with such baseline values.

Process evaluation in an RCT on a complex intervention in dementia

Table 4 Quality of subprocesses of Community Occupational Therapy in Dementia Programme performance in 54

Alzheimer's disease patients (German completers of the experimental intervention)

Performance*

Subprocesses Good (%) Poor Main problems

01 Receiving medical information 52 (96) 2 Received wrong phone number or no detailed medical information

02 Making appointments with participants 49 (91) 5 Participants had other appointments

03 Travelling to participants 46 (85) 8 Long travel to patient's home (some >40 km)

04 Meeting the participants 50 (93) 4 Participants forgot to cancel the date and were late or not at home

05 Contacting and providing confidence 50 (93) 4 Patient was sceptic or abrasive

06 Informing about the procedure 50 (93) 4 Patient could not understand procedure, misunderstood procedure as test for nursing home placement

07 Observing the time frame 42 (78) 12 Participants (mainly carer) had a great need to tell and talk

08 Explaining clearly, responding to questions 50 (93) 4 Patient could not understand the explanations, owing to communication deficits or mood swings

09 Mastering conflicts and problematic situations 39 (72) 15 Patient had severe mood swings or additional cognitive deficits or was not aware of deficits; carer was overstrained, abrasive or placed sole responsibility on therapist; family conflicts existed for a long time

10 Interviewing patient with OPHI 38 (70) 16 Patient was unable or hardly able to tell, had anomia or severe deficits in biographic memory or was disorientated

11 Observing patient activity with Volitional Questionnaire, if OPHI not done 5 (71*) 2 Patient not motivated to demonstrate activities; *Volitional Questionnaire not necessary in 47 cases, because OPHI was done

12 Interviewing carer with Ethnographic Interview 47 (87) 7 Carer had only little understanding of dementia or felt very burdened

13 Observing activities of patient and carer 43 (80) 11 Patient did activity incompletely, was very passive or was fraught when being observed; carer was demanding or impatient

14 Setting therapy goals with patient and carer 41 (76) 13 Participants negated need for change or could not specify goals

15 Defining occupational therapy problems 43 (80) 11 Patient had no activity limitations; participants could not understand the relevance of problems; problems were very complex or became clearer only later during intervention or were related not to dementia but to depression or physical limitations

16 Educating patient in new skills and compensation capability 25 (46) 29 Patient was not or hardly motivated in training, additional symptoms such as dyspraxia, depression, apathy, attention deficit disorder hampered the training; carer or family were not supportive

17 Adapting physical environment 24 (44) 30 Participants refused or hesitantly accepted necessary adaptations

18 Adapting social environment 25 (46) 29 Participants were reluctant to change social environment; informal social support or care services were lacking

19 Training of carer's competence in instruction and interaction 32 (59) 22 Carer could not change behaviour as being very burdened or impatient or bound in firm habits; was not willing to take responsibility or was missing sessions

20 Training of carer's competence in problem solving 29 (54) 25 Carer was not willing to undertake the responsibility of problem solving or not able to do so owing to high burden; carer would have needed more time or further support to undertake the responsibility for independent problem-solving

*Number of cases, in which the performance of this subprocess was rated as unproblematic (=good) or problematic (=poor).

OPHI Occupational Performance History Interview.

Differences in design

The German trial design included a comprehensive consultation as active control intervention which approximately represents the non-pharmacological standard care in Germany. This was in order to evaluate

the possible benefits of COTiD in addition to standard care. Compared with the waiting-control-group design of the Dutch original trial, the German active control intervention may have reduced the group differences in daily functioning after the treatment. Compared with

Process evaluation in an RCT on a complex intervention in dementia

Table 5 Utilisation of further healthcare resources of patients with Alzheimer's during the intervention period of intense occupational therapy compared with a single session control intervention (German completers of the Community Occupational Therapy in Dementia Programme and control group)

Community Occupational

Therapy in Dementia Programme

(n = 54) Control (n = 50)

Healthcare resources Mean (SD) Range Skewness Mean (SD) Range Skewness

Medical consultations per week

General practitioner 0.28 (0.40) 0- 2.33 3.211 0.18 (0.17) 0- 0.67 0.828

Neurologist or psychiatrist 0.04 (0.09) 0- 0.33 2.092 0.03 (0.09) 0 0.50 3.508

Other medical expert 0.14 (0.26) 0- 1.33 2.778 0.15 (0.24) 0 1.00 1.825

Hours for therapy per week

Individual therapy 0.14 (0.32) 0- 1.00 2.029 0.11 (0.29) 0 1.00 2.597

Group therapy 1.05 (2.92) 0- 14.00 3.064 0.88 (3.32) 0 -16.00 4.163

Hours for nursing or domestic home care 1.33 (3.23) 0- -15.08 3.074 1.87 (5.32) 0 25.54 3.173

per week

No of technical aids provided at home 0.15 (0.49) 0- 3.00 4.306 0.06 (0.24) 0 3.00 3.821

Comorbidity (Cumulative Illness Rating Scale24) 3.15 (3.20) 0- 13 1.101 2.42 (2.60) 0 11 1.374

No of patients with acetylcholinesterase-inhibitor medication

De-novo treatment or increased dose 4 (7%) 3 (6%)

Decreased dose or medication ceased 2 (4%) 1 (2%)

Constant level 34 (63%) 26 (52%)

No acetylcholinesterase-inhibitors 14 (26%) 20 (40%)

the Dutch therapists, the therapists in Germany had less experience with COTiD before their study involvement (NL: 240 h vs GER: 0 h), less seminar and training time in the study preparation phase (80 h vs 40 h) and fewer COTiD patients per therapist during the treatment period (34 vs 5).

DISCUSSION

The process evaluation of our multicentre RCT on community occupational therapy in Alzheimer's disease revealed that the characteristics of the German participants at baseline did not mediate patients' daily functioning after treatment as indicated by the mean change to baseline in the IDDD and PRPP. Some subprocesses, which were deemed to be active components of the applied complex psycho-social experimental intervention, were performed poorly. However, variances in the performance were not associated with patients' mean change to baseline in the IDDD and PRPP. The utilisation of further healthcare resources was equal in the experimental and control groups. Based on exploratory analyses of process data, we can reject the hypothesis that group differences in participant characteristics and variances in the treatment performance or the utilisation of further healthcare resources had a confounding influence on the primary outcome within the German study sample. The analyses were limited by the restriction of range in the IDDD baseline data within the German sample. However, the variance of the IDDD baseline data in the German sample was higher than in the Dutch sample (German sample: experimental group, mean 15.4 (SD 9.9); control group, 14.1 (10.1); Dutch sample: experimental group 23.5 (7.9); control group, 24.5 (8.7)).

Using the same eligibility criteria, the German sample showed much less room for improvement in daily functioning than the Dutch sample. In Germany, patients' daily functioning at baseline was much better. Most German patients still performed better at the end of the study irrespective of group assignment than the successfully treated Dutch group, in which patients had lower baseline scores and improved significantly. This underlines the importance to pay attention to the needs of the patients and care givers specific to the target country.

In the German study, the self-reported performance of active intervention components was not associated with the primary outcome. The small sample size and the method of self-rating are limitations for detecting such associations. Although an exploratory regression analysis is vulnerable to misinterpretation,22 we also performed this type of analysis, in order to detect any signs of an influence of treatment performance on the primary outcome. However, we found only minor rates of correlation and explanation, which makes any meaningful association between variances in the treatment performance and the primary outcome unlikely. Self-rating can be a feasible approach for evaluating adherence in dementia research,28 in order to deal with limited resources.29 However, therapists tend to overestimate their own performance.30 Although the therapists were explicitly asked to be critical when judging their own performance, for further studies it is recommended that there be an additional external monitoring of treatment performance. This may reduce possible bias introduced by overestimation or overcriticism in self-rating. Furthermore, it might help to find appropriate onsite

Process evaluation in an RCT on a complex intervention in dementia

Table 6 Responsiveness of specific activities of daily living after an identical treatment period of 5 weeks in the Dutch and the

German sample

Dutch Community German Community

Occupational Occupational

Therapy in Dementia Therapy in Dementia

Programme Dutch control Programme German control

N Mean T0-T1 SD N Mean T0-T1 SD N Mean T0eT1 SD N Mean T0eT1 SD

IDDD items with low responsiveness in the Dutch sample

Washing oneself T0 55 1.42 0.67 1.37 57 2.04 -0.07 1.63 54 0.74 0.15 1.20 50 0.78 -0.12 1.27

T1 55 0.75 0.95 57 2.11 1.58 54 0.59 1.00 50 0.90 1.30

Dressing T0 55 1.73 0.71 1.47 57 2.09 -0.17 1.49 54 0.98 0.07 1.34 50 1.06 -0.06 1.32

T1 55 1.02 1.24 57 2.26 1.51 54 0.91 1.26 50 1.12 1.44

Combing one's T0 55 0.67 0.29 1.11 57 1.04 -0.05 1.40 54 0.70 0.00 1.16 50 0.60 -0.06 1.14

hair and brushing T1 55 0.38 0.87 57 1.09 1.43 54 0.70 1.19 50 0.66 1.15

one's teeth

Eating T0 55 0.18 0.03 0.67 58 0.43 0.03 0.98 54 0.28 0.02 0.74 50 0.40 0.04 1.07

T1 55 0.15 0.45 58 0.40 0.95 54 0.26 0.78 50 0.36 0.92

Using the toilet T0 55 0.42 0.17 0.88 57 0.96 0.12 1.48 54 0.37 0.07 0.85 50 0.58 0.22 1.09

T1 55 0.25 0.65 57 0.84 1.39 54 0.30 0.74 50 0.36 0.94

Handling finances T0 47 3.89 0.10 0.60 55 3.87 0.00 0.39 54 3.09 0.00 1.35 50 2.78 0.12 1.66

T1 47 3.79 0.69 55 3.87 0.61 54 3.09 1.35 50 2.66 1.60

Overall mean T0 55 1.33 0.34 0.73 57 1.73 -0.01 0.92 54 1.03 0.05 0.77 50 1.03 0.02 0.85

T1 55 0.99 0.59 57 1.74 0.88 54 0.98 0.78 50 1.01 0.84

IDDD items with high responsiveness in the Dutch sample

Making tea or T0 55 2.05 1.14 1.39 57 2.18 -0.26 1.45 54 1.33 0.13 1.66 49 1.04 0.20 1.46

coffee T1 55 0.91 0.91 57 2.44 1.31 54 1.20 1.59 49 0.84 1.21

Shopping T0 53 3.62 1.19 0.71 57 3.33 -0.06 1.19 54 2.00 -0.15 1.70 50 2.08 0.08 1.64

T1 53 2.43 1.17 57 3.39 1.18 54 2.15 1.71 50 2.00 1.60

Using the phone T0 55 2.04 1.04 1.53 54 2.28 -0.26 1.46 54 1.43 0.26 1.51 50 1.18 -0.24 1.29

T1 55 1.00 1.23 54 2.54 1.53 54 1.17 1.34 50 1.42 1.54

Preparing a meal T0 54 3.22 1.26 1.16 55 3.09 -0.18 1.44 54 2.15 0.11 1.80 50 1.92 0.30 1.77

T1 54 1.96 1.32 55 3.27 1.37 54 2.04 1.74 50 1.62 1.68

Cleaning the T0 54 3.15 1.43 1.24 56 3.18 -0.36 1.22 54 2.31 0.43 1.65 50 1.62 0.06 1.50

house or doing T1 54 1.72 1.27 56 3.54 0.93 54 1.89 1.64 50 1.56 1.59

minor repair work

Overall mean T0 55 2.81 1.21 0.78 57 2.81 -0.22 0.96 54 1.84 0.16 1.29 50 1.56 0.08 1.56

T1 55 1.61 0.74 57 3.04 0.86 54 1.69 1.24 50 1.50 1.19

IDDD, Interview for Deterioration in Daily Living Activities in Dementia (in each item the need for assistance is rated from 0=neverto 4=always).

T0, IDDD score on entry to the study; T1 , IDDD score at week 6 after 5 weeks treatment. T0-T1 20% improvement ( $0.8; shown in bold) is

defined as clinically relevant change.

coaching strategies. These strategies should aim at high-quality treatment performance, even though the complexity of psycho-social interventions induces variance—especially in multicentre RCTs.

Data on the association of treatment performance and primary patient outcomes, although encouraged,5 31 are scarce in RCTs studying complex interventions. Teri et a¥s implemented an external video rating of therapists' adherence to protocol but found no associations between this rating and any outcome variable. Similar studies in the field32 33 did not operationalise the impact of treatment performance on the primary outcome as the British Medical Research Council had strongly recommended.1 Within the original Dutch RCT, the large number of patients treated by two highly motivated therapists from the same study site suggests an excellent quality of treatment performance.2 However, its association with the patients' outcome was not quantified. The

experiences with the subsequent Dutch implementation of the guidelines revealed that novices had difficulties in adapting to this highly complex intervention in a concrete treatment setting. Prior to cross-national implementation of complex interventions, a successful national transfer from a single-centre setting with highly motivated specialists to a multicentre routine setting with therapists varying in competence and motivation seems to be appropriate.

CONCLUSION

Our process evaluation revealed that the participants in the study may have had insufficient need for the applied treatment and that active components of the complex psycho-social intervention were poorly performed. Also, an interaction can be considered in the sense that little need for assistance can make a less intensive, one-session treatment appropriate, as applied to the control group.

Process evaluation in an RCT on a complex intervention in dementia

These recent experiences suggest that cross-national transfers are best prepared by a pilot study in the target country exploring specific patient needs, the feasibility of inclusion criteria, the usability of measurement instruments and the applicability of the complex intervention by therapists in routine care settings.

Author affiliations

1Department of Occupational Therapy, Centre of Geriatric Medicine and Gerontology Freiburg, University Hospital Freiburg, Freiburg, Germany 2Alzheimer Centre Nijmegen, Scientific Institute for Quality in Health Care and Department of Rehabilitation-Occupational Therapy, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands 3Department of Social Psychology and Methodology, University of Freiburg, Engelbergerstr, Freiburg, Germany

4Section of Gerontopsychiatry and Neuropsychology, Centre of Geriatric Medicine and Gerontology Freiburg, University Hospital Freiburg, Freiburg, Germany

5Alzheimer Centre Nijmegen, Department of Geriatrics, Radboud University Nijmegen Medical Centre, Nijmegen, The Netherlands 6Alzheimer Centre Nijmegen, Scientific Institute for Quality in Health Care, Radboud University Nijmegen Medical Centre, Kalorama Foundation, Nijmegen, The Netherlands

Acknowledgements We thank all participants and therapists for their contribution. We acknowledge C Mayers, Research Fellow, Faculty of Health & Life Sciences, York St John University, UK, for critical reading and English correction.

Funding German Federal Ministry of Health, Reference Number: IIA5-2508FSB111/44-004.

Competing interests MOR: consultancy for Jansen-Cilag and Numico. MH: grants from Wyeth, Pfizer and Medivation; payment for lectures from Wyeth, Pfizer and Merz.

Ethics approval Ethics approval was provided by the Medical Ethics Committee of the University Hospital Freiburg (no 110/08).

Contributors SVR, MG, RL, MH and MVD contributed to the study conception and design. SVR conducted the data and study management, and prepared the statistical analysis. RL performed the statistical analysis. SVR drafted the manuscript. MG, RL, MH, MOR and MVD revised the manuscript critically for important intellectual content. All authors read and approved the final manuscript.

Provenance and peer review Not commissioned; externally peer reviewed.

Data sharing statement Data sets can be provided on request for fellow researchers in the context of collaborative projects and publications.

REFERENCES

1. Medical Research Council. Developing and evaluating complex interventions: new guidance. 2008. http://www.mrc.ac.uk/ complexinterventionsguidance (accessed 29 Jun 2011).

2. Graff MJ, Vernooij-Dassen MJ, Thijssen M, et al. Community based occupational therapy for patients with dementia and their care givers: randomised controlled trial. BMJ 2006;333:1196.

3. Voigt-Radloff S, Graff M, Leonhart R, et al. A multi-centre RCT on community occupational therapy in Alzheimer's disease: Ten sessions are not superior to one consultation. (prepared for submission).

4. Oakley A, Strange V, Bonell C, et al; RIPPLE Study Team. Process evaluation in randomised controlled trials of complex interventions. BMJ 2006;332:413-16.

5. Spillane V, Byrne MC, Byrne M, et al. Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. J Adv Nurs 2007;60:343-52.

6. Herbert RD, B0 K. Analysis of quality of interventions in systematic reviews. BMJ 2005;331:507-9.

7. Hawe P, Shiell A, Riley T. Complex interventions: how 'out of control' can a randomised controlled trial be? BMJ 2004;328:1561-3.

8. Teunisse S, Derix MM. The interview for deterioration in daily living activities in dementia: agreement between primary and secondary caregivers. Int Psychogeriatr 1997;(9 Suppl 1):155-62.

9. Chapparo C, Ranka J. The PRPP System of Task Analysis: User's Training Manual. Research Edition. Sydney: OP Network, 2006.

10. Williams JW, Plassman BL, Burke J, et al. Preventing Alzheimer's Disease and Cognitive Decline. Evidence Report/Technology Assessment No. 193. (Prepared by the Duke Evidence-based Practice Center under Contract No. HHSA 290-2007-10066-I.) AHRQ Publication No. 10-E005. Rockville, MD: Agency for Healthcare Research and Quality, April 2010. http://www.ahrq.gov/downloads/ pub/evidence/pdf/alzheimers/alzcog.pdf (accessed 29 Jun 2011).

11. Voigt-Radloff S, Graff M, Leonhart R, et al. WHEDA study: effectiveness of occupational therapy at home for older people with dementia and their caregivers-the design of a pragmatic randomised controlled trial evaluating a Dutch programme in seven German centres. BMC Geriatr 2009;9:44.

12. Hsieh HF, Shannons SE. Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277-88.

13. Mayring P. Qualitative Content Analysis. Forum: Qualitative Social Research. 2000. 1(2), Art. 20. http://www.qualitative-research.net/ index.php/fqs/article/view/1089/2386 (accessed 24 Nov 2010).

14. Vernooij-Dassen M, Wester F, Auf den Kamp M, et al. The development of a dementia process within the family context: the case of Alice. Soc Sci Med 1998;47:1973-80.

15. Graff MJL, Vernooij-Dassen MJ, Zajec J, et al. How can occupational therapy improve the daily performance and communication of an older patient with dementia and his primary carer? A case study. Dementia 2006;5:503-32.

16. Mayring P. Kombination und Integration qualitativer und quantitativer Daten. Forum: Qualitative Social Research. In German: 2001; 2(1), Art. 6. http://www.qualitative-research.net (accessed 29 Jun 2011).

17. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995;311:376-80.

18. Wong LP. Focus group discussion: a tool for health and medical research. Singapore Med J 2008;49:256-60; quiz 261.

19. Ivanoff SD, Hultberg J. Understanding the multiple realities of everyday life: basic assumptions in focus-group methodology. Scand J Occup Ther 2006;13:125-32.

20. Van der Vorm A, Rikkert MO, Vernooij-Dassen M, et al; EDCON Panel. Genetic research into Alzheimer's disease: a European focus group study on ethical issues. Int J Geriatr Psychiatry 2008;23:11-15.

21. Iliffe S, De Lepeleire J, Van Hout H, et al; DIADEM Group. Understanding obstacles to the recognition of and response to dementia in different European countries: a modified focus group approach using multinational, multi-disciplinary expert groups. Aging Ment Health 2005;9:1-6.

22. Hair JF, Black WC, Babin BJ, et al. Multivariate Data Analysis. 7th edn. Upper Saddle River, NJ: Pearson, 2010.

23. Wimo A, Nordberg G. Validity and reliability of assessments of time. Comparisons of direct observations and estimates of time by the use of the resource utilization in dementia (RUD)-instrument. Arch Gerontol Geriatr 2007;44:71 -81.

24. Salvi F, Miller MD, Grilli A, et al. A manual of guidelines to score the modified cumulative illness rating scale and its validation in acute hospitalized elderly patients. J Am Geriatr Soc 2008;56:1926-31.

25. Alexopoulos GS, Abrams RC, Young RC, et al. Cornell scale for depression in dementia. Biol Psychiatry 1988;23:271-84.

26. Vernooij-Dassen MJM, Persoon JM, Felling AJ. Predictors of sense of competence in caregivers of demented individuals. Soc Sci Med 1996;43:41-9.

27. Radloff LS. The CES-D scale: a self-reported depression scale for research in the general population. Appl Psychol Measurement 1977;1:385-401.

28. Teri L, McCurry SM, Logsdon R, et al. Training community consultants to help family members improve dementia care: a randomized controlled trial. Gerontologist 2005;45:802-11.

29. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 2005;6:134-47.

30. Carroll KM, Nich C, Sifry RL, et al. A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug Alcohol Depend 2000;57:225-38.

31. Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurs Res 2004;53:63-6.

32. Teri L, Gibbons LE, McCurry SM, et al. 2003. Exercise plus behavioural management in patients with Alzheimer disease. A randomized controlled trial. JAMA 2003;290:2015-22.

33. Gitlin LN, Winter L, Dennis MP, et al. A biobehavioral home-based intervention and the well-being of patients with dementia and their caregivers: the COPE randomized trial. JAMA 2010;304:983-91.

Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT

Sebastian Voigt-Radloff, Maud Graff, Rainer Leonhart, Michael Hüll, Marcel Olde Rikkert and Myrra Vernooij-Dassen

BMJ Open 2011 1: originally published online August 9, 2011 doi: 10.1136/bmjopen-2011-000094

Updated information and services can be found at: http://bmjopen.bmj.com/content/1/1/e000094

These include:

References Open Access

Email alerting service

This article cites 25 articles, 10 of which you can access for free at: http://bmjopen.bmj.com/content/1/1/e000094#BIBL

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

Receive free email alerts when new articles cite this article. Sign up in the box at the top right corner of the online article.

Topic Articles on similar topics can be found in the following collections

Collections Health services research (665) Neurology (229) Rehabilitation medicine (127) Dementia (6)

Memory disorders (psychiatry) (7) Clinical trials (epidemiology) (16) Other rehabilitative therapies (2)

To request permissions go to:

http://group.bmj.com/group/rights-licensing/permissions

To order reprints go to: http://journals.bmj.com/cgi/reprintform

To subscribe to BMJ go to: http://group.bmj.com/subscribe/