From Addressing Barriers to Learning,
Vol. 2 (1), Winter 1997
Emerging Issue - Challenge to the Concept of SYSTEMS OF CARE
The term system of care has become popular in discussing
comprehensive and collaborative approaches for serving youth
with serious emotional problems. It also has relevance to any
effort to provide cohesive assistance to clients. In a journal
article entitled "Delivering effective children's services in the
community: Reconsidering the benefits of system interventions" in
Applied & Preventive Psychology (v. 6, 1997), Mark Salzer and Leonard
Bickman examine research on systems of care. They conclude that while
systems of care produce important system-level changes, early results
suggest these systems changes do not impact clinical outcomes. They
argue that the primary direction to improving children's mental health
services should be through effectiveness research, in contrast to
continued large-scale investments in systems research and development.
We noted their argument in the February edition of our Center's ENEWS.
Given the status of these authors and the nature of the data they
muster to support their position, their views are likely to receive
considerable attention from policy makers. We think it essential that
this issue be discussed widely and all positions aired. Let us know
your thoughts, and we will try to synthesize and report back the
responses. (If you want to communicate directly with the authors,
contact Mark Salzer at the Center for Mental Health Policy, Vanderbilt
Institute for Public Policy Studies, 1207 18th Avenue South, Nashville,
TN 37212.)
Leonard Bickman was out of the country and our ENEWS just caught up to him.
His responses to Al Duchnowski and Mike Furlong's comments about the 1997
article by Salzer and Bickman are included below. Incidentally, we mistyped
Mark Salzer's name in the ENEWS (fortunately it was correct in our newsletter).
We apologize for the error and thank Bickman for bringing it to our attention.
Bickman embedded his responses within the text of the two sets of comments,
so we are reproducing them here in that format.
Albert J. Duchnowski stated:
"interested readers are referred to a special issue of the Journal
of Mental Health Administration, Winter 1996, Volume 28 (1). This issue
has 16 articles describing the study, one of which challenges the
interpretation of the results offered by Leonard Bickman and his
colleagues."
Bickman's response:
"It would be nice if the readers were aware that concerns were responded
to in that special issue and that Duchnowski does not refute them. In
addition a more comprehensive review of this study can be found in the
May 1997 issue of the American Psychologist.
Mike Furlong (From UCSB) stated:
"Regarding Bickman. One needs to examine his methods very carefully. They
are not without some serious questions (for example using T-scores from
the CBCL instead of raw scores)."
Bickman's Response:
"I wish the author of this comment could carefully explain what he means.
In our data set, the raw score and the T-scores correlate .97. This sort
of shoot from the hip criticism does the field little good. If Furlong
thinks there are serious questions then why doesn't he raise them in a
scholarly and serious fashion?"
Furlong:
"While we need to take the outcome of the Fort Bragg study into account,
there are some very fundamental issues regarding systems of care that need
to be resolved before one jumps to the conclusion that Bickman has. In the
Santa Barbara County System of Care, we are trying to play a small part in
examining the outcome for our data in a couple of ways:"
Bickman:
"Again, it is a little disappointing Furlong equates almost a decade of
research as 'jumping to a conclusion'."
Furlong continues:
"We have conducted cluster analysis of our early cases and found
four distinguishable clusters of youths who have entered our system:
Troubled, Troubling, Troubled and Troubling, and At-Risk. [Furlong has a
paper in press and on this and will report preliminary outcomes at the U.
of South Florida conference.] Our logic is that Bickman grouped all youths
together in examining outcomes by cluster. In addition, one obviously
needs to conduct analyses that examine if (a)services were provided that
address the specific needs of each youth and (b)if the outcomes show
improvement in the targeted areas."
Bickman's response:
"Cluster analyses are not complicated to do--we have done many, but they
have to be theoretically driven otherwise you have a fishing expedition.
There is no theoretical argument proposed by system change advocates that
says certain groups would be more affected by system level changes than
others. However, even a brief review of our studies would show that we
conducted several subgroup analyses with the same outcome - system changes
had no differential effects on client outcomes. Second, an extensive
implementation study was done that demonstrated that the system was
implemented as planned and as expected. Finally, I would challenge
Furlong or anyone else reading this to tell us how to determine if
"services were provided that address the specific needs of each youth".
We have tried several ways and there is no way to match needs with
services, if you follow basic scientific principles. But I would love to
be show that I am wrong on this."
With regard to target outcomes, again our analyses included two ways of
examining this. First we examined only the subscales that related to the
child's presenting problem, second we examined the poorest subscale over
time. These outcomes were similar to the broader outcomes.
Furlong:
"In addition, before one concludes that systems of care do not work,
one needs to show that the treatment provided for each client had fidelity
with the tenets of the system of care change theory. Bickman has not done
this."
Bickman:
"First, we did not conclude that systems of care do not work. We said
very carefully that systems affect system level outcomes such as access
and cost but are unlikely to affect clinical outcomes for several reasons.
I will again refer to the implementation study that has been published.
If Furlong means that we have not matched individual level services with
individual level needs then he is correct. The study was a system study
and not a services or treatment study so that was not something we could
measure. But even if we had the foresight and funds to do so we do not
know how to do this. At this time there is no system for even describing
services provided other than by location (e.g. home, office) and time
(hours, days). So the challenge to all of us is to be better able to
describe the services that are actually delivered in the real world."
Furlong:
"His work is analogous to early work that declared psychotherapy or personal
counseling ineffective. All types of theory were co-mingled with all types
of clients presenting problems and need. What Bickman has done is only a
first step in evaluating the conditions under which systems of care are
and are not effective."
Bickman:
"Yes, that is the nature of the real world. Yes, I agree more research is
needed. However, it needs to be directed more at the services level and
not look to system reforms to produce better outcomes in the face of
inadequate services."
Len Bickman
Bickman@ibm.net or
Len@psychiatry.uq.edu.au
(use both to make sure he gets the message)
University of Queensland,
Mental Health Centre,
Royal Brisbane Hospital,
Herston Queensland 4029 Australia
School Mental Health Project-UCLA
WebMaster: Perry Nelson (smhp@ucla.edu)