From Addressing Barriers to Learning,
Vol. 2 (1), Winter 1997

Emerging Issue - Challenge to the Concept of SYSTEMS OF CARE

The term system of care has become popular in discussing comprehensive and collaborative approaches for serving youth with serious emotional problems. It also has relevance to any effort to provide cohesive assistance to clients. In a journal article entitled "Delivering effective children's services in the community: Reconsidering the benefits of system interventions" in Applied & Preventive Psychology (v. 6, 1997), Mark Salzer and Leonard Bickman examine research on systems of care. They conclude that while systems of care produce important system-level changes, early results suggest these systems changes do not impact clinical outcomes. They argue that the primary direction to improving children's mental health services should be through effectiveness research, in contrast to continued large-scale investments in systems research and development.

We noted their argument in the February edition of our Center's ENEWS. Given the status of these authors and the nature of the data they muster to support their position, their views are likely to receive considerable attention from policy makers. We think it essential that this issue be discussed widely and all positions aired. Let us know your thoughts, and we will try to synthesize and report back the responses. (If you want to communicate directly with the authors, contact Mark Salzer at the Center for Mental Health Policy, Vanderbilt Institute for Public Policy Studies, 1207 18th Avenue South, Nashville, TN 37212.)

Leonard Bickman was out of the country and our ENEWS just caught up to him. His responses to Al Duchnowski and Mike Furlong's comments about the 1997 article by Salzer and Bickman are included below. Incidentally, we mistyped Mark Salzer's name in the ENEWS (fortunately it was correct in our newsletter). We apologize for the error and thank Bickman for bringing it to our attention.

Bickman embedded his responses within the text of the two sets of comments, so we are reproducing them here in that format.

Albert J. Duchnowski stated:
"interested readers are referred to a special issue of the Journal of Mental Health Administration, Winter 1996, Volume 28 (1). This issue has 16 articles describing the study, one of which challenges the interpretation of the results offered by Leonard Bickman and his colleagues."

Bickman's response:
"It would be nice if the readers were aware that concerns were responded to in that special issue and that Duchnowski does not refute them. In addition a more comprehensive review of this study can be found in the May 1997 issue of the American Psychologist.

Mike Furlong (From UCSB) stated: "Regarding Bickman. One needs to examine his methods very carefully. They are not without some serious questions (for example using T-scores from the CBCL instead of raw scores)."

Bickman's Response:
"I wish the author of this comment could carefully explain what he means. In our data set, the raw score and the T-scores correlate .97. This sort of shoot from the hip criticism does the field little good. If Furlong thinks there are serious questions then why doesn't he raise them in a scholarly and serious fashion?"

"While we need to take the outcome of the Fort Bragg study into account, there are some very fundamental issues regarding systems of care that need to be resolved before one jumps to the conclusion that Bickman has. In the Santa Barbara County System of Care, we are trying to play a small part in examining the outcome for our data in a couple of ways:"

"Again, it is a little disappointing Furlong equates almost a decade of research as 'jumping to a conclusion'."

Furlong continues:
"We have conducted cluster analysis of our early cases and found four distinguishable clusters of youths who have entered our system: Troubled, Troubling, Troubled and Troubling, and At-Risk. [Furlong has a paper in press and on this and will report preliminary outcomes at the U. of South Florida conference.] Our logic is that Bickman grouped all youths together in examining outcomes by cluster. In addition, one obviously needs to conduct analyses that examine if (a)services were provided that address the specific needs of each youth and (b)if the outcomes show improvement in the targeted areas."

Bickman's response: "Cluster analyses are not complicated to do--we have done many, but they have to be theoretically driven otherwise you have a fishing expedition. There is no theoretical argument proposed by system change advocates that says certain groups would be more affected by system level changes than others. However, even a brief review of our studies would show that we conducted several subgroup analyses with the same outcome - system changes had no differential effects on client outcomes. Second, an extensive implementation study was done that demonstrated that the system was implemented as planned and as expected. Finally, I would challenge Furlong or anyone else reading this to tell us how to determine if "services were provided that address the specific needs of each youth". We have tried several ways and there is no way to match needs with services, if you follow basic scientific principles. But I would love to be show that I am wrong on this."

With regard to target outcomes, again our analyses included two ways of examining this. First we examined only the subscales that related to the child's presenting problem, second we examined the poorest subscale over time. These outcomes were similar to the broader outcomes.

"In addition, before one concludes that systems of care do not work, one needs to show that the treatment provided for each client had fidelity with the tenets of the system of care change theory. Bickman has not done this."

"First, we did not conclude that systems of care do not work. We said very carefully that systems affect system level outcomes such as access and cost but are unlikely to affect clinical outcomes for several reasons.

I will again refer to the implementation study that has been published. If Furlong means that we have not matched individual level services with individual level needs then he is correct. The study was a system study and not a services or treatment study so that was not something we could measure. But even if we had the foresight and funds to do so we do not know how to do this. At this time there is no system for even describing services provided other than by location (e.g. home, office) and time (hours, days). So the challenge to all of us is to be better able to describe the services that are actually delivered in the real world."

"His work is analogous to early work that declared psychotherapy or personal counseling ineffective. All types of theory were co-mingled with all types of clients presenting problems and need. What Bickman has done is only a first step in evaluating the conditions under which systems of care are and are not effective."

"Yes, that is the nature of the real world. Yes, I agree more research is needed. However, it needs to be directed more at the services level and not look to system reforms to produce better outcomes in the face of inadequate services."

Len Bickman or
(use both to make sure he gets the message)
University of Queensland,
Mental Health Centre,
Royal Brisbane Hospital,
Herston Queensland 4029 Australia

Newsletter Menu
Home Page

School Mental Health Project-UCLA
WebMaster: Perry Nelson (