An Online Knowledge Base on the 27 Cooperative Agreement Projects: A Concept Paper for the Steering Committee to summarize findings, accumulate knowledge, and observations of HRSA/HAB's SPNS Cooperative Agreement Projects

Online Knowledge Base

An Online Knowledge Base on the 27 Cooperative Agreement Projects: A Concept Paper for the Steering Committee

G. J. Huba, Ph.D.
The Measurement Group

This memorandum discusses an Online Knowledge Base to summarize the findings, accumulating knowledge, and observations of projects comprising the 1994 HRSA SPNS Cooperative Agreement. The memorandum is in response to a request from HRSA to develop a general way of making as many evaluation results and background materials from the Cooperative Agreement Projects available as widely and as rapidly as possible. The Knowledge Base described below would be developed by the EDC in collaboration with the projects of the Cooperative Agreement, with the primary work done by TMG. The development of the Knowledge Base would be reviewed by appropriate members and committees of the Cooperative Agreement.

A Knowledge Base may be thought of as a "database of findings" or a database of "facts." Within the context of the HRSA SPNS Cooperative Agreement, the Knowledge Base could contain a variety of elements. Some examples of elements, or "facts," would be:

Quantitative results from a single project

Qualitative results from a single project

Observations from a single project

Quantitative results from a group of relevant projects (or all projects)

Qualitative results from a group of relevant projects (or all projects)

Observations from a group of relevant projects (or all projects)

Materials – including those for clinical practice and training – developed by a single project

Materials – including those for clinical practice and training – developed by a group of relevant projects (or all projects)

Within a Knowledge Base, it is important to differentiate a number of different items about each "fact." Some of the things that need to be presented about each "fact" are the following.

What is the basis for the "fact?" That is, what quantitative or qualitative data were used to draw the conclusion and how were these supporting data collected? How were the supporting data analyzed? How confident can we be in the findings? How reliable and generalizable do we believe the findings to be?

For a given statement of a "fact," who is the specific audience? Is this audience one of policy makers, medical professionals, scientific researchers, legislators, consumer groups, or other target groups? While the "fact" remains the same no matter which group is the target for the information dissemination, the mode of presenting information may change depending upon the method usually employed by that group to exchange knowledge.

What is the range of application for the "fact?" Can it be used to support one type of conclusion but not another?

How would the "facts" for the Knowledge Base be developed? We propose to develop the "facts" for the Knowledge Base in different ways for each of the major types of facts delineated above.

Quantitative results from a single project: Results would be abstracted from analyses developed by the individual project or through TMG analysis of the data from the project. Before results are put into the Knowledge Base, the individual project would have the opportunity to review and comment upon them. TMG – since it will be analyzing fairly parallel datasets from the projects – can conduct similar analyses in those projects which have parallel data while at the same time working with the differences in data and data collection methods among projects.

Qualitative results from a single project: Results would be abstracted from analyses developed by the individual project or through TMG interviews and/or focus groups with project staff and/or clients as well as the review of archival records from the project. Before results are put into the Knowledge Base, the individual project would have the opportunity to review and comment upon them.

Observations from a single project: Results from staff observations will be contributed by the project and formatted for the Knowledge Base by TMG. The project would have the opportunity to review and comment upon such materials put into the Knowledge Base.

Quantitative results from a group of relevant projects (or all projects): TMG would cluster projects by a variety of criteria including having similar programs, similar expected outcomes, and similar datasets, using the general logic of the modular evaluation strategy implemented for these 27 projects. These clusters of projects will be different for various issues (for instance, all projects with data on case management might be used for analyses related to case management practices; all projects with nutrition assessments might be used for another cluster; projects in which protease inhibitor therapies were used and outcomes measures are available might form another cluster). Clusters of projects will not be mutually exclusive and will vary as different issues are probed. TMG will conduct analyses and submit them for review and comment to appropriate projects before they are entered into the Knowledge Base.

Qualitative results from a group of relevant projects (or all projects): As issues are identified that lead to clusters of projects that can speak to certain types of strategies or outcomes, they will be clustered and qualitative data will be collected and synthesized by TMG from interviews and/or focus groups with staff and/or clients as well as the review of archival records from the projects. Before results are put into the Knowledge Base, they would be submitted to appropriate projects for review and comment.

Observations from a group of relevant projects (or all projects): Observations contributed by projects about various clinical, training, staffing, and infrastructure change issues will be synthesized by TMG into a consensus position. Such "consensus observations" would be submitted to appropriate projects for review and comment before being entered into the Knowledge Base.

Materials – including those for clinical practice and training – developed by a single project: Materials submitted to TMG will be formatted and submitted to that project for review and comment before being entered into the Knowledge Base.

Materials – including those for clinical practice and training – developed by a group of relevant projects (or all projects): Materials submitted to TMG will be formatted and submitted to the contributing projects for review and comment before being entered into the Knowledge Base.

How would the "facts" for the Online Knowledge Base be presented? The TMG web site is currently a preliminary and incomplete "pilot" version of the Knowledge Base we would like to ultimately have. The web site now has information of many kinds including quantitative and qualitative data from individual projects (captured in SPNS/Faxes, individual project presentations, and professional conference summaries). It also includes instruments, primarily those from the national evaluation, as well as certain clinical and training materials developed for individual projects. We have not, of course, started to analyze the national database, nor have we tried to systematically capture qualitative information about the projects that would supplement the emerging cross-cutting quantitative results.

The current organization of our web site is not intended to be a Knowledge Base, although much of the material currently contained there could be fairly easily reorganized to form a Knowledge Base with the addition of a lot of formal evaluation results on program outcomes, characteristics, and research findings. We propose to develop a special section of the TMG web site that would be the formal Knowledge Base.

The more complete Knowledge Base would: a) systematically look at the major trends in the quantitative data from the national evaluation both in a collective and an individual project way; b) supplement national evaluation data analysis results with project-specific data and analyses; c) capture qualitative information on many of the same issues handled with the quantitative data and organize it so as to explain and add context to the quantitative results; d) present resource and supplementary materials such as clinical manuals, training materials, data collection instruments and procedures, statistical analysis methods, detailed statistical printouts, and other information needed to explain the importance and robustness of the findings to different kinds of audiences; and e) present the findings and supporting evidence in different ways which are the most natural way of communicating for different constituent groups (e.g. policy makers, scientists, medical providers, community organization staff, academics, etc.).

We believe that a key to the first generation of the Knowledge Base would be to present it online on the World Wide Web where all of the different elements can be hyperlinked. That is, on reading about a fact, there could be a hyperlink to the detailed characteristics of the sample, another hyperlink to the exact statistical procedures used to generate the conclusion, another hyperlink to the instruments used to collect the data, another hyperlink to ways of estimating confidence in the result, another hyperlink to a bibliography or literature review on the topic, and another hyperlink to pictures of the program. From the Online Knowledge Base various paper materials could be developed including the equivalent of a loose-leaf volume which is the full Knowledge Base, a series of one-page fact sheets, various scientific publications and book chapters, and other printed materials depending upon the audience for whom the information is intended.

How would the "facts" for the Knowledge Base be organized? At the present time we do not have a "rigid outline" in mind for the way of organizing the Knowledge Base facts, and we do believe that a better outline will emerge as we assemble the facts for the Knowledge Base. As a preliminary suggestion, we believe that there may be five broad topics that would help organize much of the information we could present. These topics would be as follows.

What is innovative HIV programming (what are the creative elements) and what outcomes can be demonstrated from innovative HIV programming?

Needs assessment – What gaps in the continuum of effective services used to exist before this grant initiative but don’t anymore, partially as a result of these findings? What elements now appear to be the most important to address the next five years after the Cooperative Agreement?

What programmatic elements in our projects have improved the nature of HIV outcomes and how? How much of an effect can we expect to see from projects of a similar magnitude, and with similar resource levels, to these national demonstration programs?

What have our grantees learned about how to add programs or create them from scratch or supplement existing efforts?

"Mini-studies" on topics that suggest themselves from the quantitative and qualitative data we have in hand.

What are the special advantages of an Online Knowledge Base? We believe that major advantages of the Online Knowledge Base are: a) it is incremental and can be implemented in small steps; b) as more data are available or more analyses are done or additional qualitative results are developed, the Knowledge Base can become richer as it is edited and revised and additional hyperlinks are added to related information; c) the Knowledge Base is a relatively inexpensive way to present large volumes of information about the 27 Cooperative Agreement Projects; d) the viewer can choose to view the information from different perspectives including that of the policy maker, the medical professional, the scientist, etc., and these perspectives can be maintained by presenting the same information with more or less technical detail, written as bullets instead of as charts, with more or less supporting information, and in different kinds of language and supplementing technical "footnotes;" e) the Knowledge Base can be reorganized as time goes by so that the same analyses, background information, and materials are used to illuminate additional issues which may become important as medical knowledge progresses.

What is the role of TMG in the Online Knowledge Base? It is our belief that an Online Knowledge Base needs to be developed, at least in first draft, by a small group of individuals who have fairly broad information about, and perspectives on, the 27 constituent Cooperative Agreement Projects. Because of the nature of the EDC and its role in relationship to individual projects as well as the fact that we have visited 25 of the 27 projects (and 19 more than one time) as well as having designed much of the cross-cutting data elements, we think that we have a fairly good perspective on the major issues that cut across projects and the individual strengths the projects so as to be able to illustrate major points. We believe that one very good way to capture the individual strengths of the individual projects is for those projects to collaborate with the EDC in generating the quantitative and qualitative "facts" about their projects that will be the foundation of the Online Knowledge Base while at the same time helping shape a conception of the cross-cutting results. We want the development of the Knowledge Base to be a collaborative effort between TMG and the projects.

Appendix I

Some Possible Clinical and Training Materials for the Online Knowledge Base

Training manuals and protocols from all projects conducting training.

Clinical procedure manuals and protocols from all projects providing individual patient/client services.

Presentations made to provider and collaborator groups.

Fact sheets.

Patient/client newsletters and other materials intended for service recipients.

Computerized pictorial tours of individual treatment facilities.

Appendix II

Some Possible Data Analyses to be Undertaken for the Online Knowledge Base

We believe that there are literally hundreds of analyses that can, and should be, done in the national database. TMG is willing to put the resources into conducting as many of the important analyses as possible.

Rather than listing hundreds of these analyses, the following analysis framework is presented. The framework, in combination with the data that will eventually be available on the projects, defines the quantitative analyses that are probably possible and probably likely to be of interest. It is our intent to use qualitative indices and observations to supplement and explain the quantitative results.

Note that most of the analyses on individual clients need to be done separately – and contrasted – for groups disaggregated by major factors such as gender, risk category, current and past substance abuse, employment status, stage of disease, and other ways of naturally grouping the participants. Most of the analyses on individuals trained need to be done separately – and contrasted – for groups disaggregated by types of training, and types of trainees. Such disaggregation would be done in typical multifactorial designs (ANOVAs, MANOVAs) or hierarchical (linear, logistic, Cox) regression designs depending upon the variables analyzed. It is obvious that multi-level analysis (hierarchical linear models and the equivalent) needs to be done to statistically "partition" site-specific results from cross-site robust results.

The major analyses we need to do follow directly from the grid of available data shown as Appendix III. In Appendix III, we show the number of data modules we have received from each project. It should be noted that as of the current time, approximately half of the available data for analysis are from the three projects for which TMG is the local evaluator.

In designing the analyses for the Cooperative Agreement under the modular evaluation model originally developed three years ago, the key is to look at three levels of analysis. Level I analyses are those which use the data from a single module and simply determine the best ways to "scale" the data to support further, and more interesting, analyses. As basic methodological research, Level I analyses will be interesting to researchers and publishable ultimately in journals, but not of as much interest to program evaluators or to policy makers.

Level II analyses are those which use data from more than one module – with more being better than less – and determine the relationships among the different indicators. Level II analyses are ones which look at the relationships among major content domains (such as demographic factors and nutrition patterns or substance abuse patterns and the pattern of keeping appointments) but do not study change from a baseline to a follow-up period nor would they use any statistical or conceptual modeling techniques that might allow one to impute "causality" or "effects" of services on various outcomes.

Level III analyses, in general, will be those which relate patterns of variables from one, or many modules, over time so as to look at patterns which might be attributed to the programming (or change from baseline to follow-up). In certain cases, Level III analyses may look at data which are collected at a single time (without follow-up) but within the context of a statistical model which clearly imputes "causality" and demonstrates the effects of treatment on the measures in question. Level III analyses will be the most interesting to policy makers and others as they demonstrate the "outcomes" of the projects.

It is our hope to be able to do as many relevant Level III analyses as possible within the dataset. These are the major analyses and the ones that we would most want to post in the Knowledge Base. It makes the most sense to post these findings in a number of different formats including brief reports and bullet points familiar to policy makers, simple charts or fact sheets for consumers and providers, and more detailed technical tables and expositions for researchers and evaluators. We would wish the bulk of our efforts to be focused on the Level III analyses and their successful dissemination to many different kinds of information users. We also believe that there are certain Level II analyses that will be key to understanding how these programs worked and we wish to also include those in the set we would most focus upon. We would call these latter analyses Level IIb ones.

Level I and most remaining Level II analyses (termed Level IIa here) are necessary as technical background for key Level IIb and Level III analyses. We would like to "crunch" through as many of these analyses as possible and simply post them in the Knowledge Base in two very general forms: a) a highly technical summary complete with statistics suitable for those who scientists and investigators who wish to evaluate the validity of the claims and the Level III analyses which are based on the information from the Level I and II analyses; b) simple fact sheets and/or charts in a summary form for other users of the information.

Here are a very few examples of Level I, Level II, and Level III analyses. We can think of literally hundreds of such analyses and would wish to put as many of these in the Knowledge Base as possible.

Examples of Level I Analyses

What are the major typologies of clients served in terms of demographics, level of medical functioning, and risk category?

What are the major dimensions of quality of life among these clients with HIV?

What are the major typologies of individuals trained in terms of demographics, previous experience with HIV, and previous training experience in HIV issues?

What are the major scales appropriate for instruments designed to measure: a) depression; b) satisfaction with services; c) satisfaction with trainings; etc.

Examples of Level II Analyses

What are the relationships among current substance abuse patterns, utilization of services, and perceived quality of life?

What are the relationships among perceived quality of services and the array of services received?

What are the relationships among the receipt of case management and the receipt of medical services for substance abusing and non-substance abusing clients?

Examples of Level III Analyses

How do service levels (and the combination of services received) impact upon changes in quality of life, disease progression, and satisfaction with services? What is the relative cost-effectiveness of different combinations of services in terms of quality of life, disease progression, and satisfaction with services? Are these estimates the same for groups formed by gender, risk behaviors, and other key classification factors?

How do training methods (and the combination of trainings received) impact upon changes in knowledge, satisfaction with training, and future medical practices? What is the relative cost-effectiveness of different combinations of training in terms of changes in knowledge, satisfaction with training, and future medical practices? Are these estimates the same for groups formed by gender, professional background, and other key classification factors?

What is the effect of adding case management to the services continuum in terms of its impact upon changes in quality of life, disease progression, and satisfaction with services? Are these estimates the same for groups formed by gender, risk behaviors, and other key classification factors?

[Technical aside: We are aware that it is necessary to estimate the Level III analyses using multi-level analysis or hierarchical linear models in order to estimate and eliminate differences among sites.]