Nic and Cji Quality Assurance Manual Implementing Effective Correctional Management in the community.pdf

• Topics: Sentencing, Parole

Download original document:
Brief thumbnail

This text is machine-read, and may contain errors. Check the original document to verify accuracy.

Implementing Evidence-Based Practice
in Community Corrections
QUALITY
ASSURANCE
MANUAL

December 27, 2005
1

TABLE OF CONTENTS
I. Introduction ……………………………………………………………………………..…….3
II. Acknowledgements …………………………………………………………………………..4
III. Quality Assurance Plan Development ……………………………………………………..5
IV. Peer Review……………………………………………………………………………….….9
Assessment Scoring and Inter-Rater Reliability…………..….… 10
Motivational Interviewing Critique………………………………15
Cognitive Behavioral Treatment Group Facilitation……………21
Case File Review……………………………………………….…..22
V. Quality Assurance Indicators………………………………………………………………32
VI. Customer Satisfaction………………………………………………………..…………….41
VII. Program Evaluation……………………………………………………...……………….50
Formative Evaluation…………………………………..………… 51
Process Evaluation…………………………………………………51
Outcome Evaluation…………………………………………….…52
Logic Models……………………………………………………… 54
VIII. Individual Performance Measurement……………………………………..………….56
IX. Bibliography………………………………………………………………..……….………62
X. Appendix…………………………………………………………………...………………...63

INTRODUCTION
This document was developed as part of a multiyear cooperative agreement between the National
Institute of Corrections (NIC) and the Crime and Justice Institute (CJI) for an initiative entitled
Implementing Effective Correctional Management in the Community. The purpose of this
initiative is to assist state systems in applying an integrated approach to the implementation of
evidence-based principles in community corrections. The project model, designed by a National
Project Team of researchers, consultants, and practitioners, maintains an equal and integrated
focus on three domains: evidence-based principles, organizational development, and
collaboration. The project vision is to build learning organizations that reduce recidivism
through systemic integration of evidence-based principles in collaboration with community and
justice partners.
The Integrated Model incorporates eight evidence-based principles that, when implemented with
fidelity, have been shown to reduce offender recidivism. One of the greatest challenges in
implementing evidenced-based practices is ensuring program fidelity. Ensuring that the
assessments and other tools are reliable and valid and that programs are accurately replicated
requires the same level of planning and staff commitment as program implementation. All too
often agencies put systems and programs in place that have proven efficacy but because of
implementation flaws, these same systems and programs fail to deliver projected results. This
can result in the “baby being thrown out with the bathwater.” Agency staff becomes disheartened
because they have changed their systems and programs but see little difference in outcomes.
Quality assurance programs are designed to support implementation efforts and to ensure
accurate replication and implementation. This manual provides a simple and straightforward
approach to implementing a quality assurance plan. An overview of the principles and
components of a quality assurance plan is provided below. Because each jurisdiction has
different goals and capacities, this is not a rigid, step-by-step formula for quality assurance.
Rather, it presents the basic components of a quality assurance plan and provides options for
developing and implementing the plan. Each jurisdiction can tailor its quality assurance plan to
meet its own unique needs.
A comprehensive quality assurance plan is an invaluable tool in implementing evidence-based
practice. The plan provides a clear blueprint of the organization’s goals and how they will be
achieved. Quality assurance should be incorporated into the implementation of evidence-based
practice from the outset, with the goal of creating a “culture of quality” in the organization. Use
this manual as a reference throughout the process to establish and achieve goals for quality.

ACKNOWLEDGEMENTS
The Crime and Justice Institute would like to acknowledge the following individuals for
their contributions to this manual:
Authors:
Meghan Howe, The Crime and Justice Institute
Lore Joplin, The Crime and Justice Institute
Contributors:
Brad Bogue, Justice Systems Assessment & Training
Nancy Campbell, Campbell Consulting
Mark Carey, The Carey Group
Elyse Clawson, The Crime and Justice Institute
Dorothy Faust, National Institute of Corrections
Kate Florio, The Crime and Justice Institute
Sally Kreamer, 5th Judicial District Department of Correctional Services, Des Moines, IA
Kim Sperber, Quest Consulting Network, LLC.
Bill Woodward, William Woodward Associates

______________________
This document was developed under a cooperative agreement award from the National Institute
of Corrections, Community Corrections Division, U.S. Department of Justice. Points of view in
this document are those of the authors and do not necessarily represent the official position of the
U.S. Department of Justice.

QUALITY ASSURANCE PLAN DEVELOPMENT
OVERVIEW
A commitment to quality assurance can be highly beneficial to an organization, but the creation
and implementation of a quality assurance plan requires effort and attention to detail.
Maintaining quality is a project in and of itself within a larger program, and should be afforded
the same level of planning and staff commitment that would be given to any other significant
project: a project manager, a committee of stakeholders, and a detailed work plan with timeline
for implementation. Subsequent sections of this manual provide greater detail on implementing
components of a quality assurance plan, but an overview of the principles and components
necessary for success is provided below.
PRINCIPLES
All key stakeholders must be committed to ongoing quality assurance, with the goal of
creating a “culture of quality.”
The process of quality assurance requires a great deal of collaborative effort to succeed.
Staff at all levels of the organization must be committed to the process of measuring and
maintaining quality on an ongoing basis.
Specific measurable outcomes and their indicators must be precisely defined.
To accurately measure progress, everyone must be on the same page about what quality
means and how it will be measured. To compare data over periods of time, definitions
must remain consistent. Therefore, it is important that precise, constant, and useful
measures are carefully defined at the outset.
Appropriate information management systems must be in place.
Staff members who are expected to report data need the means to do so quickly and
easily. This includes an efficient system for maintaining records on individual offenders,
as well as an efficient reporting system. In order for the data collected to be useful,
qualified staff and appropriate technology must be in place to compile and analyze data,
and present it in a concise, comprehensible format to stakeholders and decision-makers.
Data should be incorporated into ongoing practice.
Quality can only be improved if procedures are in place to incorporate quality assurance
data into quality improvement practice. Once data is collected and analyzed, timely
decisions must be made about how this data will impact policies, procedures, and
expectations at all organizational levels.

COMPONENTS
• Convene a program steering committee.
Many individuals and programs are impacted by the decision to implement and monitor
evidence-based practices in community corrections. Each affected individual will have different
needs, concerns, and ideas related to implementation and monitoring of processes. To be sure
that diverse viewpoints are represented and diverse ideas are brought to the table, the quality
assurance process should be overseen by a committee rather than an individual. It will not be
possible to involve every stakeholder in the decision making process. However, steering
committee members should be carefully chosen for their ability to represent the viewpoints of
various constituents, and to exercise leadership in implementing various phases of the project.
In addition, a project lead should be chosen to manage the quality assurance efforts. The
quality assurance process is a substantial project in and of itself, and an experiences manager and
leader is required to see it through.
Choosing Indicators

• Develop a program logic model.
A logic model is a schematic diagram that outlines what
resources are being put into a program, what is expected
to happen in a program, and what outcomes are expected
in the program. Creating a logic model compels decision
makers to clearly define the steps of the program, and
also to answer the questions of why the program should
work (i.e. why would a job training program likely lead
to more jobs for offenders?) and how the outcomes
would be measured (i.e. how will we know if more
offenders are employed?). Once completed, a logic
model provides a quick reference of “the big picture” for
all stakeholders, a guideline for program implementation
and evaluation, and a benchmark for program success.
Logic models are discussed in more detail in the Program
Evaluation section of this manual.
•

Determine the program’s outcomes, and the
indicators of outcome achievement.
Stakeholders cannot know if they’ve achieved quality
without a definition of what quality is. It is important to
define the goals of the program (outcomes), and then to
define how those goals will be measured (outcome
indicators). For example, is a successful outcome for
offenders with substance abuse problems staying clean,
not recidivating, or both? If the chosen outcome is

Often, there are many
indicators that could be used
to measure a certain
outcome. When choosing
which to measure, consider
the following factors:
• Timeline
Some data will be available
more quickly than others.
• Ease of Reporting
The easier the data is to
gather and report, the more
likely it will be reported
quickly and accurately.
• Predictive Value
Some data is more accurate
than others in truly informing
progress towards a longerterm outcome, i.e. there is a
true correlation between the
two factors. This can be
obtained through existing
literature, or can be
measured over time.

recidivism, then will that be measured only during probation, or for a period of time after
probation? Will recidivism include new arrests, or only new charges? These definition
decisions must be made before any data can be collected.
• Develop an action plan for measuring key indicators.
Implementing a data system for measuring indicators is a complex, multi-step task that will only
be successful with careful planning and accountability for implementation. Once indicators have
been chosen, an action plan should be created to measure each indicator. The action plan should
include steps for creating, implementing, and maintaining the data collection system, including
who is in charge of each step and a timeline for completion.
SAMPLE ACTION PLAN
INDICATOR: Number of properly completed offender assessments in case files.
IMPLEMENTATION
STEP

TIMELINE

PERSON
RESPONSIBLE

Develop Peer Review form
for reviewing assessments.

06/04

Mary Smith

Develop tracking form.

06/04

Mary Smith

Discuss and amend form with
Tx Team and QA Team

07/04

Mary Smith

Train Peer Review Team on
using the review form.

07/04-08/04

Ricardo Vasquez

Develop sampling methodology
for selecting case files

06/04-08/04

Andrew Morgan

Educate officers on the Peer
Review process

08/04-09/04

Ricardo Vasquez,
Sarah Stein

Train officers on conducting
assessments.

01/04-08/04

Ricardo Vasquez,
Sarah Stein

Complete pilot sample and discuss
with Peer Review Team

08/04-09/04

Mary Smith
Andrew Morgan

Amend procedures, forms

09/04-10/04

Mary Smith

•

Educate stakeholders on the program’s desired outcomes, and the processes that will be
used to achieve those outcomes.
Organizational change can be a time of confusion and stress as well as pride and excitement, and
the atmosphere in the organization is heavily dependent on how the process is managed.
Everyone who will be participating in the process must be kept informed on the changing
organizational process and goals, and what their roles are. These stakeholders must be given the
opportunity to learn the new information, ask questions, and express their opinions if they are
ultimately expected to commit to the process.

•

Develop a procedure for peer review of staff performance and progress towards
outcomes.
Peer review provides a supportive environment to assess the progress of individual officers and
departments, and to provide feedback and coaching to promote quality improvement. For the
process to work effectively and efficiently, procedures must be in place for the selection and
training of peer reviewers, and for regular review and feedback for officers.
• Develop a procedure for assessing customer satisfaction.
Interventions intended to reduce recidivism are only useful if the offenders involved benefit from
the supervision and treatment that is provided. For example, if an offender did not find a
substance abuse group relevant to his or her addiction issues, then it is unlikely that he or she
will incorporate any information or skills from the group into his or her lifestyle. (This is true
whether the lack of interest is a result of the program content or the offender’s lack of
motivation. Either way, there is something that is preventing the offender from benefiting from
the program.) Therefore, it is important to survey offenders to determine their satisfaction with
probation services and the extent to which they benefited. This survey data offers one measure
of the effectiveness of service provision.
• Conduct ongoing program evaluation.
Program evaluation encompasses the measurement of outcome indicators, discussed above, as
well as process measures, which measure program implementation. The two types of data taken
together determine whether a program is meeting its goals, and which components of the
program are operating effectually or ineffectually in pursuit of that goal. Both types of measures
are important in the implementation of existing evidence-based practice or the creation of new
evidence-based practice.
• Conduct ongoing appraisals of staff performance.
The performance of individual staff has a significant effect on the quality of services that are
being provided. Ongoing staff appraisal keeps both staff and supervisors informed on the level
of an employee’s performance, areas of strength and weakness, and ongoing training needs.
When appraisal is ongoing, supervisors have the ability to provide regular feedback and coaching
to the benefit of the employee and the department.
• Incorporate quality assurance data into practice.
A key role of the steering committee or a designee must be to determine how the data will be
used. For example, if data indicates that less that half of offenders who should be referred to
substance abuse treatment are receiving treatment, does that mean that officers need more
training on determining treatment needs, or policies on referral follow-up need to be revised, or
that more partnerships with treatment providers need to be developed? Decisions must be made
about how to determine what the root cause of the issue is, and what steps will be taken to
remedy it.

PEER REVIEW
Overview
Ongoing quality assurance requires periodic review of the supervision practices and services
being provided to offenders. This review compares actual practices and service provision to the
benchmarks of quality established by the organization. A review can be conducted by someone
internal or external to the organization. An internal peer review process can be highly beneficial
when staff are well-trained in the process of peer review and industry standards for effective
assessment. This process employs a cross-section of staff, as opposed to only supervisors or
management, to conduct case file and service audits. Peer reviewers may participate voluntarily
or be assigned to the role, and the assignment may be ongoing or on a short-term, rotating basis.
Internal reviewers are familiar with the officers, the offender population, and departmental
procedures, and have a sense of the context in which the review is being conducted. In addition,
when deficiencies are identified, officers may be more accepting of constructive criticism and
coaching from a peer. Implementing the process can be difficult if officers are suspicious of the
process and how the information is going to be used; therefore, it is important to involve staff in
the process of creating a culture of quality in the organization.
A variety of methodologies can be used for the peer review process, but they each involve a
standardized review process to record data on key indicators. In the implementation of the eight
guiding principles for reducing recidivism, four components of peer review are instrumental:
review of scoring and inter-rater reliability on assessment tools; critique of motivational
interviewing skill balance; evaluation of cognitive-behavioral treatment groups, and review of
case files. These processes can identify achievements and deficiencies and inform quality
improvement efforts regarding individual and organizational capacity for accurate risk
assessment, the building of intrinsic motivation in offenders, and the provision of or referral to
appropriate treatment.
Principles
An internal review process must be peer-driven.
A cross-section of staff must be involved in the entire process, from determining the
relevant outcomes to designing the assessment tool to analyzing results. Input from
various staff levels will increase the relevance of the process and the results, as well as
increasing staff commitment.
The process must be support and coaching-oriented.
To reduce resistance and increase the chance of success, staff should view peer review as
an opportunity for professional development, not as a punitive process. Feedback from
the process should be supportive and constructive, and staff should be given the
opportunity to learn, practice, and be coached to improve performance.

The process should create a culture of learning.
The implementation and evaluation of quality, evidenced-based practice is ongoing, so
staff members never reach the point of “perfection.” This idea of a never-ending process
may be frustrating for some, so the peer review process must create an environment that
promotes the value of ongoing learning and continuous improvement.
The process should include a feedback loop.
Peer review is only useful if the data is applied. Therefore, the process needs to be
designed so that individuals and workgroups receive well-organized, timely data that can
be applied to practice. Practitioners must be able to communicate changing data needs to
the peer review team, as well as request additional feedback and evaluation as needed.

Components
Assessment Scoring and Inter-Rater Reliability
Many validated assessment tools are available for determining an offender’s risk of recidivism,
as well as identifying the criminogenic needs that influence recidivism risk. When used
correctly, assessment tools can identify individual supervision and treatment goals for an
offender, as well as defining risk levels for a group of offenders. Most tools are administered by
interview, and there is a significant potential for error when the interviewer is inexperienced or
poorly trained. For the individual offender, this could mean assignment to inappropriate levels
of supervision and treatment, and potentially an increase in recidivism potential. On a larger
level, inaccurate scoring affects the statistical measures of risk in a population, and skews
decisions about risk level and resource allocation.
There are two important factors to consider in quality scoring: reliability and validity:
•

Reliability is the extent to which everyone interprets questions and scoring in the same way.
For example, if fifty offenders are asked how many previous offenses they have committed,
some may interpret the question as meaning all of the offenses that they have committed,
while others may interpret it as only those for which they were arrested. When the question
can be interpreted in different ways, it lacks reliability. Inconsistent scoring also affects
reliability. If an offender lists all of his offenses and some scorers count them all and others
count only convictions, then the scoring lacks reliability.

•

Validity describes whether or not a tool truly measures what it is supposed to measure. For
example, offenders who are scored as “low-risk” on an assessment should have lower
recidivism rates than those who are “high-risk.” If the assigned risk level is predictive of the
offender’s behavior, then the tool is valid. Because each population has a different
distribution of risk levels, an instrument’s validity must be reassessed with every new
population (i.e. the instrument must be “validated”).

The first component of reliability is ensuring that all officers are trained to administer and score
the assessment in the same way. Ongoing reliability can be measured by case file review, direct
observation of interviews and scoring, or tape review of an interview. The resources available
and the type of information desired will influence the review method chosen. For example,
interviewing skills cannot be assessed through a case file review, but the file review may be
faster and less expensive than tape review.
Peer reviewers can check for several components of reliability:
• Was the interview guide used?
• Were all appropriate questions asked, and were complete responses recorded?
• Did the officer demonstrate good interviewing skills (open-ended questions, etc.)?
• Were the answers verified when possible (arrest records, other data sources)?
• Did the scoring reflect the answers given?
Peer reviewers must also be accurate and precise in their review process (for example, different
reviewers might have different definitions of “good” interview skills). Therefore, reviewers
should receive standardized training and be given a detailed checklist of what and how to assess.
The amount of time and resources devoted to scoring reviews will vary. The system may consist
of a random sample of interviews selected on a monthly or
quarterly basis in order to assess a cross-section of
interviews. Alternately, all officers could be assessed when
Maintaining
they first begin administering the assessment (either once or
inter-rater
multiple times), and then on a yearly basis thereafter.
reliability ensures
Regardless of the sampling methodology, the officers should
that assessment
receive the results of the assessment, and have opportunities
for feedback, coaching, additional training, and reinterview and
assessment, if necessary.
scoring is the

same for all
offenders,
regardless of the
interviewer.

Inter-rater reliability is a measure of whether or not different
officers would score an assessment in the same way. This is
important so that all offenders in the same community are
having their risk level and treatment needs assessed in the
same way. The results of an assessment should depend only
on the offender, not on the probation officer to whom he or she is assigned. For example, if
Officer A consistently scores offenders 4 points higher than Officer B, then Officer A will likely
have several more offenders that are considered medium and high risk, and they will be
supervised at a higher level than if they were on the caseload of Officer B. This makes it nearly
impossible to consistently match offenders to appropriate supervision and treatment.
Inter-rater reliability can be measured very easily by presenting multiple officers with the same
interview, having each score the assessment, comparing the scores, and discussing and resolving

any discrepancies. This process should be done in all assessment training, and can be used for
ongoing review in two ways:
•

To measure inter-rater reliability in a training environment, officers can view and score
a validated sample interview. The interview could be viewed on a videotape or DVD, or
interactively online. The scores can be submitted to the peer review team for
comparison to the validated scores.

•

To measure scoring in the field, officers could record an interview and submit the tape
and their scoring. A team of peer reviewers could independently score the interview
and compare their ratings to the officer’s.

After the review, the officer should be provided with feedback and coaching from the peer
review team or a designated coach, with opportunities for training and re-assessment. If there is
significant diversity of answers, policies and procedures may need to be revised, and training on
a wider scale may be needed to clarify scoring expectations.

Observation: In Person, Audio or Video?
Officers can be observed in three ways: direct observation during an interaction
with a client; an audiotape of an interaction; or a videotape. Any of the methods
can convey the basic information needed for an evaluation, but each method has
its benefits and drawbacks. It is up to the agency to decide which method is best
for providing the desired information while limiting the intrusiveness of the
observation. In some cases, there are industry standards for review. For example,
the standard for MI reviews is audiotape, while review of cognitive behavioral
treatment groups requires video or in-person observation.
Direct Observation
• May be more expensive if reviewer travel is required.
• No technological concerns (i.e. tapes with poor audio, etc.).
• More reviewer control of which interaction is reviewed.
• Only one opportunity to view and take notes on the interaction.
• Often considered most intrusive by officer and offender.
• Relationship with reviewer may impact officer’s behavior.
Audio
• Often considered less intrusive than video.
• Less expensive technology than video.
• Only allows evaluation of words and verbal cues.
• Allows for replay and feedback review with officer.
• Officer has more control over which interaction segment is reviewed.
Video
• Often considered more intrusive than audio.
• Most expensive technology.
• Allows for evaluation of officer’s words, verbal cues, and body language.
• Allows for replay and feedback review with officer.
• Officer has more control over which interaction segment is reviewed.

Sample: Interview Audit Form
Reviewer:

Date:

Interviewer:

1 = Poor 2 = Fair/Needs Improvement 3 = Good 4 = Excellent
1. Explanation of the purpose of the interview.

2. Established structure for the interview.

3. Adequate use of open-ended questions.

4. Avoidance of double-barreled questions.

5. Avoidance of biased/leading questions.

6. Adequate use of follow-up questions.

8. Interviewer overcame problems such as silence or excessive talking.

9. Interviewer used the interview guide.

10. Notes were made indicating why items were or were not scored.

11. Adequate documentation in the case of an override.

12. Treatment plan clearly relates to information captured in the assessment. 1

7. Avoided barriers to listening (such as moralizing, disagreeing,
Blaming, shaming, reinforcing).

Total score: ______ divided by _______ =
Reviewer Comments:

5th Judicial District, Department of Correctional Services, Des Moines, Iowa
13

Sample: Level of Service Inventory-Revised
Review Protocol
The following are the most common errors that assessors make when completing an LSI-R
assessment. When reviewing, you should look for the following items:
1.

Lack of interview guide (i.e. failure to use an interview guide)

Yes/No questions and answers (i.e. when reviewing the interview guide or notes, look for
answers other than “yes”, “no”, “good”, “fine” et. This will usually indicate that the
interviewer is relying on closed-ended questions and is not tapping for qualitative
answers.)

Ancient History (i.e., review the questions marked as “current” to be sure that the
information recorded is based on the most recent information.)1

Collateral information (do you see discrepancies between the LSI-R and the Pre-sentence
investigation? Has the rap sheet been reviewed for scoring of the criminal history
section? Is there family information in the file that supports or refutes the scoring of the
LSI-R?)

Inconsistencies between the case plan and the scoring of the LSI-R (i.e. a criminality
class is recommended, but the person was given a “2” or “3” in the area of
attitude/orientation. The offender has no contact with family members, but the
family/marital section is rated as pro-social. The person is on supervision for forgery and
the financial situation is rated as pro-social. These are areas to explore.)

Failing to mark all of the boxes with either a “X” or circling the number.

Adding the numbers incorrectly

Blatant Scoring Errors
The following is a list of “default” scoring rule violations that most commonly are
missed:
a.
Person is on for a sex offense and item 10 is scored “no”.
b.
Person is unemployed and question 18, 19, 20 are not scored “0”.
c.
If a person’s parents are deceased, question 24 must be scored as “0”.
d.
If the assessment is done at time of discharge from jail or a residential facility and
the person does not know where he or she will be living, item 27 must be scored
“yes”.
e.
Question 29 must be scored yes if the person is residing in a residential center or
is incarcerated.
f.
If item 32 is scored yes, then items 35 and 36 must be scored yes.
g.
Questions 41-45 must be scored as “no” if there has been no usage within the last
year.
h.
If question 47 is scored “yes”, then question 46 must also be scored “yes”
5th Judicial District, Department of Correctional Services, Des Moines, Iowa

“Ancient History” refers to old or out-of-date information.

Motivational Interviewing Critique
Motivational Interviewing (MI) is a directive, client-centered counseling style for helping clients
explore and resolve ambivalence about behavior change. When used by a skilled officer, MI can
help increase an offender’s motivation to comply with supervision requirements, participate in
treatment, and address their criminogenic needs. When a more traditional and confrontational
interaction style is used, the officer can increase resistance and decrease motivation to change.
For many officers, learning and implementing MI is challenging, therefore ongoing support and
feedback are needed to ensure successful incorporation of the skills.
The Motivational Interviewing Treatment Integrity (MITI)
Code is a highly structured feedback system that is used in
MI research, the training of trainers, and as part of ongoing
professional development. A MITI critique uses a video or
audio tape of an MI interview at least 20 minutes in length.
The rater tracks the methods used in the interview, including
MI-adherent and non-adherent behavior, and the frequency
of these interviewer behaviors are used to compute a “skill
balance.” Raters also judge the tape based on global ratings,
such as genuineness and empathy. The interviewer receives
detailed feedback on all components of their interview, as
well as a skill balance rating and an explanation of the rating
form. In addition to the written feedback, the rater provides
verbal coaching to the interviewer. This type of critique has
been proven effective in improving the skills of MI
practitioners. To use the MITI correctly, peer reviewers
would need to participate in special training.
Another option is for the peer review team to develop their
own instrument for evaluating MI skills. Though this
instrument would not have the research support of the MITI,
it may better meet the needs of the peer review team and the
officers. At minimum, it should address:
•
•
•
•

What MI adherent techniques are being used, and
how often?
What MI non-adherent techniques are being used,
and how often?
What is the demeanor of the officer? Is he/she
showing empathy?
What is the demeanor of the offender? Is he/she
showing resistance? Motivation to change?

Regardless of the review
method chosen, the
same general peer
review guidelines apply:
•

A standardized review
format must be
developed and all peer
reviewers must be
trained to use it.

•

Reviews should be
done on a regular
basis (monthly,
quarterly, yearly),
either for all officers or
a sample of officers.

•

All reviewed officers
should receive
feedback, coaching,
and opportunities for
additional training and
re-assessment.

•

Widespread issues
should be addressed
with policy changes
and training for all
officers.

Specific feedback is very beneficial to someone learning MI, so any format that is used should
incorporate detailed feedback on specific skills.
15

VERSATILE MOTIVATIONAL INTERVIEWING CRITIQUE

Open Questions (Raw# =9)

26%

Closed Questions (Raw# = 1)

Affirmations (Raw# = 3)

Reflections (Raw# = 10)

29%

Summarizations (Raw# = 1)

Elicitations (Raw# = 6)

18%

Teaching/Advice (Raw# = 4)

12%

Confrontations (Raw# = 0)

SKILL BALANCE RATING

65%

65%
Agency
Avg. (n=1)

Agency Avg.
(n=1)

Date of Tape Review:
Time Length of Session:
Relationship of Participant(s) to
Client:

Officer

Probation Officer
Community Corrections
11/01/04
#VALUE!
#VALUE!

Officer

Q-Strings

CLINICAL SKILL MEASURES

Therapist Name:
Agency:
Date of Session:
Offender JIS ID:
Tape Reviewed By:

3+ Q's In a Row (Per Hour)

2.3

Largest Consequtive Q-String

4.0

GLOBAL RATINGS

Low
Acceptance

4.0

Egalitarianism

4.0

Empathy

3.0

Genuineness

5.0

Warmth

3.0

Spirit

5.0

10%

20%

30%

40%

11/11/04
26
#VALUE!

50%

60%

70%

80%

90%

7 High

MIC - VERSATILE Page 1 of 5

100%

Change Talk Rates (Officer)

190.0
140.0
90.0
40.0
-10.0
-60.0

Desire
Per Hour

Ability
Per Hour

Reasons
Per Hour

Need
Per Hour

Commit
Per Hour

Tk. Steps
Per Hour

Total Chg. Talk
Per Hour

P. Officer

76.2

9.2

11.5

46.2

32.3

0.0

175.4

Average

11.1

6.0

11.3

11.4

10.0

9.5

70.8

MIC - VERSATILE Page 2 of 5

NOTES
General Comments
Officer, at 65, your skill balance shows that you have a good understanding of all of the MI skills and are able to use most of them in appropriate proportions. You
did a great job of remembering to reflect and affirm, asking many questions that produced enthusiastic change talk from the client. Nicely done!

Elicitations:
Your specific questions about the client's steps and strategies for change really helped him to express his desire for lasting change. The strength of a client's
comments expressing desire, ability, reasons, need, commitment, and taking steps regarding change are highly predictive of long-term positive behavior
modification, so offering clients the opportunity to express these sentiments is great. There were so many times that you were able to do this in quite a brief
interview. Nicely done!

Reflections/Summarizations:
One third of your interactions with the client were reflections, which is a great start! Ideally, reflections would make up about 56% of an MI interview because
reflections allow clients to know you are listening, allow them to hear what they have said to decide if that is how they really feel, and prompt them to give more
information that you need without making them feel interrogated by too many questions. Also, you were able to do one summary, which showed that you have a
good attention to detail. Using summaries a bit more often may help you to get clients to understand what you are asking for. For example, you were trying to get
him to say what his treatment needs were, but he kept talking about the same things. Summarizing with, "So you want your treatment to consist of staying
connected to your family and church friends, remembering and spending time with your daughter, and being open with me. What other structures will help?" may
help you to focus the client on the additional things you are looking for, in addition to letting him know you've heard him.
Affirmations:
About half of your affirmations really identified something that was important to the client and expressed your appreciation in ways that really built him up. These
were so well done! The other half were conditional: "I respect you if you are sincere" or "You seem to be off to a good start, I would like to see you make it and I
really think you can...if you can hold this attitude." While you are a probation officer and it is your job to make sure that sentiment translates into behavior,
conditional affirmations undermine the goal of the skill: to convey unconditional positive regard so that the client lowers defensiveness so you get better information.
You can express concern that the client keeps up his good start by using other skills, but when you affirm, see if you can stay with comments that are positive and
empathetic. That way the client will understand the message that you care about him and that it is important that he does well, not "I care about you only when you
do well." The latter feeling tends to make clients more reluctant to reach out to you when they are having a hard time.
Global Measures:
I gave you a mix of above and below average rankings for Global Measures because on one hand, you reflected well, remembered to affirm, elicited lots of change
talk, and avoided confrontation. On the other hand, some of your affirmations were conditional, your percentage of teaching was a bit high, and I am not sure if you
realize how flat and unvaried your tone was throughout. While you may deeply care about your clients, when they do not hear your voice go up and down with
emphasis, it may be hard for them to believe that you care and they may be less motivated to give you good information about themselves that you will need to help
them succeed. I encourage you to vary your tone and give a few more unconditional affirmations to establish rapport with clients, "earning the right" to correct them
later as needed.

Summary of Recommendations
Overall, you gave a good demonstration of the majority of the MI skills and showed that you are able to elicit change talk well. In future interviews,
focus on reflecting and affirming more and varying your tone so that client's sense your regard for them. Keep up the great work!
More Of….
Reflections, Affirmations, Summarizations

Less Of…..

Interesting….
No Comment

Teaching

MIC - VERSATILE Page 3 of 5

Skill Balance Detail*

SKILL BALANCE DETAIL: Probation Officer
OPEN (9)

CLSD (1)

REF (10)

SUM (1)

AFRM (3)

ELICIT (6)

TEACH (4)

CONF (0)

30%

33%

10%

20%

12%

(1)

MODIFIED
DISTRIBUTION

(2)

COLLAPSED
DISTRIBUTION

33%

12%

(3)

CATEGORY
PENALTIES

5.3%

23%

0.07

0.00

(4)

OPEN:CLOSED
RATIO PENALTY

(5)

PENTALY
TOTAL

(6)

FINAL SKILL
BALANCE

[O/C] 0.05 + [REF] 0.23 + [S/A/E] 0 + [TEACH] 0.07 + [CONF] 0 + [O:C RATIO] 0 = 0.35

1 - 0.35 = 0.65

Explanation of Skill Balance:

(1) Modified Distribution:
For purposes of the "Skill Balance", The first six MI categories: Open, Closed, Reflections, Summarizations, Affirmations and Elicitations are isolated from the
last two (i.e., the percentages in these categories will add-up to 100%. Regarding the latter two categories, "Teaching" and "Confrontations", the distributions
are calculated as a proportion of all interactions (i.e., across all 8 categories).

(2) Collapsed Distribution:
The following categores are collapsed and their percentages combined/added: (1) Open Questions and Closed Questions and (2) Summarizations,
Affirmations and Elicitations.

(3) Category Penalties
3 OPEN/CLOSED CATEGORY:
Any value in excess of 28% is assigned as a penalty (no penalty for falling short of 28%).
3 REFLECTIONS CATEGORY:
Any value short of 56% is assigned as a penalty (no penalty for exceeding 56%).
3 SUMMARIZATIONS/AFFIRMATIONS/ELICITATIONS
Any value short of 16% is assigned as a penalty (no penalty for exceeding 16%).
3 TEACHING:
Any value in excess of 5% is assigned as a penalty.
3 CONFRONTATIONS:
Any confrontation value is assigned as a penalty.

(4) OPEN:CLOSED Ratio Penalty
3 OPEN:CLOSED RATIO
When analyzing open & closed questions seperately, closed questions (ideally) should not exceed 30%. For any value in excess of 30% a
1 point (1%) penalty for every 5% of the excess is assigned.

(5) PENALTY TOTAL:
The sum of all penalties noted above.
(6) Final Skill Balance:
One (1.0) less the sum of all penalties.

MIC - VERSATILE Page 4 of 5

GRAPH KEY

Open Questions (Raw# = 44)

Agency Avg.
(n=43)

Open Questions (Previous Interview Score)

This green line represents
the "National Average"
score.

Interviewer

This column presents the specific skill percentage of
the interviewer's present critique score for the given
category ("open questions", in this example). It is
depicted in graph form (in gray) to the right.

72%

53%

This column presents the average skill percentage of the
interviewer's agency. The number of individuals comprising the
average is noted in the heading (n=). This value is depicted in
graph form (in red) to the right.

10%

20%

If the interviewer has submitted a previous tape to be critiqued, the red
marker here indicates the interviewer's most recent score. In this
example, the interviewer's previous score was 34%. If the interviewer
has not previously submitted a tape, this yellow-shaded section will not
appear on the report.
30%

40%

50%

60%

70%

80%

90%

100%

The gray-shaded area represents the Interviewer's score
for the present critique. It corresponds with the grey
numeric value on the left, which, is 72% in this example.
The thin red line here indicates the average score of all individuals in the same
agency as the interviewer. In this example, the agency average is 53%, which
corresponds to the red numeric value to the left.

Explanation of Global Rating Categories
Acceptance (also called unconditional positive regard)
Interviewers high on this scale communicate acceptance and respect to the client. Acceptance is person-focused (unconditional positive regard) and
should not be confused with acceptance/approval of the person's behavior. Interviewers at the low-end of this scale may be perceived as judgmental,
harsh, disrespectful, labeling or condescending.

Egalitarianism (the opposite pole is authoritarianism)
Interviewers high on this scale emphasize by the words and manner the client's personal autonomy, choice, and responsibility. They may offer their
expertise when asked for it, or after obtaining the client's permission to inform or advise. Interviewers low on egalitarianism take a more authoritarian
approach of directing, ordering, blaming, threatening and confronting. There is a quality of the interviewer asserting a "one-up" position that implies, "I
know best. Listen to me."

Empathy (also called understanding)
Interviewers high on this scale are able to attain and communicate an accurate understanding of their client's perceptions, situation, meaning and feelings
through high-quality reflective listening. Their manner shows an active interest in an effort to understand the client's perspective, and their responses
actively express an attentive understanding of the client's perspective and experience. They probe to understand more fully, and reflect their
understanding back to the client. Interviewers at the low-end of this scale show little interest in or appreciation of the client's perspective, little overt
understanding or reflection of what the client is experiencing. They evidence little effort at seeking a deeper understanding of the client's perspective.
Interviewers low in empathy may ask many questions to gain factual information or pursue their agenda, but do not seek to understand the client's own
perspective.

Genuineness (also called congruence)

Interviewers high on this scale are perceived as open, responsive, and honest. The interviewer appears to be saying what he or she is experiencing in
this session. They show a quality of congruent transparency, saying what they feel and perceive in the moment. Their response to the client is individual
and personal. Do not confuse this with other scales such as acceptance or warmth. High genuineness, for example, can include expression of negative
affect or criticism. Interviewers low on this scale do not appear to be responding honestly and openly to the client, and may appear unresponsive or
phony. If they self-disclose, it may have the quality of talking about personal history rather than relating in the present. Their response may have flat,
closed, or technical-business quality, or may appear to be rote or mechanical.

Warmth
Interviewers high on this scale are perceived as warm, friendly, engaged, compassionate, helpful, caring and concerned. Interviewers at the low end of
this scale present an impression of being cold, distant, detached, or unfriendly, showing little overt evidence of helpful concern and compassion.

Spirit
This is an overall, global rating of the extent to which the interviewer manifests the fundamental spirit of motivational interviewing. It should not be
regarded merely as an average of the other scales, but rather the rater's judgment of the extent to which the interviewer "gets it", evidencing a grasp of the
"music" and not just the words and techniques. Interviewers high on this scale manifest a directive, client-centered style of facilitating, coaching, and
negotiating. The interviewer honors and values the client's perspective. There is a naturalness, comfort, and loving or artistic quality to the interviewer's
style. The interviewer is attuned to the client, and actively "mines for the client's own motivation." Interviewers low on this scale show a lack of the
balanced directive, client-centered style, erring on the side of passivity or of overcontrol (or both). On the passivity side, the interviewer misses or is
inattentive to significant client material, and may seem indifferent, isolated, ignoring, preoccupied, or detached. On the overcontrolling side, the
interviewer may communicate mistrust, disrespect, disregard, or simply the pursuit of the interviewer's own agenda without sufficiently involving the client.

MIC - VERSATILE Page 5 of 5

Cognitive-Behavioral Treatment Group Facilitation
Evidence-based practice tells us that cognitive-behavioral programming is one of the most
effective treatment modalities for encouraging behavior change in offenders. Many evaluated,
successful curricula are available for use with offenders in addressing a range of issues from
anger management to substance abuse to job skills. These curricula can be implemented in
community corrections settings, and can be facilitated by community-based treatment providers
or by community corrections officers.
In order for cognitive-behavioral programming to be effective, the facilitator must be welltrained, and the curricula must be implemented with fidelity to cognitive behavioral principles,
as well as the specific curriculum. For example, a job skills curriculum may have been proven
effective in one jurisdiction when offenders participated in twenty hours of programming and
used extensive role-plays. If the program is replicated in another jurisdiction in ten hours by
replacing role-plays with lectures from the facilitator, the program is unlikely to be successful.
The peer review process can be used to assess the implementation of cognitive-behavioral
groups, with several benefits: the facilitator can receive feedback on his or her skills; the
curriculum can be assessed for fidelity to the model; multiple models can be reviewed to
determine which is the most appropriate for client referral; and officers can better understand the
services that their clients are receiving. Peer review of treatment groups provides a reciprocal
benefit to the treatment provider and the officers who are referring clients to treatment.
Like other forms of peer review, the methodology must be chosen carefully depending on the
comfort level of the facilitator, the resources available, and the information that is being
assessed. There are models in the field for this type of assessment, such as the Correctional
Program Assessment Inventory (CPAI). Please see the appendix for a comprehensive example
of another assessment form. Like other assessments, review of cognitive programming should
include the following steps:
•
•
•

Use of a standardized assessment tool that addresses content, environment, and facilitation.
Well-trained reviewers conducting in-person or video review (audio is usually insufficient)
on a regular basis.
The opportunity to give feedback to the facilitator and the agency on the group.

Because treatment provision is multifaceted and often provided by a variety of providers, a team
approach can be very effective for this type of peer review. The team can be composed of staff
from state and local community corrections agencies and private providers. This way, the review
is conducted by a group with extensive and diverse experience, and all treatment providers,
public or private, will feel that they are being reviewed by a peer. This process can be used to
coach new facilitators, to evaluate new treatment providers and curricula, and to ensure quality in
programming over time. Like any other new type of appraisal, the facilitators must be involved
in developing the program and understanding its benefit, so that the peer review does not seem
punitive, or as a competitive process between different providers. At the same time, an
organization must have an expectation of consistent quality in the services that it offers.

Case File Review
The key component of evidence-based practice is ensuring that the level of supervision and
treatment that an offender receives is commensurate with his or her level of risk. The most
efficient way of assessing this is through periodic review of a random sample of case files. Wellmaintained case files or databases should contain the offender’s case history, assessment
interviewing and scoring, case plan, supervision notes, treatment referrals, and treatment
attendance, as well as any other documentation required by local laws and policies.
A peer review team is able to assess whether:
• Officers are maintaining adequate documentation.
• Offenders are receiving appropriate assessment, supervision, referrals, and treatment.
• Overall services are in compliance with the department’s vision, mission, and goals.
• Case plans are based on assessment data.
• Case plans are followed, and goals are met.
• Ongoing training, support, policy revision, etc. is necessary.
Case file review provides much of the data that are used as barometers of quality performance.
The process of case file review can be tailored to the size and needs of the organization,
following these basic steps:
1. A standardized case file review form is adapted or created,
taking into consideration the range of cases that will be
reviewed (i.e. juvenile and adult, offenders assigned to
various supervision types, and offenders participating in
various forms of specialized treatment). A cross-section of
staff should be involved in the creation of the instrument.
The instrument must be designed to measure the quality
assurance indicators that are being tracked.
2. The peer review team is recruited and trained.
3. A sampling plan is developed: How often will reviews
take place? How many files should be reviewed? In
general, the larger the sample, the more information in
garnered. Resource limitations must be considered.
4. Case review is completed, and the data is compiled and
compared to benchmarks (i.e. 80% of offenders referred to
sex offender treatment will complete the treatment).
Trends and deviations from the benchmarks, both positive
and negative, are noted.
5. The peer review team works with management to address
trends and deviations, as discussed below.

The Peer Review
Team

1. The team should be
drawn from the
population of officers
that they are reviewing.
2. Peer review can be the
reviewers’ sole
responsibility, or a part
of their job.
3. Participation in peer
review can be an
ongoing job
responsibility, or a
large group of staff can
serve on a rotating
basis.
4. The team should be
well trained on the
instruments that they
are using.
5. Team members need to
be as objective as
possible in reviewing
their peers, and cannot
review themselves.

The process of peer review involves more than collecting data;
the process needs to be monitored for efficacy, and the data needs to be put to work.

Sample: University of Cincinnati Risk Assessment
Note: This tool provides an “estimate” of an offender’s risk/needs assessment score, based on demographics and
history data. It can be used to quickly re-assess an offender’s risk level for comparison to the score determined by
the probation officer. This tool was validated for use with a population of offenders in Ohio. As always, this tool
needs to be validated on the population with which it is being used.

Client ID __________________
Risk Factor
Age
Education
Marital Status
Psychological Problems
Indicated
Alcohol Problem Ever
Drug Problem Ever
Unemployed at Arrest
Prior Arrests

Prior Incarcerations
Prior Conviction for
Violent Offense
Prior Conviction for Sex
Offense
Previous Community
Control Violation
Current Felony Degree
Current Offense Type

Categories and Weights
17-22
23-36
32+
< H.S. Graduate
> H.S. Graduate
Single
Married
Yes
No
Yes
No
Yes
No
Yes
No
2+
1
0
2+
1
0
Yes
No
Yes
No
Yes
No
3rd, 4th, 5th
2nd
1st
Drug, Property, Sex
Person or Other

Total
High Risk (76-115)
Moderate Risk (55-75)
Risk Category: _____________

Score

16.9
7.2
0
7.6
0
7.5
0
1.9
0
4.7
0
9.0
0
6.5
0
12.3
2.9
0
22.8
6.6
0
3.5
0
5.8
0
6.9
0
22.8
6.6
0
5
0

Low/Moderate Risk (38-54)
Low Risk (0-37)
Interviewer: _______________
23

Sample: Case Management Audit Instructions
NOTE: This case management audit pertains to case management issues only. Other issues related to
supervision but not to case management can be audited according to local policy.
1. Were the LSI, CMC/Jesness, and case plan completed within the prescribed time frames?
a. Focus: 60 days field and 30 days residential from date of assignment.
b. Source: Compare date of assignment and dates on LSI, Jesness, and case plan.
2. Is the problem prioritization consistent with LSI and CMC/Jesness?
a. Focus: Problems listed on front page of case plan should reflect results from LSI
and CMC/Jesness. Were the Big 4 (criminal history, anti-social companions, antisocial personality, and attitudes) considered in prioritization?
b. Source: Case plan, LSI, CMC, Jesness, generic notes, ask officer to explain rationale
for prioritization (SAQI, ranking, Big 4).
3. Is the problem behavior and the need that it serves correctly identified?
a. Focus: Behavior that is illegal or which leads directly to illegal behavior. What are
needs being met by illegal behavior or conditions under which illegal behavior
occurs?
b. Source: LSI interview notes, PSI, case plan, generic notes from meeting negotiating
case plan, ask officer to articulate needs and/or conditions, observation of
negotiation session.
4. Does this section contain the undesirable results of the offender’s behavior as articulated by the
offender (their motivation to change)?
a. Focus: Look at offender’s entire experience with CJ System – what is significant
negative result of this experience. From offender’s perspective!
b. Source: LSI interview guide, PSI, generic notes describing meeting when case plan
negotiated, observation of negotiation session, other evaluations.
5. Is the goal reasonable and measurable?
a. Focus: Medium or long term behavior change that is a result of intervention. Must
fit offender’s capabilities. Stated in a way that attainment of goal can be measured.
Internalized, not superficial. Include timeframes.
b. Source: LSI interview guide, Jesness, PSI, chronos describing meeting when case
plan negotiated, observation of negotiation session, other evaluations.
6. Do offender interventions, tasks, activities appropriately relate to goals? Do the interventions, tasks,
activities contain the methods, techniques, resources, and timeframes the offender will use to achieve
the stated goal?
a. Focus: Are tasks/activities consistent with and supportive of goals and
interventions? Does officer spell out how task will be completed and what resources
will be used? Are there time frames? Are they realistic?
b. Source: Case plan, chronos documenting sessions where tasks are discussed.

7. Does the benefits section clearly show meaningful/positive behavioral changes/rewards for the
offender? Are the benefits in contrast to undesirable results and do the benefits bear a relationship to
the prioritized need?
a. Focus: Congruency between undesirable behavior and the prioritized needs. Is the
benefit meaningful to the offender?
b. Source: LSI, CMC/Jesness, case plan, generic notes, observation of negotiation
session, ask officer to articulate.
8. Does the case management plan reflect intervention and supervision/monitoring strategies consistent
with the CMC/Jesness?
a. Focus: congruency between interventions, techniques, strategies, resources, goals,
tasks.
b. Source: CMC/Jesness type, wording of case plan components, specificity of goals,
tasks, expectations. Generic notes describing negotiation session. Observe
negotiation session. Ask officer to articulate how CMC/Jesness type specifically
applies to offender.
9. Do interventions target the criminogenic need in the right intensity and does the intervention use the
methods and strategies needed by the offender based on risk, need, and responsivity? Are contacts
appropriate and meaningful given the risk, need, and responsivity issues of the offender?
a. Focus: Intensity of interventions and supervision, amount of detail in tasks, degree
of planned follow-up/documentation, type and amount of planned contacts with
offender. How many contacts are personal vs. collateral? How long are personal
contacts? What is discussed in personal sessions? How often are contacts? Do all
of these things fit with CMC/Jesness type?
b. Source: LSI, CMC/Jesness, case plan, generic notes, observation of meetings, ask
officer to describe issues mentioned in focus section.
10. Is the case reviewed on an ongoing basis and does officer make appropriate ongoing case
adjustments, i.e. minor violations, rewards, case plan revisions, techniques, strategies?
a. Focus: Regarding case review: does officer go back periodically and review entire
case to get big picture of case performance. Review should emphasize congruency
between LSI, CMC/Jesness, case plan, response to supervision and treatment. Has
case stayed on the course first set out by case plan? Should it have? Regarding
ongoing case adjustment: focus is not on major responses to major events. Rather,
as officer learns more about offender and offender exhibits behavior during course
of supervision, does officer make appropriate adjustments in approach, type of
contacts, content of contacts, motivational techniques, rewarding techniques, etc.
b. Source: Generic notes, case plan, LSI, CMC/Jesness, staff case with officer (ask
what has learned about offender and how have made case adjustments).

11. Are critical incidents handled timely and appropriately, given risk, need, responsivity, and other
circumstances?
a. Focus: Critical incidents refer to those events that relate to case management issues.
Are they handled in a timely manner with risk, need, and responsivity factored into
immediate and subsequent responses.
b. Source: LSI, CMC/Jesness, chronos, ask officer.
12. Is the officer communicating with appropriate sources in order to have current, relevant knowledge of
the offender’s performance in programs?
a. Focus: Are lines of communication opened and maintained? Look at frequency and
content of contact and whether responsivity is considered.
b. Source: Generic notes, ask officer, ask program staff, attend staffings.
13. Does the officer have a basic understanding of the programs in which the offender is participating and
is this knowledge reflected by reinforcing programming goals in meetings with the offender?
a. Focus: Does officer understand programs well enough to have meaningful
conversations with program staff about performance and reinforcement? Does
officer understand programs well enough to have meaningful conversations with
offender about progress in program and to reinforce what ought to be reinforced?
b. Source: Chronos, training records, staffings, observe meetings, ask officer to explain
program to supervisor.
14. Is there a relapse prevention plan that is understood by the officer and discussed with the offender
during meetings?
a. Focus: Relapse prevention plan in global sense. Does a plan exist in some form?
Does officer understand dynamics of offender’s criminal behavior and what
behaviors to watch for to determine if offender is staying on right path or starting
down path to illegal behavior? Is officer anticipating behaviors based on his/her
understanding of offense dynamics? Are they initiating interventions as early as
possible and are they reflective of risk and responsivity? Is the plan discussed with
offender at meetings? Is there follow-up and verification?
b. Source: Chronos, staffings, discuss with officer, observation of meetings.
(auditing instrument follows on next page)

5th JUDICIAL DISTRICT DEPT. OF CORRECTIONAL SERVICES (Des Moines, Iowa)

Sample: CASE MANAGEMENT AUDIT
Case Manager Name:

Date:

Offender Name/ ICON Number:
Reviewer____________________________________ Date of Review_____________________
1. Were the LSI, CMC/Jesness, and Case Plan completed within the prescribed time frames?
5

CMC/JESNESS/LSI and Case Plan
completed with 30 days (residential), 60
days (field) of case assignment or
reassessment.

CMC/JESNESS/ LSI and Case
Plan completed within 45 days
(residential), 75 days (field) of
case assignment or
reassessment.

CMC/JESNESS/ LSI + Case
Plan completed later than 45
days (residential), 75 days
(field) of case assignment or
reassessment

Score

(1)

2. Is problem prioritization consistent with LSI and CMC/Jesness?
5

Problem prioritization consistent with
needs identified by LSI, CMC &
JESNESS.

Ranking questionable or not
supported by officer or
documentation.

Problem ranking inconsistent
with identified needs in
assessments.
(2)

Base ratings on items 3 through 7 on all active action plans.
3. Is the problem behavior and the need it serves correctly identified?
5

Problem behavior and need it serves
clearly identified and articulated on plan.

Only one criterion met or not
clearly articulated on plan.

Neither criteria met and not
clearly articulated on plan.
(3)

4. Does this section contain the undesirable (worst) results of the offender’s behavior as
articulated by the offender? (their motivation to change)
5

Results identified clearly specify the
undesirable consequences of the
behavior as articulated by the offender.

Results identified however the
relationship to offender’s stated
consequences is unclear.

Results not identified and/or
bear little if any relationship
to offender’s stated
consequences.
(4)

5. Is the goal reasonable and measurable?
5

Goal focuses on the behavior changes
the offender is capable of achieving and
expected to make as an outcome of the
case management interventions. Clearly
stated so as progress or completion can
be measured.

Goal focuses on merely
attendance/ completion not
behavior change or not clearly
stated thus progress hard to
measure.

Goal does not meet either
criteria.

(5)

6. Do interventions, tasks, and activities appropriately relate to goals?
Do the interventions, tasks, and activities contain the methods, techniques, resources, and
time frames the offender will use to achieve the stated goal?
5

Interventions, tasks and activities detail
how the offender will achieve and verify
the behavior changes. Resources are
identified and the frequency/duration of
the tasks/activities are identified.

Only one of the criterion is met.

Interventions, tasks and
Activities section meets
neither criteria.
(6)

Benefits contain meaningful positive
behavioral changes, which are in direct
contrast to the undesirable behavior and
are clearly tied to the prioritized need.

Only one of the criterion is met.

Benefits do not reflect
positive behavioral outcomes
and do not relate to the
prioritized need.

(7)

8. Does the case management plan reflect intervention and supervision/monitoring strategies
consistent with the CMC/Jesness?
8

Case plan interventions, supervision and
monitoring activities are consistent with
those most effective with the identified
CMC/Jesness type.

Only one criterion is met.

Neither criterion is met.

(8)

9. Do interventions target the criminogenic need in the right intensity and does the
intervention use the methods and strategies needed by the offender based on risk, need,
and responsivity? Are contacts appropriate and meaningful given the risk, need, and
responsivity issues of the offender?
10

Criminogenic need targeted in right
intensity. Intervention uses appropriate
methods and strategies.

Intensity okay or
methods/strategies okay.

Neither intensity nor
strategies appropriate.

(9)

10. Is the case reviewed on an ongoing basis and does officer make appropriate ongoing case
adjustments, i.e. minor violations, rewards, case plan revision, techniques, strategies?
10

Case adjustments ongoing and
appropriate.

Some problems with timeliness
and /or appropriateness.

Significant problems with
timeliness or
appropriateness.

(12)

11. Are critical incidents handled timely and appropriately, given risk, need, responsivity and
other circumstances?
9

Handled timely and appropriately with
all-important factors considered.

Some timeliness and/or
appropriateness problems.

Significant problems with
timeliness or
appropriateness.

(13)

12. Is the officer communicating with appropriate sources in order to have current, relevant
knowledge of the offender’s performance in programs?
9

Ongoing communication. Knowledge
of performance relevant and current.

Sporadic communication.
Problems with quality of
knowledge of performance.

Little or no communication.
Knowledge of performance
not current.

(14)

13. Does the officer have a basic understanding of the programs in which the offender is
participating and is this knowledge reflected by reinforcing programming goals in
meetings with the offender?
9

Thorough understanding of programs.
Goals reinforced in meetings.

Some problems with degree of
understanding and/or goals
reinforced sometimes.

Little understanding of
programs and/or goals not
discussed or reinforced.
(15)

14. Is there a relapse prevention plan that is understood by the officer and discussed with the
offender during meetings?
10

Plan exists, is understood by officer, and
discussed with offender.

Plan exists but understanding
insufficient and/or not often
discussed with offender.

No plan, little or no
understanding, little or no
discussion with offender.
(16)

TOTAL

SAMPLE PEER REVIEW AUDIT SHEET (FY 2004)
Open Case ο

Section 1: Demographics
Client Name:
Client Codap Number:
Date of Review: ___/___/___
Site: ❏ Program 1 ❏ Program 2
❏ Program 3
Primary CSP:
Client Race/Ethnicity:
❏ Caucasian ❏ African American ❏ Hispanic-American
❏ Native American/Eskimo ❏ Asian American
❏ Pacific Islander ❏ Biracial ❏ Other
Section 2: Intake/Assessment
Yes
No
1. Consent to treatment is complete within
2 business days of admission.
❏
❏
2. The client is appropriate for the program and should
have been admitted.
❏
❏
3. Intake section of record is complete, in proper sequence,
and documented in a timely manner.
❏
❏
4. Release/agreements section of the record is complete, in
proper sequence, and documented in a timely manner. ❏
❏
5. Chemical dependency assessment is complete within
14 days of admission to program.
❏
❏
Total
____ ____
Section 3: Treatment Planning
Yes
No
6. Treatment plan is appropriate to assessment.
❏
❏
7. Treatment goals are pertinent to assessed needs.
❏
❏
8. Treatment goals are written in measurable/objective
terms.
❏
❏
9. Individualized treatment plan is written within 7
days of completion of the assessment and contains:
❏
❏
a) signatures and dates ___
b)target dates for objectives/goals ___
c) reviewed and signed ___
10. Treatment plan revisions are appropriate and
completed every 30 days.
❏
❏
Total
____ ____

Closed Case ο

Section 4: Treatment Management
Yes
No
11. There is adequate documentation in clinical record
describing client status and service interventions.
❏
❏
12. Continued treatment and receipt of program services
are appropriate.
❏
❏
13. Adequate quantity and quality of physician and other
types of consultation according to recommendations
within service plans are documented.
❏
❏
14. Clinical section of the record is complete, in proper
sequence, and documented in timely manner.
❏
❏
15. Service Delivery section of record is complete,
in proper sequence, and documented in timely
fashion.
❏
❏
16. Individual progress notes reflect treatment plan
implementation.
❏
❏
17. All documentation is signed and the appropriate
credentials are held by the person providing or
supervising the service.
❏
❏
18. The service provider is only providing services
authorized by the governing authority.
❏
❏
Total
____ ____
Section 5: Discharge Planning
Yes
No
19. Appropriate continuity of care provided from
assessment through treatment, to discharge.
❏
❏
20. Discharge summary is complete and contains:
❏
❏
a) dates ___
b) signatures and credentials ___
c) reviewed and signed ___
d) level of care/services provided ___
e) client’s response to treatment ___
f) recommendations/referrals ___
g) client ID ___
Total

____ ____

Section 6: Miscellaneous
Yes
No
21. Movement/sign logs are accounted for and in proper
sequence.
❏
❏
Total
____ ____
Section 7: Employment/Financial/Vocational
Yes
No
22. Employment section of record is complete, in proper
sequence, and documented in timely manner.
❏
❏
23. Vocation section of record is complete, in proper
sequence, and documented in timely fashion.
❏
❏
24. Financial section of record is complete, in proper
sequence, and documented in timely fashion.
❏
❏
Total
____ ____
Section 8: Records
Yes
No
25. Group record is complete, in proper sequence, and
documented in timely fashion.
❏
❏
26. Urinalysis record is complete, in proper sequence, and
documented in timely manner.
❏
❏
27. Discipline record is complete, in proper sequence, and
documented in timely manner.
❏
❏
28. Medical record is complete, in proper sequence, and
documented in timely manner.
❏
❏
29. Referral record is complete, in proper sequence, and
documented in timely manner.
❏
❏
Total
____ ____
SCORING:
Total number of Yes Answers = _____
(Total number of Yes answers/29) x 100 =

Section 9: Reviewer’s Comments
Reviewer Comments:

Reviewer Signature

If record review is less than 100% accurate, an action plan must be
submitted that informs when the deficiencies within the record will be
corrected. Forms are due back to Peer Review Committee 10 working
days after receipt of form.

Date Due: ___/___/___ Return to:
Action Plan:

Supervisor Signature
Service Provider Signature

_____/_____/_____
Date

_____/_____/_____
Date
_____/_____/_____
Date

SAMPLE: PROGRAM/UNIT NAME
PEER REVIEW TRACKING FORM
For the Month of: _________________
Client ID

Reviewer

Primary PO

Date Peer Review

Date Due

Date Returned

QUALITY ASSURANCE INDICATORS
Overview
When evidence-based practice is implemented on a large scale, there may be hundreds of
outcomes that indicate success in various facets of the program, and thousands of pieces of data
that indicate progress towards those goals. If too much data is collected, it can be overwhelming
and indecipherable, as well as resource-intensive. If too little data is collected, then program
staff does not have an accurate sense of their progress and areas needing further development.
Developing a set of key indicators and institutionalizing a process for monitoring these measures
will assist organizations in gauging their progress towards implementation of evidence-based
practices. Ensuring that the fidelity and quality of service are in or at least moving toward
alignment with the organization’s plan and goals is essential to maintain momentum toward
change. Developing a small number of key indicators that can act as windows onto the
landscape of fidelity and quality of service helps to build accountability and maintain the
integrity of organizational goals.
Principles
Identify key measures
Choosing the right measures to monitor is essential to tracking progress towards goals.
The program logic model should be used to identify key program processes and short,
intermediate, and long-term outcomes. Then the data that are necessary and sufficient to
measure those outcomes and process should be selected for inclusion in the assessment.
Institutionalize measurement monitoring and discussion.
To collect accurate data on indicators in a timely manner, staff members must be
committed to the process and incorporate it into their daily routine. For this to happen,
staff must receive constant exposure to the process, the reasons for it, and their role in it.
This includes reference to quality assurance in staff meetings, supervision, and
performance reviews.
Share the information with all levels of staff
Staff members will not be able to support and participate in a process that they do not
understand. Provide formal and informational training, communication, and modeling so
that all staff understands the reason for the quality assurance process, as well as how the
process is working at the organizational level.
Provide positive feedback and celebrate achievement
In the quality assurance process it is easy to become focused on deficiencies, or on goals
that are several years from attainment. To maintain morale and keep staff members
committed to the process, it is important to celebrate quality work (as reflected in
performance data) and achievement of short-term outcomes.

Components
• Use a logic model to identify appropriate measures
The program logic model (discussed in detail in the Program Evaluation section of the manual)
provides information on the processes and outcomes that are necessary to achieving a program’s
goal. For example, if a goal is to have all sex offenders complete a community-based treatment
program, intermediate measures may include making
Sample Measures
contact with potential providers and referring offenders
to the treatment. Measuring one of these “checkpoints”
1) Assessment
can provide information on progress towards program
o % of population with
goals. The logic model should be used to identify
completed assessment.
intermediate measures, and choose the information that
o % of population with recan be tracked on a regular basis to indicate that
assessment completed
progress. In our example, referrals to a treatment
within time frame identified
program might be easy to track, and may be a strong
by policy
indicator of the number of offenders who will complete
o Gain Score (changes in
protective measure score)
the program.
•

Develop regular and easily accessible reporting
capabilities
For data to be useful in monitoring quality, these data
must be accessible to the staff members responsible for
reporting, and the means of reporting must be easy to
use. The more difficult the reporting process, the less
likely that the required data will be reported on a
regular basis.
o Choose indicators that are easy to report on a
regular basis.
Sometimes this will mean
developing a plan to make data more accessible.

2) Case Plans
o % of medium to high-risk
offenders that have case
plans
o % of case plans that
address the top 3
criminogenic needs
according to the
assessment.
3) Average length on
supervision by risk level.

Develop standardized reporting instruments, pilot
test them, and solicit feedback from those who
will have to use them. Involve staff from across
the agency to increase buy-in and commitment.

4) Revocations
o Number of technical
violations resulting in
revocation to jail.
o Number of technical
violations resulting in
revocation to prison.

Increase technological capabilities. Web-based
reporting tools can increase the ease of reporting,
compiling, and analyzing data, but only if staff
has access to computers and the internet.

5) Treatment
o % of high-risk offenders
referred to treatment.
o % of high risk offenders that
attended treatment.

Provide
comprehensive
training
documentation on the reporting process.

and

% of total population that
attended treatment that are
high-risk.

o Adhere to deadlines. Reward staff members or departments for on-time and accurate
reporting and enforce consequences for those that don’t adhere to deadlines.

SAMPLE: EMPLOYMENT DATA
BENCHMARK: 85% OF CASELOAD WILL HAVE EMPLOYMENT DATA ENTERED
INTO CIS:
PERCENT OF MULTNOMAH COUNTY
CASELOAD WITH EMPLOYMENT DATA ENTERED
100%
90%
80%
BENCHMARK =
70%
85%
60%
50%
40%
30%
20%
10%

N
ov
D 02
ec
Ja 02
nFe 03
bM 03
ar
Ap 03
r-0
M 3
ay
Ju 03
n0
Ju 3
l-0
Au 3
gSe 03
pO 03
ct
N 03
ov
D 03
ec
Ja 03
nFe 04
bM 04
ar
Ap 04
r-0
M 4
ay
Ju 04
n0
Ju 4
l-0
Au 4
gSe 04
pO 04
ct
N 04
ov
D 04
ec
Ja 04
nFe 05
bM 05
ar
Ap 05
r-0
M 5
ay
-0
5

Series1

Nov-02

Feb-03

Mar-03

May-03

Jun-03

Jul-03

Aug-03

Sep-03

Dec-03

Mar-04

Jun-04

Feb-05

May-05

64%

75%

72%

73%

80%

90%

84%

85%

84%

94%

87%

94%

Multnomah County Department of Community Justice

SUGGESTED NEW EMPLOYMENT BENCHMARK:
XX% OF ACTIVE CASELOAD WILL BE EMPLOYED
MAY 2005
PERCENT OFFENDERS
EMPLOYED

PERCENT WITH DATA
ENTERED

OVERALL

34%

94%

OFFICE 1

28%

98%

OFFICE 2

38%

97%

OFFICE 3

50%

91%

OFFICE 4

98%

OFFICE 5

22%

94%

OFFICE 6

35%

97%

OFFICE 7

32%

95%

OFFICE 8

35%

97%

OFFICE 9

39%

96%

OFFICE 10

10%

93%

OFFICE 11

33%

94%

OFFICE 12

25%

99%

OFFICE 13

23%

91%

OFFICE 14

52%

94%

Multnomah County Department of Community Justice

UNIT ACTIVITY: TREATMENT MODULE
BENCHMARK: 50% OF CASELOAD WILL HAVE AT LEAST ONE ENTRY INTO
THE TREATMENT MODULE
PERCENT OF ACTIVE CASELOAD
WITH TREATMENT MODULE ACTIVITY
100%

no activity
module activity

80%

60%

BE NCHM ARK =

40%

20%

OFFENDER REFERRAL TO. ENTRY INTO,
PARTICIPATION IN AND EXIT FROM TREATMENT
2500
2 2 0 2 2 19 5

2000

Mar-03

Sep-03

Dec-03

May-04

Jan-05

Apr-05

Feb-04

2021

19 8 3
18 3 8

17 7 6

17 8 7

1500

1000

500

2 9 2 3 17 2 8 5

202

14 6 17 3

9 7 12 1

18 8 14 5
13 7

226
18 9
13 7 14 6
13 0 16 9
12 7

84 97

referral

entry

in tx

Multnomah County Department of Community Justice

exit

UNIT ACTIVITY: TREATMENT MODULE
April 2005

OVERALL

Referred

Entered

Open

Exit

173

1787

146

OFFICE 1
OFFICE 2

109

OFFICE 3

339

OFFICE 4
OFFICE 5

112

OFFICE 6

OFFICE 7

101

OFFICE 8

OFFICE 9

OFFICE 10

119

OFFICE 11

109

OFFICE 12

OFFICE 13

OFFICE 14

500

OFFICE 15

Multnomah County Department of Community Justice

UNIT ACTIVITY: NEEDS ASSESSMENT AND CASE PLAN
NEEDS ASSESSMENT AND
CASE PLAN UTILIZATION
(RST AND MTDC EXCLUDED)

90%

Feb-05

May-05

78%

80%

71%

71%
70%
61%
60%

54%

54%
51%
50%
42%
40%

30%
22%
20%
11%
10%

0%
percent of offenders
with needs assessment

Definitions:

•
•

percent of offenders
with timely needs
assessment

percent of offenders
with timely case plan
review

percent of offenders
with open case plan

percent of offenders
with active case plan

Timely = every six months
Open case plan = case plan that has at least one objective or requirement with no initial
action date. An objective is a referral or direction to Offender in regard to objective.

Active case plan = plan that has an initial action date on at least one objective that has not been
completed. An objective is a referral or direction to Offender in regard to objective.

Multnomah County Department of Community Justice

UNIT ACTIVITY: NEEDS ASSESSMENT AND CASE PLAN
May 2005
PERCENT OF
OFFENDERS
WITH NEEDS
ASSESSMENT

PERCENT OF
OFFENDERS
WITH ACTIVE
CASE PLAN

PERCENT OF
OFFENDERS
WITH TIMELY
CASE PLAN
REVIEW

PERCENT OF
OFFENDERS
WITH TIMELY
NEEDS
ASSESSMENT

PERCENT OF
OFFENDERS
WITH OPEN
CASE PLAN

OVERALL

78%

54%

22%

71%

54%

OFFICE 1

75%

56%

34%

60%

55%

OFFICE 2

88%

80%

38%

81%

63%

OFFICE 3

68%

32%

19%

62%

53%

OFFICE 4

84%

63%

76%

66%

OFFICE 5

72%

19%

66%

51%

OFFICE 6

81%

64%

16%

76%

45%

OFFICE 7

73%

51%

11%

66%

60%

OFFICE 8

89%

68%

38%

84%

79%

OFFICE 9

79%

74%

31%

73%

64%

OFFICE 10

80%

71%

10%

76%

59%

OFFICE 11

70%

50%

13%

57%

45%

OFFICE 12

87%

37%

248%

85%

33%

OFFICE 13

76%

54%

65%

42%

OFFICE 14

87%

57%

40%

83%

57%

Multnomah County Department of Community Justice
39

RISK ASSESSMENT
BENCHMARK: 80% OF RISK ASSESSMENTS WILL BE COMPLETED WITHIN ONE
MONTH OF DUE DATE.

PERCENT OF ACTIVE CASELOAD WITH NEEDS ASSESSMENT COMPLETED
(MTCB, MTDC AND MTDV DSP CASELOADS EXCLUDED)
90%

77%
80%

73%

84%

83%

68%

70%
60%
50%
40%
30%
20%
10%
0%

Dec-03

Mar-04

1
Aug-04

Feb-05

May-05

Multnomah County Department of Community Justice
40

CUSTOMER SATISFACTION
Overview
Research has shown that offenders are more likely to change their behavior and less likely to
recidivate if they receive positive reinforcement. Assessing customer (i.e. offender) satisfaction
with the experience while on probation gives a sense of whether or not the offender feels that he
or she was treated with fairness and respect, and whether positive reinforcement was provided
when the offender maintained compliance with conditions. Therefore, customer satisfaction
surveys are one way of measuring alignment with evidence-based principles.
The basic procedure for assessing customer satisfaction is very simple: develop a questionnaire,
choose a sample of offenders, administer the questionnaire, and analyze the results. However,
each step has its challenges. A survey questionnaire must be carefully designed to elicit the
desired information, and careful sampling and administration are required to ensure that a
representative sample of offenders responds to the questionnaire. Finally, it is important that
feedback from customers (offenders) is considered and incorporated into practice where
appropriate.
Principles
The sample of offenders surveyed must be representative.
All offenders must have an equal opportunity to be selected for the sample, and to have
an opportunity to complete a survey. This includes offenders who are incarcerated or
who drop out of treatment before completion.
Questions should be consistent with quality assurance indicators.
Questions should be carefully chosen to provide data necessary to measure quality,
without the survey being too long and collecting irrelevant information.
Offenders must be able to respond honestly without fear of retribution.
The data is only useful if it is honest, so responses should be anonymous. Offenders must
also trust that their responses are truly anonymous, and that there will not be
consequences for negative feedback.
Results should be used to improve program quality and efficacy.
Results should be made available to all of the stakeholders charged with program
improvement and quality assurance. This data should be considered in program
evaluation and decisions on program and policy changes.

Components
•

Who will be surveyed?
A sample size is determined based on available resources: will all offenders be surveyed, or
only a sample? Will each offender be surveyed only once, or multiple times during his or her
involvement with community corrections? When choosing a sample, it is important that all
offenders involved with community corrections have the opportunity to be included in the
sample (i.e. select participants from the entire census, not just a “convenience sample” from a
certain jurisdiction or treatment group). Once a sample is selected, it is important that all
sampled offenders are given the opportunity to complete the survey, regardless of the reason
they leave probation (i.e. re-arrest or incarceration). Otherwise, only offenders who
successfully complete their probation will be sampled, and that group may not have the same
viewpoint as all offenders. If a certain group is unavailable for survey, then their exclusion
must be reported.

•

When will the surveys be completed?
This relates to the issue of sampling. If multiple surveys are going to be used, when should
they be administered? This will likely vary based on the average length of probation for
potential participants. If only one survey is being used, it could be administered universally
on a certain date (a “snapshot” of satisfaction across the agency on that date), or at a certain
point in the offender’s probationary period. Surveys administered at the end of probation are
often called “exit interviews.” Exit interviews are common because they allow an offender
to evaluate his or her entire experience with less fear of retribution. Also, administering a
single survey such as an exit interview requires fewer resources than multiple surveys.

•

What questions will be included?
As with all other instruments, the questions should reflect the indicators of quality for the
department. For example, did offenders feel they were referred to treatment in a timely
manner? Did they feel heard and part of the decision
making process? Have they re-offended while on
Survey questions
probation? Questions must be chosen carefully so that
the survey is not too long (which would decrease
must be chosen
completion rates), but still offers enough information to
carefully and
be meaningful.

pilot tested with
the target
audience.

Developing a survey instrument that meets the needs of
the agency and its customers can be complicated.
Careful consideration must be given to the instructions,
questions, and format of the survey. For example, respondents with low literacy levels or
speakers of other languages may have difficulty with complex language. The survey will
also need to be pilot tested for reliability and validity. A staff member or consultant with
experience in survey development should participate in developing the tool.

•

How will the survey be administered?
The way in which the survey is administered will influence whether the offender receives the
survey, completes it, and returns it in a timely manner. Also, the method must ensure
confidentiality. There are many methods for administering surveys, but the one that lends
itself to confidential exit interviews is self-administered mail surveys. This type of survey
can either be given to the offender in person or mailed to his or her home. The offender is
given the survey and a self-addressed, stamped envelop in which to return it. The survey will
not identify the offender by name, but it may use a number to track which surveys are
returned and which are not.

Mail Surveys: Benefits and Drawbacks
Benefits:

•
•
•
•

Confidentiality is ensured when no identifying information is on the survey.
Offenders can complete the survey at their own pace.
Mail surveys are less resource-intensive than in-person interviews or phone
surveys, so a larger sample size can be used.
Treatment providers can distribute the surveys as well as probation staff.

Drawbacks:
•
•
•

•
•

•

Mail surveys have low response rates compared to other survey methods,
especially when no reminders are given.
Literacy or language issues may prevent an offender from responding.
Surveys sent to offenders’ homes might not reach them.
Offenders may not feel motivated to respond honestly, completely, or at all.
(This is a risk with any survey; a compelling cover letter or other incentive
may be needed.)
Without tracking the survey, there is no way to determine whether certain
groups of offenders do not respond. (For example, results would be skewed
if offenders who are subsequently incarcerated never respond to a survey
about treatment effectiveness.)

How will the results be used?
The survey questions should be designed to inform the quality assurance process, determine
if the agency is providing professional, respectful services, and to determine if supervision
and treatment are addressing an offender’s criminogenic needs and preventing re-offending.
The results should be analyzed and reported in a useful way, ideally as part of the overall
data management system. The stakeholders in the quality assurance process, including the
peer review team, management, and any advisory groups, should use this data to inform their
review of key indicators and development of new best practice.

Interview #____

Interviewers Name: _____________
Interview Questions

What is your gender?
Male: _____

Female: _____

Please indicate your age: ______

What is your relationship status?
Not in a relationship: _____

In a relationship: ____ How long: _____

Which do you most strongly identify with?
1. ___ White (Non-Hispanic)

8. ___ Hispanic – Puerto Rican

2. ___ Black (Non-Hispanic)

9. ___ Other Hispanic

3. ___ Native American

10. ___ Asian

4. ___ Alaskan Native

11. ___ Southeast Asian

5. ___ Asian or Pacific Islander

12. ___ Mixed

6. ___ Hispanic – Mexican

13. ___ Other: ___________

7. ____ Hispanic - Cuban
5)

Is English your primary language?

5a)

If no, which language is? __________

Was language a barrier during your supervision process?
Yes: _____

Yes: _____

No: _____

What is the highest grade you have completed? ____________________
1. ___ Less than High School

4. ___ Some College

2. ___ High School Degree

5. ___ College Degree

3. ___ GED Completion

6. ___ Trade School Certificate
44

7a)

Were you home schooled?
How many years?

No: _____

_____

Have you ever had an immediate family member in the corrections system?
Yes: _____

Yes: _____

No: ______

Before your most recent arrest, did you have any close friends involved in the corrections
system?
Yes: _____

10)

No: _____

At what age did you first associate with friends or family in the Criminal Justice system?
_____

11)

Are you on… Probation: ______ or Post-Prison Supervision (Parole): ________?

12)

What crime were you most recently convicted of? ______________________

13)

Is this your first time on supervision?
Yes: _____

13a)

No: _____

How long have you been on supervision? (since the last conviction)
_____ months

14)

Have you been convicted of or found in violation of your supervision conditions
( this cycle) ? Yes: _____

14a)

No: _____

More than once: _____

What was the sanction (what happened)?
1) Jail _____ (# of days) _____

5) Counseling _____

2) Community Service

6) House Arrest _____

_____

3) Forest Project _____

7) Increased Reporting _____

4) Work Release _____

8) Revoke _____
45

15)

Were you employed at the time of your arrest?
Yes: _____

15a)

15b)

16)

No: _____

What is your current employment status?
1. ____Full-time (35 hours or more)

3. ____Irregular (less than 17 hours)

2. ____Part-time (17 – 34 hours)

4. ____Not working or looking for work

If unemployed, what is your reason for unemployment?
1. ___ No desire to work

7. ___ In treatment

2. ___ Student

8. ___ Incarcerated

3. ___ Homemaker

9. ___ Seasonal Worker

4. ___ Retired

10. ___ Temporarily Laid-Off

5. ___ Physical reasons

11. ___ Supported by other

6. ___ Mental illness

12. ___ Looking for work

What programs have you been involved with while under supervision?
1. _____Counseling

If yes, what type? ____________

2. _____Community Service

8. _____ Forest Project

3. _____Victims Panel

9. _____ Drug/Alcohol Treatment

4. _____Cognitive Restructuring

10. _____The Learning Center

5. _____One-Stop Employment

11. _____Other

6. _____Day Reporting Center

12. _____ None

7. _____Work Release

13. _____ Pending Referral

17)

17a)

How long have you been with your current P.O.?

__________(months)

On a scale of 1-4 please rate your current Probation/Parole Officer’s performance.
1 = Poor, 2 = Fair, 3 = Good, 4 = Very Good
Timeliness
1 - 2 - 3 - 4
Availability
1 - 2 - 3 - 4
Understanding of your situation
1 - 2 - 3 - 4
Ability to relate to you in a respectful manner
1 - 2 - 3 - 4
Informative about programs
1- 2 - 3 - 4

18)

Have you had more than one Probation/Parole Officer during this supervision?
Yes: _________________________ No: ___________________

19)

If yes, has it been problematic?

20)

Regarding your experience with the supervision process in general, would you describe it
as…

Positive ____ or

Yes: _____

No:_____

Negative _____

Why? __________________________________________________________________

20a)

Has your perception of the supervision process changed since the beginning of your
current supervision?

Yes: _____

No: _____

21)

Currently, what is your level of concern in the following areas?
Please answer low, high or none.

Low:_____

Finding a Job
High:_____
None:_____

Managing your finances
Low:_____

High:_____

None:_____

Returning to a neighborhood where criminal activity is common
Low:_____

High:_____

None:_____

Community rejection
Low:_____

High:_____

None:_____

Family rejection
Low:_____

High:_____

None: _____

Facing situations where criminal behavior might be tempting
Low:_____

High:_____

None:_____

Facing situations where drugs or alcohol are easily available
Low:_____

High:_____

None:_____

22)

On a scale of 1-4 please rate how your supervision was helpful in the following

areas……………………..
Reducing attitudes that might lead to crime.
1–2–3–4
Increasing educational opportunities.
1–2–3–4
Reducing interest in hanging out with individuals that might be involved with crime.
1–2–3–4
Offering job assistance or employment training.
1–2–3–4
Learning new problem solving skills.
1–2–3–4
Developing an understanding of your impact on the community.
1- 2 – 3 – 4
Developing an understanding that even if family and friends commit crime their actions
should not be followed.
1–2–3–4
Please rate your over all experience with the people in probation and parole
1–2–3–4
23)

Was there anything that you feel you need to add, or that you feel we might have left

out?_________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
Multnomah County Department of Community Justice/Portland State University
49

PROGRAM EVALUATION
Overview
Program evaluation is beneficial in several ways:
• It allows a program to be “tracked” from development through implementation, to ensure that
it has been implemented faithfully.
• It requires that stakeholders carefully define “success” and decide how to measure it.
• It measures a program’s progress towards its goals and objectives, letting stakeholders know
when success has been achieved, or when change is necessary.
• It allows new “best practices” to be created, as organizations can cite evidence to prove their
success.
Evaluation methodologies too numerous to count are available to an organization that wishes to
define and measure success. An evaluation strategy can be tailored to the type of program, its
size, goals, and available resources. Evaluations are most often used in three ways: to pilot test
new or adapted programs or materials, called formative evaluation; to test the implementation of
a program, called process evaluation; and to measure progress towards outcomes, called outcome
evaluation. The design of a comprehensive evaluation plan is beyond the scope of this manual,
but a general overview of the process is provided.
Principles
Evaluation planning should begin when program planning begins.
The evaluation process is much easier, more comprehensive, and more accurate if
evaluation activities are incorporated into the program from the outset. Also, the
systematic process of designing and implementing an evaluation often results in ongoing
program improvement.
All key stakeholders should be involved in the development of the evaluation plan.
Input from many people will help ensure that the optimal evaluation design is chosen,
and that staff and resources will be committed to following through with the evaluation.
A knowledgeable evaluator should guide the process.
Evaluations can be very complex, and many decisions must be made to balance the needs
of stakeholders, available resources, and sound research. An experienced evaluator is
required to do this successfully.
The more rigorous the evaluation methodology, the more reliable the results.
The more carefully the evaluation is conducted, the more meaningful the results are. This
can have implications not only for determining if the program was successful, but also for
evaluating the fidelity to the program model, whether or not a new “best practice” has
been created, and whether the program is a good candidate for funding or replication.

Components
• Formative Evaluation
Formative evaluation, or “pilot testing” is the process of testing program components on a small
group and soliciting feedback before the program is implemented on a larger scale. Formative
evaluation is an important step when implementing a new
program, or when adapting an existing program for a new
Qualitative Data
population. Though it requires more resources initially, formative
are the answers to
evaluation saves resources in the long term by finding and
open-ended
addressing problems that would reduce a program’s efficacy. For
questions.
example, if a small group of officers is trained on an assessment
Ex.: What aspects of
tool and then can give feedback on the training, then the training
the treatment did you
can be improved before all officers are trained.
find helpful?
Any response is
Formative evaluation is often done by collecting qualitative (openpossible. Results
ended) data, so that respondents are able to give whatever
are analyzed by
feedback they feel it appropriate. Sometimes the feedback
grouping similar
provided is completely unexpected, so it is important that
responses.
respondents are not limited in their response options. The data is
usually collected through focus groups, individual interviews, or
written questionnaires after participating in the program being tested. As with survey data, it is
important that the questions are carefully written.
Evaluators could test anything from a worksheet to an entire curriculum. Participants in a
formative evaluation are usually a convenience sample, such as probationers from one region,
rather than a random sample. A convenience sample is less resource intensive and allows
evaluators to target a specific group (for example, testing readability of materials with offenders
for whom English is a second language).
• Process Evaluation
Process evaluation measures the implementation of a program, including whether it was
implemented as intended, whether the intended audience participated, and whether participants
were satisfied with the program.
The first step of a process evaluation is choosing what questions should be answered. In many
cases, the process evaluation questions will align with quality assurance indicators. Are
assessments and case plans being completed for all offenders? Are offenders attending
treatment? Are offenders satisfied with treatment? The number of process questions asked
depends on what is being measured and the resources available to track the answers.

The second step is determining how the data is going to be collected. Most process data is
collected using paper or electronic tracking forms that are completed and submitted on a regular
basis, either by the staff member completing the task or an
evaluator reviewing the task. For example, an officer may
Quantitative Data
track how many intake assessments he or she does each
are the answers to
month, then a member of the peer review team may track the
closed-ended
accuracy and completeness of the assessment. Some process
questions.
data may be tracked by survey, such as participant
Ex.: On a scale of 1 to 5,
satisfaction with an activity. Process data may be qualitative
how would you rate the
or quantitative (close-ended). In most cases, quantitative
instructor’s level of
data is collected because it is much easier to gather, analyze,
knowledge?
and report. Some process measures, like client satisfaction,
are amenable to qualitative data collection if resources
Responses are limited.
permit.
Results are analyzed
by assigning
numerical values to
The final step is determining how the data is going to be
responses and using
used. The data that is gathered must be able to be compiled
statistical analysis.
and reported in a timely manner, so that it can be used in a
timely way. Electronic information systems are very helpful
with this. Then, as with all other data, stakeholders must decide how it will be applied. The
stakeholders may also decide to change the evaluation process if the data that they are receiving
is not meeting their needs. However, once evaluation methods have changed, new data cannot
be compared to old data.
• Outcome Evaluation
Outcome evaluation measures whether a program achieved its stated goals. The first step in
conducting this type of evaluation is determining which of the program’s outcomes should be
measured by the evaluation. This could be a short-term outcome, such as an increase in offender
knowledge after one session of group therapy, or a longer-term outcome, such as recidivism. In
general, it is more difficult to measure longer-term goals, and to prove that they are a result of
the program. For example, an evaluator may have to choose between measuring the goal “the
offender will not recidivate during the period of probation” or “the offender will not recidivate
during the two years following probation.” The program manager may want to know whether or
not the program has an impact beyond probation. However, it is much more difficult to keep
track of offenders after they leave probation, to accurately measure their rates of recidivism, and
to be sure that their lack of recidivism is due to the program and not due to something that
happened after probation. These longer-term evaluations provide valuable data, but are much
more resource intensive. So, depending on the resources allocated to the evaluation (and the
timeframe in which results are expected), realistic outcomes should be chosen.
The second step is determining how the data is going to be collected. There are many designs to
choose from and many possible methods of collecting data. Some methods are very inexpensive
but provide questionable proof that a program works, such as a convenience sample where

probation officers occasionally ask offenders if the have re-offended. Others are very
methodologically rigorous, but more resource intensive. For example, evaluators could follow a
group of offenders who participated in a treatment program and a similar group of offenders who
did not participate in treatment, and on a monthly basis interview
the offenders and check police reports and court documents for
Good programs
evidence of re-offense. This would be more reliable, but also
become
requires the dedication of more resources. In addition, there are
“Evidence-Based
ethical considerations in design, as the methods cannot violate
offender rights. However, some very simple, effective measures
Practice”
are available, such as conducting a pre- and posttest to measure
through good
the amount of knowledge participants gained in a treatment
outcome
group, and comparing offenders who completed treatment to
evaluation.
those who did not.
Again, the final question is how the data will be used. If the results indicate that the program
was successful and implemented according to plan, the evaluation results could be used to
publicize new best practices. If the outcomes were not achieved, the results can be used to
examine why. In most cases, one set of evaluation results will present more questions, and lead
to another, more detailed evaluation. For example, if an evaluation finds that male offenders
have more success with a program than female offenders, additional evaluation could uncover
why.
• Choosing an Evaluator
Clearly, there are many decisions to be made in designing and implementing an evaluation,
whether large or small. A team of stakeholders should be involved in decision-making, but the
team must be lead by a knowledgeable evaluator capable of guiding the process. The team must
decide whether this task should fall to an internal staff member, or if an external consultant
should be brought in. Either choice could be appropriate, but the following should be
considered:
•
•
•
•
•

The individual should have the necessary knowledge to design and implement the plan.
The individual should have the time to manage the evaluation. It may not be possible for
a staff member to balance an evaluation with other responsibilities.
The individual should be involved from the program’s inception, and should have a good
working knowledge of the program.
The individual should have a good relationship with the team and be able to exercise
leadership.
The individual should be aware of personal interests in the results, and conflicts of
interest should be avoided.

A good evaluator, who contributes to effective program development and implementation as
well, is a key member of the project team. The evaluator’s role should be clarified at the onset of
the program.

• Logic Models as Evaluation Tools
To measure whether a program was effective and why, there must be an understanding of how
the program is supposed to work. This includes both the mechanics of the program and the
theories behind its effectiveness. For example, in developing a cognitive-behavioral job skills
program, each lesson would include a chance to practice a job skill such as arranging a job
interview. The reason for this is that the program designers know that if they create the activity,
provide the materials for it, and run it (the mechanics), that offenders are more likely to learn and
use the skill because they have the chance to practice it (the theory). This chain of events is
often implicit, but each link in the chain must be in place in order for the program to be
successful.
Activity Æ Materials Æ Activity
Designed
Obtained
Implemented

Æ Offenders Æ Offenders Æ Offenders
Participate
Learn Skill
Practice Skill

Any of these links in the chain could mean the difference between whether a program works or
does not work. When a program is evaluated, it is important to understand why it did or did not
work. In the example above, even if the activity is perfectly designed, it may not be effective
without enough time to practice. Or, maybe the skill is not one that offenders really need, so
they don’t have an opportunity to use it. When a program does not work, designers need to
know what link in the chain broke down so that it can be fixed (or what link was missing in the
first place). When a program does work, designers want to know which links need to be
replicated and which can be eliminated or redesigned. To achieve that level of understanding, all
of the links in the chain must be explicitly mapped out.
A logic model, also called a process map, is a flexible tool for mapping the mechanics and theory
of a program. Creating a logic model forces designers and stakeholders to identify the resources
that are available for a program, describe what will occur in the program, and describe the
desired outcomes. The model can then be used to design the program itself, to design a program
evaluation, and to analyze evaluation results and determine what made a program succeed or fail.
The model also gives everyone involved in the project an idea of the “big picture.” A great deal
of literature is available on the creation of logic models, and a sample is available in this manual.
Components of a Logic Model
Inputs: The resources being put into the project (staff, materials, etc.)
Activities: What is being done in the project (treatment groups, etc.)
Outputs: Direct product of activities, usually numerical targets (# of offenders
trained on a skill, etc.)
Initial Outcomes: Short-term changes, such as an increase in knowledge.
Intermediate Outcomes: The next step in the change process, often applying new
knowledge or skills.
Long-Term Outcomes: The ultimate goal of the project, often a behavior change.

SAMPLE: LOGIC MODEL
Program: County Community Corrections Substance Abuse Intervention
Outcomes
Inputs
•

•

•
•

•

Department will
provide probation
officers to make
referrals and meeting
space.
Tx provider will
provide certified
facilitator and
educational materials.

Activities
•

•

Department will
provide probation
officers to make
referrals.
AA/NA will provide
meeting structure.
Several local churches
will provide space for
meetings.

•

Department will
provide probation
officers, facilities, and
materials for drug
testing.
Local lab will contract
to provide urinalysis
screening and report
results.

•

Probation officers will
complete referrals to
TX group within two
weeks of completing
case plan.
Tx provider will
complete intake
within 2 weeks of
receiving referral.
Offender will attend
groups twice weekly
for 20 weeks.
Probation officers will
complete referrals to
AA/NA within two
weeks of completing
case plan.
Offenders will attend
weekly meetings for a
minimum of 24
weeks.
Probation officers will
begin urine testing of
offenders at first
check-in.
Probation officers will
conduct testing
according to offender
risk level at check-ins
thereafter.
Officers will respond
to test results
according to policy.

Outputs
•

•

100% of offenders
with substance abuse
Tx on their case plan
will be enrolled in the
Tx group.
80% of offenders who
enroll in the Tx group
complete it.

Initial
•

•

100% of offenders
with substance abuse
Tx on their case plan
will be referred to
AA/NA groups
80% of offenders who
are referred will
attend weekly AA/NA
meetings for 24 weeks

•

100% of high-risk
offenders with
substance abuse Tx on
their case plan will be
urine tested at every
check-in.
100% of medium-risk
offenders with
substance abuse Tx on
their case plan will be
urine tested at least
twice monthly.

•

Offenders will gain
knowledge of the
impact of their
substance use.
Offenders will gain
skills for addressing
triggers of substance
use.
Offenders will
increase motivation
and self-efficacy to
discontinue use.
Offenders will
develop a prosocial
support network.
Offenders will gain
knowledge of the
impact of their
substance use.

Offenders will gain
knowledge of the
consequences of
continuing substance
use.

Intermediate

•

Long-term

Offenders will address
triggers for substance
use in prosocial ways.

•

Offenders will utilize
prosocial supports to
avoid substance use.

Offenders will
discontinue substance
use.

•

Offenders will remain
substance-free for the
duration of their
probation.

Offenders
experiencing relapse
will utilize Tx options
to regain sobriety.

INDIVIDUAL PERFORMANCE MEASUREMENT
Overview
To fully incorporate evidence-based practices into the culture of an agency, the language and
methodology of EBP must be incorporated system-wide, and staff must be supported and held
accountable for implementing, maintaining, and measuring those practices. This includes
incorporating evidence-based practices into the performance measurement system. An effective
performance appraisal measures what the organization values. Thus management and line
officers must be evaluated on how aligned their management or line practices are with EBP
practices and principles. As job expectations change, the measures against which performance is
evaluated also need to change, so that everyone is on the same page about what constitutes good
performance, and so that staff members have an opportunity to be acknowledged for what they
do well and to learn how to prepare for advancement opportunities.
It is important that individual performance reviews are designed to support managers and
officers to meet organizational expectations and goals. If performance is being judged by out-ofdate criteria that are not in alignment with the organization’s desire to implement EBP, there is
little or no extrinsic motivation to incorporate EBP into daily practice. Management and line
staff must be supported and held accountable to meet these new expectations. Just as line staff
must focus on what offenders do well, so must managers support staff to continue practices that
align with EBP and to change behaviors that are not consistent with EBP. Following-up on areas
for change with performance improvement plans, providing skill development opportunities, and
when necessary, taking disciplinary action are all necessary steps in the appraisal process.
Principles
Performance reviews should be ongoing.
To allow staff to incorporate feedback and improve performance, supervisors should
provide ongoing assessment and reinforcement. It does not benefit the officer nor the
agency to ignore behavior, positive or negative, until an annual review.
Performance criteria should be explicit and measurable.
Staff should have clear expectations of the job responsibilities and expected outcomes.
The process for measuring achievement should be clearly defined and consistent.
Performance criteria should align with desired outcomes.
To encourage commitment to evidence-based outcomes, staff at all levels should be
measured by their ability to meet desired outcomes rather than unrelated, outdated, and
possibly contradictory criteria.

Reviews should focus on positive behavior and provide opportunities for improvement of
skill deficiencies.
Everyone is more responsive to positive feedback. Procedures should be in place to
recognize and reward positive performance. Organizations must be willing to follow-up
on negative evaluations through skill development opportunities, performance
improvement plans, or discipline when necessary.
Distinguish between performance review and coaching.
When implementing new practices, staff needs an opportunity to practice without fear of
repercussions. Over time, however, everyone must expect that they will be evaluated on
their use of EBP, and that they may face consequences for failing to use those practices.
Components
•

Establish a regular schedule for formal supervision, encompassing observation and
performance reviews.
Performance appraisals should not be a once a year occurrence. At the annual appraisal meeting,
the supervisor and employee should create a learning contract for the year. This can include
continuing and enhancing current practices to changing and adopting new practices. This
contract should be discussed on a regular basis and no less than once a quarter. By observing
employee performance, reviewing performance indicators, and meeting with employees
regularly, both positive and negative performance feedback is more immediate and useful.
Problems regarding behavior, skill, and attitude can be addressed without delay through
performance improvement plans and opportunities for additional skill development if
appropriate. Leaving performance issues unattended to until an annual performance evaluation
is scheduled leaves the organization suffering from possible poor productivity and employees
blindsided at review time. There should be no “surprises” in a performance evaluation.
• Create an environment open to ongoing communication, learning, and feedback.
Organizations implementing evidence-based practices must constantly collect data about their
progress, analyze that data, and make changes and course adjustments based on that feedback.
The same is true at all levels of the organization, including that of the individual employee.
Without open and ongoing communication, performance feedback, and learning between
supervisors and employees, progress toward providing improved services and reducing
recidivism is stalled. Information sharing and communication at all organizational levels are
critical to achieving performance improvement.
• Staff should evaluate their own performance.
A self-evaluation should be the starting point for the appraisal. The supervisor should always
review the employee’s self-evaluation with the employee in person. This creates more active
participation in the review process, and may increase the likelihood of reaching agreement on
what changes may need to occur. The self-evaluation provides valuable insight and information
regarding the employees’ perceptions of their job responsibilities, whether they are completing

them adequately, and whether they have the training and resources necessary to fulfill their
professional goals. Many performance failures are rooted in a lack of understanding and
agreement by supervisors and employees regarding what constitutes effective performance.
• Staff should provide feedback on their supervisor’s performance.
The relationship between an employee and their supervisor, including the quality of the
communication and the ability of the supervisor to train, model, and give feedback on skills can
have a significant influence on ability of an employee to fulfill his or her responsibilities.
Employees should be given an opportunity to evaluate supervisors so that they can give feedback
on the quality of that relationship. (In a way, this is a customer satisfaction survey of the
supervisor/employee relationship. The “services” provided by the supervisor are only effective
if the employee finds them beneficial.) This can be done informally in concert with an
employee’s review, or through a formal written (and preferably confidential) survey
administered by the peer review team or management.

Performance Appraisals:
The Questions

Rewarding
Exceptional Performance

While ongoing performance review
can be an informal process, an
annual performance appraisal
should be done using standardized
guidelines. Both the employee and
the supervisor should be clear on
what is being assessed.

A positive performance appraisal is
most often equated with a salary
increase or a promotion. However,
there are many ways to formally
and informally recognize quality
work.
•

Publicly acknowledge the
contributions of individual
employees or teams (for staff who
enjoy public recognition).

•

Offer incentive gifts, special
activities, or time off.

•

Provide advanced professional
development opportunities.

•

Shift workloads: offer smaller,
specialized caseloads, or more
leadership opportunities with less
administrative work (do not reward
good work with even more work)!

•

Provide opportunities for
leadership and participation in
organizational development
activities.

Potential questions are:
•

•

What was the employee expected to
accomplish (i.e. his/her job
description)?
What did the employee
accomplish?
How was the employee able to
achieve these accomplishments
(i.e. skills and strengths)?

•

What goals were not met, and why?

•

What potential for improvement
exists, and what is the plan for
professional development?

Sample: Employee Observation Evaluation
Employee Name:

Activity Observed:

Observation Time (amount):
Behavior

Start Time:

Below
Needs
Expectations Improvement

Stop Time:

Place of Observation:

Meets
Exceeds
Comments on Direct
Expectations Expectations Observations

Uses cognitive-behavioral
language during encounters with
clients.
Models appropriate language
and behavior to clients.
Includes: Speaking positively
about program, law, courts, etc.
Does not use derogatory
language/jokes op sarcasm.
Avoids power struggles with
clients (e.g., does not argue with
clients, raise voice at clients,
antagonize clients)
Consistently applies appropriate
consequences for behaviors
(both positive and negative)
Identifies thinking barriers in
clients in value-neutral way
Overall Score

Sample: Supervisor Evaluation
Supervisor ______________________________________ Date: ___________________
This evaluation is confidential. It is important to have your honest opinion so that we can address strengths and
needs of all staff as well as provide needed training. Thank you for taking time to assist in bettering our team.
1.

My supervisor is firm and fair and has effective use of authority.
_____strongly agree _____agree

My supervisor models and reinforces pro social behaviors through positive and negative reinforcers.
(Knows how to reward appropriately and how to stop negative behavior without damaging morale or
the dignity of staff.)
_____strongly agree _____agree