Tn Doc and Cca Comparative Evaluation Guidelines 2001
Download original document:
This text is machine-read, and may contain errors. Check the original document to verify accuracy.
APPENDIX G COMPARATIVE EVALUATION GUIDELINES For the Comparative Evaluation of the South Central Correctional Center RFP and Contract Performance November 2, 2001 Table of Contents 1: OVERVIE\V 3 2. METHODOLOGy 5 2.1. PERFORMANCE COMPARISON 2.1.1. PERFORMANCE EVALUATION MEASURES T)'PES OF MEASURES 2.1.2. 2.1.3. SCORING 5 5 6 6 3. PERFORMANCE EVALUATION INFORMATION COLLECTION 7 4. PERFORMANCE EVALUATION INFORMATION VALIDATION 8 S. VALUE AND WEIGHTING OF PERFORMANCE MEASURES 8 6. CONDUCTING THE I>ERFORMANCE EVALUATION 8 7. COST COMPARISON 9 2 1. OVERVIEW The Public Acts of 2001, Chapter 132 amended TCA Title 41, Chapter24 relative to the comparative evaluation process for private contracts for correctional services. The new language charged the State of Tennessee with establishing objective performance criteria and cost criteria for both the State and the private contractor. These performance and cost criteria are to be used as the basis for a comparative evaluation between the operations of the state's facilities, as set out in the Request for Proposal, and the privately operated facility known as South Central Correctional Center (SCCC). This comparison is to be made after the end of the second full year of operation, but before renewing the initial contract at the end of the three-year contract term. The performance evaluation is to be performed by the Select Oversight Committee on Corrections. The cost comparison is to be compiled by the Fiscal Review Committee. The contract can only be renewed if the contractor is providing essentially the same quality of services as the state at a cost of 5% lower than the state, or if the contractor is providing superior services (greater than 5%) at essentially the same cost as the state. TCA 41-24-105 requires the following: a) Before the awarding of any contact to provide correctional services as defined in Section 41-24-1 04(2)(F), the state shall establish objective performance criteria and cost criteria for both the state and private contractor. The performance criteria shall measure the quality of management, security and safety, personnel training, inmate programs, and treatment and other topics deemed appropriate. The performance criteria and cost criteria shall be established and incorporated as requirements in any proposed request for proposal and any contract and shall be used as the basis for any comparison between the state and any contractor. b) For any contractor to provide correctional services as defined in Section 41-24102(2)(F), the initial contract term shall be for a period of three (3) years in order to allow the contractor sufficient time to demonstrate its performance and to provide sufficient information to allow a comparison of the performance of the contractor to the performance of the state in providing similar services. Provided, however, to allow sufficient time for completion and review of any comparative evaluation, the initial term may exceed three (3) years by a period of up to four (4) months as necessary to make the end of the initial term coincide with the state's fiscal year. The initial contract may include an option to renew for an additional period of two (2) years, subject to the provisions of this section. c) After the first two (2) years of operation, but before renewing the initial contract, the performance of the contractor shall be compared to the performance of the state for similar services as set out in the contract. The contract may be renewed only if the contractor is providing essentially the same quality of 3 services as the state at a cost of five percent (5%) lower than the state as set out in the contract, or if the contractor is providing services superior in quality to those provided by the state at essentially the same cost as set out in the contract. For the purposes of this statute and comparison, "essentially the same" shall mean the difference is no greater than five (5%). For the purpose of this statute and comparison "superior" shall mean a difference greater than five percent (5%). The methodology for determining the measurement of five (5%) differences shall also be set out in the request for proposal and contract. d) The quality of services provided by the contractor and by the state shall be compared by the select oversight committee on correction, or, in the absence of such committee, a committee designated by the speakers of the senate and house. The committee shall determine the quality of services provided by the contractor and the state by applying the performance criteria set out in the request for proposal and contract pursuant to subsection (a) and provide a numerical score for the state and the contractor. The committee shall report its determination to the parties responsible for determining whether the contract should be renewed. e) The fiscal review committee, or, in the absence of such committee, any other committee designated by the speakers of the senate and house, shall compare the cost measures as established in this chapter and set out in the request for proposal and the contract for service and provide a prisoner per day cost for the state and the contractor. The committee shall report its determination to the parties responsible for determining whether the contract should be renewed. As a means of satisfying the statutory requirement to develop objective performance and cost criteria, the Select Oversight Committee on Corrections (SOCC) director, the SOCC consultant, a member of the Fiscal Review Committee, as well as Department of Correction employees with a wide range of correctional expertise, initiated the development of a performance evaluation process and assessment and scoring instruments. It was decided that Northeast Correctional Complex (NECC) and Northwest Correctional Complex (NWCC) are the most comparable state facilities to evaluate and compare with the South Central Correctional Center (SCCC). The two state operated institutions have been selected for previous comparisons because of the similarity in age of the facilities, design of the facilities, and inmate populations. These two facilities continue to be the most comparable. Since the early institution comparisons were made, consolidation of state institutions resulted in other facilities being combined administratively with both NECC and NWCC. These consolidations add some challenges, such as adjusting for differences in the number of inmates, the comparability of inmate populations, etc.; however the evaluation process and instruments have been developed to compensate for these differences through the use of per capita ratios and other scoring measures. 4 As the process evolved, the methodological guidance of the Vanderbilt Institute for Public Policy Studies was sought and received. The co-director of the Center for Evaluation and Research Methodology, Dr. Mark Lipsey, assisted in reviewing the evaluation instrument for consistency of methodology. He confirmed that the approach adopted by the evaluation group was a fair way of comparing the performance of all three prisons because the criteria for the comparison apply equally to all three institutions and because the processes necessary to obtain the data required to support the resulting measurements are in place. Resulting from these meetings is the Performance Evaluation process and instrument that is included as a part of the Pro Forma Contract of the Request for Proposal under Appendix G. This instrument, identified as a Performance Evaluation data collection instrument is included with this Overview as Exhibit "A". 2. METHODOLOGY Since renewal of the contract is dependent upon an evaluation of the quality of services as well as the cost to provide those services, it is important that both performance and cost measures be comparable and quantifiable. The following methodology was developed with that in mind. 2.1. PERFORMANCE COMPARISON 2.1.1. PERFORMANCE EVALUA nON MEASURES The comparative evaluation is framed around four (4) major categories. Each category has a principal goal that expresses the desired result(s) for that particular category. The categories and goals are as follows: a) Security and Control The goal of the security and control category is that the community, staff, and inmates are protected from harm b) Inmate Care The goal of the inmate care category is that the institution provides for the basic needs, appropriate care, and programming of the inmate population c) Institution Safety & Physical Plant Conditions The goal of institution safety and plant conditions is that the institution environment meets national and state fire and safety codes, health codes and requirements and 5 provides for safe living and working conditions. with order and efficiency Emergencies are responded to d) Institution Administration The goal of institution administration is that the institution is managed in a professional and responsible manner The measurement of how'well the goals have been achieved will be measured in two (2) ways: Outcome Measures and Mandated Practices. Most terms or phrases used in the Outcome Measures and Mandated Practices are defined in Department of Correction policies. An easy reference document with a selection of these definitions is provided with the Performance Evaluation instrument as well. Requests for clarification of terms should be made during the Pre-proposal Conference or should be requested in the form of a Written Comment during the Request for Proposal process. The State's written responses will become part of the final contract. 2.1.2. TYPES OF MEASURES Outcome Measures: Outcome measures measure the results of the institution's operation. Outcome measures look at specific expected results within each major category. Mandated Practices: Mandated practices are those areas of institution operation, which must be effectively accomplished in order to achieve the goal. These mandated practices are items, which are done on a continual basis and are regularly reviewed for compliance with policies, procedures, or other regulations. 2.1.3. SCORING The scoring for the items in the performance evaluation instrument utilizes a deductive method. In this method, each institution starts with the same base score in each of the four categories of interest. Each institution may lose points if it does not meet baseline standards, but each institution may also gain points if they are in full compliance with the mandated items. In the outcome measure section, points are deducted from the base score according to the severity of the institution's deficiency. The scores in these sections generally range from 0 to -4, with 0 assigned to the accepted range based on the average level of the three institutions' previous performance on a particular item and -4 assigned to extremely deficient performance on the item. On some items in this 6 section, an institution may score greater than --4 points. This range of scoring was reserved to the most serious items, such as events of escapes, riots, hostages being taken, homicides, and suicides. In the mandated practices section, the institutions may be given additional points. The scores in this section are either 0 or 2, with 2 points awarded for items if an institution is in 100% compliance and 0 points awarded on those items if an institution is not in 100% compliance based on the latest annual inspection. The last category, Institution Administration, is the only exception. In this category, institutions are awarded 3 points for 100% compliance on each of the mandated items. This was done in an effort to standardize the percentage of total section points an institution may gain from its excellent performance on mandated items. At the end of each category, points from the outcome measure section will be deducted from the base score and points from the mandated section will be added to the score. The resulting score will count for a specified percentage of the entire score. Category Security and Control Inmate Care Institutional Safety Institution Administration TOTAL: Value Percent 35% 30% 20% 15 % 100 % This evaluation instrument will penalize institutions for poor (lower expected) performance but reward institutions for consistently superior performance on the day-to-day operation of their institution. In essence, this method of scoring takes into consideration that problematic events may occur even at the best run institutions, and this instrument seeks to recognize this fact in its attempt to balance the outcome measures and mandated items. A list of variables and the distribution of scoring of items is included with this appendix as Exhibit B. The scoring formula to be used is included as Exhibit C. 3. PERFORMANCE EVALUATION INFORMATION COLLECTION The source of the information needed to document performance will be from the following primary sources: a) The 2003-2004 Annual Inspection Report b) Tennessee Department of Correction records c) Contractor records 7 d) Tennessee Department of Correction and Contractor reports (weekly, monthly, and annual) To ensure fairness during the Fiscal Year 2003/2004 Annual Inspection process, appropriate representatives of both the state and the contractor will participate on the inspection teams at all three institutions, which are involved in the comparison. The Select Oversight Committee on Corrections and the Office of the Comptroller will send a representative to monitor the proceedings. Detailed guidelines for the inspection process shall be drafted by the Department of Correction with assistance from representatives from the Select Oversight Committee on Correction and the contractor as soon as practicable after the awarding of the bid. The guidelines shall include the make up of the membership of the inspection team, the clarification of TDOC policy and terms used in the performance instrument, and the details of the inspection process. The office of the State Comptroller shall review these guidelines. 4. PERFORMANCE EVALUATION INFORMATION VALIDATION The information supplied by the Department of Correction and the Contractor will primarily be verified from one of two sources: a) Fiscal Year 2003/2004 Annual Inspection b) TOMIS Reports This validation method has been used in previous comparative evaluations. Reliability on this method will be increased with the use of both state and contractor representatives on the inspection teams that review all three institutions. 5. VALUE AND WEIGHTING OF PERFORMANCE MEASURES Each of the four (4) Major Categories was given the aforementioned proportional value based upon the relative importance of that category in comparison with the others. These relative percentage values were arrived at through several discussions involving the SOCC director and consultant, select Department of Correction staff, and a representative for the Vanderbilt institute for Public Policy Studies. 6. CONDUCTING THE PERFORMANCE EVALUATION TCA 41-24-1 05(d) states that the quality of services provided by the contractor and the state are to be compared by the Select Oversight Committee on Corrections. 8 The SOCC will use the performance criteria established in the Request for Proposal and included in the Contract. The Performance Evaluation instrument and Scoring instrument will be used for this purpose. The state and the contractor will supply any information deemed necessary by the Select Oversight Committee on Corrections to complete the performance portion of the comparative evaluation. The state and the contractor will each appoint one person who will serve as the contact person for the SOCC. Those appointed individuals will be responsible for gathering whatever information is required and transmitting it to the SOCC. The performance evaluation will result in a numerical score for each institution in compliance with TCA 41-24-1 05(d). An average score for the two state institutions will be calculated to arrive at a single state numerical performance score. The state score will then be compared to the contractor score to determine if the statutory mandates established in TCA 41-24-105(c) have been met. An opportunity will be given to both the State and the Contractor to present facts or evidence to clarify any misunderstandings and correct any perceived misrepresentations of facts and data. The final draft report will be given to both the State and the Contractor for comment before it is delivered to the Select Oversight Committee on Corrections. If either the State or the Contractor choose, they can submit a written response to the final report which will be included when the report is submitted to the SOCC. 7. COST COMPARISON TCA 41-24-105(e) requires the Fiscal Review Committee to provide a prisoner per day cost for the State and the Contractor based upon cost measures set out in the Request for Proposal and the Contract. Those costs are to be used in the evaluation to determine if the Contractor is providing essentially the same quality of services as the state at a cost of five percent (5%) lower than the state, or if the Contractor is providing services superior in quality to those provided by the State at essentially the same cost pursuant to TCA 41-24-105(c). The financial information to be compared will be for the Fiscal Year 2003/2004. This is necessary in order to comply with the statutory mandates which state that the comparison is to be made after the second year of the Contract, but before any renewal can occur at the end of the third year. The FY04 information would be the most current information available at the time of the comparison evaluation and will match the review period that will be used for the performance evaluation. The institutions included in the Contract for comparison with the South Central Correctional Center (SCCC) are Northeast Correctional Complex (NECC) and Northwest Correctional Complex (NWCC). The two state operated institutions have been selected for previous comparisons because of the similarity in age of the 9 facilities, design of the facilities, and inmate populations. These two facilities continue to be the most comparable. Since the early institution comparisons were made, consolidation of state institutions resulted in other facilities being combined administratively with both NECC and NWCC. In order to restore a reasonable level of comparability, adjustments will be made for staff and operating costs. The cost comparison will review the full costs of the Contractor with the full costs of the State's comparable facilities (NECC and NWCC). The costs attributable to the Contractor will include any costs of monitoring the Contract incurred by the State, which would not have been incurred by the State otherwise. In addition to monitoring costs, other adjustments and allocations will be made. The cost comparison will be for the period of July 1,2003, through June 30, 2004. Allocations will be based on the following: a) Divide Central Office or Overhead costs between activities involving residential prisoners and other activities based on direct expenditures for residential facilities versus direct expenditures for other activities to obtain a percentage of Central Office or Overhead expenditures applicable to residential facilities. b) Allocate the amount of Central Office or Overhead expenditures applicable to residential facilities based on the census for each residential facility to the total census for all residential facilities. c) Expenditures for revenue generating activities such as commissary, inmate labor, inmate telephones, inmates fines, recycling, and art and craft sales at institutions are to be included in facility expenditures and will be offset by total revenues collected. Costs will be allocated to the South Central Correctional Center for: d) The pro rata costs of the Tennessee Offender Management Information System (TOM IS), which are applicable to the handling of information on prisoners assigned to the SCCC facility e) The amounts expended by the State for monitoring the Contractor's operations during the 2003/2004 fiscal year f) The amounts expended by the State for the benefit of the Contractor during the 2003/2004 fiscal year g) Any other amounts expended by the State (including any state agency) which would not have been expended by the State in the absence of the Contract 10 h) State overhead items determined not to be applicable to SCCC will not be added to the Contract cost Adjustments will be also made for: i) Year-end supply inventories j) Equipment items purchased for use in the facilities with a cost in excess of $1,000 will be deducted from the total cost of operations for all facilities. Equipment purchased for use by the State's monitors at SCCC will not be deducted from the State's cost of operating SCCC k) Expenditures for the use of motor vehicles and motorized equipment purchased for use by NECC and NWCC will be reduced by the depreciation/replacement factor included in the reimbursement rate to the Department of General Services The Fiscal Review Committee will further adjust the reported costs to ensure comparability in making the comparison of the relative costs of operating the facilities for the period of July 1, 2003 through June 30, 2004. This would include, but not be limited to the following: I) Any costs that appear to be made ahead of the time needed or are deferred to a subsequent period if, in the opinion of the committee staff, such costs are in an amount sufficient to materially affect the comparison m) The State's or the Contractor's costs for any program or functional areas which it determines to be not substantially comparable to the operations of the facilities bein"g compared n) Any cost items not accounted for in a similar manner 0) Necessary adjustments for population variance to include fixed and variable cost items for payroll and operational support expenditures p) The medical component of cost will be adjusted to equalize the costs of each facility due to the $4,000 stop-loss provision for medical care in the Contract Requests for clarification should be made during the Pre-proposal·Conference or should be requested in the form of a Written Comment during the Request for Proposal process. The State's written responses will become part of the final Contract. As required by the Contract, the Comptroller of the Treasury will review all accounting information submitted to Fiscal Review by the Department of Correction, and all accounting information provided by the Contractor to Fiscal Review is to be 11 analyzed by an independent accounting firm. The reports generated by those reviews will be utilized during the evaluation process. The Fiscal Review Committee staff will calculate the State's and the Contractor's cost per inmate day. The final draft report will be given to both the State and Contractor for comment before it is delivered to the Fiscal Review Committee. If either the State or the Contractor chooses, they can submit a written response to the final report that will be included when the report is submitted to the Fiscal Review Committee. 12