Performance Standards Questions and Answers Prepared December 23, 2003
Question: "With the three new UI performance standards (entered employment, job retention and earnings gain) when will the first measurements be available for right of first selection (which month of 2004)? Since there is such a delay in getting this information will we be able to get preliminary results from the 1st and 2nd quarter of UI data?"
Answer: First, some background: Participants who meet one of the three criteria for inclusion in the denominator in the 1st quarter of 2004 (this is termed their "event quarter") will be matched against UI wage data from the 2nd quarter of 2004 to determine whether they had earnings (this is counted as an Entered Employment [EE]). Although UI wage data from the 2nd quarter of 2004 will be available at 95% accuracy in the first week of November 2004, the PS45 Advisory Workgroup concluded that the data should be at least 98% accurate; as a result, UI wage data (which improves with age, so to speak) from the 2nd quarter of 2004 (which are being used to evaluate participants with a 1st quarter 2004 "event quarter") will actually be available in the first week of February 2005.
Similarly, UI wage data from the 3rd quarter of 2004 (which are being used to evaluate participants with a 2nd quarter 2004 "event quarter") will actually be available in the first week of May 2005.
With respect to contract performance as it pertains to Right of First Selection, then, it is expected that participants with a 1st quarter 2004 "event quarter" will have available UI wage data. Agencies will be able to get preliminary results for participants with a 1st quarter 2004 "event quarter" through some type of report in mid-December 2004. Final results (for the 1st quarter 2004) should be completed by mid-March 2005.
Note that this only pertains to Entered Employment. Job Retention (JR) and Earnings Gain (EG) both require UI wage data from three quarters after a participant’s "event quarter," so, for example, a JR/EG report on participants with a 1st quarter 2004 "event quarter" won’t be available until mid-August 2005.
When the information is available it will be expected that an agency will meet the base level as indicated throughout the contract period.
Question: "RFP Appendix B, Performance Standards, Use of UI Data: For the three standards linked to UI data, Entered Employment, Job Retention, and Earnings Gain, is the start date for individuals to be considered part of the denominator from the beginning of the contract?"
Answer: By "start date" I’m assuming that this means "employment start date." As indicated in the answer to the previous question, an individual who meets one of the three criteria for inclusion in the denominator in the 1st quarter of 2004 (this is termed their "event quarter") will be matched against UI wage data from the 2nd quarter of 2004 to determine whether they have (or whether they had) an Entered Employment. Therefore, the answer to this question is "No," because the first cohort (i.e., those with a 1st quarter 2004 "event quarter") won’t be analyzed to look for an ENTERED EMPLOYMENT until the 2nd quarter of 2004.
Question: "Information Only criteria – What does the asterisk denote after the work completed on both the Barriers Screened and Barriers Assessed?"
Answer: This was an error and the notation was omitted accidentally. The asterisk indicates that the term, completed includes those participants that complete the entire screening tool and it will also count participants who decline the screening tool and have the appropriate coding filled out on the screening tool.
Question: "RFP Appendix B, Performance Standards, Use of UI Data: Three criteria are listed under the heading that begins, ‘The denominator for the Entered Employment…’ Please describe the distinction between items two and three."
Answer: The third criteria listed for the denominator in Appendix B was accidentally truncated and is erroneous. It should read:
3. W-2 participants who, in a given quarter, (1) moved from a non-CMF or non-CMU placement to a CMF or CMU placement or (2) were never in W-2 (or not in W-2 in the prior month) and went straight to a CMF or CMU placement (participants moving back into a non-CMF/non-CMU placement in any of the 3 following quarters are excluded).
The distinction between items 2 and 3, then, involves whether the participant moves into a CMF/CMU placement (third criteria) or leaves W-2 entirely (second criteria) after being in a non-CMF/non-CMU placement.
Question: "RFP Appendix B, Performance Standards, Performance Standards Criteria Table, Contract Performance, Paragraph Six: Will DWD provide Reports of the UI database?"
Answer: DWD is prohibited legally from releasing or publishing wage data for individual PINs. However, DWD can, upon request, inform an agency of whether or not (yes or no) a particular PIN (or set of PINs) had an Entered Employment, a Job Retention or an Earnings Gain.
Question: "In Appendix B, are the Literacy and Numeracy Gains and Customer Satisfaction Survey both contract-to-date measures?
Answer: Yes, this will be measured as contract-to-date.
Question: "Is the customer satisfaction an average of 24 months of scores?"
Answer: Actually, the customer satisfaction is an average of 23 months of scores. The survey is (or survey is completed) in March for all cases that are selected that meet the requirements as of the end of February. This is the way it is being done for the 2002-2003 Contract, and the Department will be measuring it in the same way for the 2004-2005 contract.
Question: "How will the State account for W-2 participants who leave due to SSI, death, marriage, or relocation out of state?
Answer: The Department will continue to remove SSI recipients from the Entered Employment denominator. However, it is expected that there will be a fluctuation in how cases leave W-2 and that is accounted for in the benchmarks. There will be no additional adjustments given for these types of situations.
Question: "Will there be a means of tracking W-2 participants statewide? For instance, will the State track a participant who leaves the caseload in Waukesha and enrolls in W-2 in Milwaukee County two months later?"
Answer: The Department tracks all W-2 participants.
Question: "When a person leaves FSET and enrolls in W-2, will the system track the continuation on a benefit program?"
Answer: In regards to calculating the performance for the UI standards when a person leaves W-2 but returns to W-2 within the 3 quarter follow up period they would be removed from the Universe. This will also be the case for cases that are on W-2 then go on to FSET and FSET cases that go onto W-2.
Question: "Contract Compliance: How is an acceptable level of performance defined in the contract compliance section relating to Entered Employment, Job Retention, and Earnings Gain?"
Answer: Entered Employment, Job Retention, and Earnings Gain that utilize Unemployment Insurance data each have a listed benchmark for an acceptable level. The reference to acceptable level also means that all the other required performance standards must be maintained at the base level for the entire contract.
Question: "Performance Standards Criteria Table, Contract Performance: Because reports PS45 #1, #2, #4, and #8 are not currently available, can DWD describe what the reports measure and the acceptable level of performance associated with each report?"
Answer: PS45 #1, #2, #4, and #8 are just an educated guess that the current reports labeled PS23 #1, #2, #4, and #8 will be changed to reflect the new contract period. The reports will be the exact same reports that are currently available under the PS23 label. The acceptable level will be the benchmark that is associated with each of the standards.
Question: "Performance Standards Criteria Table, Basic Education Activities: What is considered an acceptable testing tool for measuring Literacy and Numeracy Gains? What is an acceptable gain?"
Answer: An acceptable testing tool will be further defined in the Information Map. We are waiting to see if the Federal Government clarifies what this means in their standards. It is expected that tests such as the TABE will be acceptable.
An acceptable gain will be set for each participant individually when taking into account barriers and current education levels. It should be set as a benchmark that is attainable within 6 months to a year. It should not be set at a level that the participant would attain it in a very short period of time, e.g., only 1 to 2 months.
Question: "Performance Standards Table Criteria, Basic Education Activities: Where will testing information be documented? Will reports measuring the Basic Education Standard be available?"
Answer: Testing information will continue to be collected on CARES screens WPED and history will be collected on WPTS. Reports will continue to be available for the Basic Education Standard. These reports will measure the success of Basic Education, English as a Second Language, Regular School, and Literacy Skills as claimed by the agency when completing WPCH. The test scores will be manually reviewed randomly.
Question: "Does the customer satisfaction survey consist of only the four questions listed in the standard?"
Answer: Yes, the current 8 questions in the 2002-2003 contract have been combined to form 4 questions for the new contract.
Question: "Will DWD provide Reports of the UI database?"
Answer: Yes, all Performance Standards will have reports.
Question: "Will agencies have the opportunity at some later time to provide input/feedback on the standards and their measurement?"
Answer: Agencies have been involved and provided feedback prior to the release of the RFP. There is also a Performance Standard Sub-Committee that is part of the W-2 Contract and Implementation Committee that is under the leadership of local W-2 agency representatives.
Question: "Customer Satisfaction Survey question #3 is a compound question. The first question asks if staff set up meeting times that fit into the participant’s work schedule. The second question asks if the participants were satisfied with service. These should be two separate questions. Question #4 also asks about overall customer satisfaction."
Answer: There was an error in question #3 on both the Required and Bonus performance measures. The second sentence of question #3 should not be included. Only Question #4 will address overall satisfaction.
Question: "Contract Performance – this is a multiple-part standard. It includes Contract Compliance, Corrective Action Plans, Financial Management, and maintaining acceptable performance on three bonus benchmarks: Entered Employment, Job Retention and Earnings Gain, thereby adding three mandatory performance standards."
Answer: Actually, this standard is measured the same way in both the mandatory and bonus levels with the exception of the benchmarks utilizing the UI data. There are several components and measurements to the Contract Performance Standard. Agencies must meet contract compliance and financial management. Agencies will also need to meet the benchmarks listed for Entered Employment, Job Retention, and Earnings Gain and there are different benchmark levels indicated for the required and bonus levels. In addition to this, agencies also need to maintain all the required performance standards at the listed base level for the entire contract and monitoring results must remain satisfactory.
Question: Customer satisfaction survey – these are same questions listed in the base criteria section, except they are numbered as #5, #6, and #7"
Answer: This was an error and they should be numbered 1-4. This will be changed in upcoming documents.
Question: "Timely and complete Processing of 24 and 60 Month Extension Requests: Will DWD provide pre-extension monitoring report for the AIWE entry standard? In other words will a report be available to show cases that need a determination?"
Answer: At this time there are no plans to develop a report of this nature. The Department may consider developing such a report in the future.
Question: Are the UI measures for Entered Employment, Job Retention and Earnings Gain point-in-time or contract-to-date measurement?
Answer: A W-2 or FSET participant who is included in the denominator (see Appendix B and associated corrections) is matched against UI wage data in a certain quarter (e.g., Jan.-Mar. 2003) to see if he or she earned a wage (greater than $0) during that quarter. In this sense, UI wage data are point-in-time data: they show whether or not an individual earned a wage (and if so, the amount of the wage) during a quarter as of the date the UI query or report is made. Like other point-in-time measures, though, each ratio (numerator/denominator) ultimately is combined over multiple quarters to arrive at a single contract-to-date ratio for each measure (ENTERED EMPLOYMENT, Job Retention and Earnings Gain).
Question: Under consortium structure, performance is measured in the aggregate. Do Iowa and Richland Counties get "border county" status along with Grant, Green, and Lafayette Counties?
Answer: All benchmarks as indicated in the RFP will remain the same regardless of consortiums that have both border and non-border counties.