RSA Fiscal Year 2008 Monitoring: State Vocational Rehabilitation and Independent Living Programs Information Guide U.S. Department of Education Office of Special Education and Rehabilitative Services Rehabilitation Services Administration State Vocational Rehabilitation and Independent Living Programs Information Guide CONTENTS Page INTRODUCTION 3 CHAPTER 1: SERVICE DELIVERY - VR PROGRAM 4 CHAPTER 2: SERVICE DELIVERY - IL PROGRAM 8 CHAPTER 3 SERVICE DELIVERY - OIB PROGRAM 14 CHAPTER 4: DATA MANAGEMENT 18 CHAPTER 5: CASE MANAGEMENT 21 CHAPTER 6: PLANNING 25 CHAPTER 7: POLICY 28 CHAPTER 8: PERSONNEL MANAGEMENT 29 CHAPTER 9: FISCAL MANAGEMENT 31 CHAPTER 10: SRC AND SILC 37 Introduction When the Rehabilitation Services Administration (RSA) monitors a state's vocational rehabilitation (VR) and independent living (IL) programs, the RSA monitoring team initially collects and analyzes a variety of information about those programs from multiple sources. The RSA team also engages in discussions on-site and off with the state agency staff, the State Rehabilitation Council, the Statewide Independent Living Council, persons with disabilities who receive program services, and stakeholders. These initial review activities are designed to enable RSA to fully understand how the state's VR, Supported Employment, State Independent Living Services, and Independent Living Services for Older Blind Individuals programs operate and perform. Following these initial activities, the RSA team works with the state to identify performance goals and strategies to achieve those goals, identify and correct compliance issues, identify promising practices, and design the technical assistance that RSA will provide to assist the state achieve its performance goals and take corrective action. The entire review process is described in detail in RSA's monitoring protocol. The State Vocational Rehabilitation and Independent Living Programs Information Guide (the Info Guide) is one of a number of sources that RSA uses to assist it to fully understand how the VR and IL programs operate within a state. RSA is sharing the Info Guide with state agencies and other interested parties in order to enhance the transparency of the review process and promote collaboration. The Info Guide addresses key program areas and its content is presented in the form of questions. The Info Guide is used by RSA: -as prompts to RSA monitoring teams to explore program areas and obtain certain key program information in the course of the review; and -in a variety of ways, for example RSA staff may pose some of the Info Guide questions during, the review or they may use the questions to direct their information gathering from various sources such as the state's: annual VR state plan; the state plan for IL; website; strategic plan; or various performance reports. In addition: -RSA shares the Info Guide with review participants as a courtesy and does not expect the state agency or other parties to submit answers to the questions in the guide unless both parties agree that it would be advantageous to do so; and -RSA's information gathering is not limited to the areas included in the Info Guide, and there may be some parts of the Info Guide (e.g. order of selection) that are not applicable in a particular state. Chapter 1: Service Delivery - VR Program I. The VR Process A. referral B. application C. assessment and eligibility D. IPE development and implementation E. employment outcomes F. due process II. The Service Delivery System A. direct B. purchased C. general III. Order of Selection (00S) I. The VR Process These questions explore how the VR agency conducts service delivery throughout the VR process: referral, assessment, eligibility, IPE development and implementation, job placement, and achievement of the employment outcome. A. VR Referral 1. How does the agency develop and maintain referral sources, and who is responsible? 2. How does the VR agency schedule new referrals, and who is responsible? 3. How does the VR agency assure that referrals for VR services are handled in a timely manner? 4. What outreach strategies does the VR agency utilize to reach individuals with disabilities from targeted populations, including transition youth, non-English speaking, individuals who are deaf or hard of hearing, individuals who need supported employment services? 5. What methods does the VR agency use to communicate with consumers in their native languages or requested mode of communication? B. Application 1. How does the VR agency ensure that application forms are widely available throughout the state? 2. How does the VR agency process applications for VR services, and who is responsible? 3. How does the VR agency orient new VR participants to the VR program, and who is responsible? C. Eligibility/Assessment 1. How does the VR agency conduct the assessment to determine eligibility and the individual's priority under an order of selection, if the agency operates under an order? 2. How does the VR agency verify the individual's eligibility under Title II or Title XVI of the Social Security Act for purposes of presumed eligibility? 3. How does the VR agency provide trial work experiences to individuals, and what kinds of trial work opportunities are offered? 4. How does the VR agency address recidivism, i.e., the issue of individuals returning to the VR agency for services? D. IPE development and implementation 1. How does the state VR agency make pertinent information available (including in accessible formats) to individuals with disabilities in order for them to make informed decisions throughout the rehabilitation process, including options for developing the IPE, information about types of services available, qualifications of potential service providers, and consumer satisfaction with services? 2. How does the VR agency ensure the timely development and implementation of individual IPEs? 3. How does the VR agency ensure the IPE is consistent with and supports the IPE employment goal? 4. How does the VR determine financial need and potential cost sharing by eligible VR participants? 5. Does the VR agency provide direct job development and placement services, purchase those services, or use other strategies for these services? 6. How does the VR agency utilize assistive technology services to enhance consumer capacity to obtain or maintain employment? 7. How does the VR agency develop resources for long-term supports for individuals in supported employment? E. Employment outcomes 1. How does the VR agency ensure quality outcomes for consumers? 2. Does the state VR agency assess job retention beyond 90 days, and if so, how and for what period? F. Due Process 1. How does the. VR agency communicate due process provisions to consumers? 2. What are the VR agency's due process procedures? II. The Service Delivery System These questions explore the nature of the service delivery in more detail from the standpoint of internal and external service delivery. Information from these questions will provide an overview of the VR agency's external service delivery process, including the identification of external service providers, specific service provider arrangements, standards, monitoring, evaluation, and reports. 'These questions will also provide information on the monitoring and evaluation of the VR agency's internal service delivery process. A. Direct 1. Does the VR agency staff provide assessment, counseling and guidance, job placement services directly through its staff, including staff of agency-operated facilities? 2. How does the VR agency monitor and evaluate its internal service delivery system? 3. How does the VR agency use input from the SRC to evaluate its internal service delivery system? 4. Does the VR agency use the results of consumer satisfaction surveys as part of the process, and, if so, how? B. Purchased Information from these questions is closely aligned with information from the fiscal monitoring questions on contracts and third party arrangements. Therefore, for the overall view of how the VR agency purchases VR services with respect to contracts and third party arrangements, information from the service delivery and fiscal areas together should be taken into account. 1. What services does the VR agency purchase from community rehabilitation programs (CRN), including assessment, job training, and job placement services? 2. What is the process used by the VR agency to identify CRPs? 3. What CRPs does the VR agency use for VR and SE service delivery? 4. Does the VR agency use other partners for VR and SE service delivery, including those used through third party arrangements, and if so, who are they? 5. Does the VR agency maintain performance standards for service providers, and, if so, what are those standards? 6. If the VR agency maintains standards for service providers, how are the standards developed, and how does the VR agency use input from the SRC in the development of the standards? 7. How does the VR agency monitor and evaluate its external service providers, in particular with respect to whether these programs result in employment outcomes and quality employment outcomes? 8. How does the VR agency use evaluations of service providers to determine the adequacy of services, whether services of the providers should be continued, or what performance goals should be required for satisfactory performance? 9. Does the VR agency require periodic reports from service providers, and, if so, what information is required and how is it used to evaluate and improve the effectiveness of service provision? 10. Does the VR agency utilize input from the SRC in the identification and evaluation of service providers? 11. Does the VR agency use the results of consumer satisfaction surveys as part of the process in the evaluation of service providers, and, if so, how? 12. What type of monitoring and evaluation systems do service providers have in place to ensure the quality of services and outcomes of VR participants? 13. What service provision reports are generated from the VR agency's MIS system and how are those report used to improve performance and/or revise policies/standards to improve service delivery? 14. Do service providers provide self-monitoring reports to the VR agency or only reports requested by the VR agency? 15. How does the state VR agency promote and maintain collaborative working relationships with the service delivery network, including employers, other state agencies, community-based programs, WIA partners, state education agencies, centers for independent living, and institutions of higher education? 16. How do service providers market their services to the VR agency? C. General (applies to both internal and external service delivery) 1. How does the VR agency market VR services to the public? 2. How does the VR agency solicit input from consumers, the public, the SRC, and the CAP about the service delivery system? 3. How does the VR agency use information solicited from consumers, the public, the SRC, and the CAP to improve the service delivery system? 4. What service provision reports are generated from the VR agency's MIS system and how are those report used to improve performance and/or revise policies/standards to improve service delivery? 5. Do state personnel policy or union contract rules allow the agency flexibility to provide performance incentives to internal and external service providers? III. OOS These questions explore how the VR agency determines the sufficiency of resources to serve all eligible individuals and how the VR agency develops and implements the OOS. 1. How does the VR agency determine the sufficiency of its resources, both fiscal and personnel, to serve all eligible individuals? 2. How does the VR agency solicit input from the SRC, the CAP, consumers, and others about the OOS and its implementation? 3. If the VR agency operates under an OOS, how does it develop alternative resources for individuals in closed priority categories? 4. How does the VR agency communicate its OOS priority categories to consumers? 5. If the VR agency is operating under an OOS, how does it manage the waiting list to better serve all eligible individuals? 7. How does the VR agency ensure its external service providers adhere to OOS requirements? Chapter 2: Service delivery - IL program I. The IL Service Delivery System A. DSU-Administered Program B. Statewide Network of Centers II. The IL Process – Services Provided by the DSU Directly A. Referral and Application B. Eligibility Determination C. ILP Development, Goal-Setting and CSR Documentation D. IL Services Provision E. IL Goals Achievement F. Quality Assurance G. Due Process III. The IL Process – Services Provided by the DSU Via Grant/Contract A. Referral and Application B. Eligibility Determination C. ILP Development, Goal-Setting and CSR Documentation D. IL Services Provision E. IL Goals Achievement F. Quality Assurance G. Due Process IV. Other Service Provider Requirements A. Recordkeeping, Access and Reporting B. Protection, Use and Release of Personal Information I. The IL Service Delivery System A. DSU-Administered Program 1. The DSU does / does not provided IL services directly: - through its specialized IL staff and/or its VR counselors - with Part B funds and/or state match funds and/or other state funds 2. The DSU does / does not provide IL services indirectly ... through grants and/or contracts - with Part B funds and/or state match funds and/or other state funds - to CILs and/or non-CIL providers for general CIL operations and/or specific IL services B. Statewide Network of Centers CIL Name:___ City, State:___ Part B Funds:___ State Match Funds:___ Other StateFunds:___ Part C Funds:___ C. Non-CIL Providers II. The IL Process — Services Provided by the DSU Directly A. Referral and Application 1. Does the state agency have written policies and procedures regarding IL referrals and applications? 2. What outreach strategies does the state agency utilize to reach unserved or underserved areas and populations, including minorities, in accordance with the approved SPIL? 3. Does the state agency have a cross-referral process involving its VR and IL programs as well as the 01B program, the CILs, the SILC and other entities? B. Eligibility Determination 1. How does the state agency determine applicants' eligibility or ineligibility for IL services? 2. Does the state agency have written policies and procedures regarding eligibility determination? 3. Do the agency's policies, procedures and practices conform to federal law and regulations, including section 6.4 of the SPIL? C. ILP Development, Goal-Setting and CSR Documentation 1. Does the state agency have written policies and procedures regarding Independent Living Plan (ILP) development and consumer service records (CSR) documentation? 2. Do the agency's ILP development and CSR documentation policies, procedures and practices conform to federal law and regulations, including section 6.5 of the SPIL? 3. Does the agency ensure that IL goals are established and documented, whether or not the consumer signs an ILP waiver? 4. Are the documented goals meaningful? That is, do they relate to independence in one or more "significant life areas" identified in the 704 Report? D. IL Services Provision 1. Are the IL services provided by state agency IL specialists or VR counselors? 2. If IL specialists provide the IL services, to what extent does the state agency's training, performance evaluation and compensation system include the IL specialists? 3. If the VR counselors provide the IL services, how is the VR counselors' IL workload reflected in the sate agency's job description, training, performance evaluation and compensation system? 4. Are the state personnel providing IL services knowledgeable about the IL philosophy, including the four core services? 5. Do the state personnel providing IL services meet the staffing requirements outlined in section 6.1 of the SPIL? 6. Does the state agency provide other IL services, as indicated in the approved SPIL? 7. Do state personnel collaborate and coordinate with CILs in the provision of IL services? 8. How does the agency ensure the quality of its IL services? E. IL Goals Achievement 1. Are the consumers' achieved goals documented in the ILPs and/or CSRs? 2. Do the achieved goals related to independence in one or more "significant life areas?" F. Quality Assurance 1. How does the state agency define a quality IL service or outcome? 2. How does the agency assess the quality of its IL services and outcomes (e.g., consumer satisfaction surveys, interviews with consumers, available outcomes data and/or efficiency measures)? 3. What are the results of the state agency's latest assessment of IL quality? 4. Is the IL program's quality assurance system separate from or part of the VR's system? G. Due Process 1. How does the state agency ensure that it meets the requirements regarding the Client Assistance Program described in section 6.6 of the SPIL? 2. Are the agency's VR due process procedures available to IL consumers? III. The IL Process — Services Provided by the DSU Via Grant/Contract A. Referral and Application 1. Does the state agency monitor its IL grantees or contractors' (i.e., CILs) outreach strategies for unserved or underserved areas and populations, in accordance with the CILs' work plans and the approved SPIL? 2. Does the state agency engage in cross-referral practices with its IL grantees and contractors? B. Eligibility Determination 1. How does the state agency ensure that its IL grantees and contractors meet the eligibility requirements outlined in section 6.4 of the SPIL? C. ILP Development, Goal-Setting and CSR Documentation 1. How does the state agency ensure that its IL grantees and contractors meet the Independent Living Plan (ILP) requirements outlined in section 6.5 of the SPIL? 2. How does the state agency ensure that its IL grantees and contractors' consumer service records (CSR) documentation policies, procedures and practices conform to federal law and regulations? D. IL Services Provision 1. How does the state agency ensure that its IL grantees and contractors provide each of the IL core services, in accordance with the IL philosophy? 2. How does the state agency ensure that its IL grantees and contractors meet the staffing requirements outlined in section 6.1 of the SPIL? 3. How does the agency assess the quality of its IL grantees and contractors' services? E. IL Goals Achievement 1. How does the state agency ensure that the IL grantees and contractors' consumers pursue and achieve goals that are meaningful (i.e., related to independence in one or more "significant life areas"), and that they are documented properly in ILPs and/or CSRs? F Quality Assurance 1. How does the state agency assess the quality of its grantees and contractors' IL services and outcomes (e.g., consumer satisfaction surveys, interviews with consumers, available outcomes data and/or efficiency measures)? 2. What are the results of the state agency's latest assessment of its grantees and contractors' IL services and outcomes? G. Due Process 1. How does the state agency ensure that its IL grantees and contractors meets the Client Assistance Program requirements described in section 6.6 in the SPIL? IV. Other Service Provider Requirements A. Recordkeeping, Access and Reporting 1. How does the state agency ensure that it meets the recordkeeping, access and reporting requirements outlined in section 6.3 of the SPIL? 2. How does the state agency ensure that its IL grantees and contractors meet the recordkeeping, access and reporting requirements outlined in section 6.3 of the SPIL? B. Protection, Use and Release of Personal Information 1. How does the state agency ensure that it meets the protection, use and release of personal information requirements outlined in section 6.7 of the SPIL? 2. How does the state agency ensure that its IL grantees and contractors meet the protection, use and release of personal information requirements outlined in section 6.7 of the SPIL? Chapter 3: Service Delivery - OIB Program I. The OIB Service Delivery System II. The 01B Process — Services Provided by the State Agency Directly A. Referral and Application B. Eligibility Determination C. Case Record Documentation D. OIB Services Provision E. Quality Assurance F. Due Process III. The OIB Process — Services Provided by the State Agency via Grant/Contract A. Referral and Application B. Eligibility Determination C. Case Record Documentation D. OIB Services Provision E. Quality Assurance F. Due Process IV. Other Service Provider Requirements A. Recordkeeping and Access B. Protection, Use and Release of Personal Information I. The OIB Service Delivery System 1. The state agency does / does not provide OM services directly: -through its specialized OIB staff and/or its VR counselors -with chapter 2 funds and/or state match funds___ and/or other state funds . 2. The state agency does / does not provide OIB services indirectly: -through grants and/or contracts -with chapter 2 funds and/or state match funds and/or other state funds -to CILs and/or non-CIL providers -The state agency provides grants and/or contracts to the following service providers: Provider Name:___ City, State:___ Chapter 2 Funds:___ State Match Funds:___ Other State Funds:___ II. The 01B Process – Services Provided by the State Agency Directly A. Referral and Application 1. Does the state agency have written policies and procedures regarding OIB referrals and applications? 2. What outreach strategies does the state agency utilize to reach unserved or underserved areas and populations, including minorities? 3. Does the state agency have a cross-referral process involving its VR, OIB and other IL programs as well as the CILs, the SILC and other entities? B. Eligibility Determination 1. How does the state agency determine applicants' eligibility or ineligibility for OIB services? 2. Does the state agency have written policies and procedures regarding eligibility determination? C. Case Record Documentation 1. Does the state agency have written policies and procedures regarding the OIB case records documentation? 2. Do the agency's case records documentation policies, procedures and practices ensure that the 7-0B information required by federal law and regulation is collected and reported? C. OIB Services Provision 1. Are the 01B services provided by state agency OIB specialists or VR counselors? 2. If OIB specialists provide the OIB services, to what extent does the state agency's training, performance evaluation and compensation system include the OIB specialists? 3. If VR counselors provide the OIB services, how is the VR counselors' OIB workload reflected in the state agency's job description, training, performance evaluation and compensation system? 4. Does the state agency provide each of the OIB services outlined in section 752(e) of the Act? 5. Are the state personnel providing OIB services knowledgeable about the OIB program? 6. How does the agency ensure the quality of its OIB services? D. Quality Assurance 1. How does the state agency define a quality OIB service or outcome? 2. How does the agency assess the quality of its OIB services and outcomes (e.g., consumer satisfaction surveys, interviews with consumers, available outcomes data and/or efficiency measures)? 3. What are the results of the state agency's latest assessment of OIB services and outcomes quality? 4. Is the OIB program's quality assurance system separate from or part of the VR and IL's system? E. Due Process 1. What due process procedures are available to OIB consumers? 2. How does the state agency ensure that it meets the Client Assistance Program requirements in 34 CFR 364.30? III. The OIB Process — Services Provided by the DSU Via Grant/Contract A. Referral and A eplication 1. Does the state agency monitor the outreach strategies utilized by its OIB grantees and contractors to reach unserved or underserved areas and populations, including minorities? 2. Does the state agency engage in cross-referral practices with its OIB grantees and contractors? B. Eligibility Determination 1. How does the state agency ensure that its grantees and contractors properly determine applicants' eligibility or ineligibility for OIB services? C. Case Record Documentation 1. How does the state agency ensure that its grantees and contractors collect and report the 7-0B Report information required by federal law and regulations? D. OIB Services Provision 1. How does the state agency ensure that its OIB grantees and contractors provide each of the OIB services outlined in section 752(e) of the Act? 2. How does the state agency ensure that the OIB grantees' and contractors' personnel providing OIB services are knowledgeable about the OIB program? 3. How does the agency assess the quality of its 01B grantees and contractors' services? E. Quality Assurance 1. How does the state agency assess the quality of its grantees and contractors' OIB services and outcomes (e.g., consumer satisfaction surveys, interviews with consumers, available outcomes data and/or efficiency measures, third party evaluations)? 2. What are the results of the state agency's latest assessment of the quality of its grantees and contractors' 0113 services and outcomes? F. Due Process 1. How does the state agency ensure that its grantees and contractors meet the Client Assistance Program requirements in 34 CFR 364.30? IV. Other Service Provider Requirements A. Recordkeeping and Access 1. How does the state agency ensure that it meets the recordkeeping and access requirements outlined in section 34 CFR 364.37? 2. How does the state agency ensure that its grantees and contractors meet the recordkeeping and access requirements outlined in section 34 CFR 364.37? B. Protection, Use and Release of Personal Information 1. How does the state agency ensure that it meets the protection, use and release of personal information requirements outlined in 34 CFR 364.56? 2. How does the state agency ensure that its 0113 grantees and contractors meet the protection, use and release of personal information requirements outlined in 34 CFR 364.56? Chapter 4: Data Management I. RSA Report Preparation and Submittal A. Responsible preparer B. Preparation process C. Accuracy II. Agency Report Preparation, Use, and Needs Determination A. Identification of reports/tables and their content B. Use evaluation and need determination C. Accuracy I. RSA Report Preparation and Submittal A. Responsible preparer Who prepares the following reports? -The RSA-113, -The RSA-911, -The RSA-722, -The 704? B. Preparation process RSA-113 1. Are RSA-113 submittals to RSA timely? 2. What is the process for preparing the 113 report data elements? 3. Are there challenges in preparing the 113 and submitting it in a timely manner? RSA-911 1. How timely are RSA-911 submittals? 2. What is the process in preparing the RSA-911 report? 3. Are there challenges in preparing the RSA-911 and submitting it in a timely manner? RSA-722 1. Are RSA-722 submittals to RSA timely? 2. What is the process for preparing the 722 report data elements? 3. Are there challenges in preparing the 722 and submitting it in a timely manner? RSA-704 1. Are RSA-704 submittals to RSA timely? 2. What is the process for preparing the 704 report data elements? 3. Are there challenges in preparing the 704 and submitting it in a timely manner? C. Accuracy 1. For all RSA-submitted reports, what methods are used to ensure the accuracy of the data? 2. How often does the agency use RSA's editing program (ERA) for the RSA911? 3. Are other editing processes to ensure the validity of the data on all RSA-submitted forms? II. Agency Report Preparation, Use, and Needs Determination Vocational Rehabilitation and Independent Living programs A. Identification of reports/tables and their content 1. What reports are prepared for management, district office heads, counselors, legislators, and others? 2. Are data reports used in preparing state goals for the state budgeting or other legislative processes? 3. Are data reports used to measure progress on state plan goals? 4. How frequently are data reports prepared? 5. What data elements are tracked? 6. Do data elements analyze numbers of individuals served and placed? 9. If so, what quality data elements are analyzed? 10. Do the tables include data from other sources including other VR agencies? 11. Do the tables generated include an analysis of services rendered by service providers? 12. Are ad hoc tables prepared for management, legislators, and/or others? If so, who prepares those tables? 13. Is there a set of standard tables prepared for management or others? B.Use evaluation and need determination 1. Is there a process for determining the use of the tables/reports by management and others 2. As a result, are standard tables deleted or added? 3. How frequently are tables adjusted? C. Accuracy 1. What accuracy checks are utilized to ensure that your data in input correctly? 2. What checks are run of the agency's database to ensure that existing data are accurate? 3. How are reports checked for accuracy? Chapter 5: Case Management I. Design A. General B. Reliability C. Ease of Use II. Usefulness as a service provision tool A. For counselors to provide quality VR services to consumers B. For supervisors to provide quality supervision and guidance to counselors III. Usefulness as a management tool A. Fiscal B. Planning and policy C. Data collection IV. Usefulness for the IL program A. Service Provision B. Planning and Policy C. Data Collection V. Other I. Design A. General 1. Which part of the case management system requires manual input? 2. Is case management system interne- based? 3. Does the new system satisfy all the agency's needs as expected? 4. Would the agency recommend it to others? B. Reliability 1. What mechanisms are in place to ensure that counselors are coding information correctly? 2. Who checks the data input to ensure the accuracy, and how? For example, do you have different staff members inputting and verifying the data? 3. What safeguards are in place to ensure the integrity of the information in case power failures or other disasters? 4. How frequently is the system backed up? 5. Are there any built-in edits to ensure that the data are reasonable – closure dates don't precede application dates, birthdates are within the last 100 years, and many others? If so what are they and how do they function? 6. How does the agency ensure that multiple closure codes are used for individuals receiving services more than once in a year? C. Ease of Use 1. What are the steps involved in retrieving the information need for counselors and supervisors to perform their duties? 2. Can counselors query the information easily themselves or do they have to get information from the IT department? For example, it is easy to find out how many individuals left the process in. June 2007 as an applicant and why? 3. How is training provided to counselors and supervisors in the use of the case management system? Is the training comprehensive and timely enough? Are there aspects of the system for which more or better training should be provided? II. Usefulness as a service provision tool A. For counselors to provide quality VR services to consumers 1. Does the system provide a tickler function that reminds counselors and supervisors of key process steps? 2. If so, what does this tickler function comprise, e.g., is it an automated system with pop-up messages that let counselors know whether critical dates are being met? 3. Can administrative assistants input data in the system? 4. Does the system meet all of the case management needs of counselors? If not, what enhancements or modifications would most help counselors to better serve consumers? B. For supervisors to provide quality supervision and guidance to counselors 1. Does the system track the consumers' movement through the vocational rehabilitation process from application to closure, including both input and output? 2. Does the system provide reports on counselor caseload management? 3. Does the system provide reports on how much money has been expended per counselor or per case or per customer? 4. Does the system provide all the management reports that supervisors need, including budget dollar and outcome comparisons? (Only relevant where district offices and, possibly counselors are budgeted.) If not, how is this information tracked? III. Usefulness as a management tool A. Fiscal 1. Is there a financial component to the system other than to track the cost of services, e.g., a financial system for use in preparing the SF-269s? 2. Does the system track fiscal information related to the IL program? If so, how? 3. Does the system track fiscal information related to the OIB program? If so, how? B. Planning and policy 1. Does the system provide the information needed for management to monitor and assess the agency's performance? If so, how? 2. Does the system compute standards and indicators data by agency and office? 3. Does the system help management make the appropriate resource allocation decisions? If so, how? 4. Does the system provide management with relevant information in a timely and useful manner? C. Data collection How does the system generate the 911 Report? If so, how? Is the process simple or complicated? IV. Usefulness for the IL program A. Service Provision 1. If IL services are provided directly by the DSU, does the system incorporate the counselors' IL case management responsibilities and activities? Is so, how? 2. If IL services are provided directly by the DSU, does the system track the case management case responsibilities and activities incorporate the counselors' IL? 3. If IL services provided indirectly via grants/contracts, does the system track the IL case management responsibilities and activities of grantees and contractors? Is so, how? 4. Does the case management incorporate the supervisors' IL case management responsibilities and activities? Is so, how? (if applicable) B. Planning and Policy 1. Does the system help management monitor, assess and make resource allocation decisions regarding the IL program? Is so, how? (if applicable) C. Data Collection 1. Does the system generate relevant IL information for the 704 Report? If so, how? (if applicable) IV. Other 1. How long has the agency had its current case management system? 2. If purchased/developed in-house it recently, what process was followed , which products did you consider, and how did you arrive at a decision? 3. What prompted the agency to buy/develop a new one? 4. How did the agency arrive at its desired needs? 5. For other agencies that are considering buying/developing a new system, what suggestions or advice would you like to offer? Chapter 6: Planning I. Agency Mission/Values II. Strategic Planning Process III. VR State Plan IV. IL State Plan V. Implementation VI. Evaluation I. Agency Mission/Values 1. What is the agency's mission statement? 2. How are the fundamental values or principles of the agency developed? 3. How are they incorporated in the planning process? 4. How does the agency use feedback from the SRC, SILC, consumer organizations, individuals with disabilities and other stakeholders to shape decisions, priorities and agency values? 5. How does the agency communicate its values and principles to its staff? 6. How does the agency communicate its values and principles to the SRC, SILC, consumer organizations, individuals with disabilities and other stakeholders? II. Strategic Planning Process 1. Does the agency engage in strategic planning? 2. If so, how does the agency define strategic planning? 3. What is the agency's strategic planning process, including when the agency last conducted the process. 4. What is the impetus for planning? (e.g. is it required or is it voluntary?) 5. Is the strategic planning process reflected in the agency's VR and IL State Plans? 6. What is the involvement of the SRC, SILC, consumer organizations, individuals with disabilities and other stakeholders in the strategic planning process. 7. How is the agency's strategic plan communicated to the SRC, SILC, consumer organizations, individuals with disabilities and other stakeholders? III. VR State Plan 1. How is the agency's VR State Plan developed? 2. When is the VR State Plan developed? 3. Do the goals identified in the agency's strategic plan and its State Plan differ? If so, why? 4. How are goals and objectives in the State Plan developed? 5. What sources of information, including the results of the statewide needs assessment, does the agency use to develop the goals and priorities contained in the VR State Plan? 6. Describe the involvement of the SRC in the development of the VR State Plan, in particular, the goals and priorities contained in Attachment 4.11(c)(1). 7. How is input into the VR State Plan obtained from the CAP, consumer organizations, individuals with disabilities and other stakeholders? 8. How is the VR State Plan communicated to agency staff? 9. How is the VR State Plan communicated to the SRC, CAP, consumer organizations, individuals with disabilities and other stakeholders? IV. IL State Plan 1. How is the SPIL developed? 2. What are the identified IL needs, and how are they incorporated in the SPIL? 3. What other sources of information do the DSU and the SILC use when developing the SPIL? 4. How do the DSU and SILC collaborate in the development (and/or amendment) of the SPIL, especially regarding the SILC resource plan, the financial plan and the goals and objectives? 5. What processes do the DSU and the SILC engage in to solicit and incorporate comment in the SPIL from the CILs, CAP, OIB program, VR program (e.g., statewide comprehensive needs assessment), and other IL stakeholders? 6. How do the SILC and the DSU assess whether a SPIL amendment becomes necessary or advisable? V. Implementation 1. How does the agency implement its strategic plan? 2. How does the agency ensure that staff actions are consistent with agency goals and priorities identified in the strategic plan? 3. How are other federal, state, or private agencies and organizations involved in the implementation of the strategic plan? 4. How are other federal, state and private agencies and organizations involved in the implementation of the VR State Plan? 5. How are other non-IL federal, state and private agencies and organizations involved in the implementation of the SPIL? 6. How are the DSU and the SILC coordinating with and leveraging the resources of CILs, the OIB program, the VR program, the AT program, Councils on Aging, DD Councils, and other public and private organizations? VI. Evaluation 1. How does the agency evaluate its progress on the goals contained in the strategic plan? 2. How does the agency evaluate the VR State Plan, in particular its progress toward the achievement of the goals and priorities contained in the Plan? 3. How does the SRC evaluate the agency's implementation of the VR State Plan, including progress on goals and priorities? 4. How is the SPIL evaluated? 5. How does the SILC assess, monitor, and evaluate the implementation of the SPIL? 6. How are CILs, CAP and other IL stakeholders involved in the evaluation of SPIL implementation? 7. What progress is being achieved in meeting the SPIL goals and objectives with regard to: reaching underserved areas and priority populations; expanding the network of centers; collaboration with other organizations; coordination with other VR and IL-related services? 8. How does the agency inform its staff of its progress on the goals contained in the strategic plan? 9. How does the agency inform the SRC, SILC, CAP, consumer organizations, individuals with disabilities and other stakeholders of its progress on the goals contained in the strategic plan? 10. How does the agency inform its staff of its progress on the goals contained in the VR State Plan? 11. How does the agency inform the CAP, consumer organizations, individuals with disabilities and other stakeholders of its progress on the goals contained in the VR State Plan? 12. How does the agency inform its staff of its progress on the goals contained in the SPIL? 13. How do the agency and SILC inform the CAP, consumer organizations, individuals with disabilities and other stakeholders of its progress on the goals contained in the SPIL? Chapter 7: Policy I. Development II. Dissemination/Implementation III. Evaluation IV. IL-Specific I. Development 1. What is the process for developing policy in the state? 2. Describe the state rehabilitation council (SRC) or commission involvement in policy development. 3. How are the CAP, consumer organizations and individuals with disabilities involved in the development of policy and/or reviewing current policy? 4. Describe the process and frequency of internal review of policies. (Development of new and review of new/old). What are the results of the review? E.g.: Are new policies developed or old policies amended? 5. How are staff involved in policy development and/or reviewing current policy? 6. Explain what mechanisms the SRC/commission has in place for review of policies? 7. Have there been any policy changes in the past year? If so, were the changes the result of a non-litigation activity or a litigation activity? How did the policy change benefit the consumers? II. Dissemination/Implementation 1. What is the process for policy dissemination, including the training of staff. How are policy manuals made available and accessible to consumers? 2. What resources are available to educate consumers about policies? (Open forums, websites, and orientations)? 3. Is there a process for requesting exceptions to limitations on the delivery of services? 4. How does the agency communicate the exception process to consumers? 5. Describe the resources available to address questions or concerns individuals may have regarding a policy. III. Evaluation 1. What processes are in place to assess the effectiveness of training and consistency of staffs interpretation of policies? 2. What feedback has the SRC/commission received on the implementation and effectiveness of agency policies? 3. What policies receive frequent inquiries in reference to intent, purpose, implementation, or explanation? IV. IL-specific 1. What are the agency's policies related to the major stages of the IL process, including: applications; eligibility; goal-setting; service provision; goals-achievement? Chapter 8: Personnel Management I. Planning for Personnel II. Recruiting Personnel III. Managing Personnel IV. Maintaining Quality Personnel V. Quality Assurance I. Planning for personnel: 1. What is the SRC's role in the agency's plan for recruitment, preparation, and retention of qualified personnel? 2. What are the current staffing requirements to meet the needs of the agency's customers? 3. What is the agency's turn over rate for Vocational Rehabilitation Counselors? 4. What is the agency's CSPD standard? 5. Is the agency able to hire below the standard, and if so, does the agency have planned objectives and activities (including training and continuing education) that reflect a timely approach to getting current and new counselors that do not meet the qualified personnel standards up to the level of those standards? 6. What percentage of Vocational Rehabilitation Counselors does the agency expect to retire within the next five years? 7. Does the agency have a succession plan in place based on its organizational mission, values, overall strategic plans and comprehensive system for personnel development? II. Recruiting personnel: 1. What are the barriers to hiring and retraining staff to the required level (e.g., pay scales, time off to complete coursework, tuition reimbursement, recruitment in remote areas, etc.), and what steps are being taken to reduce the barriers? 2. What is the recruitment process for hiring qualified personnel? 3. What coordination and facilitation efforts exist with institutions of higher education and professional associations to recruit qualified personnel, including personnel from minority backgrounds and personnel who are individuals with disabilities? 4. What strategies and incentives are used to attract and retain qualified individuals? III. Managing personnel: 1. How is the counseling staff organized within the district and local offices? 2. How is other agency staff used along with counselors in the VR process? IV. Maintaining quality personnel: 1. What coordination and facilitation efforts exist with institutions of higher education and professional associations to prepare and retain qualified personnel, including personnel from minority backgrounds and personnel who are individuals with disabilities? 2. What data analysis and data systems training are provided to agency staff? Who receives this training? 3. Do counselors receive routine training on agency policies? 4. What kind of special training do new counselors receive to acclimate them to the agency's policies and organizational culture? 5. What support does the agency provide to enable staff to pursue professional development and certification training? 6. How does the agency link its in-service training grant to CSPD efforts? V. Quality assurance: 1. Does the agency maintain a system to collect and analyze, on an annual basis, data on qualified personnel needs and personnel development? 2. Does the agency identify the number of DSU personnel with respect to type of position, ethnic diversity and disability status? 3. How well do current staffing patterns reflect the population of the state in terms of ethnicity? 4. What is the process for individual counselor performance evaluation? 5. Are there promising practices related to the agency's recruiting strategies that can be identified for dissemination to the field? Chapter 9: Fiscal Management I. Planning A. Fiscal Planning II. Funds Into the Agency A. Drawdown B. Program Income C. State Match D. Carryover E. Resource Management III. Funds (Expended/Obligated/Managed) Out of the Agency A. Administrative Costs B. Cost Allocation C. 3rd Party Arrangements D. Contract Administration E. Maintenance of Effort (MOE) IV. Reporting V. Quality Assurance A. Audits I. Planning A. Fiscal Planning 1. What is the nature of the agency's fiscal planning? 2. Who is involved and what are each person's roles? 3. When does it take place? 4. How could the fiscal planning process be improved? 5. How does the fiscal planning process ensure that ample funds will be available throughout the course of the entire year? 6. How does the case management system provide information to the fiscal planning process? 7. What is the role of the DSA in fiscal planning? 8. How is fiscal planning coordinated and managed between the DSU and the DSA? 9. How are these processes applied to the IL and 01B programs? II. Funds into the agency A. Drawdown of Funds 1. What is the process for drawing down funds? 2. Who is responsible for the drawdown of funds? 3. How could the drawdown process be improved? 4. How often/what is the usual pattern for drawdowns? 5. How does the agency determine the amount of funds to be draw down? 6. How are these processes applied to the IL and OIB programs? B. Program Income 1. How is the program income generated, tracked, and expended? 2. Who is responsible for managing program income? 3. What are some strategies for generating more program income? 4. What internal reports are generated on program income? 5. How does the agency track receipt and expenditure of these funds? 6. Is there an internal report that is utilized to provide program income data for the fiscal reports the agency submits, or, is there additional ledgers used to compile this figure to fulfill all report responsibilities? 7. What avenues of program income does the agency have and what is the breakout (percentage-wise) by source? 8. How has the program income fluctuated overall, and by source, over the last five years? 9. What does the agency believe is the reason for these trends? 10. What is the agency's projection for program income for FY 2008 and how does it compare to previous fiscal years? 11. How can the current sources of program income be enhanced, or current sources not utilized be accessed, to increase the availability of funds for program operations? 12. Is Social Security reimbursements usually used in the program in which it is generated or is there transfers made to other VR, IL and OIB programs? If transfers occur, what percentage of the total reimbursements in Social Security reimbursements end up being transferred to another program? C. State Match 1. What are the sources of State Match and who is responsible for tracking, applying, and reporting it? 2. What is the process for monitoring match to ensure sufficient funds are available to meet all match requirements? 3. What are the sources of match and the percentage breakout of each source of match for the VR, IL and OIB programs? 4. Is the direct state allocation remaining constant over the last 5 years, increasing, or decreasing? 5. What record keeping is done to track and report state match? 6. Is the match required to meet additional federal funds through the reallotment process, computed at the beginning of the year of allocation or determine in the last quarter? 7. How much of the state appropriations, used in meeting the state match is "borrowed" from the next year's State budget, to meet the federal matching requirements? 8. What does the agency do to supplement the direct state allocation if a decrease in this funding stream is projected? 9. If there are multiple sources of state match, how are these areas monitored or strategically planned for, to ensure all federal funds are appropriately matched? 10. In the fourth quarter of the year of appropriation, what percentage of the state match is unliquidated obligations? 11. What amount/percentage of the match portion of the unliquidated obligations result in outlays? D. Carryover Funds 1. How are carryover VR, IL and OIB funds managed, tracked, reported, and who is responsible for these tasks? 2. What internal fiscal reviews does the agency perform on carryover funds and what is frequency of those reviews? 3. How has the agency's history on the carryover of funds over the last five years effected resource management? 4. Does the agency have a written policy or plan for what is an acceptable level of carryover? E. Fiscal Resources Management 1. How is Resources Management administered in the agency and what staff is responsible for its coordination and/or how fiscal resources are applied? 2. What fiscal resources does the agency utilize in the administration of the Rehabilitation Programs? 3. What are the procedures for resource management and what staff tracks the agency's expenditures? What is done with that information? III. Funds out (expended/obligated) of the agency A. Administrative Costs 1. What are the agency's procedures for managing and controlling administrative costs? 2. Who is responsible for carrying out these procedures? 3. When did the agency last receive an approve Indirect Cost Rate from the cognizant agency? 4. What cost centers do the agency charge to administrative costs? 5. How are these processes applied to the IL and OIB programs? B. Cost Allocation 1. What is the process and responsible staff in administering the agency's Cost Allocation process? 2. How are staff salaries/fringes allocated to the benefiting grants/funding sources, through payroll/timekeeping systems? 3. What is the Cost Allocation Plan for the agency, and when was the last time it was updated? 4. What are the other programs operated by the agency other than those funded through the Rehab Act? 5. What is the on-going process how staff charges their time to the VR grant? 6. What documentation is kept for those staff working solely on the VR grant? How often is the documentation for staff working 100% on the grant gathered? 7. If Time Studies are used, how often are those studies done and what is the process and scope of those studies? 8. What staff is responsible for cost allocation and how are the fiscal and program operations coordinated in this process? 9. How are these processes applied to the IL and OIB programs? C. Third Party Cooperative Arrangements 1. What analysis is done to ensure that the services detailed in the arrangements are VR focused, new, meet all aspects of the State Plan, and available only to VR applicants or consumers? 2. What percentage of the state match comes from the agency's cooperative arrangements? 3. How has the percentage of cooperative arrangements used to in state match changed over the last five years? 4. What internal audit/monitoring process does the agency have in place to verify the reported certification of areas from 3rd Party Arrangements such as salaries and supplies? 5. Is there a service component to the agreements in the 3" Party Arrangements? D. Contract Administration 1. What are the policies, procedures, practices, and staff responsible for the agency's contract administration? 2. What are the agency's procurement and fee determination policies and procedures? 3. How are contracts developed and monitored? 4. What percentage of the services delivery system is expended by using outside vendors and/or contractors? How has this percentage increased or decreased over the last five years and what has been the resulting impact on performance? 5. When was the last time that procurement was done to contract out for services? 6. What kind of contract format is used by the agency? 7. Who was involved in the development of the contract and what was the process for developing the contract? 8. Who monitors the fiscal and programmatic (internal and external) contract performance? 9. What is the monitoring schedule? 10. How often are the contract administration procedures reviewed/updated to correct deficiencies and improve the effectiveness of the monitoring tool/process? 11. If monitoring identifies overpayment for services and/or non-compliance with the contract, are there written policies for recovering the funds, developing corrective action plans and/or terminating the non-compliant contractors? 12. What contracts management and oversight practices are applied to ensure the proper use of federal funds by IL and OIB service providers, including CILs receiving Part B or state match funds? 13. What contract management and oversight practices are applied to ensure the proper use of federal funds by the SILC in a manner that respects the SILC's autonomy? 14. What contract management and oversight practices are applied to federal funds that support SPIL-related activities (other than IL services and SILC operating expenses)? 15. If the agency has determined a set level of fees to pay for each service provided, how were these set fees determined? 16. If the fees are developed on a case-by-case basis, how does the agency decide on the reasonableness of the fee for the service provided? 17. Did the agency have the state's legal counsel approve the language of the contract and ensure that the payment process did not violate any state's fiscal policies? 18. What is the step-by-step process for contractor payment from the time a service is approved, until the service provider receives payment for that service? 19. Is the Debarment/Suspension certification for each contractor completed at procurement or at contract execution? Does this certification include a no cost verification of the self-certification on the GSA website? How often is the certification updated? 20. How are IL services provided indirectly via grants/contracts and does the system track the IL case management responsibilities and activities of grantees and contractors? E. Maintenance of Effort (MOE) 1. How does the agency ensure meeting their MOE requirement and who is responsible for this compliance management? 2. What record keeping system is used for managing/tracking the agency's MOE requirement each year? 3. What checks and balances are utilized by the agency to successfully meeting the MOE each year? 4. How has the level of MOE differed over the last four years? What has contributed to this trend? 5. How does the agency manage its annual state match funds to ensure compliance with Maintenance of Effort Requirements? IV. Reporting A. Fiscal Reporting 1. What are the procedures and practices for developing the (internal/external) fiscal reports and what staff is involved in their preparation? 2. What part does the Case Management System play in tracking and reporting costs? 3. What are the steps taken in preparing the SF-269 and RSA-2, and what staff is involved in its preparation? 4. Describe the financial component to the case management system and does it perform functions other than to track the cost of services, e.g., a financial system for use in preparing the SF-269s? 5. How does the case management system track fiscal information related to the IL program? 6. How does the case management system track fiscal information related to the 01B program? 7. How are the costs of services reconciled to those shown on the agency's accounting record, for preparing the second page of the RSA-2 (consumer services)? 8. How is the RSA-2 generated from your case management system, or is it developed separately? 9. How does the agency track, contrast and compare expenditure progress to the budgets for the agency, district offices, and counselors? 10. What are internal financial reports used in the fiscal management of the agency and how are they used? 11. What is the current process for re-checks of figures in the fiscal system before they are entered into the reports? And, rechecks after reports are completed? 12. What are the steps taken in preparing the SF-269 and RSA-2, and what staff is involved in its preparation? 13. Is there a current printout from the fiscal system that provides the necessary information to prepare the two fiscal reports or does it take the development of subsidiary ledgers or spreadsheets to produce the required reports? 14. How are these processes applied to the development of the 704 Report, the 7-0B Report and the SPIL? V. Quality assurance A. Audits 1. In addition to complying with the Single Audit Act, what other external or internal program or fiscal audits/reviews is conducted and who are the responsible staff? 2. What internal fiscal reviews does the agency perform and what is frequency of those reviews? 3. When was the last Single Audit completed? What were the material findings in that report? What findings were repeat findings from previous years? 4. Are there any outstanding and/or unresolved audit findings from previous audits? 5. How does the agency incorporate the audit process as part of their internal control system of checks and balances? Chapter 10: SRC and SILC I. SRC Organization and Management II. SILC Organization and Management I. SRC Organization and Management 1. Is the agency fully constituted in accordance with federal law and regulations? 2. If not, what membership positions are vacant? 3. Have nominations to fill vacant positions been made to the governor's office? 4. What methods does the SRC use to recruit new members? 5. How is the resource plan for the SRC developed? 6. Does the SRC have a full or part-time executive director? 7. What administrative support does the agency provide to the SRC? II. SILC Organization and Management 1. Where is the IL program placed in the DSU's organizational structure? 2. How does the SILC manage its own budget and expenditures? 3. Where is the SILC placed within (or outside of) the state system enabling it to function effectively and independently? 4. What nomination and appointment processes does the SILC have to ensure continued compliance with composition requirements? 5. Does the current SILC membership meet all composition requirements? 6. What DSU administrative support services are provided to IL? For example, administrative support staff assigned to the SILC; direct IL services staff; contracts management/oversight staff; in-kind such as office space, supplies, training and technical assistance services? 7. How does the SILC manage its own staff? 8. What written policies and procedures exist to protect the independence of the SILC, with respect to: DSU staff providing administrative support to the SILC; SILC management of its own staff, budget and programmatic priorities?