Monitoring Process and Protocol for the Vocational Rehabilitation and Independent Living Programs

FY 2007-09

Introduction

Section 107 of the Rehabilitation Act of 1973, as amended (Rehabilitation Act), requires the Commissioner of the Rehabilitation Services Administration (RSA) to conduct annual reviews and periodic on-site monitoring of programs authorized under Title I of the Rehabilitation Act to determine whether a state vocational rehabilitation (VR) agency is complying substantially with the provisions of its state plan under Section 101 of the Rehabilitation Act and with the Evaluation Standards and Performance Indicators established under Section 106.  RSA has developed this monitoring protocol to fulfill the requirements of Section 107 of the Rehabilitation Act.  In addition, this protocol will assist the Commissioner in assessing the degree to which programs offered under Title VI of the Rehabilitation Act are substantially complying with their respective state plan assurances and program requirements.  In order to achieve this purpose, RSA will collect and analyze information related to performance and compliance.  

The monitoring process will also encompass the State Independent Living Services (SILS) and the Independent Living Services for Older Individuals who are Blind (OIB) programs authorized under chapters 1 and 2 of Title VII of the Rehabilitation Act.  RSA will also monitor the performance of the designated state units and the Statewide Independent Living Councils (SILC) to determine compliance with and performance on the assurances contained within the state plan for independent living.

Scope and nature of review

RSA will conduct periodic reviews of all Title I, VI and VII Part B programs beginning in fiscal year (FY) 2007, and continuing through FY 2009.  Approximately one-third of all states and territories will be monitored in each of the three fiscal years.  RSA will provide the state VR agencies and stakeholders with a three-year monitoring plan containing each state and the year the periodic review will be conducted.    

States will be selected for periodic monitoring with attention to a balance of states across the three years.  At the same time, circumstances may require RSA and a State Agency (SA) to conduct a review of a state agency more than once every three years for a variety of reasons including, but not limited to: the SA requesting more immediate assistance; the SA experiencing issues that require immediate attention, such as confusion over fiscal requirements; inability to meet the standards and indicators over several years; or the adverse impact on an SA’s operations resulting from catastrophic natural disasters.  A non-scheduled full review is less likely than is the provision of substantial technical assistance to meet specific and pressing agency needs.  

If there are two agencies in a state providing VR services, one to individuals who are blind and visually impaired and the other to individuals with all other disabilities, both agencies will be reviewed in the same year.  In those years when a state program is not to be monitored, that state will receive an annual report covering the previous fiscal year.  In addition to the periodic onsite reviews, RSA will issue annual Section 107 reports for all other state agencies.

Focus areas for this review will be individually tailored to the state under review.  RSA will design instruments that will be used to identify, i.e., screen, for issues specific to the states under review, taking into account the unique demographic, economic and geographic circumstances of the state.  RSA will also identify promising practices and models that can be shared with all states when appropriate.  

While focus areas may vary, RSA’s review protocol is governed by those program and fiscal areas highlighted in Section 107, which includes compliance with and performance on the state plans for VR and IL.  

While only RSA staff will carry out monitoring activities, the review process will include collaboration with key state agency staff as well as other stakeholders.  Input will be solicited from the state rehabilitation councils, statewide independent living councils, client assistance programs, consumer groups, service providers, and other stakeholders whenever appropriate.  RSA will consult with state stakeholders to solicit input during the review.
Overview of the review process

This protocol provides state agencies and stakeholders with the steps in the process that RSA will take to: 1) identify any issues of compliance and areas of low or high performance; and 2) identify any appropriate action steps state agencies should take to achieve compliance or improve performance.  

RSA will focus its review on those aspects of the programs identified in Section 107 of the Rehabilitation Act, which states:

In conducting reviews under this section the commissioner shall consider, at a minimum:

(A) state policies and procedures; 
(B) guidance materials; 
(C) decisions resulting from hearings conducted in accordance with due process;
(D) State goals established under section 101(a)(15) and the extent to which the State has achieved such goals; 
(E) plans and reports prepared under section 106(b); 
(F) consumer satisfaction reviews and analyses described in section 105(c)(4); 
(G) information provided by the State Rehabilitation Council established under section 105, if the State has such a Council, or by the commission described in section 101(a)(21)(A)(i), if the State has such a commission;
(H) reports; and
(I) 	budget and financial management data.

RSA will draw on a variety of sources when seeking to identify issues warranting further investigation.  In accordance with Section 107 of the Rehabilitation Act and RSA monitoring practices, these sources can include, but are not limited to:

(1) the State Rehabilitation Council (SRC);
(2) the Statewide Independent Living Council (SILC); 
(3) RSA data; 
(4) the state plans for VR and IL;
(5) the state agency’s policies and procedures; 
(6) public hearings or other discussions with consumers and advocates; 
(7) individual case files, including individualized plans for employment and ineligibility determinations; and
(8) meetings with qualified VR counselors and other personnel.  

As noted above, because the purpose of the monitoring process is to assess a state agency’s performance and whether the agency is complying with the state plan, the performance and compliance issues that will be reviewed will be particular to each state.

RSA will evaluate an agency’s performance on:  
(1) Standards and indicators; 
(2) The goals, objectives, and priorities included in the VR and IL state plans; and
(3) RSA’s VR and IL strategic performance plan goals and objectives.

Performance is broadly defined as the effective management of available resources to achieve quality employment and independent living outcomes.

-RSA will evaluate compliance with the VR state plan by determining, in part: 
(1) whether the assurances that the state agency provides in its state plan are valid, with particular attention to eligibility, development and implementation of the Individualized Plan for Employment (IPE), and the provision of services; and 
(2) if the state is under an order of selection, verification that it is following requirements regarding the order of selection set forth in Section 101(a)(5)(A).

-RSA will also monitor compliance with the assurances contained in the IL state plan.  In the case of the State Independent Living Services (SILS) program, Title VII of the Rehabilitation Act gives the SILC joint responsibility with the state agency in developing, signing and submitting the State Plan for Independent Living (SPIL).  Several SPIL assurances relate directly to the SILC membership and operation, including its resource plan and staff.  Therefore, RSA’s on-site review of the state’s compliance with SPIL assurances will necessarily involve a review of the SILC as well as the state agency.  RSA will review the relationship between the SILC, Centers for Independent Living (CILs), and the SPIL.  

-RSA will gather information in order to understand how the agency’s different systems function and work together to maximize performance and ensure compliance with the state plan and the program’s requirements, particularly as they relate to the VR and IL processes.  

-After consulting with the state agency, within 45 days after the completion of all review activities, RSA will issue a report that identifies:  
(1) the areas of performance that either require improvement or may serve as model performance practices; 
(2) areas of compliance that require the agency to take corrective action in order to be in compliance with the agency’s state plan; 
(3) the steps that must be taken to improve performance, expand model practices, and correct findings of noncompliance; and  
(4) technical assistance resources to assist the agency to improve its performance and to respond to corrective actions.   

-Following the issuance of the report, RSA will continue to provide technical assistance and work jointly with the state agency to: 
(1) improve its performance; and 
(2) take corrective action to address findings of noncompliance.

In addition to assessing the progress toward achieving state plan goals and priorities, the technical assistance given in the course of the review may also provide state agencies with valuable input toward refining state plan goals and priorities or the strategies to achieve those goals.  

The next section of the monitoring protocol describes each step in the review process, with attention to the participants in each step, and the resources on which the review will rely.  

Description of the Monitoring Process

The following is an outline of the process the state teams will follow through the course of the monitoring cycle.  The state liaisons, in collaboration with their state teams, will take the lead in the monitoring efforts, and the functional units in the State Monitoring and Program Improvement Division (SMPID) will provide input and expertise as the process advances.  

The state team representative from the VR, IL, Technical Assistance (TA), Fiscal, and Data Collection and Analysis units will be responsible for identifying issues and resources pertaining to the work of their respective units.  By utilizing a screening tool, the units will assist their team members in identifying resources, highlighting issues to pursue, seeking sources of information, analyzing the information gathered, and suggesting solutions and strategies for addressing any areas of compliance or performance that require further investigation and possible corrective action.  The state team may utilize the input from the different units to identify focus areas for the state review.  

This process is intended to allow for maximum input from stakeholders throughout the course of the monitoring cycle.  Stakeholders will serve as both a resource of information as the state teams carry out their review and will, whenever appropriate, contribute suggestions to the manner in which state agencies can improve performance.

Step 1:  State team planning meeting 

At this initial planning meeting, the state team, led by the state liaison will:

-develop a work plan for the periodic review, which may include timelines, work assignments, and other managerial details;
-develop a list of available data and information resources at the beginning of the process; and
-discuss possible VR, TA, fiscal or IL issues based on in-hand information.

The team will review the steps in the process and make certain that each member understands his or her role.  The state liaison will inform the state agencies and other stakeholders that the process is underway and that the liaison will be routinely contacting them when appropriate as the review progresses.
  
Stakeholders other than state agencies include:

-State Rehabilitation Councils (SRC); 
-Statewide Independent Living Councils (SILC);
-Centers for Independent Living;
-Designated State Agencies (DSA); 
-Community Rehabilitation Programs;
-Independent Commissions;
-Consumer advocacy groups, including state coalitions of citizens with disabilities;
-Individual VR and IL consumers;
-State Workforce partners;
-Governor-appointed disability councils; 
-Client Assistance Programs (CAP);  
-The Protection and Advocacy for Individual Rights (PAIR) program and other protection and advocacy grantees; and
-Other pertinent state agencies and organizations involved in the rehabilitation and independent living of individuals with disabilities, including state agencies for individuals who are deaf and hard of hearing, state special education agencies, and developmental disability councils. 



Step 2:  Organization of information and development of a state profile

a. The liaison will gather and collate the information provided by the team members and, with the team’s input, will identify potential performance and compliance issues to be reviewed with the agency.

All materials used for analyzing performance and compliance are maintained on RSA’s Management Information System (MIS) from which each unit will obtain information needed for its contribution to the team’s monitoring plans.  Much of the data is generated from the databases produced from the completed forms and reports submitted to RSA, such as the Financial Status Reports (SF-269), Annual VR Program/Cost Report (RSA-2), Quarterly Cumulative Caseload Report (RSA-113), Case Service Report (RSA-911), and other program and fiscal reports.  Most data are contained in a continually updated database maintained for each state.  RSA makes public all data through the sharing of information with relevant stakeholders and by publishing annual reports, monitoring reports, and corrective action plans on the RSA website.  

b. During this stage of the review, the State team will analyze the general topics listed below, using the identified resources.  Other sources may be utilized to screen for issues, but those identified below will provide a comprehensive picture of the general state of VR and IL activities.  The units and teams will then further analyze all available resources as issues are identified. 

Topics:

-eligibility, service provision and order of selection; 
-progress toward achieving Goals and priorities in the VR and IL state plans; 
-relationships among VR and IL state partners, including the designated state agency, designated state unit, state rehabilitation council, and statewide independent living council;
-personnel and staffing issues, including progress in meeting Comprehensive System of Personnel Development (CSPD) requirements; and
-practices that facilitate the agency’s ability to comply with program requirements, meet performance standards, and achieve quality employment and independent living outcomes for individuals with disabilities.

Resources:

* RSA performance data, e.g., standards and indicators data;
* state VR written service policies, procedures, and forms;
* state IL written policies and procedures involving the state agency and the SILC;
* transition policies and practices;
* findings resulting from state program and fiscal audits required by OMB Circular A-133;
* VR and IL outcome data; 
* interagency and third-party cooperative agreements; 
* the comprehensive statewide assessment report;
* state economic indicators;
* statewide disability demographics; 
* the SRC Annual Report;
* the Annual Client Assistance Program (CAP) report (RSA-227); 
* state agency annual reports;  
* results of consumer satisfaction surveys;
* the RSA Management Information System (MIS);
* state visit summaries;
* records of technical assistance (TA) provided to the state;
* section 107 corrective action plans;
* quarterly cumulative caseload reports (RSA-113);
* RSA monitoring tables; 
* VR and IL state plan goals and priorities; 
* annual Section 704 Part I and II Performance Reports;
* input from stakeholders;
* order of selection information; and
* grant award, match, and maintenance of effort information.  

c. The liaisons will inform state agencies and other stakeholders of the information gathered and the potential issues identified at this stage of the process, and request any updates as appropriate.  

Step 3:  Screening of topic areas to identify issues of concern and areas of high performance

The functional units will develop screening tools and analyze information to identify issues of concern related to the agency’s performance and compliance with the state plan.  In addition, the functional units will use the screening tools to determine areas of high performance, defined as any area in which the state is improving its performance compared to previous years and with other states similarly situated.

Similarly situated means:

-state agencies that are comparable with respect to the amount of their grant awards, the number of individuals served, and the number of staff;
-states that share similar geographic characteristics, such as the division among urban and rural areas;
-challenges related to delivering services across great distances;
-states with similar disability and ethnic demographics; or 
-states with similar economic indicators, such as unemployment rates, wages, and job growth sectors.

The goal of looking at states similarly situated is to provide a tool of analysis, which may lead to identifying effective policies and practices best suited for a given state's unique characteristics.

As part of the screening process, the State team will gather information in order to understand how the agency’s different systems function and work together to maximize performance and ensure compliance with the state plan and the program’s requirements, particularly as they relate to the VR and IL processes.  These systems include, but are not limited to:

-the comprehensive system of personnel development;
-the case management system;
-the service delivery system;
-the fiscal management system;
-the data collection and analysis system; and 
-the quality assurance system.

In addition, the state team will also gather information on the agency’s policy and planning processes, and its various procedures for internal and external communication.


a. The SMPID functional units will implement their respective components of the screening tool and share the results with the state team via the unit representative.  The state liaison will be responsible for compiling any compliance and performance issues identified by the units.

The VR unit, in addition to analyzing information related to the systems referred to above will use other information resources to screen for issues related to performance on standards and indicators and other measures from the RSA monitoring tables.  This will enable the state team to uncover possible explanations for a state agency’s performance on a given indicator, and will also allow for analysis of other issues that will have an impact on an agency's performance, such as allocation of fiscal and human resources.  

The fiscal unit will identify issues of concern and areas of high performance by reviewing the results of predetermined compliance and performance measures significantly at variance with computed national averages.  Fiscal unit staff will analyze data gathered over several fiscal years to determine performance trends on such issues as:

-grant information including carryover
-an analysis of the sources of matching funds
-an analysis of indirect costs to total grant amount
-an analysis of program income to total grant amount
-the percentage of reported matching funds from un-liquidated obligations
-funds relinquished through the re-allotment process
-maintenance of Effort (MOE) penalties
-prior fiscal and program audit findings
The data unit also will assist the state team to identify issues of concern and areas of high performance by reviewing the results of pre-determined compliance and performance measures significantly at variance with computed national averages.  The data unit staff will analyze data gathered over several fiscal years to determine performance trends on such issues as: 

-types of services provided and expenditures on these services;
-staff composition;
-year-end statuses for the individuals participating in the VR program;
-Measures of Efficiency;
-number and types of employment outcomes;
-disability populations served; and
-specific categories of individuals served, including transitioning youth and individuals receiving social security benefits related to their disabilities.

The IL unit will conduct an analysis of the systems referred to above and identify issues of compliance and performance related to the agency’s delivery of IL services, such as:

-SILC autonomy, independence, and performance of statutory duties;
-Working relationships between the state agency, the SILC, the CILs, and other entities;
-compliance with assurances related to service provision, eligibility, consumer service records, SILC membership, and financial accountability;
-implementation of the SPIL financial plan;
-appropriate uses of Part B funds (as authorized under Title VII, Chapter 1 of the Rehabilitation Act); and
-amendments to the SPIL.

b. The liaison will share with the state agency and appropriate stakeholders the results of the initial screening and will solicit feedback and clarification.

Step 4:  Finalization of compliance and performance issues for review

a. The team, after incorporating feedback from stakeholders, will create a final list of focus areas that will constitute the major components of the review.  

b. The liaison will share with the state agencies the list of focus areas of the review to be further analyzed through the following steps of this process. 

Step 5:  On-site and other monitoring activities 

The state teams will meet to identify the materials and methods best suited to gather the information needed to review the programmatic and fiscal issues identified through the use of the screening tools.  


Monitoring activities may include:

-use of topical monitoring modules from RSA's monitoring and technical assistance guides (MTAG) developed in prior years;
-review of information developed by the state agency, for example, the results of quality assurance activities conducted through the course of the review year;
-on-site activities further described below; 
-review of VR and IL written policies and procedures;
-the development of new topical modules; and
-any other activities germane to the review.

On-site activities may include:

-targeted service record reviews; 
-public hearings; 
-meetings with state rehabilitation councils, statewide independent living councils, and VR and IL staff, including counselors and supervisors;
-visits to community rehabilitation programs or other contracted service providers; 
-meetings with special education officials, teachers or parents of children with disabilities; and
-examination of supporting documentation for submitted SF-269s and RSA-2s, as well as discussions regarding cost-allocation plans, OMB payroll requirements and other fiscal issues.

Step 6:  Analysis of monitoring information and development of written report

a. Each functional unit will prepare an analysis resulting from the review of issues pertaining to their area of expertise.  The state liaison will compile all the analyses into one report.  

This preliminary draft analysis will contain any findings, outstanding issues and technical assistance, and highlight areas of high performance and promising practices identified through the monitoring activities referenced in step 5.  

b. The liaison will share results of the analysis with state agencies and offer an opportunity for the agencies to provide feedback on these preliminary results.  


Step 7:  Identification of actions to address issues of concern and expand areas of high performance

State teams will work with state agencies to jointly develop action steps in response to any compliance findings or recommendations for the improvement of performance generated by the review process.  The Technical assistance unit team representative will compile all technical assistance needs and develop a TA plan which will identify TA resources and suggest mechanisms for delivering TA needs.
 
RSA intends this step in the process to be as collaborative as possible, as is appropriate to the nature of the issue.  

Action steps may include:

-the updating of goals and priorities as reflected in the VR or IL state plans, and the strategies for achieving state goals and priorities; 
-amending policies and written procedures; 
-adoption of quality assurance procedures; 
-structural changes, such as agency organization, staffing, nature and scope of relations with other entities; and
-amendments to the SPIL or VR state plan.


Step 8:  Finalization of plan to implement action steps to address issues of concern and expand areas of high performance  

State teams, in collaboration with state agencies and stakeholders, will finalize a plan to implement action steps to address issues of concern and expand areas of high performance.  Agreed upon action steps will be incorporated into a monitoring review report that will be issued 45 days following the conclusion of the review.  The review is not necessarily concluded following an on-site visit.  The report will incorporate results of the review of both VR agencies in those states that have established a separate agency to serve individuals who are blind or visually impaired.  

The monitoring report will contain a description of the review process, the issues of concern and areas of high performance identified through the review, the state agency’s response to any findings and recommendations, and the action steps jointly developed by RSA and the state agency necessary to address the compliance findings and recommendations.   

Step 9: Development and implementation of a technical assistance plan

a. Once the monitoring report is finalized, the RSA technical assistance unit representative to each state team will compile all identified technical assistance needs from the report.  

b. In collaboration with the state team, the TA unit will develop a TA plan for each agency reviewed.  The plan will identify both TA needs and the resources to meet these needs.  

c. Once the TA plan is implemented, the TA unit will facilitate and track the implementation of the plan.  


Role of Technical Assistance Unit

Throughout the monitoring process, the Technical Assistance Unit focuses on the identification, development, and provision of technical assistance resources to respond to the TA needs of the state VR and IL programs.  To assist the state teams to carry out the steps of the monitoring process described above, the TA Unit will:

-identifies stakeholder contacts in each state, from which team members can solicit information in the planning and conduct of monitoring activities; 
-compiles technical assistance needs identified during the course of monitoring and collaborates with the Division of Training and Service Programs and others to focus training and technical assistance resources on those needs;
-in collaboration with the state team and the state agency, develops and coordinates the implementation of a Technical Assistance plan to meet the TA needs identified during monitoring-- establishes the TA delivery process, matches TA needs with available TA resources, makes linkages between problems identified and appropriate TA remedies;
-develops, identifies, and, as appropriate, provides TA resources to address the TA needs, such as the SRC online tutorial, comprehensive needs assessment training materials, and quality assurance technical assistance;
-tracks progress on the implementation of TA plans; and
-compiles promising practices identified and posts on ed.gov website.

Summary

In carrying out its obligation to conduct periodic monitoring of state agencies in accordance with Section 107 of the Rehabilitation Act, RSA has developed a collaborative monitoring process, throughout which it will seek input from state agencies and stakeholders.  The monitoring process consists of the following steps:

Step 1:  Teleconference between RSA State teams and VR agencies, SRCs, and SILCs 

The RSA state liaison will contact the VR agency director, SRC chair, and SILC chair to schedule a teleconference during which the liaison will introduce the state team, review the monitoring protocol, and discuss possible times for on-site visits.  This step will be completed by December 15, 2007.

Step 2:  Sharing of preliminary state profile data and issues 

RSA liaisons will share with state agencies and other stakeholders the state profile compiled at this stage of the review, highlighting preliminary issues of concern and areas of high performance.  Agencies will have an opportunity to provide input to the RSA state team, and update any data utilized in the review.  This step will be completed by February 1, 2008.  

Step 3:  Screening of topic areas to identify issues of concern and areas of high performance 

The State team will work with VR agencies to gather information in order to understand how the agency’s different systems function and work together to maximize performance and ensure compliance with the state plan and program requirements, with particular attention to the VR and IL processes.  Such systems may include: fiscal management, case service management, data collection, professional development, service delivery, and quality assurance.  This step will be completed by March 15, 2008.  

Step 4:  Finalization of compliance and performance issues for review

The liaison will share with the state agencies and SRC and SILC chairs the finalized list of focus areas of the review to be further analyzed through the following steps of this process.  This step will be completed no later than March 27, 2008.  

Step 5:  On-site and other monitoring activities 

RSA state teams will conduct review activities in collaboration with VR agencies and IL partners.  This includes all on-site review activities.  This step will be completed by July 1, 2008.  

Step 6:  Analysis of monitoring information and development of written report

Following the completion of the analysis of the data gathered through the review process, the state liaison will compile the analysis of the VR, supported employment, and IL programs into a monitoring report.  The liaison will share the draft report with state agencies to obtain comment on the factual content.  This step will be completed during August 2008.  

Step 7:  Identification of actions to address issues of concern and expand areas of high performance

State teams will work with state agencies to jointly develop action steps in response to any compliance findings or recommendations for the improvement of performance generated by the review process.  This step will be completed by August 15, 2008.  

Step 8:  Issuance of final monitoring report-- 

Agreed upon action steps will be incorporated into a monitoring review report that will be issued 45 days following the conclusion of the review.  The review is not necessarily concluded following an on-site visit.  The report will incorporate results of the review of both VR agencies in those states that have established a separate agency to serve individuals who are blind or visually impaired.  This step will be completed no later than the end of the fiscal year 2008. 

Step 9: Development and implementation of a technical assistance plan

Once the monitoring report is finalized, the RSA technical assistance unit representative to each state team will compile all identified technical assistance needs from the report, develop a TA plan for each agency reviewed that identifies TA needs matched with resources, and facilitate and track the implementation of the plan.  

Following the issuance of the monitoring reports, state agencies and stakeholders will have the opportunity to comment on the implementation of this monitoring process through an evaluation survey available on RSA’s Management Information System.  Individuals who participated in the process will receive instructions for accessing the survey when they are notified that the reports have been issued.  RSA will use the information obtained through the survey to improve the process in future monitoring cycles.   





1


14
Revised – 10/26/2007

Revised – 10/26/2007