Evaluation of Non-Degree Granting Units

Professor speaks to his class in language lab, CAS
Professor Scott Hartley, wearing protective glasses, talks with students in the lab, CAS

Overview

In response to the 2015 Ohio Taskforce on Affordability and Efficiency Report’s call for each Ohio public higher education institution to “continuously review its administrative operations,” Miami University recently established a program review process for its non-academic units.

The goals of this process are to:

  1. Promote data-driven, continuous and constructive improvement and efficiencies;
  2. Align the review expectations and reporting requirements with the University’s strategic plan and the vice-presidential divisional mission and priorities;
  3. Advance the criteria articulated in the Higher Learning Commission’s accreditation process;
  4. Review units once every five years.

The units and schedule of reviews are determined by the Provost. 

The program review consists of two phases: (1) a self-study created by the staff or leadership of the program, office, or center, and (2) an evaluation by reviewer(s).

The provost or dean establishes a schedule for those units in their division.  

To initiate the self-study process, the unit head and the designated representative in the provost’s or dean’s office meet to discuss the following:

  • Overall review process and timeline;
  • Goals, objectives and metrics to be addressed in the review;
  • Format and content of the self-study;
  • Data needs (to be supplied by Office of Institutional Research, Budget Office and others);
  • Process used to identify review team.

Review Process & Timeline

Two semesters in advance of the review visit:

  1. The provost (in consultation with the appropriate dean) selects units (offices, centers, programs) to be reviewed.  The provost may choose to form a “cluster” of several units based on similarity of mission for combined review (e.g., several offices within Global Initiatives may undergo a combined review).
  2. The provost (and dean, if relevant) and unit leader identify 2-3 goals and objectives for the review as well as metrics for each objective.
  3. In consultation with the Provost Office, the Office of Institutional Research & Effectiveness (OIRE), and the Budget Office, the unit leader creates a plan for data collection.

 One semester in advance of the review visit:

  1. In consultation with the unit leader, the provost (and/or dean) chooses external and internal reviewer(s) for the unit or cluster to be reviewed.  Reviewer(s) are generally selected 4-6 months prior to the site visit.
  2. In consultation with Office of Institutional Research, conduct satisfaction survey (if appropriate).
  3. Each unit in consultation with the provost (and dean, if applicable) develops a self-study that includes key issues and a strategic plan. The self-study document should be completed and submitted to the provost by the end of the semester and no later than at least six weeks prior to the campus visit.
  4. The provost and/or dean may decide to provide input on the self-study.  If needed, a revised self-study should be completed four weeks prior to the site visit.

During the semester of the review visit:

  1. One month prior to the visit, the provost (or dean, if applicable) sends the self-study and other information materials as appropriate to the reviewer(s). 
  2. The itinerary for the review visit and travel arrangements is coordinated by the unit leader.  Meetings with the provost or dean (or designee) should be scheduled at the beginning and end of the visit.
  3. The review team conducts the campus visit and a careful review of self-study.
  4. The reviewer(s) generate a report within four weeks of the visit.  The report will be shared with key stakeholders (e.g., provost, leadership and staff of the unit).
  5. The unit leader and provost (or dean, if applicable) conduct a fact-check on the report within a week of its receipt.  If necessary, they ask for corrections of errors in facts.
  6. The unit provides a written response to the review within one month of receiving the report.
  7. The unit leader meets with the provost or dean to discuss the results of the review.
  8. Reports of the review team and responses are submitted to the Senate Executive Committee and the President’s Executive Council.

The unit being reviewed pays for all expenses including honoraria for the eternal reviewers.

Self-Study

Although the specific format and content of the self-study will be determined collaboratively by the leadership of the unit and the provost or dean (or his or her designee), units should include the following elements in the self-study:

  • History and Overview: Description of the fundamental purpose, functions and activities of the unit, including its mission, goals and objectives and how they relate to divisional and university goals. A table should be included that illustrates the staffing organization as well as an employee roster by position title as an appendix. The appendix should include a staff list and job description for each staff member.
  • Internal Environment: An overview of the strengths and weaknesses of the unit informed by the data identified below. This description should include any meaningful metrics from prior evaluations, external associations, or professional societies that may provide additional context to the unit’s performance and operations.

Units should include the data listed in the table below for the last four fiscal years. Please note: The Office of Institutional Research and Budget Office will provide much of these data, as indicated in the table.

Data needed from the last four fiscal years:
Type of Data Provided by Notes
Budget and actual revenue and expenditures Budget Office/BI Tool Derived from Banner tables and BI Tool; focusing on non-instructional staff salaries & related benefits, support costs of unit, graduate assistantships (if applicable)
Count of employees (FTE) by standard occupational codes Office of Institutional Research (OIR) and Budget Office November 1 snapshot of headcount and FTE
Span of control OIR, Academic Personnel and Budget Office Derived from multiple sources, including ABC data and state auditor methodology
Space (square footage) used by the unit Registrar and Physical Facilities
Efficiency Administrative Head Count Ratio (administrators/total employees) OIR and Budget Office Data will include faculty (if applicable
Administrative Expense Ratio (administrative salaries/total salaries) OIR and Budget Office Total salaries will include faculty
Trends in outputs for each of the department’s programs Unit (OIR as resource for student/faculty/staff data - Academic Personnel on searches and hires) To de determined at the beginning of the process, but may include: the number of: searches and hires, “individuals” served by unit; publications, grants, major fellowships and awards; educational programs; students enrolled; diversity of faculty/staff
Satisfaction survey for those individuals/groups the unit serves Unit in consultation with OIR Could use data (if relevant) from existing surveys such as College Senior Survey, NSSE, Your First College Year Other units may need to contact OIR on constructing and delivering their own survey as well as summarizing results
Productivity measures for the unit’s services or programs indicating the level of output divided by the number of employees and total number of students served Unit in consultation with Budget Office, OIR
Comparison of employees and budget for the unit to peers at Bowling Green State University, Kent State University and Ohio University as well as other institutions as identified by the unit Budget Office Use standard template developed in collaboration with other Ohio public universities as well as other benchmark data
Unique technologies used to support programmatic activities Unit If applicable. Focus only on major applications and technologies
Professional development and other learning resources made available to staff Unit
Major external funding sources Unit (OARS, University Development) Federal and other grants, etc., if applicable
  • External Environment: Description of the potential opportunities and challenges facing the unit from outside the University that are likely to have an impact on its activities. External forces may include labor market, technology changes, and the regulatory or legal environment.
  • Planning and Assessment: List of goals and objectives for the next five years including indicators of success, notes on strategies and tactics that will be used to pursue them, and financial implications for the unit. This should include a quality improvement plan that is based upon the assessment and planning portion of the review and that includes a timeline to meet the specified goals and objectives.  The plan should be updated annually.

Self-Study Outline

Executive Summary

Briefly summarize the self-study components and any guiding questions for the reviewers to address.

History and Overview:

Describe the fundamental purpose, functions and activities of the unit, including its mission, goals and objectives and how they relate to divisional and university goals.

Internal Environment:

Provide brief general description of overall budget, including any external funding sources.

Budget

  • Operating Budget

[INSERT BUDGET TABLE(S) WITH ACTUAL EXPENDITURES & REVENUE PROVIDED BY BUDGET OFFICE AND OIR]

  • Endowment Budget

[INSERT BUDGET TABLE(S) WITH ACTUAL EXPENDITURES & REVENUE]

  • Salaries

[INSERT TABLE PROVIDED BY OIR OF ADMINISTRATIVE EXPENSE RATIO (ADMINISTRATIVE SALARIES/TOTAL SALARIES)]

  • External Funding Sources
  • Discussion

Provide assessment of strengths, weaknesses and challenges of budget plan, including efficiency measures.

Staffing

Provide brief description of and rationale for staffing structure. Include:

  • Staffing Structure
  • Staff Development
  • Discussion

Space & Technology

Provide overview of the physical space, facilities and unique technologies of the unit.

Physical Space & Facilities
  • Technology
  • Discuss

Programs & Services

Overview

Assessment

  • Productivity Measures
  • Surveys & Other Measures
  • Comparison with Benchmark Institutions

Discussion

External Environment:

Describe the potential opportunities and challenges facing the unit from outside the University that are likely to have an impact on its activities. External forces may include labor market, technology changes, and the regulatory or legal environment.

Planning and Assessment

List of goals and objectives for the next five years including indicators of success, notes on strategies and tactics that will be used to pursue them, and financial implications for the unit. This should include a quality improvement plan that is based upon the assessment and planning portion of the review and that includes a timeline to meet the specified goals and objectives.  The plan should be updated annually.

Appendices

Appendix A: Employee roster by position title

Appendix B: List and job description for each staff member

Data Needs

The unit leader will make requests of data from the appropriate offices, such as Office of Institutional Research, Academic Personnel, Budget Office, etc.

Review Team

Reviewers are selected by the provost or dean in consultation with the leader of the unit.  The review team will typically include internal and external members. The unit should generate a list of ten possible members (seven external and three internal), and the provost will make the final selection of team members that will draw upon the list as well as his/her own knowledge of experts. The goal of the review team is to evaluate the unit’s principal activities in comparison with similar and leading units in other universities, to assess its trajectory and to recommend changes for improvement, under the assumption of constant resources.  Generally, review teams consist of 3-5 persons with at least one member selected from a peer institution with expertise in the key areas of the unit. 

Establishing the date and time for the on-campus visit; developing the charge, questions, and objectives for the review team; and setting the agenda are all critical components that require communication within the unit and between the unit and provost. The opening and closing meetings with the reviewers include the provost (or designee).  They do not include members of the unit being reviewed

Review Report

After the review visit is complete, the reviewer(s) should prepare a critical evaluation of the unit’s self-study and outcomes of the site visit.  The report is due to the provost within one month of the conclusion of the site visit.

The report should be prepared collaboratively among the reviewers with one person taking the lead as the main editor and ensuring that there is a consensus on the contents of the final version of the report.

The review report generally should include the following:

  • Executive Summary (no more than one page)
  • Analysis of the activities and performance of the unit since the last review (strengths, areas of improvement, its response to challenges and changing contexts)
  • Analysis of the unit’s planning and assessment
  • Evaluation and discussion of external factors impacting the unit’s performance
  • Conclusion, including summary of recommendations (under the assumption of constant resources) and any additional impressions, including staffing, resources and viability.

Following receipt of the report, the unit being reviewed will pay honoraria for the external reviewers. Internal reviewers do not receive honoraria.

Units to be Reviewed

  • Art Museum (conducted during review of Department of Art)
  • Center for Teaching Excellence
  • University Libraries (both Oxford and Regionals)
  • Miami Online (Oxford & Regionals)
  • Global Initiatives
  • Graduate School
  • Howe Center for Writing Excellence and Howe Writing Initiative
  • Liberal Education
  • Office of Research and Innovation
  • Honors College