partner reporter

Is the public expenditure really worth it?

November 14, 2008
Cost of Tracking Schools Is Said to Be $130 Million
By JENNIFER MEDINA, NY TIMES

The Education Department is spending roughly $130 million this year to track the performance of schools in New York City, according to a report published on Thursday by the city’s Independent Budget Office.

Efforts to establish accountability, like performance bonuses for principals and teachers, sophisticated student data systems and report cards that assign letter grades for schools, have been a cornerstone of the Bloomberg administration’s efforts to overhaul the school system.

While the budget office said that the figures in the current fiscal year included some spending that the Education Department does not explicitly count as projects aimed at keeping schools accountable, the report estimates that the city will spend $105 million on similar initiatives next year.

The report found that while private money initially paid for many of the accountability efforts, the same kinds of measures were increasingly being financed with city dollars. Many of the figures cited in the report had already been released, but this was the first attempt to detail the overall costs of Chancellor Joel I. Klein’s accountability efforts.

Betsy Gotbaum, the city’s public advocate, had asked the budget office to review the Education Department’s spending on accountability measures amid complaints from some parents, educators and public officials that the chancellor and his team were overly focused on standardized tests.

Mayor Michael R. Bloomberg has asked the department to make cuts in its $20 billion budget of $181 million from the school system this year and another $385 million next year. Ms. Gotbaum, who has been an outspoken critic of Mr. Klein, said that education budget cuts should be made to the accountability office rather than directly to the classroom.

Ms. Gotbaum said during a news conference at her downtown office that because of the current state and city budget cuts, the Education Department should “re-evaluate the accountability system from top to bottom” to ensure that it is “cost effective and producing real results in the classroom.”

“There are hundreds of millions of dollars that are being spent without real assurance that our schools are getting any better,” she said.

She took particular issue with a student data program called the Achievement Reporting and Innovation System, developed by I.B.M. and a group of subcontractors. The $80 million system has not functioned for much of the school year, even for principals, and is still unavailable to parents.

The report also found that the department spent $2 million on municipal report cards evaluating individual schools. The report card grades were released in September for elementary and middle schools, and this week for high schools. The city has also spent $6 million over three years on parent, teacher and student evaluations of individual schools, according to the report.

Mr. Klein said in a statement: “I believe the dollars we’ve invested in this work are some of the smartest dollars we’ve spent. Leading experts in education in America and around the globe agree and are working to create accountability measures in their schools and districts like the ones we’ve created.”

He added, “The report released Thursday misunderstands the purpose and importance of accountability.”

Thursday’s report also provoked a fresh round of criticism over how the Education Department reports information on its spending to the public. Unlike other city agencies, the department does not make a line-by-line accounting of its expenses available to outside groups like the budget office. Ms. Gotbaum and others have called for tightening the guidelines next year when the State Legislature considers renewing the mayor’s control over the schools, which he won in 2002.

The Independent Budget Office said that its findings on how much the city spends on accountability measures were somewhat subjective, in part because of having to rely on information the Education Department provided. The department repeatedly argued that the accountability efforts were narrower, and said it would spend $37.1 million on the initiatives this fiscal year and $48.5 million in 2009-10.

“It’s been a very difficult process,” said Doug Turetsky, the communications director for the Independent Budget Office. “We’ve had a lot of back and forth over some things that seemed straightforward.”

One of the most contentious points was whether to include as spending on accountability the money that is used to pay for “periodic assessments” four times a year.

The tests are given at all schools across the city, but the results have no effect on a student’s or a school’s evaluation; they simply provide teachers with an idea of how well students are performing. The budget office found that the spending for such tests was $4.3 million in the 2008 fiscal year, which ended June 30, and that, under a new contract, it ballooned to $26 million in the current fiscal year.

The department was protesting some of the numbers in the report as late as Thursday morning. After the report was released, David Cantor, an Education Department spokesman, said it was unfair to include a $30 million grant program as an expense, because it used money that the city received from the federal government. The grants are awarded to schools that are struggling under state and city standards.

Among several initiatives that were initially paid for with private dollars but that are now publicly financed is a school review program, which sent consultants from Cambridge Education, a London-based company, to visit and evaluate schools. The $19.1 million contract is set to expire next August, after which officials from within the Education Department will conduct the site visits.

The School Accountability Initiative: Totaling the Cost

SUMMARY

The city’s public school's accountability initiative was launched in April 2006 as part of the Bloomberg Administration’s Children First reform program. Under the initiative, “progress reports,” “quality reviews,” and other assessment tools were put to use to hold principals accountable for the academic progress of their students.

The accountability initiative has become one of the more controversial components of Children First. Some public school parents and elected officials contend the initiative has contributed to the over-testing of students and that the system of school ratings that has been developed is sometimes at odds with other ratings schools receive from state or federal monitors. There has also been a question about how much of the department’s budget is going towards various accountability pieces. Given this concern, Public Advocate Betsy Gotbaum asked IBO to estimate the cost of the accountability initiative.
The Department of Education does not have a category in its budget labeled accountability initiative. There are a range of activities associated with accountability, making it difficult to draw a clear line between costs of the initiative and the cost of other efforts related to monitoring and improving school performance. Based on our reading of education department documents explaining their various accountability efforts, IBOBO has included a broader set of activities under the umbrella of the accountability initiative then the department says should be incorporated.
Among our findings:
IBO estimates accountability initiative spending totaled $129.6 million last year. We project the initiative will cost $105.0 million this year. Because IBO includes more items as part of accountability than does DOE, our estimates of the program’s costs are substantially higher than those reported by the department.
While the budget for bonuses for schools with the greatest performance increases is rising from an estimated $3.5 million in 2008 to a projected $20.0 million this year, supplementary funding for low-performing schools is phasing out, dropping from an estimated $29.9 million in 2008 to a projected $15.6 million this year.

Some accountability costs were initially covered with private funding, but these same costs are now increasingly being paid with city dollars. For example, funding for Cambridge Education’s school quality reviews was originally provided by the Fund for Public Schools. The $19.1 million contract is now publicly funded.

Some of last year’s higher cost for the accountability initiative was due to one-time start-up, systems development, and related spending. Assuming the initiative continues in its current form, much of the $105.0 million in projected spending this year would be ongoing and make the initiative a significant expenditure in coming years.

NEW YORK CITY INDEPENDENT BUDGET OFFICE
The Accountatability Initiatative

The New York City schools accountability initiative was officially launched in April 2006 as part of the Bloomberg Administration’s Children First reform package. Its main goal was to hold principals responsible for the academic progress of every student as demonstrated by standardized testing and other objective measures. Beginning in the 2007–2008 school year, all principals were held accountable for meeting a “statement of performance terms.” In these documents, principals agree to meet academic performance goals as outlined in a performance review, and to work with the Office of Accountability to utilize tools such as “progress reports” and “quality reviews.”

Over the past two years, significant school resources have been devoted to the accountability initiative. In response to a request from the Public Advocate’s Office, the Independent Budget Office conducted a review of the costs associated with the accountability initiative, including the costs of testing and other school evaluations, information technology costs, contracts with vendors, and personnel costs.
As with many analyses of Department of Education (DOE) spending, the limited information available to outside agencies through the city’s financial data systems made it difficult to independently collect the necessary information. As a result this report depends to a large extent on information that was supplied by DOE at the request of IBO. This general problem was compounded in this case by differences over how broadly to define the department’s accountability initiative.

Because of the wide range of activities associated with accountability, it is not possible to draw a clear line between costs of the accountability initiative and other costs associated with monitoring and improving school performance. Accountability spending is distributed across a wide range of department functions, making it difficult to define costs of the initiative. In discussions with IBO, the department has taken a relatively narrow view, suggesting that these costs should include only the expenses of developing and producing the school progress reports, learning environment surveys, and quality reviews; school bonuses based on these assessments; and the cost of operating the Office of Accountability. IBO takes a broader view that also includes some school-based staffing costs and related items such as school support organizations (SSOs), performance bonuses, and other items contributing to the larger accountability mission.

The difference results from judgments made by IBO to include spending on programs and initiatives that may not be formally under the purview of the department’s Office of Accountability, but appear to us to be closely related to the larger accountability mission.
For example, in soliciting outside organizations to contract as SSOs, the first item in the list of services to be provided was the following: “The provider shall assist in the effective implementation of NYC DOE assessment and accountability requirements.” The importance given to having the SSOs support school leaders operating in the new accountability regime led IBO to count a third of the spending for SSOs towards the accountability initiative.

Using the education department’s definition, IBO estimates that accountability costs totaled $37.1 million in 2008 and will total $48.5 million in 2009. (Except where otherwise noted, years are New York City fiscal years, which correspond to school years.) Using IBO’s broader definition, we estimate accountability costs of $134.9 million in 2008 and $104.7 million in 2009. Note that IBO’s 2008 figure includes $20 million in capital spending for the Achievement Reporting and Innovation System (ARIS) data management system.

Costs of Accountability
Progress Reports.
Progress reports for schools were designed to reinforce standards associated with the accountability initiative. All schools were given a progress report during the 2007–2008 school year. The reports give each school a letter grade (A–F) based on things like student attendance, survey responses, and student performance. DOE indicates that they spent almost $1.6 million in 2007, when approximately 1,200 out of 1,500 schools were graded; every school was graded in 2008 and costs increased proportionately, to $2.0 million. Projected costs for 2009 will decline sharply to $195,000.

These reports are designed to measure both student performance and student improvement. Credit is given for any improvement, even less than a full level, and extra credit is given for students who are in the most need (level 1). For example, under this system, a school whose students who scored at the bottom of level 2 when the year begins and at the top of level 2 at the end of the year will get a higher grade than a school whose students scored at the top of level 2 throughout the year.

Learning Environment Survey. During the 2007–2008 school year, the education department began to solicit feedback from every teacher, parent, and 6th to 12th grade student through a “Learning Environment Survey” that could be completed on paper and online forms. While the survey was optional, the education department strongly encouraged participation.

NEW YORK CITY INDEPENDENT BUDGET OFFICE
The surveys covered such areas as safety, respect, community engagement, and academic expectation. Survey results contributed 10 percent of a school’s progress report grade.
In the first year of the surveys, costs were about $2 million and covered, among other things, development, analysis, printing, distribution, communication, and survey reporting by KPMG and its subcontractors. The KPMG contract is worth about $3.3 million and set to expire in November of 2009 (it is not clear if the current contract will be renewed). Actual spending on the surveys has been approximately $6 million over three years, significantly more than the KPMG contract. The department bore some translation, copying, and distribution costs.

Additionally, the Office of Accountability worked with survey coordinators to oversee shipments of surveys that were sent directly to schools. Survey coordinators are expected to work with the Children First Intensive team and the ARIS team (see page 5 for information on these teams), as part of an evaluation and performance reporting unit. Since these survey coordinators were current employees and only a small fraction of their time was spent on the surveys, IBO did not include this position in the estimate of costs.

Quality Reviews.
Quality reviews started in 2006 and were conducted over several consecutive-day visits to each school site. Educators were asked to observe how well schools were able to use and absorb data to guide (and improve) their instructional methods. The results are 5 to 10 page reports that describe how familiar the instructional staff is with their students’ performance, the school’s ability to set learning plans for its students, the school’s ability to follow through on the plans and how well the school can track its students. All these factors result in an overall quality score. Quality scores are reported on school progress reports.

The quality review rates schools as outstanding, well developed, proficient, under developed with proficient features, and under developed. The first round of reviews was conducted by Cambridge Education, LLC, a London-based, outside consultant. The consultants were originally paid from private donors through the Fund for Public Schools. Ultimately these consultants were asked to train internal Department of Education district administrators to continue the site visit exercises. The Cambridge Education LLC consultants now have a publicly funded $19.1 million contract set to last until August 2009 for “‘provision of quality review and training services.” To date more then $15 million has been encumbered on this contract.

Central Costs.
The Division of Assessment and Accountability, the department’s arm for administering evaluative tools and research, was originally responsible for the accountability initiative.
It is now called the Office of Accountability. The office is located in the Tweed building and had budgeted 79 positions as of June 2008. Much of the staff had previously consisted of education administrative analysts but also now includes computer analysts and a number of technology positions previously located in the Division of Instructional and Information Technology as well as clerical and secretarial support. Also included are a number of knowledge management positions previously located in the Division of Teaching and Learning.

As the accountability initiative gathers more steam, the demand for staff in the accountability office has grown. Aggregate salaries for the office cost roughly $11.1 million in 2008, including fringe benefits, out of a total budget of $23 million (accounted for elsewhere).

Achievement Reporting and Innovation System. The ARIS computer system is intended to provide access for administrators to data about student performance. The education department wanted a single system that principals and teachers could use to retrieve all relevant data, including data from other systems and databases such as the High School Scheduling and Transcript system and Automate the Schools. The system would also allow educators to follow students across grades and school districts. Access for parents is planned for some time during the 2008-2009 school year.

IBM was hired to create ARIS as part of an $81 million contract that expires in 2011. In addition to system development, the agreement also calls for the company to maintain and support the system. Out of this total amount, $22 million is operating expenses and $59 million is capital expenses. DOE indicated that $4.6 million was spent in 2008 as operating expenses and another $4.7 million will be spent in 2009; approximately $39 million of the capital expenses were incurred in 2007 and the remainder in 2008. In 2008, an additional $4.7 million in operating funds was spent on laptops for use with the ARIS system.

ARIS is an example of accountability spending which was initially paid for with private funds but now come out of city tax dollars.

PERIODIC ASSESSMENTS AND ACCOUNTABILITY

Observers of the city’s public schools often lump the department’s periodic assessment tests with the accountability initiative. Periodic assessments are defined by the department as “no stakes assessments that give regular and timely feedback on student strengths and weaknesses to help guide instruction and increase student achievement. They provide early indicators of student performance on state tests and measure student progress.”

The results of these tests are not reported in school report cards or other public assessments of school performance and carry no consequences for schools or school staff, so they cannot be considered accountability costs in the same sense as other expenditures discussed in this report. On the other hand, the periodic assessments are carried out by the Office of Accountability, they are widely perceived as being connected with the accountability mission, and they are sometimes included in the department’s public discussions of the accountability initiative. For instance, the assessments were mentioned by Chancellor Joel Klein in a September 2, 2007 New York Daily News op-ed on accountability. While IBO opted not to include periodic assessments in our estimate of accountability spending, we recognize that others may take a different view. Therefore, we provide some discussion here of the costs involved.

The main periodic assessment tools are Acuity from CTB/McGraw-Hill, online assessments by the Scantron Corporation, a packaged option from the Teachers College Reading Writing Project and various software platforms from Wireless Generation. CTB/McGraw-Hill has a five-year, $57 million contract with the DOE to provide schools with access to the Acuity diagnostic and predictive assessments. Approximately $20 million was spent on this contract in 2008 and $16 million will be spent in 2009. Scantron’s contract with DOE for the periodic assessment program is for $12 million. Spending on the Scantron contract is estimated at $1.6 million for both 2008 and 2009. Wireless Generation has a $6 million contract for assessment testing. All three of these contracts expire in 2010; Scantron also has a small contract for testing materials that expires at the end of 2009.

Alternatively, schools can opt for their own assessments, known as DYOYO (design your own) assessments. The Office of Accountability provides funding to schools that select this option, but because schools may supplement these funds with money from their own budgets, it is possible that costs of DYOYO assessments exceeded the centrally funded budget amount. During 2008, 148 schools used this option and received a total of $685,000 and IBOBO expects this amount to grow to about $1 million in 2009.
In total, IBO estimates that spending for periodic assessments was $4.3 million in 2007 but grew to more than $26 million in 2008 with a $22 million projection for 2009.

NEW YORK CITY INDEPENDENT BUDGET OFFICE
The Michael and Susan Dell Foundation committed $2.4 million in private funds during the Request for Proposal evaluation process, which included the initial evaluation, vendor presentations, and vender selection. IBOBO has not estimated the long-term maintenance and upgrading costs for either ARIS or the laptops but these costs will be recurring for as long as ARIS is used.

Diagnostic Testing. Testing is at the core of the education department’s accountability measures. Elementary students take tests for English language arts, math, social studies, and science. High school students are required to pass Regents exams. There are also entrance exams for the specialized high schools, gifted and talented screening tests for entering kindergarteners, advanced placement exams, and others. These kinds of tests would be given regardless and therefore these costs are not considered here. The new periodic assessments are also not included in our main cost estimate, but because they are arguably connected to accountability, their costs are discussed in the sidebar on page 4.
School-Based Personnel. In addition to costs associated with centrally based positions the department funded certain school-based positions on a per session basis. In 2008, $11.6 million was spent on Inquiry Teams composed of school staff, and an additional $3.1 million on school-based data specialists. Of this expense, $13.6 million was paid for with state Contract for Excellence funds though the “Children First Intensive” professional development program; the remainder was paid out of city funds. In practical terms, the school-based inquiry team works to identify appropriate actions to address the needs of the struggling students at a school by looking at data such as school work, progress reports, assessment results, and so on.

The allocation of more than $16 million in 2008 was intended to be used to pay for per session costs of the supervisors, principals, teachers, and guidance counselors who generally make up the school-based inquiry team. Each school was given an allocation equal to a minimum of 88 hours at a rate of $40.29 per hour. The 2009 allocation is slightly smaller at $13.3 million.

Data specialist allocations were also issued as per session funding. Data specialists were required to commit to two additional hours of work per week over and above time spent as children first inquiry team members. The school-based data specialists are responsible for streamlining the data collection process and providing ARIS support by being the main advocate for data integrity. Each school was funded for up to two hours per week for 39 weeks of the year at an hourly rate of $40.29. The final allocation totaled about $4.0 million in both 2008 and 2009.

School Support Organizations. The support organizations provide support, coaching, and guidance to school administrators. Under the increased budgetary discretion granted to principals, schools can purchase services from any of a number of SSOs, some provided by the education department and some by outside vendors. While SSOs provide a variety of services for schools, one of their primary roles is to assist schools in complying with the various accountability requirements.

Because SSO services are provided as a package, the exact portion attributable to accountability cannot be determined, but given the close connection between SSOs and the accountability initiative, IBO believes assigning one-third of total SSO costs to the initiative is a reasonable estimate. During the 2007–2008 school year, the education department employed 12 school support organizations, including five internal organizations staffed by education department employees. IBO determined that for the 2008–2009 school year, SSO costs—based on the listed prices for SSO services and the number of schools using SSOs—totaled $57 million. This is approximately the same as was budgeted in 2007–2008. Assuming that one-third of SSO outlays are attributable to accountability, annual costs are approximately $18.5 million.

Performance Bonuses. The accountability initiative would have limited effect on performance if there were no system of rewards and penalties. Most people are familiar with federal No Child Left Behind Act performance rankings, but the New York public schools administration has raised the stakes even higher. Schools with Progress Report grades of A or B and Quality Review scores of “Well Developed” are eligible for bonuses.

Financial commitments were originally made from several private sources, including the Eli and Edyth Broad Foundation and the New York City Partnership, in 2006–2007, the first year during which performance bonuses were awarded. In the following year, the education department originally expected to divide $20 million in bonus money among at least 200 schools, but only 135 schools were eligible at a total cost of $3.4 million.1
In the 2008–2009 school year, bonuses totaling about $20 million will be awarded to schools, with the full cost borne by the city. The education department once again anticipated that 200 schools would hit their targets, but in reality only 89 schools did. This includes $5 million that will be given to principals and assistant principals with the strongest progress report scores.

Principals are the beneficiaries of separate bonuses under their latest collective bargaining agreement, which expires in 2010. The Chancellor has sole discretion over these performance-based awards, which can be as much as $25,000. In 2007–2008, the top 1 percent of principals earned these awards, totaling $300,000. IBO projects that these awards will total $350,000 in 2008–2009.

School Success Grant Program. At the other end of the scale, the education department offers supplementary funding to low performing schools. These grants are paid for with state funds, but IBO includes them under accountability spending because they are distributed primarily based on the department’s accountability standards. The grants go to schools that receive progress report grades of D, F, or three consecutive Cs, as well as to those identified as low performing by state standards. Schools with these grades are subject to leadership change after two years and restructuring or closure after four years. These citywide sanctions are layered on top of federal and statewide performance mandates. To avoid penalizing low-scoring schools that later improve their performance, the funding is guaranteed for three years from the year of the initial award.
During the 2007–2008 school year roughly 270 schools received the success grant for a total allocation of almost $30 million. In 2008–2009 the education department will only spend about $16 million on this effort. Spending is reduced because the grant is being phased out and schools that received a grade of D or F this year for the first time will not be awarded success grant funding.

Past and Future Costs. Although the accountability initiative began with some privately funded pilot programs, a progress report evaluation tool, and the ARIS system (to combine a few legacy systems), over the course of its first year, the growth of internal staff, multiyear vendor contracts, performance awards, and supporting technology led to a quick rise in expenditure levels.

Successful pilot programs, initially funded with private grants, have found their way into the full initiative and into the public budget. Cambridge Education best demonstrates the transition from private funding to public funding. While Cambridge’s diminishing role as an outside consultant means that the cost for their contractual services could be capped, the education department picked up the cost by employing their own staff to do the job going forward.

Conclusion
Under IBOBO’s broad definition of accountability costs, we estimate that $130 million was spent on the accountability initiative in fiscal year 2008 and $105 million will be spent on accountability in 2009. Much of the 2009 expenditures are recurring because the initiative requires commitment of full-time staff.
Although IBO believes that the items mentioned above include all major accountability expenditures, there are significant uncertainties about the full cost of the initiative. Accountability requires the use of internal staff, external vendors, software and other assessment projects, not to mention the use of pilot programs to test the utility of a product or service. This makes it difficult to pin down the overall single cost of the accountability mission. But it is clear that the initiative will continue to be a significant expenditure in coming years.

This report prepared by Yolanda Smith

Endnote
1 Districtwide initiative reported in the 2008 –2009 proposed citywide Contract For Excellence Plan. Note, however, that this spending may be cut as a gap-closing measure.