Performance Management of RAP3

2.1. Overview

RAP3 was one of the first large scale Results Based Programmes, designed to link payment of invoices directly to measurable results achieved on the ground. In RAP results included 97.5 kilometres of DRCN roads built, 2,200 kilometres of DRCN roads maintained per year, 5,573 million employment days generated and 18,433 training days delivered over the six years of the programme.

The advantage of such clear cut numbers is that results can be precisely measured and progress convincingly demonstrated when set against clear and explicit targets. A physical achievement such as1,983 kilometres of roads maintained against a target of 2,000 kilometres is a credible and verifiable result at 99.3% of target, the equivalent of a “bullseye” in archery or darts where the aim is to get as close to target as possible.

As a measure of the success of its projects which are not results based, DFID conventionally uses a scoring system from ‘C’ (“outputs substantially did not meet expectation”) to ‘A++’ (“outputs substantially exceeded expectation”). More results for a given budget clearly represents good Value for Money but for a Results Based Programme any substantial deviation, whether over achievement or under achievement from forecast, is both “off target” and risks being “off budget”. Instead accurate delivery against forecasts determines good performance. This is a well understood practice by accountants and company board rooms

Accuracy in meeting targets has been measured by RAP management as follows:

Score

Deviation from Target

Remarks

A++

Less than +/- 2.5%

Substantially Exceeded expectations of accuracy

A+

+/- 2.6% to 5.0%

Exceeded expectations of accuracy

A

+/- 5.1% to 7.5%

‘A’ means that output targets are achieved[1].within expected accuracy

B

+/- 7.6% to 10%

Poor accuracy and requires remedial action

C

+/- 10.1% to 15%

Sound the Alarm bell.




[1] Reviewing and Scoring Projects, DFID Smart Guide, June 2014, unpublished.

2.2 Overall Result

The overall weighted score for RAP3 performance over six years using this system is 103%, or the equivalent of an A+ in DFID’s conventional scoring system. In circumstances of remoteness such as where RAP has been working in western Nepal, accurate and precise delivery against forecasts can be difficult. For instance, it took four days to simply walk from the nearest airport to one work site. However, remoteness has been mitigated by the use of Community Contracting whereby RAP has employed members of households living along this road corridor to build the road using labour based approaches that are in harmony with the local construction environment.

As reported by the parliamentary Independent Commission on Aid Impact (ICAI) in its October 2018 Performance Review of DFID’s Transport and Urban Infrastructure Investments (p19) “The Rural Access Programme in Nepal has performed well at both output and outcome level, achieving all of its current targets. It has constructed 97.5km of new roads and maintained 2,200km, improving all-weather access for 2.1 million people. The strong emphasis on road maintenance contributes to sustainability and value for money. A mid-term review found that roads maintained by the programme were used more regularly by local communities, with evidence of increased usage of health posts along the route. The programme uses workers from poor communities as construction and maintenance crews, creating employment for 8,100 people. Another mid-term review found that participating households spent more money on their children’s education and health, had better access to credit and spent more on acquiring productive assets, such as tools and livestock.

 

2.3. Outcome Results

Led by British consultant itad who have had an independent, third party contract for the Monitoring Evaluation and Learning (MEL) component of the RAP3 programme, the LogFrame of the main IMC contract is presented in detail in Figure 1.2 on the next page. As can be seen there have been a number of changes to the LogFrame over the past six years producing two distinct LogFrames. The first applied from 2013 to 2017 and is called here the PRE 2017 LogFrame while the second is called the POST 2017 LogFrame. Where indicators are common to both LogFrames they are designated as PRE/POST indicators as presented in Figure 1.1 below, which shows the 17 Outputs that feed into the 10 Outcomes used in the RAP3 LogFrames both before and after 2017.

Outcomes are not directly controlled by the programme but are the result of outputs designed to have an impact on them. We would therefore expect that Outputs would better match forecasts than Outcomes. Discounting Outcome 10 which is linked to the CONNECT pilot and was not paid by results because it was experimental, Outcomes overall exceeded expectations by 108% while Outputs achieved a better result at 103% of targets.

All outcomes met or exceeded their targets with the single exception of employment days created, which is probably the result of basing forecasts derived from RAP2 figures where works were overwhelmingly focussed on 400km of new construction. By contrast, in RAP3 less than 100km of new construction was envisaged employing some 8,000 workers as against 2,300km of roads maintained by RAP which employed only 1,600 workers.

The measurement of the numbers of people benefiting from road works was originally based on the zone of influence but was later simplified to include the entire population of VDCs benefitting from RAP3 road works.

2.4. Output Results

Variations in performance during the six years of the programme are presented in Figure 1.3 below and show that overall weighted performance was 103%, slightly ahead of forecast but comfortably within an acceptable accuracy range of forecast targets. However, during the first three years, variations were greater than in the second half of the programme.