Taftie guidelines on performance indicators for evaluation and monitoring


To make any activity profitable a company must strengthen its cost and income structures. The purpose of R&D and innovation activities is to create new products, processes, methods or services in addition to or as a replacement for existing ones. These new innovations should reduce costs and/or improve revenues in the short, medium and/or long term.

The rate of return for investment in R&D can basically be calculated in same way as for any other investment. The characteristics of R&D investments differ mostly in risk and in the difficulties in accounting for all costs and revenues involved. The elements of profitability are costs, expected and realised revenues associated with uncertainty or risk. These will have to be accounted for during the expected lifetime of the innovation.

There are some principal hypotheses that are usually attached to the effect of public funding of R&D with respect to profitability. These are:

  • Public funding encourages companies to take higher risks to undertake projects with higher potential, which are rewarded by higher financial returns to companies.

  • Public funding can make long term and otherwise unfeasible projects feasible for companies.

How can the profitability be assessed? The answer is basically quite simple, since basic financial issues are being dealt with. The obvious and well known indicators of profitability for any investment are net present value (NPV) and rate of return on investment (ROI).

To be able to calculate the financial rate of return we will need estimates of all relevant costs and revenues for the duration of the R&D-project and the subsequent commercialisation. As future costs are uncertain and as the uncertainty may differ among projects, upper and lower decentiles of probability of outcome should preferably be assessed. The sensitivity of the financial rate of return due to critical risk factors should be assessed.

Whether the computed financial rate of return is satisfactory or not will depend on the company's next best alternative use of the resources provided for the realisation of the project. Many companies will have some rule of thumb on these figures. Branch studies might identify some average, median or 'best practice' values. The support agency might also have some cut-off criterion, but will most often look at the financial rate of return as just one among several considerations for decision.

6.1 Costs and revenues

6.1.1 Scope

Calculations of appropriate costs and revenues refer to the complete set of activities the company has to carry out in order to be able to achieve the final increase in revenues and/or productivity gains (reduced costs of production). Most often the R&D/Innovation project as supported by the public agency will only be part of this overall set of activities. Information given in support applications will not usually suffice. Additional information will have to be collected for all complementary activities. The term 'project' as used here refers to this complete set of activities. This complete project has to form a meaningful unit for result calculation (ref. attribution discussion in Ch. 10)

If the scope of the project is limited to the company or to a business unit within a concern, no problem should arise. If, for practical reasons another scope delineation has to be selected, the problem of how to treat indirect costs (overheads) arises. Here agencies have different rules of thumb; examples are 0.16% of annual wage, wage + 20%, etc. In reality what matters is to what extent the project pushed demand for common services beyond existing capacity levels. Marginal projects might very well be served by means of excess capacity, and thus escape overhead costs.

Ideally, such a concrete assessment at the margin should be carried out in each particular case. There is no 'correct' rule of thumb for 'short-cuts'. Due to the lack of these, the actual formula applied by each agency should be spelled out, and sufficient information provided to transform one calculation into another for comparability studies.

Agencies will also often have different administrative rules regarding cost items allowed in the calculation of support basis. Due to this, project cost data in agencies' project data bases might be incommensurable. What matters in the financial rate of return calculation is the companies complete resource employment in the project, regardless of agencies' acceptance rules. For this reason additions might be necessary to existing project data.

A particular problem in deciding the scope of the project is the extent to which previous activities upon which the current project is built, should be taken into consideration. In general, previous costs that are sunk, should not be included. However, if the current project is just a step in a series, where the decision to start the previous project clearly included the presumption of continuation with the current one, the proper scope should include the previous activity and its results. In this case the current project is not by itself an individual decision.

How to define real costs is then the next problem, especially if the company is executing several projects simultaneously. Recommendations of the Oslo Manual could be used to overcome this problem. It might be necessary in order to achieve transparency and comparability between Agencies and other public bodies to use a taxonomy of costs of innovation according to the type of innovation activity. According to the Oslo Manual, the following taxonomy could be used:

  • R&D expenditure (intramural and extramural)

  • acquisition of disembodied technology and know-how

  • acquisition of embodied technology

  • tooling up, industrial engineering, industrial design and production start-up, including pilot plants and prototypes

  • training linked to the project

  • marketing for technologically new or improved products

6.1.2 Timing

Internal rates of returns are heavily dependent on the time profiles of revenue and cost streams, in particular early years variations. Thus, simplified procedures asking for annual averages might lead to greatly biased estimates. Most often it will not suffice to ask for time to market, time in market and average annual costs and sales. The correct procedure is to assess financial figures at sufficient points in time to establish proper time profiles of overall costs and revenues. How many time samples will be needed will depend on the curvature of the profiles in each case.

For comparison of alternatives equal durations have to be applied. Thus the rate of return of an investment with 6 years lifetime, cannot directly be compared to an alternative investment with e.g. 8 years (different life span). Reinvestments at different future points will have to be taken into account up to a commensurable common time period. (4 cycles of 6 years could be compared to 3 cycles of 8 years). Normally indefinite (eternal) reinvestments will be assumed, calculating the equivalent annuities. If the next investment cannot be expected to be a simple replication of the current one, the issue of explicit consideration of reinvestment alternatives arises. In particular, considerations of path dependencies should be included, investments keeping more options open being preferable, everything else being equal.

6.1.3 Attribution of project costs and revenues

Financial rates of return have to be calculated on a company basis. If the project is carried out by a consortium of companies, and possibly by R&D-institutions, the problem of attribution of project costs and revenues to each individual partner comes up. More than one partner may make use of the same results from the project for its own further work. R&D-project outputs may have multiple use options.

This problem is more severe for the economic assessment, than for the financial one. Financially speaking, what matters for the companies is the cash flows involved. Take as an example: A consortium of 5 partners develops a new technology at 1/5 of the costs to each. Only one of these finds the outcome sufficiently good to go on building a prototype and finally market the product. In the financial rate of return calculation, only the one fifth of the costs should be included for each company, while in the economic calculation the rate of return should include the full costs. Thus, one company might have a positive financial rate of return, while the four other incur negative ones. The economic assessment should be made for the consortium as a whole, giving some in-between economic rate of return.

The opposite perspective may also give rise to attribution problems: the R&D-projects given public support may be just one of many company activities leading up to the successful new product or process providing the financial return. This attribution problem is however only relevant for the assessment of the additionality of public support. This problem is resolved by a proper calculation of the relevant alternative project, not by trying to assess additionality directly as a marginal change in the supported project. Thus the financial rate of return calculation avoids this attribution problem by having a complete company-wide scope for both alternatives.

6.1.4 Spillovers and pricing of resources

Many companies will have more than one project going on in parallel. There might be mutually beneficial or hampering interactions between such projects. An example might be the pulling of a particular valuable researcher out of some other project causing lower performance of this project than would otherwise have been the case. The detrimental effect might be larger than that reflected in this researcher's wage rate as entered in ordinary project calculations. The real costs to the company for bringing this researcher into the project will have to be adjusted by the additional loss of revenue (or incurred costs) hampering the other activity. In general, all resources employed should be valued at their shadow price; i.e. revenue foregone and/or extra costs arisen in the alternative employment of the resource.

Similarly, positive indirect effects may occur if a researcher learns something in one project that can also be successfully exploited in another project. The benefit can be calculated at the opportunity costs required to achieve the same knowledge if the project had not been there.

6.2 Risks

Perceived risk is one minus the assessed probabilities of success at each project step times the accumulated costs which are not of value for an alternative project. The risks can be identified separately with all project steps and with various aspects, like technology, market or management. Risks are associated similarly with both costs and revenues.

Risk assessment is done by the company since it has the best available knowledge for the purpose. Since policy tends to favour higher risks, the company also tends to present the risks at being higher than they actually are. This is a problem that needs to be addressed at the Agency in order to have a more realistic picture of the actual uncertainties involved.

There are commonly used methods for performing risk assessments which are available and are used by the companies to perform these kinds of analyses. The investment decision will typically depend on the expected realistic and worst cases, in which the concentration is focused on the immediate period of pay-back. Longer lifetime analyses are often not performed very thoroughly. Thus the best cases - which are often displayed to the Agency with the high risk option - are often grossly overestimated.

6.3 Data structures and variables

The basic information consists of the following items:

  • Costs per year (investments and operations)

  • Revenues per year

  • Probability of success per project step

  • Termination value per project step.

Up to present year all costs and revenues are actual figures. Future values are given by optimistic (best case), expected, and pessimistic (worst case) estimates. Best and worst cases are assessed as decentiles, i.e. 10% probability of an even more extreme outcome. Expected estimate is one with 50% probability of better and 50% probability of worse outcome.

Estimates of costs and revenues refers to the successful path of project development.

The risk assessment refers to the chances of having to close down the project, i.e. not being able to continue. The probability of success is equal to one minus the probability of failure at each project step.

The companies taking part in the supported R&D/Innovation project form the primary analysis population. The companies carrying out exploitation projects form a secondary population, which may partly overlap with the first one. A tertiary population will be the exploitation companies' customers (final users of the innovation).

Analyses will be relevant for full surveys or sampled surveys among all these three populations.

Financial rates of return may be calculated for pure R&D- or development companies - selling or licensing their technology. Financial rates of return might be calculated for exploiters turning the technology into commercial innovations. Financial rates of return for users of the provided innovations may also be calculated.