Taftie guidelines on performance indicators for evaluation and monitoring


Apart from direct effects of the R&D project that become visible as results in the financial statements of the participants in the consortium, there are also other benefits to the society, either within or outside the participating companies in the project. These effects which are realised through what was earlier called the social needs mechanism are called externalities. Some of these externalities can have an influence on the innovation system (ref. conduciveness), and are therefore important for innovation policy makers. Other effects like the environment or well-being have effects, usually indirect, on innovation policy as such, but can be considered politically very important.

The usual hypotheses attached to public funding with regards to the social benefits are the following:

  • R&D creates knowledge and innovations which diffuse widely leading into spin-offs which benefit the society. Public funding induces companies to invest more in R&D activities.

  • Public funding can be used to ensure controlled structural changes in industry and the economy and thus ensuring social well-being and economic growth.

  • Public funding can be used to solve social problems and to improve social conditions.

The main issues concerning externalities are: the effective mechanisms, the types of externalities and the value of externalities to society. The diffusion or spin-off mechanism is one very typical mechanism of impact. The effects of R&D have some impact on the company or companies undertaking it. The next effects are then realised in the companies and organisations in the networks with these primary companies. Then there are successive levels of impact depending on the innovation and its commercial exploitation. In some cases the diffusion can be seen clearly, but in others it is very difficult to visualise. Sometimes the impact is largest close to the original innovation and decreases as the diffusion progresses, but in other cases - especially when the social impact is important - the impact can increase as the innovation reaches the society more widely.

A few words about the value of externalities should be said. The value of spin-off's of an innovation is basically something that can be presented in monetary terms, but the problem becomes difficult when all costs and income are being considered. For an innovation spin-off to come to the market usually some R&D activities are required. Now, this R&D can be accounted for if the companies involved can be found. This will however require the Agency to monitor each innovation and to be able to build a map of innovations and spin-off's, which is difficult and requires a lot of resources and basically access to all R&D performed nationally/regionally and even internationally.

Improvements to the innovation environment can be measured in the way described in Chapter VIII. The employment effects are discussed more thoroughly in the following text, but what was said about the diffusion mechanism earlier applies to this in many ways.

The social effects also have some features of the diffusion effect, but what differentiates them from the other effects is the difficulty of assessing their value. The concept of social rate of return can be defined to apply to all externalities described here or it can be defined to consist of social effects only. Anyway, the measurement of social rate of return is a difficult problem. There have been experiments in some evaluations to assess this, but the usual outcome is that the numbers are very high, thus creating a lot of disbelief and criticism.

Externalities are (by definition) primarily effects which are not readily presented in monetary terms and this makes the measurement of these effects difficult and challenging. Most of the data cannot be obtained by the normal project consortium data collection, but must instead be estimated by more indirect methods.

9.1 Employment effects

Maintenance of existing employment and creation of additional em-ployment are often primary aims of national governments, and form the underlying base of technology poli-cies: increased innovativeness, improved techno-economic structure and increased economic activity is expected to form the foundation for the long term security of employment levels. Determination of the employment effects of technology programmes may therefore be of interest for technology policy makers, and could therefore be part of an integral evaluation of technology programmes.

Policy makers are, although they use stimulation instruments at the micro-economic level, primarily interested in effects at the macro-economic level: results at country level, not at firm level. With the questionnaire method is macro-economic data can not be gathered. In order to determine the mechanisms of employment creation and destruction, data on employment at the firm (and project) level however may be useful.

The employment effects can be divided in the following way:

  • employment during the R&D phase

    • direct employment in the company
    • indirect employment in the R&D network around the company

  • employment after the R&D phase

    • direct employment in the company
    • indirect employment in the business network around the company
    • employment elsewhere in the society caused by the diffusion of commercial results of the innovation
    • employment elsewhere in the society caused by increased consumption capacity of those employed by the other means described here

So, there are both direct and indirect effects which can occur both at and after the actual R&D phase. The measurement of the direct effects is rather easy although there are some problems with it. One can basically measure the number of people employed in the businesses on which the innovation has an impact. One problem is that if other businesses are not doing as well, people are transferred to those that are doing well. On the other hand if employment is measured on a company basis, the impact of the innovation is underestimated (or overestimated) if other businesses decline (or grow) at the same time.

Indirect employment effects can be accounted for to some extent in the immediate network around the company especially if most of those companies are included in the original R&D consortium. In addition to the problems described in the last paragraph, there is the problem of changes in the network. In the case where the original business network changes as a result of R&D activities and new innovations, there are obviously some winners and some losers. The winners will probably increase their workforce and the losers will have to find other markets and customers or their workforce will inevitably decrease. How can this be accounted for? Maybe the indication of change and it's growth effect is enough to give a rough estimate of the possible negative effects.

Continuing the analysis outside the network the Agency loses its capacity to measure the employment effect routinely on a company basis. The effects on employment which are realised through commercial use of the developed technology require deep analysis that must be performed on a case by case basis. This type of analysis is time consuming and costly, but might in some specific Agency Actions prove to be useful. This could also give some idea of the magnitude of effects in similar types of cases (products, technologies, services, etc.). These estimates of magnitudes could then be used to estimate the impact of other activities giving some idea of the total effect.

An employment analysis which has been done a number of times in various contexts is the estimation of employment elsewhere in society, due to increased power of consumption by those that are employed directly or indirectly in the business activities resulting from commercialisation of the innovation. These studies have put forward different numbers outlining how many people are being employed elsewhere in the economy because of one new sustainable job created in the industry or in a specified industry sector. These are perhaps the only means of assessing this type of employment effect.

A lot of research effort has been spent to determine the employment effects of technology stimulation . Conclusions however are diverse. An important division is generally made between product innovations and process innovations. In general it is acknowledged that, at the firm level, there is a positive correlation between product innovation and employ-ment, while there may be a negative correlation between process innovation and employ-ment. However there are studies that contradict these results. In order to obtain data that are useful for further evaluation it is therefore necessary to make this division also in the monitoring system.

But there are also qualitative employment effects: effects on the level of education of the people working as a result of the project. A separate indicator should be used for these effects. This indicator could be based on blue-collar vs. white-collar workers, which is perhaps the best simple measure of employment structures in specified industries and services (the field in which the Agency operates). Comparative figures can usually be obtained from the national/regional statistics.

9.2 Effects on public services

One of the basic objectives of public funding is to improve social well being and one indication of that is the improvement in public services. It may be quite easy to define which public services are directly benefiting from some innovation or R&D activity, but the more difficult task is to estimate the value of this improvement.

A number of technologies can be applied to public services, but only a few projects are targeted primarily for this purpose. Thus there are both primary and secondary effects of public funding on public services. It is probably wise to separate these when developing indicators.

The most important public services are health and other social services including education. These are typically the largest single expenses that the society has to face and thus subject to the greatest pressure in developing more effective ways to implement these services. Also the quality of life depends a lot on the availability and costs of these services.

The common problem with the measurement of the impact on these services is the methods of assessing their monetary value. The society has some costs involved in these but, depending on the trend in national/regional policy, the cost structures and available services are changing. The other more severe problem is credibility. Since the costs are high, especially with innovations targeted for global markets the estimates on public savings sometimes come up to very high numbers which are hard to believe. The third additional problem is how to value improvements in quality in terms of money. This has to do with public priorities which in some cases differ if looked at from a public decision making or from a customer point of view. Examples of these can easily be found in public health services where the final customer might see the benefits, but can not get them because there are no cost benefits to the public organisation covering the costs.

9.3 Environmental effects

Measurement of the environmental effects of Technology Policy is also of political importance. In the past, technological development has, in some cases, led to declining environmental quality. Modern technological development, however, takes environmental effects into account: negative environmental effects do often have negative financial effects because of taxation, increased utilities use, etc.

Quantification of the nature and size of these effects is possible, at least to some degree of reliability. A central performance indicator must be based on the different environmental effects which it is possible to measure, e.g. global warming, ozone depletion, acidification, edp, eca/ect, ap, pocp, ht and np. Increased awareness of environmental problems and the extra cost for Society (e.g. because of environmental problems that cannot be traced to a specific source) must also be taken into account when developing performance indicators.

Quantitative determination of the above mentioned parameters is very complex and therefore probably only considered useful for the evaluation of environmental programmes aimed at solving these specific environmental problems. In general a qualitative performance indicator might be more useful, e.g. effect of a project on the environment is ++, +, 0, - or --.

9.4 Spin-offs of the project beyond the participants in the project

Technological results of projects will be exploited primarily by partners in the consortium. There are however three types of effects that can be considered externalities.

The first are the economic effects that are obtained from the technological effects of the project outside the consortium (e.g. the turnover of companies that have bought a licence for the technology developed). As a performance indicator the multiplier effect of the technology or in some cases financial rate of return to customers and users can be used.

The second effect of importance especially when regional objectives are present in programmes, is the non-technological spin-off (i.e. the general impact of the increase in any type of local industrial or other activity creating new jobs and economic growth in the region) e.g. for every 400 people that are employed an extra teacher is necessary, an extra supermarket, etc. (ref. employment). In literature multipliers for these types of effects are given. These can be used when there is a special demand in a certain programme for the measurement of these effects.

The third effect is the diffusion of knowledge. For every R&D and innovation project it is expected that the stock of knowledge of the company increases. This new knowledge can be kept secret or can be added to the existing stock of knowledge, which is an externality.

The position of the company in the network determines how the knowledge flows. The following types of knowledge flows can be distinguished , :

  1. intra-firm / intra-organisational flows : involving the movement of knowledge at individual, group, departmental and establishment level between sites and countries within the same organisation
  2. inter-firm flows / vertical : covering the links between staff from firms in different industries collaborating on the same product, process or in the same technological field (associated with co-makership, buyer-supplier, vendor and subcontracting links).
  3. inter-firm flows / horizontal : relating to the movement of knowledge and staff and contacts between different firms working in the same industry (competitors)
  4. inter-institutional flows : involving links and staff flows between firms, higher education institutes, public research establishments and other intermediaries.

If support is given to a consortium one can also define "intra-consortium flows", flow of information within the consortium (of type 2, 3, 4 or any combination, depending of the consortium). This type of flow can be an indicator of the quality of co-operation.

Taking into account the concept of regional innovation system (RIS), we can even go a step further and define "intra-RIS and inter-RIS" flows depending on the location within a specified region or nation of the different actors.

The direction of the knowledge flow determines if the action is improving the competitive position of the company or on the contrary making competitors or other industries profit from the spill-overs. A knowledge flow towards the company or consortium improves the competitive position. A knowledge flow out of the consortium gives rise to externalities. As public support nowadays is expected to create externalities as well as an improvement of the competitiveness of the companies involved, a good balance has be to found.

By participating in government programmes project participants often change the way they are working in order to better comply with the objectives of the government programme and therefore to increase their chance of receiving funding (see e.g. Cameron et al., 1996 ). A lot of programmes (esp. EU-programmes, but also more and more national programmes) require co-operation between partners. The rationale behind this requirement is threefold: co-operation increases the efficiency of the research (co-operation promotes access to the existing stock of knowledge and therefore diminishes duplication of research that has already been done), co-operation with market parties promotes the market orientation of research institutes and co-operation increases the possibility of multiplication of the results of the project.

It is important to notice one other aspect of knowledge flow. This is sometimes called the network effect and is defined e.g. by Katz and Shapiro in the following way. Network externalities arise when the advantage of using a certain technology grows as the number of users of this technology increases. This effect is technology specific and is thus difficult to transform into general indicators.

Performance indicators regarding the innovation system were already discussed in Chapter VIII.3. Other spin-off indicators are typically based on multipliers and have already been referred to in the text.

9.5 Other social effects

Other social effects include safety, the situation of the disabled, non-discrimination against women and minorities and social equilibrium. The weight that must be given to these effects is very dependent on political priorities. These priorities vary from country to country