Taftie guidelines on performance indicators for evaluation and monitoring


The Social agenda is concerned with the two issues described earlier and the issue attached to the market mechanism is to some extent similar to the financial rate of return to a company. Here, however, instead of dealing with a company the cost, income and risk features must be assessed according to the national/regional economy. Also with regard to the timing of the income of R&D investments on a national/regional level, the actual approach must be somewhat different.

The main issues concerning the national/regional economy are the structure of the economy and the conduciveness of the environment to increased investment in R&D and production. The structure of the economy at present is the basis for short term improvements in the national/regional economy. The longer term improvements in the economy include the renewal of this structure through the birth of new companies and the growth of existing ones concentrated on the most important areas with regard to the national/regional economy. The regulatory environment and incentives together with supportive R&D and networking facilities are decisive factors in creating a conducive environment especially for high-tech investments.

The hypotheses attached to public funding in creating a conducive environment and facilitating improvements in the structures and operability of the national/regional economy are the following.

  • A versatile economy can endure sector specific recessions well. Public funding can be used to ensure versatility of the economy.

  • Future investments and placement of high-tech industries and businesses will be decided by the conduciveness of the environments. Public measures are vital in creating conducive environments.

  • Strong export balance is a major factor in sustaining and improving a strong national/regional economy. Public funding can be used to encourage companies to invest in developing products for global markets instead of domestic markets.

The improvement of the national/regional economy depends on the structure of the economy and its capacities and the environment for growth. The issues dealing with these questions are discussed below.

8.1 Structure of economy

The structure of the economy under free market conditions tends to develop towards strengthening the strong sectors. The birth of new strong sectors in the industry or economy can be enhanced by public measures. The basis of this is the ongoing analysis of the structure of industry and of the economy the trends of and predictions for their change. The measures consist of activities directed towards creating new companies from existing ones (MBO's etc.), from university research or by other mechanisms. Other activities are typically aimed at finding new synergies between existing national clusters and trying to enhance the development in these areas. At the same time all existing strong sectors or clusters should maintain their strong position by sufficient renewal and change.

The current structure of the national/regional economy is something that can be analysed through national/regional statistics. These statistics include the normal indicators of economic conditions and structures, where the most important ones are the industry and services sectors. The important indicators for industry are first of all domestic and global market oriented sectors and clusters. What are the most important export and import related clusters and how are they connected to each other? How does the university and research institute structure support these clusters and is there a good match in terms of focus on a national/regional basis?

Furthermore, there is the question of other national/regional economic priorities. How do they effect the current situation and any future public investment? Are there regional or other policies that require a focus other than what is written down in the technology policy? Are all public measures in line or do some other public incentives direct companies activities elsewhere from what would be desirable from a technology policy point of view?

Any technology policy actions require good foresight or vision work to support it in setting up the necessary goals. What are the main goals based on the foresight? Does the foresight work give any insight on the future clusters or is it based mostly on the existing ones? Where do the new important growth sectors come from and what strengths or weaknesses does the nation/region have in those areas?

The public funding is of course only one instrument in implementing the technology policy and its effect on the structure of industry and the economy is therefore not easy to show. The common approach is to look into the effect of R&D and innovation activities collected by some R&D and/or innovation statistics and combine that to reach some measure of additionality. One of the possible approaches is to look into the growth areas and companies over a suitable time scale and to try to see if public investment has preceded the growth that has occurred. This can then be complemented by more and deeper analysis on companies to find out the actual effect of public funding in these cases and to draw the appropriate conclusions from those results.

The other simple and frequently used approach is to study the effect of public R&D incentives on the overall investment in R&D in companies. The public funding should have an increasing effect on overall R&D investment made by the companies and thus help alleviate the underinvestment problem.

One of the most important aspects of the structure of the economy is the versatility especially of the industry and of the services which support it. Versatility is a good way to survive recessions in specific sectors of industries. Versatility can be improved mainly by facilitating the birth of new businesses and the growth of existing ones. New businesses can be created from existing companies (e.g. MBO's), from universities and research institutes and from other sources. There are a number of activities in various countries/regions to activate the birth of new companies. The most difficult task is to try and create companies in new sectors of industry. It is always easier to create new companies if there are larger companies that are demanding clients for their services/products.

The other side of versatility is the growth of existing companies. Since one of the main requirements of growth and especially of growth in global markets is the companies ability to create innovations, public funding for R&D is one of the main instruments by which policy makers can in fact enhance versatility. Economic growth is further discussed later.

The indicators for the structure of economy should be based as much as possible on commonly used economic indicators. This helps the discussion with policy makers who have to make decisions concerning the whole economy and who must be able to relate public funding for R&D to other uses of public funds in ensuring that the markets will operate in a desired way.

8.2 Economic growth

Exports are a major mechanism by which a nation/region collects wealth to be used by the policy makers for the benefit of the society. Public funding is frequently used to enhance R&D which is directed to global markets in order to strengthen the export balance.

The position in global markets includes factors like export/import structure and balance, strength and presence in the fastest growing or declining global markets, focus of public funding, etc. The potential of growth sectors is something that can be assessed to some degree as an aggregate of companies' R&D activities and expected outcome combined with an analysis of cross-sector benefits.

The structures of industry and economy are changing dynamically and the public incentives must be able to react to these changes. There are usually a few important trends that are dominant in the global markets and some or all of these are relevant also on the national/regional level. In addition to these megatrends, there are usually some specific national/regional trends that are important. Identification of these trends and of their expected impacts is important in policy making and policy implementation.

Identifying the most important sectors of industry and economy is one of the tasks of the Agency. The national/regional policy is not usually directed according to specific industrial sectors, although in some countries/regions there are some sectors that are more important in the economy and also in policy.

The problem in many cases is how to show the impact of public funding on something that must have had all related issues solved before it is able to grow. What has been the impact of public funding to this success story? The important thing is to measure or monitor any impact on-line. Ex-post evaluations will always run into credibility problems, even when the data is collected on-line.

As with the structure of economy, the indicators that relate economic growth to public funding must consist of commonly used economic indicators. The focus should be placed on identifying and showing the new and emerging future clusters and sectors together with their potential and the public investment. This is to emphasise the longer term dimension which differentiates R&D investments from other more immediate public investments.

8.3 Conduciveness of the innovation environment

This part of the TAFTIE Guidelines will have to be developed further. The present version is no more than a selection of very preliminary ideas to give some direction and inspiration in this regard. The project portfolio of the TAFTIE agencies should be looked upon as a great, but under-utilised, information resource. Much learning about the conduciveness of the various national systems can potentially be drawn from focused and well-structured monitoring of agency projects.

A conducive innovation environment reduces innovation costs and increases the propensity to innovate. These are the basic reasons why the future investment and placement of the high-tech activities of companies will be decided by the conduciveness of the environment. Public funding of R&D offers one specific incentive together with other public measures to ensure the conduciveness.

At the macroeconomic level the 'conduciveness' parameter will correspond to the 'productivity parameter A in the model Y=A*L1-a*Xa. Thus, with a higher A one country will achieve a larger product (Y) with the same use of labour and capital. More informaiton on the lack of macroeconomic models with endogenous R&D a lot more practical indicators related to the 'ease of life' of entrepreneurs, the efficiency of support agencies, availability of laboratories etc. Most agencies have some rudimentary indicators of this type already in use, but a more consistent and comprehensive way of characterising national (or regional) innovations systems at aggregate (or macro) level is lacking. An important component of the innovation environment will be indirect regulations, i.e. tax rates, company laws etc. Many countries have R&D tax support schemes. (Comparison of the degree of net support between countries is difficult and has been addressed by the OECD and Canada in the so called B-index method.)

The conduciveness factor may be the most difficult to assess and certainly will require intensified efforts and new research by public support agencies. TAFTIE may be a relevant forum for such tasks.

Some of the possible ideas for measurement of conduciveness are:

  1. Average or modal time required to set up innovation network
  2. Average or modal interaction costs per innovation
  3. Average or modal time required to achieve financing of innovative projects
  4. Percentage of innovative project finance supplied by venture capital or similar sources
  5. Average or modal financing costs as percentage of total costs of innovative projects
  6. Geographical density of R&D-institutions taking part in innovation projects
  7. Locational density of venture capital or similar financing sources
  8. Average level of direct public subsidies to innovation projects

Of course a series of other indicators might be equally well suited. This is an underdeveloped research field. Experimentation (and innovation !) will be required. The demand for meaningful indicators at the aggregate level is at present clearly not available nor are they properly defined. Most studies of national systems of innovation consist of (endless) descriptive statistics a far cry from the provision of comprehensive indicators of the general conduciveness of the system. However, the field is gradually and slowly improving. One major source is the European Union innovation survey.

The simple analysis will be to calculate the above indicators for the complete portfolio. The analytical problems mentioned under methodological issues above call for careful interpretation and control for third variables. Stratified sub-sampling will be required in most cases.

Those indicators that relate directly to the efficiency of support agencies will of course have to be addressed by these. For the other more systemic features of the innovation system, the agencies have the choice either to go on struggling against current obstacles, or to channel more resources into obstacle removals, maybe by advice to ministries, opinion raising activities, joint actions with public and industrial bodies etc.