Evaluation in general is the process of collecting and interpreting results of a specific action in order to measure the outcome and find ways to improve the action in the future. In these Guidelines the results of three years of work within the TAFTIE Evaluation Network are given. The TAFTIE Agencies have a common focus on the stimulation of market driven R&D aimed at industrial innovation. The Agencies are responsible for various (government) incentive schemes. Evaluation in the TAFTIE sense is therefore defined as the measurement of the effectiveness and efficiency of government programmes.

The TAFTIE Evaluation Network was started in 1994 in order to exchange ideas and experiences in the field of evaluation. Members of the Network were people within the TAFTIE organisations either with responsibilities in the field of evaluation and assessment or programme managers with a strong interest in evaluation . In the first 18 months of the Network the focus was on evaluation methodology in general. the last two years the focus has been on the determination of a set of performance indicators that can be used for evaluating government programmes. These Guidelines are the result of the activities of the Network in the last two years. They have been adapted from current experiences and experiments among the TAFTIE member Agencies. One point of departure of the work has been the OECD 'Oslo Manual' on innovation surveys.

Although this text is titled "Guidelines…", it does not intend to be a step-by-step manual on which an evaluation of a programme can be based. A complete step-by-step evaluation manual may have been the original idea, but it has since become obvious that for various reasons this is not something that the Network should aim for at this point. However the work on the guidelines, in our opinion, has been contributing greatly to the exchange of ideas on evaluation between the members of the Evaluation Network.⁴

The aim of the TAFTIE Guidelines is to provide a basis for improved transparency and comparability of performance measurements and assessments in reviews and evaluations of TAFTIE partners' activities. The Guidelines can also be used as a toolbox when designing evaluations. Lots of suggestions for performance indicators and parameters including considerations for their applicability, reliability and measurement are given.

The contents of the Guidelines are described briefly below.

Chapter 1 provides with an introduction to the rationale of public funding and the role of a TAFTIE Agency.

Chapter 2 contains the context and policy framework which begins with a brief background of technology policy and continues with a description of the innovation environment from both the agency's and the company's viewpoint. This is followed by a description of the Agency's impact mechanisms. Then the public funding for R&D is discussed in terms of the different types of funding and schemes. Finally, the chapter describes the role of evaluation and monitoring in this context and ends up with some remarks on the design of evaluations.

Chapter 3 discusses the types and uses of performance indicators and considers briefly the concepts of cost-benefit analysis and portfolio analysis.

Chapter 4 provides an introduction to the next five chapters (V-X) by laying out the ground for the core issues relating to the use of public incentives for market oriented R&D aimed at industrial innovation.

Chapter 5 discusses the main rationale for the Agency's existence, the additionality of public funding provided through the Agency based on expert assessment. The issues of efficiency and customer satisfaction (quality) are also discussed in relation to the Agency's operation.

Chapters 6 and 7 describe the issues most involved in company motivation for investing in risk oriented R&D and innovation activities, namely financial rate of return and competitiveness.

Chapters 8 and 9 contains discussions of issues relating mainly to Society's motivation to invest in public incentives for R&D and innovation activities, namely strengthening of national/regional economy and social rate of return.

Chapter 10 discusses the main questions related to measurement problems.

Chapter 11 finishes the document by providing brief concluding remarks and some suggestions for future work.

The Evaluation Network hopes that this Guidelines document will continue to be a living document that captures the development and new ideas in evaluation among the TAFTIE Agencies. It is meant to enhance the evaluation culture within the Agencies and to help improve evaluation as an efficient tool for organisational learning and as a basis for the improvement of Technology Policy implementation Activities.

The TAFTIE Evaluation Network