Outranking Methods - NAIADE

Extended Version

Introduction

Naiade (Novel Approach to Imprecise Assessment and Decision Environments) is a MCA method able to deal with both qualitative and quantitative information concerning policy socioenvironmental evaluation problems. Naiade was developed by G. Munda and technical information on his theoretical background and application procedure can be found in Munda (1995). Naiade can provide the following information: (a) Ranking of policy alternatives according to a set of evaluation criteria, (for instance, a compromise solution/s); (b) Indications of the distance of the positions of the various interest groups (e.g., possibilities of convergence of interests or of coalition formation; (c) ranking of the alternatives according to the actors' impacts or preferences.

Naiade also performs an equity and conflict analysis in order to identify those alternatives which could reach a certain degree of consensus or would provide a higher degree of equity among different interests groups. It is a very flexible method suitable for real−world applications, and in particular, for situations where fuzzy uncertainty or indeterminacy is recognised. In particular, fuzzy uncertainty regards not only to the difficulties to set probabilities about the occurrence of a particular event but especially to the difficulties to describe the event itself in an unambiguous manner.

Methodology

The application of NAIADE entails the construction of an impact (evaluation) matrix which includes, on one axis, a limited set of given policy alternatives, and on the other, a limited set of different criteria by which such policy alternatives are to be evaluated. Inside the matrix measurements of all the policy alternatives with respect to each evaluation criterion are given and the different alternatives are assessed by means of pairwise comparison. This aggregation procedure takes into account both the number of the criteria in favour of each alternative and the intensity of the actors' preferences. In principle, the determination of the criterion scores is independent from the actors' preferences; for instance, impact on employment can be chosen or used as a criteria to evaluate a given set of policy alternatives but the score of each criterion may not depend on the actors' preferences

Process

NAIADE compares a finite set of alternatives (hence it is a discrete method) on the basis of various pre−defined criteria. NAIADE does not allow giving weight to the criteria: all criteria are given the same weight. The consecuence is that the dimensions are given weight according to the number of criteria belonging to each of them. NAIADE generates a ranking of alternatives (problem formulation) through pair−wise comparisons. The ranking can be complete preorder, when only relations of preference or indifference between the alternatives exist, or partial preorder, when imcomparability relations may also exist. The aggregation process also includes some indicators on uncertainty and compensability. NAIADE can perform two kinds of evaluations:multi−criteria analysis and equity analysis.

The multi−criteria analysis is based on a comparison algorithm of the alternatives. It is performed according to the following steps:

1. Impact matrix with score values assigned to the alternatives' criteria;

2. Pair−wise comparison of alternatives using preference relations;

3. Aggregation of all criteria;

4. Ranking of alternatives.

The equity analysis is performed by the completion of an equity matrix where a similarity matrix is calculated. It sheds light upon the level of decision conflicts among the different interest groups and highlights the possible formation of coalitions (building adendrogram of coalitions) showing the impact of each alternative, as perceived by the social actors. In this way, NAIADE gives the following information: (a) Distance indicators between the interests of the different social actor groups, as an indication of the coalition formation possibility, or interest convergence; (b). Rankings of alternatives for every coalition, in accordance with the impacts over the social groups, or the social compromise solution.

Review

Evaluation of results

Policy processes

MCA is a useful tool which can contribute to assess the nature of the policy problem and identify the conflicting values or criteria which affect the different possible options. As other MCA methods, Naiade can supports structuring the problem and guiding the search for further relevant information, and can also be used in investigating the conflicting assumptions of relevant stakeholders. In this manner, Naiade can con tribute to the analysis of policy proposals by making explicit the impacts table of the various alternatives. Naiade can support the selection of the most optimum policy option, or the ranking of the best ones, according to the different selected criteria. However, Naiade is not suitable to give support in later stages of the policy processes, and in particular, those related to the implementation of already assessed or selected options.

Sustainability aspects

Naiade can deal with sustainability assessment problems because it can collect an deal with information on the three dimensions of sustainable development, the economic, the environmental and the social one. Naiade can incorporate several sustainable development aspects as separate criteria to compare alternative policies, such as the following: intergenerational effects (e.g. on the loss or natural or social capital), (de−)coupling aspects, adaptability and (ir−)reversibility. MCA is suitable to compare impacts independently of the gauge year. MCA can incorporate the impacts on different groups/sectors/regions as separate categories and give a clear overview of the differences for these sectors. MCA is suitable to compare impacts independently of the spatial dimension, as long as the spatial dimension of the separate criteria are comparable. Distributional effects and global impact can be accounted for as a criterion to assess the various alternatives.

Operational aspects

Cost for the application of Naiade vary very much depending on the context, but relatively cheap, as the tool is readily available on the net; As a whole SMCE can be applied in 4−10 man−months, depending very much on the depth of the assessment, issues at stake and availability of data. Time for making the assessment can vary at about 120−300 days to collect the information, but once the impact matrix is made, and therefore the whole relevant information is available, the assessment can be done in a few days. Defining the criteria against which the alternative are to be evaluated as well as the evaluation process itself requires a large amount of structured information to ensure the quality of the assessment. NAIADE is able to deal with both qualitative and quantitative information to assess the impact of the different alternatives and with different degrees of uncertainty, including fuzzy sets (where the issues are not defined in an unambiguous way). Results are given in the form of a ranking of alternatives according to each criteria and actors' preference, as well as a dendrogram of the coalition formation process (as outcome of theequity analysis). Although MCA analysis does not require computational processing per se, NAIADE is usually applied with its attendant computer software.

However, caution needs to be taken during its application:

  1. The outcome evaluation will always be highly dependent upon the initial contextual settings.
  2. When defining the criteria and how to evaluate them, one must keep in mind that NAIADE works on the basis of a pair−wise comparison (i.e. relative evaluation).
  1. One of the main difficulties lies in defining the preference thresholds (for each social actor and in agreement with the conjunction of the participatory process) for quantitative criteria. Condensing different visions and aims in one number may deliver biased evaluations. A sensitivity analysis is therefore recommendable in view of assessing the robustness of the evaluation as well as the influence of each criterion in the final outcome.
  2. In order to avoid biased outcomes, high levels of transparency and participation are required throughout the evaluation process, and especially when defining the referential parameters.

Furthermore, it is important to note that the equity analysis of NAIADE (which aim is to represent the position of each stakeholder with regard to each alternative), departs from the assumption that each stakeholder is equally powerful, a situation which does not occur in real world situation. In this regard, utility analysis can be understood as an acceptability analysis but it should be counterbalanced with an institutional analysis of real power relationships, which occur within the policy−making process.

Experiences

The use of Naiade in Social Multicriteria it has demonstrated its suitability and effectiveness in various decision−making processes. In particular, Naiade has been used in a participatory way in several occasions, by means of combining the use of MCA software with input from focus groups and interviews. Examples of this combination can be read in De Marchi, et al. (2000, Funtowicz et al., 1998) for the case of water management in the municipality of Troina, Sicily, and the exploration of alternatives to development of near−by area around a National Park in the Pyrenees under threat of sky resort expansion (Munda, 2004; Tabara, 2003). The inclusion of the later was intended to provide insights of the relevant criteria and stakeholders' preference systems to be considered in the assessment. Another recent application of this approach has been carried out for the assessment of plausible electrification alternatives in a rural area covering the Biosphere Reserve of El Montseny nearby Barcelona (Russi, D. 2004). Munda (2004) goes beyond the simple application of Naiade in order to encompass and deal with the issues of social processes and evaluation, thus developing and calling its approachSocial Multi−criteria Evaluation process (SMCE) method.

A recent Phd dissertation using MCA in Colombia and in particular Naiade (Vargas Isaza, 2004), shows that some options which could be considered by the MCA matrix at the local or regional level −such as ecotourism− may not be feasible in a country at war, so the final assessment of the alternatives of action may be dependent upon institutional and social factors which lay beyond the boundaries of the problems considered by a MCA. Other analytical tools and procedures −such as institutional gap analysis, or studies on public perceptions− may be needed to complement the results of a MCA. Furthermore, while MCA can help, in a very relevant manner, to structure the relevant information −both quantitative and qualitative−, its assessments need always to be read not in a static way but as the result of a dynamic broad social process in which the MCA tools, as other tools, can play a significant role.

Combinations

There are possible MCA links and combinations with other tool teams of the Sustainability A−Test. For instance, valuation methods can be used to assess the economic criteria that will be used in MCA. Such combinations are not restricted to a particular method, but can usually be used in compensatory, outranking or non−compensatory methods. Among such possible relationships are those related to scenarios, although it must be very clear that the impacts matrix or the option alternatives are not scenarios in any manner. In this guise, many foresight exercises utilise simultaneously several distinct types of methodologies: for instance, the combination of MCA and "Scenario Analysis"[1]. An example of this sort of "hybrid" methodology could be the Deliberation Matrix concept formalised in the GOUVERNe Project (Guidelines for the Organisation, Use and Validation of information systems for Evaluation aquifer Resources and Needs) where the Naiade was partially applied. This Deliberation Matrix has then been adapted for application within the PEGASE project, as a framework for the presentation of scenarios in a multi−criteria, multi−stakeholder perspective. The scope of VIRTUALIS was to exploit these concepts, and adapt them in the full development of a multi−player game with ICT−mediated interactivity. The VISIONS Project was a groundbreaking endeavor in the development of scenarios and integrated visions based around an "integrated" multi−disciplinary scenario exercise with stakeholder participation. So, MCA frameworks in "Scenario Composites" (VISIONS, GOUVERNe, PEGASE, VIRTUALIS, etc.) that, as a "fuzzy future"[2], are felt to traverse sufficiently the spectrum of driving forces, policy degrees of freedom and governance issues. Also, considering the narrative characteristic of the scenarios' development, qualitative criteria scoring, as provided by Naiade, seems more consistent than using projections or forecasting to valuate criteria. Also, the equity analysis provided by Naiade offers an opportunity to scenario analysis in assessing coalition possibilities and inter−stakeholder agreement for various scenarios.

Therefore, Naiade has been used in a participatory way in several occasions, by means of combining the use of MCA software with input from focus groups and interviews. input from stakeholders is usually used to define both the criteria and the alternative sets in order to render the outcomes of the MCA application more transparent and relevant. Then, as suggested by Munda Naiade can be used in a participatory way as part of a larger social Multicriteria policy assessment. In this way, MCA can become Social Multicriteria Evaluation (SMCE) and it application process entails the following steps: 1. Structuration of the problem, where the main boundaries and components of the issues at stake are defined; 2. Institutional analysis, aimed at examining the main interests involved; 3. Generation of the policy options; 4. Development of the Multicriteria impact matrixes, e.g. via Naiade; 5. Application of the mathematical procedure to aggregate and discriminate the alternatives; 6. Sensitivity analysis, to assess the robustness of the analysis (e.g. to see when changing parameters the final ranking result vary. If understood as such, SMCE can be seen more as a general approach to policy evaluation, where it is thought that the pluralism of conflicting interests, values and options have to be taken into account, rather only a isolated method.

In this regard, P. Paneque et al. (2004) while assessing the use of Naiade in the evaluation of alternatives for the use of scarce water resources in an area in South Spain stressed the importance of a full integration of such approach with other procedures of a deliberative kind during the whole process of its application:

'when the stakeholders were presented with the final list of alternatives that had been worked on [via SMCA and Naiade], they suggested the inclusion of other alternatives initially proposed by a small number of stakeholders, but not included in the final evaluation exercise precisely because they did not enjoy extensive support [¡]. This shows that the latter phase in which all the stakeholders meet is crucial, because it can significantly change the final result' (P. Paneque et al., 2004: 87).

The use of participatory tools are enhance the social robustness and relevance of its results. MCA tools can also use data from both physical (e.g. Material Flow analysis) or monetary sources (e.g.: Cost−Benefit Analysis) as scores for the different criteria to assess the limited set of policy alternatives.

Strengths and weaknesses

MCA tools are mainly analytical tools which therefore are dependent on the way the problems at stake are defined in the first place. The availability of data to carry out the first depiction of the situation, the way the boundaries of the decision−making process are set, or even the language or rhetorics used in the qualification of the different criteria influence the final results. Furthermore, as it is the same when measuring social perceptions, the values assigned to different alternatives or criteria can change over time as a result of the learning process. Also, different constituencies which operate at different scales (e.g. local, national) may provide different assessments of those finally selected by the modeller or some crucial groups may be omitted in the process.

Among the strengths and potentialities of Naiade it is possible to mention that Naiade:

  • Is able to assess the impact of a given alternative in their original unit.
  • Can use information affected by different types and degrees of uncertainty, including fuzzy data (where the phenomena to deal with cannot be defined in an unambiguous manner,e.g. indeterminacy).
  • Can attribute values to the alternatives' criteria expressed in the form of either crisp (exact and certain), stochastic, fuzzy numbers or linguistic expressions.

While amongst the disadvantages, NAIADE shows:

  • Complexity of the model renders it little intuitive: many parameters need to be determined (aggregation and compensation coefficients like alfa, gamma, etc.)
  • Notable level of technical expertise is required for its sound use and correct understanding.
  • Fastidious participatory and social process: 4 preference and indifference thresholds to be defined in agreement with all social actors involved.
  • No weights can be applied.
  • No possibility of directly using ordinal criteria (converted in qualitative or crisp criteria).

 

Another disadvantage of these MCA methods is that that they can less suitable for processing qualitative data. In practice, this disadvantage is not very significant because the pluses and minuses used for qualitative assessments are often derived from underlying classes of quantitative data. With a well−chosen method of standardization this underlying quantitative scale can be used in the weighted summation of these scores.

Further work

The approach is being used in several policy research applications by the students of G. Munda at UAB both in Europe and Latin America. Within the Sustainability A−Test, further work could be done to assess to in what ways Naiade can be combined with other tools and in particular to be applied at large scales. In particular it would be worth exploring to which extent Naiade could be further developed to serve as multicriteria appraisal for the evaluation of specific current ongoing EU policies with impact on the global scale such as climate change.

References

Munda, G. (1995), Multicriteria evaluation in a fuzzy environment. Theory and applications in ecological economics, Physica−Verlag, Heidelberg.

Munda, G. (2004). Social Multi−criteria Evaluation (SMCE): methodological foundations and operational consequences, European Journal of Operational Research, in press.

De Marchi, S. O. Funtowicz, S. Lo Cascio, G. Munda. (2000) 'Combining participative and institutional approaches with multicriteria evaluation. An empirical study for water issues in Troina, Scilly. Ecological Economics, 34:267−282.

Joint Research Centre, JRC, 1996. NAIADE Manual. JRC, Ispra, Italy, available at: http://www.aiaccproject.org/meetings/Trieste_02/trieste_cd/Software/NAIADE/naiade.PDF

Russi, D. (2004) Social Multicriteria Evaluation of rural electrification. Master's thesis supervised by G. Munda. Dept. Economics and Economic History: UAB. Barcelona.

Tabara, D. (2003). 'Participacion cualitativa y Evaluacion Integrada del Medio Ambiente. Aspectos metodologicos en cuatro estudios de caso' (Qualitative participation and Integrated Environmental Assessment. Methodological issues in four case studies). Documents d'Andalisi Geografica., 42.

Funtowicz, S.; Martinez−Alier, J.; Munda, G. & Ravetz, J. (1999). Information tools for environmental policy under conditions of complexity, European Environmental Agency, Experts' Corner, Environmental Issues Series, No.9.