Receipt date: 10/01/2022 / Revision date: 10/05/2022 / Acceptance date: 23/08/2022

Introduction

The development of this research arose from the need and importance of implementing a formal evaluation procedure at the end of each EPH project to validate the achievement of the project's products and detect the causes of any deviations in costs, scope and execution time that may have occurred, and to propose evaluation criteria to be applied in future projects.

This work will show that the evaluations are a valuable source of information and that, based on the data collected throughout the evaluations, they demonstrate the accuracy of the company's projection and formulation of projects, the convenience of using the same suppliers or materials for future projects, and would allow for corrections or adjustments to internal procedures if necessary, among others. 

The goal of this research is not the comparative analysis between different companies in the sector, but to generate for itself a project learning from the lessons learned in the case study. This is why the analysis is focused in-house so that the organization can know its strengths and weaknesses in the design and management of projects.

Efficiency evaluation is defined as "a comparative analysis between the components that were planned to be executed according to the pre-investment study that led to declaring the project as viable and the components actually executed" (JICA and MEF, 2012, p.195), analyzing various factors such as the achievement of the outputs obtained, execution time, costs, overall efficiency and ex post project sustainability.

Method

The work was developed through the use of a non-experimental research design of the projective type with a mixed approach from the collection of information whose data collection instruments consisted of in-depth interviews conducted with key project personnel along with documentary review.

These instruments were validated based on expert judgment to ensure their suitability.

In order to evaluate the project, use was made of the completion evaluation included in the ex post evaluation methodology proposed by the Japan International Cooperation Agency and the Ministry of Economy and Finance of Peru (JICA and MEF, 2012, p.196), which analyzes the efficiency and sustainability of the project. In addition, lessons learned can be obtained to subsequently generate action plans for corrections or improvements, and subsequent follow-ups and measurements are suggested to analyze the project's performance in the medium term and confirm the achievement of the project objectives established in the early stages of the project.

Results

The results and justifications obtained for the Efficiency Evaluation concept will be presented below:

The PH under study has been divided into six main components. These components were evaluated individually to corroborate whether they had been fully completed or not, and to confirm whether they provide the service for which they were built.

 

Analysis of project deliverables

Table 1

Quantitative comparison of the main outputs of the project

Main project components	Component unit	Quantity (Number, physical dimension, etc.)
		Preview	Current	execution % (Actual/ Planned)
DAM AND INTAKE WORKS
Dam and intake works	global	1	1	100%
3000 PSI concrete and 15 kg/cm2 cyclopean concrete	m3	907	1,452.09	160%
CONDUCTION PIPE
Conduction piping	global	1	1	100%
Total length	m	2,872.00	3,215.90	112%
PRESSURE TANK OR LOADING CHAMBER
Pressure tank or loading chamber	global	1	1	100%
Concrete 280 kg/cm2	m3	515.	514.	100%
PRESSURE PIPE
Pressure piping	global	1	1	100%
Total length	m	654	654	100%
POWERHOUSE
Powerhouse	global	1	1	100%
Construction area	m2	330.91	455.	138%
LIFTING SUBSTATION
Lift substation	global	1	1	100%

 

Note. Adapted from JICA and Ministry of Economy and Finance of Peru. ( 2012)

 

Table 1

Analysis of implemented project components

Main project components	execution% Execution% Execution% Execution% Execution% Execution% Execution	Information extracted from interviews
Dam and intake works	160%	Additional dam placement, requiring an increase in the volume of cyclopean concrete and 3000 PSI concrete of approximately 550m3.
Conduction piping	112%	During the execution of the works, it was necessary to go around obstacles, which led to an increase in the length of the pipeline. In addition, it was necessary to use a considerably higher amount of blasting than estimated because the soil found on site had different characteristics than expected. There is a lack of an adequate geological study.
Pressure tank or loading chamber	100%	The design was slightly modified.
Pressure piping	100%	It was carried out as planned.
Powerhouse	138%	The powerhouse was redesigned to house an additional turbine, requiring an increase in area, and the construction method of part of the powerhouse was also modified to minimize the impact on the budget.
Lift substation	100%	The use of a transformer bank was required to reduce the unbalance between the phases of the National Interconnected System circuit that hindered the synchronization of the plant and put the plant's equipment at risk.

 

Analysis of the time efficiency of the project

JICA and MEF (2012) indicate that project time efficiency is obtained by making a comparative analysis between the execution time foreseen in the pre-investment study with which the project was approved and the time it actually took for the project to be completed.  For this section, the project was divided into seven main components. The data were obtained from the schedule presented in the feasibility study and the final construction schedule provided by the PH supervising company.

 

Table 2

Comparison of Planned and Actual Execution Time

	Scheduled			Current			Expected/ Actual Ratio	Degree of efficiency
	Home	End	Period (months)	Home	End	Period (months)
Overall execution time	01/11/2017	31/03/2019	17.	10/12/2017	15/01/2021	37.	0.46	Inefficient
Execution time by main activities
Preliminary activities	01/11/2017	30/04/2018	6.00	10/12/2017	09/04/2018	4.	1.	Very Efficient
Dam and intake works	01/02/2018	30/09/2018	8.	01/02/2018	30/03/2020	25.00	0.	Inefficient
Conduction piping	15/01/2018	31/01/2019	12.	10/12/2017	20/05/2020	29.	0.43	Inefficient
Pressure tank or loading chamber	01/05/2018	31/07/2018	3.	09/11/2018	30/03/2020	16.	0.	Inefficient
Pressure piping	01/06/2018	28/02/2019	9.	09/04/2018	13/06/2020	26.	0.	Inefficient
Powerhouse, Overhead crane, Turbines and generators	01/03/2018	31/03/2019	13.	07/07/2018	23/09/2020	26.	0.	Inefficient
Transmission line, Lift substation and Electrical and control system	01/07/2018	31/10/2018	4.	10/02/2020	15/01/2021	11.	0.	Inefficient

 

Note. Adapted from JICA and Ministry of Economy and Finance of Peru. (2012)

                
Project cost efficiency analysis

According to JICA and MEF (2012), project cost efficiency is the:

Comparative analysis between the total investment cost of the project foreseen in the pre-investment study with which it was declared viable and the total cost of the project in its execution, at the level of each product, component or package of contracts. (p.73)

 

Table 0

Comparison of Expected and Actual Cost

	Preinvestment Study (Thousands of USD)	Amount Executed (Thousands of USD)	Expected/ Actual Ratio	Degree of efficiency
Overall Cost of Execution	10,944	18,481	0.	Inefficient
Cost per product (Components)
Pre-investment, land purchase and construction administration	515	1,349	0.	Inefficient
Indirect costs (revolving fund, financial and legal expenses)	877	3,265	0.	Inefficient
Development costs, preliminary activities, temporary activities, access and miscellaneous works	534	988	0.	Inefficient
Dam and intake works	1,129	1,509	0.	Inefficient
Conduction piping	4,326	5,515	0.	Inefficient
Pressure tank or loading chamber	204	623	0.	Inefficient
Pressure piping	866	1,296	0.	Inefficient
Powerhouse, Overhead crane, Turbines and generators	1,999	3,426	0.58	Inefficient
Transmission line, Lift substation and Electrical and control system	495	511	0.	Inefficient

Note. Adapted from JICA and Ministry of Economy and Finance of Peru. (2012)

 

Analysis of overall project efficiency

This analysis of the overall efficiency of the project is based on the level of achievement of the outputs, on the time required to execute it and on the investment costs required to complete it. 

Figure 1. Calculation of Overall PH Efficiency

 

Sustainability analysis

According to JICA and MEF (2012), the action is defined as:

Evaluate the factors indicated in the pre-investment study that declared the project's feasibility, which guarantee that the project will generate the expected benefits and results throughout its useful life, and analyze whether and to what extent these factors are maintained or vary.  Of special interest is the identification of problems that occurred during the execution period, as well as the possible risks in operation and maintenance. (p. 76)

This evaluation was divided into 3 main dimensions, namely:

Financial Sustainability

The purpose of this section is to evaluate, first of all, the accuracy of the O&M, administrative and financial costs and expenses foreseen in the feasibility study against those actually incurred during the operation period up to the preparation of this report.

 

Table 3

Comparison of projected and actual O&M, Administrative and Financial costs from December 2020 to August 2021

	Planned (USD)	Real (USD)	Planned/Actual Ratio	Degree of efficiency
Projected Operating & Maintenance and Financial Costs and Expenses for Dec - 2020 and Jan - Aug - 2021	1,034,333.33	775,490.12	1.	Overall, the costs and expenses are within the budget estimated in the feasibility study.
Breakdown of costs and expenses
Projected operating, maintenance and administrative costs and expenses for Dec - 2020 and Jan - Aug - 2021	635,000.00	203,512.34	3.	These costs and expenses DO fall within the budget estimated for this purpose in the feasibility study.
Projected financial expenses for Dec - 2020 and Jan - Aug - 2021	399,333.33	571,977.78	0.	These costs and expenses are NOT within the budget estimated for this purpose in the feasibility study.

Secondly, to make an initial assessment to confirm whether the costs of all operations, maintenance and administration are being covered by the income received from the operation of the plant.

 

Table 4

Verification of the coverage of costs and expenses for the production to be invoiced*

Amount to be Invoiced* (USD)	O&M, Administrative and Financial Costs and Expenses (USD)	Invoicing/Cost & Expense Ratio	Remarks
794,488.19	775,490.12	1.	Costs and expenses to produce and cover financial commitments are covered with a reduced margin of slack.

Note. *This column shows the sum of the amounts to be billed between December 2020 and August 2021. It is worth mentioning that this amount does not necessarily coincide with the actual invoicing, nor with the payments received from the client.

Operational Sustainability

The purpose of this dimension is to confirm whether the components are capable of providing the service or function for which they were built, thus asserting the operational sustainability of the executed project.  Data were acquired through structured interviews and documentary analysis. 

 

Table 5

Operability of delivered components

Component	Expected function or service	% of compliance with the objective (0% to 100%)	is it possible to improve the function or services currently provided by the delivered components?		Proposals for improvement	Additional remarks or comments by the Plant Operations Manager
			Yes	No
Dam and intake works	Accumulate water in the riverbed and divert part of the water to the pipeline.	80%	X		Construction of a sand trap. Raise the height of the dam to increase the pressure at the penstock inlet.	The desander is missing in the dam. The river's flow is not being used.
Pipeline	Gravity conveyance of water from the dam and intake works to the loading chamber.	80%	X		Improve pipe supports. Internal painting of piping.	Settlements of up to 25 cm in sections of the pipeline. Small reverse slope sections. There is more friction than estimated. It recommends painting the inside of the pipeline to protect it from corrosion, improve internal friction and prevent the release of rust particles that could affect the turbines in the long term.
Pressure tank or loading chamber	Stabilize the water level before entering the pressure pipe. It allows managing load oscillations during operation and controlled overflow of water during load stops or load rejections by means of an overflow controlled by a side spillway.	100%	X		Verify if there is a way to improve the operation of the desander.	The desander is not working satisfactorily.
Pressure piping	Convey the operating flow at the pressure required for the operation of the powerhouse turbogenerator units.	100%	X		Access for inspection required (Manhole)	It works satisfactorily.
Powerhouse	To protect the turbines, generators and regulation and control elements of the power plant.	100%		X		It works satisfactorily.
Turbines and generators	Transform the kinetic energy of a stream of water into mechanical energy and subsequently transform the mechanical energy into electrical energy.	100%	X		Modification of Unit 1, pressure release was performed on the rear part of the turbine shell to normalize the pressure that the water exerted on the bearings. (Factory defect)	Bearing wear caused the turbine to be used at only 60% of its capacity, wasting water resources during the rainy months and thus affecting production. Not yet received.
Lift substation	Raise the required voltage level before delivering power to the transmission grid.	100%		X		It works satisfactorily.

 

Table 9 shows the detail of the net energy expected to be produced according to the modeling of the hydrological study used as reference in the feasibility study and the net energy that the plant is actually producing and its respective ratio.

 

Table 6 

Net energy production (kWh) actual versus forecast

Month - Year	Production - Net energy (kWh)		Actual/ Planned Ratio	Degree of efficiency
	Scheduled	Real
Production from Dec -2020 to Aug -2021	10,088,657.00	7,107,263.53	0.	Inefficient
Production breakdown
December -2020	993,657.00	653,400.86	0.	Inefficient
January - 2021	791,000.00	805,133.87	1.	Very Efficient
February - 2021	515,000.00	491,089.04	0.95	Inefficient
March - 2021	437,000.00	387,696.57	0.	Inefficient
April - 2021	302,000.00	451,171.42	1.	Very Efficient
May - 2021	195,000.00	614,664.00	3.	Very Efficient
June - 2021	2,302,000.00	1,255,254.42	0.	Inefficient
July - 2021	2,498,000.00	1,006,072.83	0.	Inefficient
August - 2021	2,055,000.00	1,442,780.52	0.	Inefficient

Note: Prepared by the authors.

Table 10 seeks to confirm the existence of a general maintenance plan and verify whether it is adequate to extend the useful life of the dam components and thus ensure its operational sustainability. 

 

Table 7

Analysis of existing maintenance plans

Area/Component	is there a maintenance plan for this area or component?		If there is a maintenance plan, is it effective?			If there is NO maintenance plan, is work currently underway to develop a maintenance plan?		Mention the challenges faced by not having a maintenance plan
	YES	NO	YES	NO	Explain	YES	NO
Electromechanical Equipment (Turbines, generators, traveling crane, etc.)	X		X		If they have details of the routine inspections to be carried out and the periodicity of revision of the components.
Civil works of the hydroelectric dam (dam, conduction and pressure lines, load chamber, powerhouse)		X				X		Measurements are taken on a daily basis.  We are working under a corrective rather than preventive maintenance modality.

Table 11 below analyzes the quality of the transfer of the project to the operator by analyzing the products delivered and the training provided to the end users.  This allows verifying whether the operators were provided with the information required to become familiar with the final product and whether they were provided with adequate training to ensure that they have the knowledge required to correctly operate the hydroelectric plant to ensure that its operational sustainability is achieved.

 

Table 8

Project transfer analysis and training of operating personnel

Delivered products	YES	NO	Rate the products delivered at the time of transfer and/or training received			Additional comments or observations
			GOOD	REGULAR	MALO
was an inventory of equipment delivered?	X		X			An inventory of spare parts and installed equipment was provided.
were the warranties for the different equipment or components delivered and their completion dates?	X					The immediate supervisor was provided with it.
were as-built drawings, quality controls and tests provided during execution?	X		X			The operations manager does not have the hydrological studies to make energy production comparisons.
were the activities and responsibilities of each member of the team that would manage the hydroelectric plant provided?	X		X			The employment contract stipulates the obligations.
Qualify the transfer of the project to the end user.			X
Rate the training received at the time of the project transfer			X			Some of the personnel operating the plant were in charge of electromechanical assembly and had previous experience in similar projects; training was provided during commissioning.
how do you consider the follow-up and monitoring after the training was adequate?			X			24/7 advice is available from suppliers.
was an operating manual for the plant components provided? (Dam, pressure and conduction lines, loading chamber, powerhouse). Please rate this manual.	X			X		They were only provided with a general manual, the detailed manual is being prepared by the personnel currently operating the dam.
was an operating manual for the electromechanical equipment provided? (Turbines, generators, traveling crane, plant management software, etc.) Please rate the manual.	X		X			From the technical point of view it is adequate, it would be advisable that it be in Spanish with friendlier terms.

 

Risks

Table 12 shows the possible socio-environmental risks that could compromise the operation of the plant and that must be taken into consideration in order not to lose continuity of operation as a result of conflicts of this nature.

 

Table 9

Socio-environmental risk analysis 

Potential socio-environmental risks	Yes	No	Additional comments
was there a process of free, prior and informed consultation with the surrounding community prior to the approval of the project and submission for state approval?	X		There is documentation that supports the rapprochement with the population from the beginning of the project through open meetings.
have the company's commitments to the population been fulfilled, for example, aid projects of some kind, such as reforestation, irrigation systems, access road improvement, etc.	X		Little by little, the agreements are being fulfilled.
is there acceptance of the plant operation by the community surrounding the project?	X		In general, yes, regardless, there is always slight disagreement from the population because they seek to take advantage of the hydroelectric plant for personal benefits.
has the monitoring of water quality in the river in winter and summer season and conducting of aquatic fauna inventory of the river, by specialist in the subject, once a year suggested by the Environmental Measures Compliance Report (ICMA) ( 2018, p.52) been carried out?		X	He is not aware of any such action to be taken.
is river flow monitored downstream of the dam and downstream of the final discharge? (The guideline value should be that of the ecological flow of the river, ICMA, 2018, p.53)?	X		Ecological flow monitoring is performed daily.
is there an indigenous population that could be affected by the operation of the dam?	X		Yes, there have already been approaches with the Maya-Chortí ethnic group.  There are employees of this ethnic group.
do they have conflict resolution and preventive conflict management strategies?		X	It is convenient to define action plans in the event of demonstrations or conflictive approaches so as not to be surprised.
has the environment surrounding the hydroelectric plant been affected in an unforeseen way?		X
does the plant have an Occupational Health and Safety Plan?	X
Mention any additional risks that may currently be present.		Due to lack of knowledge of the operation of the hydroelectric plant by the population, conflicts may arise.   The operations manager suggests inviting the population to learn about the power generation process to validate that it is a clean and environmentally friendly way of generating electricity.

 

Discussion and conclusions

This section presents the interpretation of the results previously presented, mentioning the possible causes that led to these results and a comparison is made with the findings of publications on evaluations of related topics. The lessons learned are also listed, consisting of both the successes and failures identified. Finally, proposals are provided to solve the problems that caused the gaps.

Efficiency: This criterion was evaluated in four dimensions: product achievement, project execution time efficiency, project cost efficiency and overall efficiency.

With respect to the achievement of outputs, it can be seen that all the components planned to be built were 100% completed, reaching a level of execution of components equal to 1 according to the calculations shown in Table 1. However, despite the fact that all the components were completed, three of them showed considerable deviations in their expected dimensions, as shown in Table 2. The dam, the pipeline and the powerhouse were the components that suffered the aforementioned differences. These differences had a negative impact on the project, as they were the source of the project's cost and execution time lags.

When analyzing the time efficiency of the project (see Table 3), it was observed that the approved feasibility study considered that the expected execution time of the project would be 17 months; however, the project was executed in 37 months in total, therefore, an efficiency of 0.46 was observed, which qualifies it as Inefficient. Referring to the table in question, it can be seen that almost all the main activities of the project required more time than programmed to be executed.

For the calculation of cost efficiency, the prefeasibility study indicated that US$10,944,000 was required to execute the project; however, US$18,481,134.69 was required, resulting in a cost efficiency of 0.59, which is classified as inefficient. Table 4 shows that all components suffered cost overruns and specifies the formula used to obtain the cost efficiency. It can be seen that the components that underwent changes were those that had the greatest impact on the final cost of the project.

The overall efficiency was calculated with a value of 0.27, which is catalogued by Montero et al. (2013) as inefficient, concluding that the overall efficiency was not adequate.

The implementation problems identified through the analysis of the documentation submitted and the interviews conducted with key personnel are summarized below:

• Vague feasibility study. The budget presented in the feasibility study, on which the decision to execute the project was based, was practically an order of magnitude estimate, as it appears to have been calculated without precise engineering data. Additionally, it was not based on a reliable geological and geotechnical study.
• Inadequate component designs for actual conditions. The fact of not having adequate technical information (information that is used as a basis for the design of the structures), led to assume different scenarios to those faced in the field, causing considerable changes in the first components executed.
• Poor planning. There was no detailed implementation program from the outset. During implementation, many unplanned activities occurred that prolonged the duration of the project due to designs based on incorrect studies and assumptions.
• Monitoring and control of passive work. At the beginning of the project, despite being aware that the project was experiencing considerable delays and changes, no decisive or effective action plans were made to help reduce the delays. It can be seen that reactive, not proactive, management was exercised.
• Absence of a technical file or definitive study at the beginning of the project. The project was initiated without having: technical specifications, accurate construction execution drawings, accurate quantity quantifications, detailed base budget, detailed execution schedule, and accurate geological and geotechnical studies.
• Inadequate risk management. Each risk entails a cost, the risk study identifies those that are most likely to materialize and a risk response plan is established to reduce the chances of their occurrence. This is evidenced by the inadequate identification of risks when undertaking a project of this magnitude, with general information and with personnel with insufficient skills to plan and lead it.
• Inadequate cost management. The budget lacked: adequate determination of the activities necessary to carry out the project, proper definition of unit prices for the activities, accurate quantification of the quantities of works in key activities and, last but not least, inadequate monitoring and control of costs, since it is not apparent that in the first year of project execution adequate corrective actions were taken when budget slippages occurred.
• Poor procurement management. First of all, not having an accurate schedule directly affects procurement planning, as errors are made in knowing what and when to procure. Secondly, the lack of a supplier selection process, both for services and supplies, does not necessarily allow for the selection of the most favorable offer in the market, and limits the customer's price negotiating power and does not allow him to compare services with companies that perform similar activities.
• Insufficient competence of key personnel involved in planning and design. The experience that the personnel may possess to manage the work is a very important point, since a competent person knows the procedures to follow to obtain adequate results and increases the probabilities of meeting the objectives. This does not mean that new personnel should not be hired, but rather that it is imperative that the people who are going to lead a project have the minimum competencies required to ensure compliance with the pre-established goals.
• Inadequate initial quality management. The EU did not have an adequate quality plan until the supervising company started its quality controls parallel to those of the contractors to validate results.
• Ineffective communications management. Personnel interviewed indicated that at the beginning there was no efficient coordination and communication to achieve synergy between the different contractors. Effective communication is a key factor for the proper performance of the work.

Ruíz's (2005) research, entitled Determining Project Success. The case study indicates that among the most common problems that arise in projects and that affect the achievement of their objectives are: poor planning, poor communication, improper monitoring of progress, incompetent project manager, among others.

Cruz (2003), in his research entitled Metodología a seguir para la planificación de un proyecto hidroeléctrico antes de su implementación en Guatemala (Methodology to follow for the planning of a hydroelectric project before its implementation in Guatemala ) indicates that:

The scope and details of engineering studies, geological explorations, designs and cost assessments in connection with feasibility investigations would have to be sufficient to ensure the reliability of the project plan and to guarantee that the project can be carried out at the anticipated cost. (p. 90)

ILPES (1997) indicates that the feasibility study should provide as precisely as possible the benefits and costs of the project and deepen the analysis of those variables that affect the project.

Ríos and Medina (2020), that it is necessary to adequately address risk management, since the materialization of one or more unforeseen risks can cause considerable impacts on the other areas of the project, usually the most affected being cost management, work scheduling, resources and procurement.

Sustainability: As mentioned, sustainability can be assessed under the probability that the components are likely to be operated and used and achieve their ultimate purpose during their useful life. Under this statement, we proceeded to validate the existence or absence of the same in the project under investigation. The valuation was made by analyzing the financial and operational sustainability and the risks under the present socio-environmental context.

With respect to financial sustainability, Table 6 shows that the maintenance costs and expenses projected in the feasibility study for the period during which the plant has operated are higher than those actually used once it is in operation. This indicates that the original projection was adequate and does not jeopardize the sustainability of the project. It is only important to mention that, as expected, financial expenses exceeded projected financial expenses due to the fact that loans for the execution of works were higher than estimated. This was offset by the fact that the expected operating, maintenance and administrative costs and expenses have turned out to be much lower than expected, and these savings have served as a "financial cushion". Another parameter evaluated in terms of financial sustainability was the coverage of costs and expenses for the production that has been invoiced, as shown in Table 7. This parameter shows a small gap between both elements, which indicates that the company may have difficulties to cover its fixed and financial costs if production declines, which places it in a latent financial risk. Currently, there is a divergence between the production invoiced and the payments received from the client, which makes the project financially unsustainable.

In assessing operational sustainability, the ability of the components to provide the intended service was addressed in the first instance. It was found that all the components are functioning, but that there are some of them that are not fully providing the service they should and preventing the maximum use of the available water resources. The above is reflected in Table 8. In addition, the adequate design of the hydroelectric plant is questionable, as there are deficiencies in the overall operation of the plant. Another factor analyzed in the context of operational sustainability was the comparison between the planned operation and the one actually produced. In order to evaluate this parameter, net energy production was used as a reference, the values obtained are shown in Table 9, in this table. an efficiency equal to 0.70, which results in a degree of efficiency classified as inefficient according to Montero et al. (2013), as only 70% of what was planned has been produced. This production percentage was affected both by deficiencies in the design of the plant, which as mentioned above, do not allow the best use of available resources, and by factory problems in one of the installed turbines. Another point considered relevant to assess in order to confirm the operational sustainability of the project was the analysis of the existing maintenance plans. Table 10 shows that the maintenance plan provided for the installed electromechanical equipment, such as turbines, generators and traveling cranes, is in place and considered effective. However, there is no maintenance plan for civil works and the operating personnel are currently working on developing one, and for now corrective rather than preventive maintenance is being carried out in certain situations. To conclude the evaluation of operational sustainability, the quality of the process of transferring the project to the end user and the training provided to operational personnel were analyzed. Under this criterion, according to Table 11, there is a general good transfer and training of personnel, with only the operating manual for non-electromechanical components being rated as regular. Based on the above, it can be concluded that the PH is currently operationally sustainable; however, it is necessary to adjust some of its components and formally prepare the civil works maintenance manual to ensure its long-term operability.

The last aspect evaluated in the sustainability criterion was that of risks related to socio-environmental conflicts. This aspect is of utmost importance because its mismanagement can lead to the temporary or permanent closure of a project of this nature. As shown in Table 12 in the social part, it can be generally seen that there has been a rapprochement with the population and that they have shown acceptance of the project, however, there are always specific situations of disagreement. It is important that the company complies with its commitments to the population to avoid confrontations. In the environmental aspect, the company is complying with the ecological flow quota and other tasks required by the environmental regulatory entity, and it is only necessary for the company to validate the need to monitor water quality and the inventory of aquatic fauna in the river, which seems sensible to do to confirm that no negative impact is being made on the tributary. Based on the above, it can be concluded that there are currently no socio-environmental risks that could compromise the operation of the plant.

The sustainability issues identified through the analysis of the documentation submitted and the interviews conducted with key personnel are listed below:

• Billing mismatch. Invoicing to the customer is differentiated from production. It is advisable to minimize the gap between the reading of the metering instruments and the issuance of the invoice to reduce the financial impact. Attempt to comply with the seventeenth registration, billing and payment clause of the contract.
• Lack of a civil works maintenance manual. In order to extend the useful life of the project and ensure the recovery of the investment, it is necessary to have a manual that indicates the actions to be carried out in an orderly, planned and permanent manner and also indicates the frequency with which these tasks must be performed.
• Deficiencies in the overall hydroelectric plant design. In general, it can be seen that the plant has deficiencies that do not allow the full use of the water resources of the area and it is necessary to make adjustments to the components built to maximize the use of the flow identified in the area.
• In the socio-environmental aspect, there is a lack of conflict resolution strategies.

With respect to social conflicts, Gamboa and Cueto (2012), in their Policy Paper entitled "Hydroelectric plants and social conflicts: recommendations for better environmental management", identify the following direct causes for the generation of social conflicts: the failure to communicate prior and timely the realization of the hydroelectric project, low quality of the Environmental Impact Studies, distrust between the local population and the operator/concessionaire of the hydroelectric project, absence of conflict resolution strategies and preventive conflict management, lack of prior, free and informed consultation with the affected indigenous peoples, etc.

Lessons learned: Based on the above findings, the following is a list of the most important lessons learned from the project under study:

1. Improve the quality of pre-investment studies. It is imperative that decisions be made based on reliable documents, which is why it is necessary to allocate a greater amount of economic resources and sufficient time to carry out the necessary studies that will reliably demonstrate the advantages and disadvantages of the project being evaluated.
2. Creation of a technical dossier. Once a project has been approved by means of an accurate feasibility study, it is imperative to allocate the time required for the preparation of a complete technical file that includes the following aspects: descriptive report, final studies and designs, final drawings, technical specifications that will govern the project, quality specifications, quantification of work quantities, preparation of a base budget with a precision that is within a range of ±5 to ±10 of the expected amount, cost sheets or analysis of unit prices or direct costs, analysis of general expenses and profit or indirect costs, a base chronogram. It is highly recommended that the personnel performing the above activities be part of the staff that will execute the project since they will have full knowledge of the project from its conception.
3. Seek advice from companies or personnel with proven experience when entering an unknown market. Experience is an added value that must be taken into consideration when undertaking a project, especially when it is the first time that a project is to be developed outside the market in which the company has previously worked. Usually, experienced companies already have and know the procedures, formats and guidelines necessary to manage, as far as possible, situations that may be out of control.
4. Perform effective project management. Effective project management allows the development of a project management plan that effectively integrates the following aspects:
• Scope, establish all the work that needs to be done to successfully complete the project and define a scope baseline to validate that the defined deliverables are being met;
• Time, establish a baseline of execution time to monitor and control during the execution of the work;
• Costs, establish a cost baseline to monitor and control during the execution of the work;
• Quality, establish the quality standards required for the project to be executed, perform quality assurance and quality control;
• Human resources, identify the personnel needed to carry out a project and when such personnel are required. In addition, define the skills and competencies that such personnel must possess. Subsequently, acquire the required personnel, improve their skills, evaluate, provide feedback and carry out conflict resolution.
• Communication, define how communications with the different stakeholders will be carried out and validate that the communication is effective;
• Risks, a preliminary study of the possible risks that could occur, defining their probability of occurrence, establishing the impact they would generate and defining whether the risk is assumed, eliminated or transferred;
• Procurement, establishment of processes for the acquisition of materials and contracting of services. These procedures must define the specifications of the materials, the minimum quality required, evaluation of the supplier in terms of compliance with contracts and finally define the selection criteria, whether based on quality or price;
• Project stakeholders, it is necessary to identify stakeholders, whether they are sponsors, users, affected people, customers, etc., in order to establish strategies for effective stakeholder involvement.
5. Creation of a database of contractors and professional service providers It is advisable to initiate the creation of a database of contractors, consultants and companies that provide professional services for hydrological, geological, geotechnical, topographical, structural design, etc. studies. In addition, record their experience, references to validate their performance in previous works, contacts and any other information considered relevant.
6. Development of a digital repository. It is important and convenient for the company to have a digital platform or shared digital folders. These folders provide access to various technical information resources, lessons learned or relevant information to be taken into account by key personnel for the performance of their functions, or for the same personnel to gradually add the information they generate, so that it is available to other colleagues and forms part of the company's intangible capital. Access restrictions can be made as deemed appropriate.
7. Validation of hydrological modeling The work was approved through the use of hydrologic modeling due to the lack of site-specific data. Given that the hydroelectric plant is in operation, it is considered a good practice to compare the actual observed versus modeled data to, firstly, certify the degree of reliability of the base modeling and simultaneously confirm whether the productions foreseen in the feasibility study will be achieved and, secondly, adequately study the area to assess whether it offers new business opportunities.
8. Bechmarking In order to ensure the sustainability and useful life of the project, an evaluation and analysis of the processes (e.g., operation and maintenance), products and/or services of companies or firms with extensive experience in hydroelectric power generation and that demonstrate "best practices" is proposed. The purpose is to compare the processes of the reference or benchmark company with the processes currently being executed and to confirm if there are opportunities for improvement and implement the changes considered appropriate. It is suggested to pay special attention to operation and maintenance practices, personnel training and socio-environmental risks. The aim is to learn from the experience of others in order to improve one's own performance.
9. Preparation of a socio-environmental risk analysisA detailed analysis of possible risks, both favorable and adverse, is important in order to establish action plans to prevent negative risks from materializing and to enhance those with a positive impact.

The general objective of this research is to analyze the causes that generated considerable deviations in costs, project execution times and the uncertainty of obtaining the expected energy production once the hydroelectric dam is put into operation. Based on the above, it can be concluded that:

• It is considered that the research has achieved its main purpose by identifying through the use of the evaluation at completion, which is the first of the evaluations stipulated within the ex post methodology, that: the inaccuracy of the technical studies used for the project design, the lack of a definitive study or technical file before starting the project, the lack of an effective project management plan, the planning and initial management of the project by personnel with limited experience, have been the origin of the main causes of the time and cost delays that have arisen.
• It has been possible to confirm that the dam has weaknesses in its design, which do not allow it to make full use of the available water resources and obtain the expected energy production. It is necessary to make adjustments to certain components to maximize their use.
• An ex-post follow-up evaluation is required within one year to verify the sustainability of the project. This sustainability will be evaluated based on operation and maintenance.
• It is important to insist on the need to document the lessons learned and proceed immediately to their dissemination for application to the project under study and to future projects to be executed.

To close this section, some recommendations for the future are proposed.

• Continue with the next stage of the Ex Post evaluation, i.e. ex post monitoring, to validate sustainability oriented to operation and maintenance within one year.
• Review of the overall design of the dam, by personnel or companies with the appropriate skills, to identify the necessary adjustments to be made in order to get the most out of it.
• Once the design review has been carried out, invest in an adequate planning for the execution of the design, establishing from the beginning baselines for comparison of scope, budget and work scheduling.
• Evaluate the operation of the sand trap built in the loading chamber since it is not working effectively.
• Continue to compile lessons learned and disseminate them to company personnel so that they can be taken into account in the planning and execution of future projects.
• Identify, as a team, the risks that threaten the project, which may have a positive or negative impact, in order to establish action plans to maximize or minimize them or eliminate them respectively.
• To reduce the financial risk of the project during its operation stage, there must be adequate coordination between the operating and administrative teams to ensure that invoicing and collections are not delayed, since there is no slack between the amount invoiced and the operating, maintenance, administrative and financial costs and expenses.
• To ensure that there are no interruptions in operations for socio-environmental reasons, bear in mind the Compensation Measures contracted with the neighboring community set out in the Qualitative Environmental Diagnosis (DAC) 2007 and the duties acquired in the Environmental Measures Compliance Report (ICMA), 2018, and validate the need to carry out the environmental control measures number 53 and 55 of said report.
• Analyze the feasibility of performing preventive maintenance to extend the useful life of the metallic pipe by using coatings that protect the pipe both externally and internally from rust and corrosion.

 

---

References

Atkinson, R. (1999). Project management: cost, time and quality, two best guesses and a phenomenon, it's time to accept other success criteria. International Journal of Project Management, 17(6), 337-342 https://www.sciencedirect.com/science/article/abs/pii/S0263786398000696 

Aziz, E. E. (2015). Project closing: the small process group with big impact.  Paper presented at PMI® Global Congress 2015-EMEA, London, England. Project Management Institute.

Condori, V. (2016). Ex post evaluation of the project "Improving the quality of life of the family from the kitchen in the municipality of La Asunta in the department of La Paz". [Master's thesis, Universidad de San Andrés] https://repositorio.umsa.bo/handle/123456789/6564

Cruz, J. (2003). Methodology to follow for the planning of a hydroelectric project before its implementation [Degree thesis, Universidad de San Carlos de Guatemala] http://biblioteca.usac.edu.gt/tesis/08/08_2319_C.pdf

Endesa Fundación (n.d.). Hydroelectric power plant. https://www.fundacionendesa.org/es/educacion/endesa-educa/recursos/centrales-renovables/central-hidroelectrica

Gamboa, C. and Cueto, V., (2012). Hydroelectric plants and social conflicts: Recommendations for better environmental management. https://library.fes.de/pdf-files/bueros/peru/09004.pdf

International Project Management Association. (2016). Individual competence baseline for project, programme and portfolio management. IPMA. 

Latin American and Caribbean Institute for Economic and Social Planning. (1993). Methodological proposal for the ex post evaluation and completion report of investment projects. Santiago, Chile. LC/IP/L.84

Kerzner, H. (2009). Project management: a systems approach to planning, scheduling and controlling. Wiley & Sons.

Klastorin T. (2005). Project management. Alfaomega.

Martinez, A. (September, 2015). what is a hydroelectric dam? Prezi. https://prezi.com/lsumjcxmdlni/que-es-una-represa-hidroelectrica/

Medianero, D. (2010). Ex post evaluation methodology. Critical Thinking, 13,71-90 https://doi.org/10.15381/pc.v13i0.9001

Ministry of Economy and Finance, Japan International Cooperation Agency - JICA. (2012). General Guidelines for Ex Post Evaluation of Public Investment Projects.JICA and MEF

Montero, J. Díaz, C., Guevara, F., Cepeda, A. and Barrera, J. (2013). Model for measuring real production efficiency and integrated information management in a mill. Oil Palm Research Center-Cenipalma.

Motoa, G. (2015). Measuring success in projects, a review of the literature. Ingenium, 9(25), 11-25 https://www.researchgate.net/publication/308084179_Medicion_del_exito_en_los_proyectos_una_revision_de_la_literatura 

Pinto, J. & Mantel, S. (1990). The causes of project failure. IEEE Transactions on Engineering Management, 37(4), 269-276.

Pinto, J.K. & Slevin, D.P. (1988). Critical success factors across the project life cycle. Project Management Journal, 19(3), 67-75. 

Pinto, J.K. & Slevin, D.P. (1988). Project success: Definitions and measurement techniques. Project Management Journal, 19(1), 67-72. 

Real Academia de la Lengua Española (n.d.). Plan. https://dle.rae.es/plan

Ríos, L. & Medina, C. (2020). Ex post evaluation of public-private infrastructure projects: Case applied to two projects at ABC Company. [Master's thesis, Universidad EAFIT].  https://repository.eafit.edu.co/handle/10784/17545

Ríos, L and Medina, O. (2020). Ex Post Evaluation of public-private infrastructure projects: Case applied to two projects at ABC Company [Master's thesis, Universidad EAFIT].       https://repository.eafit.edu.co/handle/10784/17545

Ruíz, M. (2005). Determination of project success. Practical case study. [Degree thesis, Universidad Politécnica de Valencia] https://riunet.upv.es/handle/10251/54584?show=full 

Saenz, A. (2012). The success of project management, a new approach between the traditional and the dynamic [Doctoral thesis, ESADE] https://www.tesisenred.net/handle/10803/117483#page=5

Shenar, A., Levy, O. & Dvir, D. (1997). Mapping the dimensions of project success. Project Management Journal, 28(2), 5-13.  https://reinventingprojectmanagement.com/material/other/7. Mapping dimensions of projects success PMJ 1997.pdf

Shenhar, A. & Dvir, D. (2007). Project management research, the challenge and opportunity. Project Management Journal, 38(2), 93-99. 

Shenhar, A., Levy, O. & Dvir, D. (1997). Mapping the dimensions of project success. Project Management Journal, 28(2), 5-13.

Thomas, G. & Fernández, W. (2008). Success in IT projects: A matter of definition? International Journal of Project Management, 26(7), 733-742.

Turner, J.R. (2009). The handbook of project-based management: leading strategic change in organizations. McGraw-Hill.

Villacís, A. (2008). Establishment of a methodology for ex post evaluation of the financial impact on agroindustrial companies benefiting from the MCA-Honduras project. [Degree work, Escuela Agrícola Panamericana] https://bdigital.zamorano.edu/bitstream/11036/136/1/AGN-2008-T029.pdf

Yaguno, K. (2017). Ex post evaluation at the completion level of the San José technified irrigation project, Azángaro province - Puno [Degree thesis, Universidad Nacional del Altiplano, Puno, Peru] http://repositorio.unap.edu.pe/bitstream/handle/UNAP/5816/Yaguno_Yucra_Kevin_Gregorio.pdf?sequence=1&isAllowed=y