{'fraction_non_alphanumeric': 0.03142234192638194, 'fraction_numerical': 0.01778461801547604, 'mean_word_length': 4.7039746403316265, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Abnormally Low Tenders in Non-pricing Criteria: the Need for Control

As public procurement accounts for approximately 10 to 15% of gross domestic product (GDP) in developed countries, tendering mechanisms should be clearly defined in order to avoid any actions that could endanger the basic principle that all bidders should be on equal terms. An Abnormally Low Tender (ALT) is defined as an offer too low to provide a normal level of profit and that cannot be explained on the basis of construction methods, the technical solution chosen, the originality of the work, or the favorable conditions of the tenderer. Public bodies are well aware of the risk of accepting an offer that cannot be carried out and despite the difficulty of detection recommendations for their prevention usually focus on the price criterion. Most tenders are awarded to the economically most advantageous tender (EMAT), which is assessed by various criteria (including price), though other criteria often have equal or greater weight in the final decision. The method used in this research study is divided into two main phases. First, the score of the bidders is obtained for criteria evaluated by formulae other than price, based on the contract terms of three case studies, after which new scores for these award criteria are obtained from ALT formulae, then, the results of both scoring methods are analyzed. This paper defends the need to control abnormally low tenders by means of award criteria evaluated by formulae other than those of price.


Introduction
Public procurement in first world countries makes up between 10 to 15% of their gross domestic product (GDP) [1][2]). In some countries these values are even greater, as for example in Hong Kong [3][4][5].
The award of a contract by an administration depends on a number of endpoints. In the US public procurement has traditionally been strictly regulated and procurement contracts are usually awarded to the lowest qualified bidder, although other methods have recently attracted interest [6][7][8].
In the European Union, Directive 2004/18/EC [9] regulates public procurement and describes the tendering criteria (such as price, quality technical merit, aesthetics and functional characteristics, environmental characteristics, running costs, profitability, customer service, technical assistance, delivery date, execution time, etc.) that allow the contracting authority to select the economically most advantageous tender (EMAT).
The EMAT, based on several criteria, is traditionally called the procedure contest, while the bid which is based solely on a single criterion, which must inevitably be the price, is the procedure traditionally known as an auction. These rules are common practice in most of the public procurement sector and are also used in many procedures in the private sector [10][11][12].
The selection of the criteria and their scoring formulae or ranges is determined by the contracting authority. They are published in the contractual documents and must be in keeping with the objective of the project tendered and obviously suited to the technical capabilities of the contracting authority.
The evaluation criteria used can be divided into two groups: the criteria evaluated by formulae and those evaluated by values judgments. For the former, various predetermined formulae can be employed, including aspects such as price, delivery time, the labor necessary for the project, etc. However, the scores for the criteria assessed by value judgments will always contain some subjective bias by the individual who performs the evaluation.
Once the bids have been submitted, the contracting authority considers the bidders' proposals. The bidders may occasionally deviate from rational behaviour and make anomalous offers (also known as outliers), at prices much lower or higher than the other bidders.
Impossibly high offers, which do not expect to win the auction, are known courtesy bids [13][14] and may be made for any number of reasons; the buyer may have little interest in the contract, or lack the resources and skills to properly submit a suitable bid, or may simply make an offer to ensure being considered in future procedures [15].
Impossibly low offers (known as Abnormally Low Tenders (ALT) [16] are those considered as disproportionate if too low to provide a normal level of profit and cannot be explained on the basis of construction methods, the technical solution chosen, the originality of the work or the favorable conditions of the bidder. There are many reasons to explain this behavior: the bidder may be in desperate need of the contract, even though it may turn into a financial loss [17]. He may lack experience in auctions or may have miscalculated the costs and the return needed to recoup its funding [18][19].
There have also been cases in which a low bid was deliberately submitted to oust a competitor or to protect a company's position in the market or to gain access to a new market [20][21][22].
In an industry as important as construction, with poor profitability and insufficient company resources, ALTs have consequences for national economies and international competitiveness [16]. The final cost of the work is in many cases above the price at which the project was awarded [23][24][25][26][27][28].
Public administrations are well aware of the risk of accepting an offer that cannot be carried out and despite the difficulty involved [29][30][31], the European Union has proposed recommendations for their detection [16], including the use of an EMAT procedure instead of auctions.
Contracting authorities, previously to score the offers, must make an analysis as to whether an offer presents abnormal or disproportionate values has been carried out. In the European Union this step is regulated according to the article 55 of the Directive 2004/18/CE and the subsequent laws o rules of the member countries, but this is not an isolated fact from the European Union as many authors have pointed in theirs works [10,[32][33]. If the contracting authority establishes that an offer can be considered as disproportionate or abnormal, the bidder must provide an appropriate justification of the offer, based on the technical, organizational and economic terms to ensure a proper execution of the contract. If this justification is considered right, the offer will be considered likewise the rest of offers and if the justification is not considered right, the offer will be rejected.
Not many studies exist in the literature on mathematical formulae or tools to determine which bids are 'abnormal' or 'risky' [9,[34][35][36][37]. Some methods detect ALTs by assessing the deviation of the offer from the average bid [38][39], while others use graphical methods [40].
Other authors have developed mechanisms based on the price that will be paid when the contractor completes the project [41] or on contracts that include a penalty for the contractors if the project does not meet the requirements [42], although these may be somewhat difficult to apply. Other works have developed formulae to assess at same time the abnormally and the weighting of the offers [43][44].
The concept of Abnormally Low Tenders (ALT) (also known as Abnormally Low Bids (ALB) has always been applied to the price criterion; consideration should also be given to the other criteria evaluated by formulae. In many cases, these criteria have as much weight as the price in the final decision. The objective of this paper is to show the need to control ALTs by criteria evaluated by formulae other than price, including the labor involved in the contract, economic improvements and the guarantee period. This paper is divided in four sections. The first section is the introduction on the topic. In section two the method is developed. In section three, the results of the control of ALTs are presented and finally in section four shows the conclusions from the article.

Method
The method used in this research study is divided into two main phases. First, the score of the bidders is obtained for criteria evaluated by formulae other than price, based on the contract terms of three case studies, after which new scores for these award criteria are obtained from ALT formulae. The results of both scoring methods are then analyzed.
The formulae used to assess ALT in the price criterion can be used to measure the abnormally low offers of other formula-assessed criteria. These formulae should take into account the importance of the award criteria, i.e. whether they are premium reduction criteria, as for example execution times, or other criteria involving financial benefits or the guarantee period of the finished project.
In this study, the selection of the criteria for outlying tenders is based on the project's material execution budget (MEB) and the number of bidders (N) [45]. We set two tests for ALT, the first based on the average of the bids submitted (B m ), which was used when the number of bidders (N) was less than or equal to 10. The second test was based on the reference base rate (B R ), consisting of the arithmetic average and standard deviation of the bids submitted, which was used when the number of bidders exceeded 10.
Both methods behave very similarly for low numbers of bidders. The first test was chosen in cases with fewer than 10 bidders, since it was quicker and easier to apply. For a larger number of bidders, calculating the reference base rate made a more precise evaluation possible.
The value of 'X' was selected on the basis of the margins as determined by the MEB (Table 1). (1)  Depending on the differential "X" selected, the ALT will be determined according to the following expressions. If the criterion rewards reductions: If the criterion rewards increments: ( )  All of the bids will be collated. If the criterion rewards reductions, bids that are above the limit are considered acceptable and if they are below, they are considered abnormally low tenders. If the criterion rewards increments, bids that are below the limit are considered acceptable and if they are above are considered disproportionate or abnormal. Where: B i is Bid i B m is the Mean Bid n is the Number of bidders L is the limit X is the percentage according to Table 1 B. For N >10:  The standard deviation of the bids is calculated according to:  To calculate the reference base rate (BR), those bids n' will be taken into account that meet the following condition:  The reference base rate is calculated taking into account the bids that have met the previous condition using the following formula:  Depending on the differential "X" selected, the ALT will be determined according to the following expressions: If the criterion rewards reductions: If the criterion rewards increments: ( )  All of the bids will be collated. If the criterion rewards reductions, bids that are above the limit are considerable acceptable and if they are below, they are considered abnormally low tenders. If the criterion rewards increments, bids that are below the limit are considered acceptable and if they are above are considered disproportionate or abnormal. Where: B i is Bid i B m is the Mean Bid B R is the base rate B h is the bid that satisfies condition (5) n is the Number of bidders n ' is the number of bidders that satisfy condition (5) L is the price limit σ is the Standard Deviation

Case Studies
This section describes three case studies to which the proposed methodology was applied. The results were analyzed both with and without the application of ALT criteria.

Case 1: Construction of 300 Burial Niches in the Municipal Cemetery of Oliva (Valencia)
This project was put out to tender by the Council of Oliva (Valencia) in February 2013 with a budget of €212,000 + VAT [46]. The execution time was three months and the EMAT identification criteria were all assessable by mathematical formulae.
The award criteria are:  Criterion A: Bid expressed in Euros as a percentage reduction (drop) of the tender price. This criterion is valued between 0.00 and 6.00 points. The maximum score (6.00 points) is awarded to the lowest bid and the rest of the tenders are scored according to the following formula: (one year contract is required) and must be expressed in additional months. The score ranges from 0 to 2 points. The maximum score (2.00) will be awarded to offers that propose a further guarantee period, and the rest of the tenders are scored according to the following formula: Where: S B is the criterion B score (expressed in points) G i is the extension of the guarantee period offered by bidder i (expressed in months) G máx is the maximum extension of the guarantee period (expressed in months)  Criterion C: Improvements related to the quality of the project, which must be described and quantified in Euros. The score will be between 0 and 2 points and the maximum score (2.00) will be awarded to the offer that submits the highest financial improvement, the rest of tenders being weighted according to the following formula: Where: S C is the score of criterion C (expressed in points) I i are the improvements offered by bidder i (expressed in Euros) I max are the maximum improvements (expressed in Euros) Table 2 summarizes the offers and their scores for each award criterion.
In this case, besides the price, there are two award criteria evaluated by formulae: improvements to the project and extension of the guarantee period. Bidder E 11 gets the highest score in both criteria, with improvements of €135,408.11 and extending the guarantee period by 384 months (32 years). Improvements to the project account for 63.91 % of the tender price and 70.43 % with respect to the offer of bidder E 11 . The extension of the guarantee period is 3200 % higher than the period required in the project. In this procedure the financial criterion (A) loses weight as award criteria in the overall decision, as the maximum score is limited to a percentage of the tender price and all or almost all bidders obtain the highest score [47][48][49][50].
When the procedure is analyzed with the proposed methodology, due to the amount of the tender price the differential X is equal to 5%; and as the number of bidders is equal to 40, the first of the conditions described in the methodology is used, with the results shown in Table 3.
The ALT scores mean 0 points for all criteria in this study. After applying the ALT criteria, companies E 7 and E 33 (72 months) get the maximum score (2 points) for criterion B (extension of guarantee period). The maximum score (2 points) for criterion C (improvements) is obtained by E 38 with improvements worth €29,150.18. E 38 gets the maximum score for all the criteria, while it was placed at Nº 23 before the application of the disproportionate price criteria. This bidder proposes to reduce the tender price by 25%, raise the guarantee period to sixty months and include improvements worth €29,150.18. As compared to E 11 this implies a reduction of the guarantee period of 84.38% (324 months) and a reduction of 78.47% in improvements (€106,257.93).

Case Study 2: Replacing Sidewalks
This project was put out to tender by the Council of Valencia in February 2009 with a budget of €2,458,279.22 + VAT [51]. The execution time was eight months and the bidder was selected by contest.
The award criteria are:  Criterion A: Volume of Manpower. The score will be between 0 and 40 points. Thiscriterion takes into account the labor needed to carry out the project as the product of the number of workers employed and the contract period.
The indicators used to calculate the volume of labor to be used in the project are: Where: S i A is the criterion A score (volume of manpower) M i is the volume of manpower of bidder i according to Expression (10) M max is the maximum volume of manpower  Criterion B: Improvements. The improvements must be related to the objective of the project and must not involve any cost to the city council. This criterion will have a maximum score of 25 points. Improvements are considered to be:  Work not covered by the project, justified by unit prices or market prices.  Environmental conditions, such as better accessibility for residents or improved conditions for traffic.  Criterion C: Execution Time. This is defined in months (eight for this project) and will have a maximum of 20 points. The biggest reduction in execution time will earn the maximum score and the other bids in proportion according to the following expression: Where: S i C is the criterion C score ET i is the execution time proposed by bidder i (in months) ET min is the minimum execution time proposed by bidder i (in months) The bids will be considered ALT if they offer to reduce execution time by more than two months. The maximum score is 10 points. The bidder will provide a working method adapted to the project, which will consider the proposed work plan, a detailed description of the activities to be performed, a work schedule and resources.
 Criterion E: Price. The largest reduction will receive a maximum of 5 points according to the following formula: Where: S i E is the score of the criterion price D i is the drop of the bidder i (expressed in %) D max is the maximum drop (expressed in %) Drop is defined as the discount or bid reduction on the tender price of a contract (P t ) submitted by a given contractor i for a particular capped tender. It is mathematically expressed as: In this procedure, apart from price, there are two criteria evaluated by formulae: reduced execution time and the volume of manpower. The former has an ALT criterion that sets the maximum reduction to two months. As all or almost all bidders offer the maximum reduction to get the maximum score, then the weight of this criterion disappears in the overall score. The bidders' scores for Volume of Manpower offers are given in Table 4. The bidder with the highest score in criterion A is Company E 1, who offered the highest volume of manpower than the other bidders and received the lowest score for this criterion. Bidder E 1 proposes using 47 existing workers and hiring 838 new employees, representing a total of 885 employees for 189 days (six months). Assuming a gross cost (salary + Social Security + Other Concepts) of €1000/month per worker, the cost of direct labor only would amount to €5,310,000, i.e. 2.57 times the budget execution material for the project. If the procedure is analyzed by the proposed methodology, since the amount of the tender price differential X is equal to 10% and as the number of bidders is equal to 26, the second method is applied. The results are shown in Table 5.
The ALT scores mean 0 points for all criteria in this study. After applying the ALT criteria, for criterion A (volume of manpower) the maximum score (40 points) is obtained by E 12, which hires 25 new workers and employs 70 existing workers. If the criterion of the proposed methodology is used, these values would reach more than €570,000, which is 27.59% of the project's MEB.

Case 3: Construction of a Sports Center
This project was put out to tender by the Sagunto Town Council (Valencia) in July 2010 with a budget of €2,786,034.48 + VAT [52]. The execution time was fourteen months and the procedure was conducted by contest.
The award criteria are:  Criterion A: Offers with improvements to the project. This criterion is valued between 0 and 40 points, 20 points for the technical quality and 20 points for the financial value of the proposed improvements.  Criterion B: Reduction of execution time. This criterion is valued from 0 to 25 points. The maximum reduction is set at 8 weeks, and gets the highest score. The remaining reductions are assessed in proportion.  Criterion C: Financial Proposal. This criterion is valued between 0 and 15 points and assesses reductions in the tender price. The maximum score is assigned to the lowest bidder and 0 points are given to the tender price, assigning scores to other offers in proportion.  Criterion D: Extension of the guarantee period. This is valued between 0 and 10 points. The maximum score is assigned to a two-year guarantee and zero points to the standard guarantee period (one year). The remaining offers are scored proportionally.  Criterion E: Study and analysis of the project. This is valued between 0 and 10 points and evaluates the bidder's knowledge of the project.
In this case, besides price, three other criteria are assessed by formulae: reduction of execution time, project improvements and extension of the guarantee period. Reduction of execution time gives the highest score to a reduction of two months and extending the guarantee gives the highest score to a two-year extension. This assumes that all bidders offer the maximum reduction of execution time and the maximum guarantee period, obtaining the highest score in both criteria, so that their weighting vanishes in the overall score.
The offers and scores of all bidders for Criterion A (Improvements) are given in Table 6. Company E 2 , who makes an offer of €568,400.77, i.e. 20.40 % over the tender price, is the bidder with the highest score in Criterion A. These improvements, at the contractor's expense, put the feasibility of implementing the project in doubt. In fact E 5 , the winning bidder (taking into account all the criteria), abandoned the project, the second, E 4 , began work but also abandoned during the earthworks phase and the third company (E 2 ) has requested changes to the contract or a higher budget. Work on the project has been suspended since July 2012.
If the unrealistic price offers are analyzed, the differential X is equal to 10%; and as the number of bidders is equal to 5 the first of the conditions described in the methodology is used, with the following results (Table 7). The ALT score means 0 points for all criteria in this study. Applying the criterion of disproportionate prices, bidder E 4 obtains the maximum scores (20 points). This offer represents a reduction in improvements of €193,077.55, i.e. 33.97% less than E 2 's offer

Conclusions
Using ALT formulae to set the lower or upper limit of an award criterion (depending on the direction of the criterion) assumes that the majority (or all) bidders are located in this range, eliminating the importance of the price criterion in the procedure. This means the final decision rests more on criteria based on value judgments, whose results may be more arbitrary.
Not using ALT formulae could be justified (especially in criteria such as improvements to projects or extending the guarantee period) by public bodies as being one of the basic principles of public procurement, ("offers are at bidders' risk"), although the real justification lies in the financial benefits obtained from such offers.
Situations such as those described in the case studies could be avoided by the use of ALT formulae, which would generate less risky bid evaluation systems closer to the core formed by most of the bidders, and reduce the risk of problems during project implementation, such as unreasonably high or low price offers, the need to modify projects during construction, work stoppages, or even the contractor abandoning the project. All these problems lead to delays in implementation, reduced work quality and budget overruns, problems endemic to public works [53][54][55].