With a diligent evaluation of the work results, project managers create the basis for a targeted resource search and the optimum assignment of suitable translators.
The following information provides an overview of the mechanisms used and the configuration options of quality assurance in OTM.
Please keep in mind that the day-to-day work of a translation agency cannot be about awarding fully objective individual ratings for individual services.
The quality ratings and their results are not shown to the translators as it is not our wish to ignite academic discussions about censorship. Their main purpose is to identify strengths, weaknesses and trends so that project managers can react appropriately in day-to-day business.
The quality assurance parameters are set in the administration area under >Resource administration >Ranking.
All results and the individual ratings for a resource can be found in >Project management >Resources >Resource details.
Under >Project management >Resources >Search and under >Project management >Jobs >Post, the ranking following from the quality ratings influences the sorting of the results lists.
Result files delivered by translators (resources) are given a quality rating in OTM. This quality rating consists of a compulsory component and an optional component.
The compulsory component is the Formal Quality Assessment (FQA). The optional component is the Linguistic Quality Assessment (LQA).
Furthermore, on completion of an order for each job, a value for the Willingness to Work Again (WWA) is saved. Here, the project manager can give their impression of the communication behavior and business practices of a resource and the degree to which they would be willing to continue working together.
In the FQA, simple Yes/No answers are entered for each result file to determine if
The project manager must record an FQA for each result file before the file can be used as a delivery file for the customer.
The optional Linguistic Quality Assessment for a result file is also recorded by the project manager.
LQA criteria are:
An LQA comes about when a result file is either identified as “error-free” or if at least one deficit criterion is activated.
The frequency with which a specific deficit criterion occurs in a document is inconsequential for the time being. The only thing registered is whether or not there were corresponding deficiencies or errors.
As well as the information on accuracy or linguistic criteria, it must be noted down in each LQA who contributed to the rating. The project manager, the customer or a corrector or a combitnation of these can be the information source that contributes to a rating.
If there is no information source activated, then an LQA rating will not be saved.
The scale on which the LQA values are illustrated ranges from 0 to 100 percent. However, the lowest value of the scale can be increased. You can, for example, specify that the maximum deduction in all criteria should not lead to a result of 0 percent, but to a result of 50 percent.
The reasoning behind this option is that with the possible, later introduction of factors for error frequencies, the values collected up until then can be associated with a medium error frequency, thus keeping the usual number values in a more differentiated range.
If a value of 100 percent shows for an individual criterion or for a facet, this means that no deficiencies were detected. In other words: the criterion or facet was rated as “error-free”.
As a WWA rating, the project manager gives a value of ‑2 (very poor) to +2 (very good) when the project is concluded. The preset value of 0 is neutral and represents neither a positive nor a negative rating.
The facets of the quality rating – FQA, LQA and WWA – are processed by OTM on three different levels. The file scores for FQA and LQA are calculated on the level of the individual result files. Job scores for FQA, LQA and WWA are calculated on the level of a job. Resource scores for FQA, LQA and WWA are calculated from the job scores.
In addition, there is a ranking score. This is an overall value which is not shown, but is an important parameter which controls the rank order in results lists when searching for and selecting resources.
Here is an overview of the recording / calculation on different levels:
Facets Levels |
FQA |
LQA |
WWA |
File scores |
Compulsory rating, PM input, criteria, with different weighting if necessary |
Optional rating, PM input, criteria, with different weighting if necessary |
- |
Job scores |
Average value of the file FQAs of the job |
Average value of the file LQAs of the job |
PM input on project conclusion |
Resource scores |
Calculation based on all job FQAs |
Calculation based on all job LQAs |
Average of all WWA values |
Ranking score |
In the results lists of the resource search, the ranking score influences the sorting according to OTM rank. It is calculated from the weighted results of the individual facets. |
The quality rating of a facet is made up of values for individual criteria. This applies both to the FQA and the LQA.
The four criteria for the FQA are ”Completeness“, “Adherence to deadlines“, “Compliance with parameters“ and “Formatting“. These four criteria can be weighted differently.
If, for example, adherence to deadlines is of utmost importance to your company, then you can configure OTM in such a way that deficiencies in this respect are given greater importance than deficiencies in formatting. A resource is thus deducted more heavily for missed deadlines than for poorly formatted documents.
The same applies to the LQA criteria. Users with resource administration rights can adjust the corresponding settings in the administration area under >Resource administration >Ranking.
The resource scores for FQA and LQA are formed on a job basis from the weighted individual criteria.
In addition, OTM calculates (also based on the job scores) overall values for the individual criteria. These are shown in the front end for the project management in the details for a resource.
For each individual criterion a value which represents the average result in this criterion is returned. Neither weighting nor scaling are involved here. Project managers can use these values to possibly identify existing accumulations of deficiencies in the individual criteria.
The ranking score is an overall value which is made up of the resource scores for the FQA, LQA and WWA facets.
The ranking score is not shown in OTM. However, it does have a decisive influence on the results list if the sort option ”OTM rank“ is active during the resource search or resource selection.
You can adjust to what extent a resource score for a certain facet should influence the ranking score and, for example, apply weighting according to your own measures in such a way that LQA results have greater influence on the ranking than FQA or WWA results.
Note for licensees who licensed OTM before 2011: An adjustment factor is applied which has the effect that resources with multiple jobs which have not yet been rated and which are dated before 2011 achieve a higher ranking score than resources which had few rated jobs.
When entering a WWA value for a job, you can choose between five options:
-2 (worst value)
-1
0 (neutral)
+1
+2 (best value)
The average value is calculated from all WWA values saved for a resource and shown for the resource. This WWA average is converted into a percentage figure from 50 to 150 percent for the OTM ranking so that it can be used in the ranking score.
This way, a resource with a positive WWA average is given a percentage figure which is greater than 100 percent, that is, a kind of bonus for the ranking. A resource with a WWA average of 0 is given a virtually neutral 100‑percent value, while a resource with a negative WWA average is given a value below 100 percent, that is, a minus point as it were.
If areas of specialization are considered as search parameters during resource search and resource selection, this will affect the sorting of the results list. The OTM rank is not solely made up of the ranking score, but is calculated while considering all parameters which are important in the specific line-up. This means both the QA values and the areas of specialization as well as the type of matches which the resources have for the areas of specialization (100% or fuzzy).
For a resource search in which areas of specialization are not considered, the results listing when sorting according to the OTM rank is derived from the ranking score.
There is the option of prioritizing resources for the ranking, e.g. when they were recently added to the OTM database. In this regard, you can use the option “Prioritize during search/selection” in >Administration >Resource administration >Resources >Preferences.
If this option is activated when sorting according to OTM rank, the resource will be named in the results list and shown high up even if the resource has no good quality results or none at all.
Sample application: New resources can firstly collect up a certain amount of quality ratings before their prioritization is withdrawn again. After this they will be found in the results lists based purely on their areas of specialization and the QA results.