A Rubric to Assess and Improve Technical Writing in Undergraduate Engineering Courses

Abstract

Although there is consensus in the literature that writing skills are important in STEM (Science, Technology, Engineering, and Mathematics) studies, they are often neglected. However, some efforts have been made to correct this deficiency, one of them being the development of assessment rubrics. This study seeks to contribute to the discussion by presenting the results of the application of a rubric designed to assess the writing skills of a group of 3rd year engineering students. This rubric, which includes linguistic and rhetorical-organizational criteria alongside the mathematical and technical, was used to assess a number of written exercises and essays submitted by students in a 15-week course. The main interest of this study was to test the efficacy of the rubric as a diagnostic tool, conceived to detect the areas of improvement in the students’ written performance and, ultimately, to also help them to achieve higher levels of competence. This goal was achieved, as one of the main conclusions of the study is that, although students usually master the technical aspects of the course, they must improve the linguistic and rhetorical aspects of their written communication. It can likewise be said that all the participants involved in the study profited in one way or another from the application of the rubric and contributed to identifying the ways in which the rubric itself can be improved for future application.

1. Introduction

The development of communicative competences in the area of STEM (Science, Technology, Engineering, and Mathematics), and more specifically, in Engineering, is often neglected in the curriculum and in the classroom. However, as noted by many researchers (see References [1,2], among many others), they are crucial in the professional career of graduates.

Many efforts have been made to try to fill in that gap in the training of future practitioners in the different STEM areas. One such effort is the development of rubrics, designed to facilitate the assessment of the oral and writing skills of undergraduate students. However, their value goes beyond the mere assessment of the progress or achievement of students. Rubrics can also be valuable tools for the diagnostic of improvement areas and to organize effective feedback.

In this article, we will focus on the need to provide training in writing skills for engineering students, as this is a critical competence in the workplace. Rubrics will be shown to be useful tools in that enterprise. Before going into the details of our study, a brief literature review is in order.

1.1. The Written Skills in Engineering Education

The importance of communicative skills, and more specifically, writing skills in STEM studies has often been stressed in the literature, for instance [3,4,5,6,7,8]. Communicative competence, both oral and written, is crucial to practitioners in the scientific areas just mentioned, to the extent that a study carried out by Sageev [9] reveals that “the average 64 percent time they spend on various types of communication validates industry’s requests that engineering schools urgently address this major “competency gap”. There is consensus among scholars that engineers (and other STEM graduates) should master communication skills, as well as all the technical aspects of their area of expertise. Researchers coincide likewise in the need to implement in the classroom didactic strategies designed to facilitate the future professional development of students and to promote abstract thinking [10,11].

Thus, it seems clear that one of the goals of engineering studies, and higher education in general, should be to enhance the communication skills of students [12], as the ability to communicate technical concepts in a clear and concise manner will be very important in their career. Lengsfeld et al. [13] cite the Engineering Criteria 2000, Accreditation Board for Engineering and Technology, Baltimore, where this ability is described as an essential engineering skill.

Now, the question is how to integrate the development of communication skills, and in particular writing skills, in the engineering curriculum. According to Davies & Cousin [6], there are three main possibilities: first, embedded, i.e., implicit in the general content; second, integrated, that is, explicit but integrated in the academic content; and third, bolt-on, that is, in modules which are separate from the technical content proper [10,14]. Missingham [3] proposes an interdisciplinary didactic approach which would involve linguistic and technical courses.

However, important the development of writing skills, it must be noted that writing is a complex process [15] which involves many cognitive strategies, and many problems emerge when instructors decide to introduce it in the syllabus. Most often they are faced with the reality that they have neither the necessary means/tools nor the time [16]. They must additionally deal with the reluctance of students, who are not aware of the importance of writing in their line of studies and, consequently, of the need to master the skills. To add to this, their writing competence is very often seriously flawed [17].

In an interesting study, Conrad [2] identifies the deficiencies which occur most often in the written production of the engineering students in her sample. They include: ambiguous or inaccurate complex sentences, imprecise word choice, mistakes in grammar and punctuation, and problems concerning the organization of ideas in the genre under study, the engineering memorandum.

These deficiencies are unacceptable in writings which should be characterized by the accuracy and precision (both mathematical and linguistic) of their contents, which must also be logically organized [18] and comply with the rhetorical conventions of the area (and specific genre) [19,20]. An optimal organization of ideas is also expected [21]. Today, this can be accomplished with the help of outlines and schemas prepared with computing programs [15]. Some authors stress also the need to take the gender perspective into consideration [22,23].

Several didactic proposals have been made to develop writing skills in the engineering classroom. One example is the creation of a wiki which is later presented orally [12], the oral presentation of different types of written contents [1], and the use of model texts for illustration [16].

Some other activities have also revealed themselves as quite useful to improve the student’s metacognition in the writing process. Among them, the following are noteworthy: pair-blind review [24], directed interviews and individual tutorials intended to improve writing skills [25], or a triple revision process involving different agents (engineering students, professors and students participating in didactic courses at the master level, for example) [26]. The intervention of the professor, as well as that of other fellow students, has proved useful in the construction of, for example, coherent argumentative texts [27,28].

It must be noted that the implementation of some of these activities may be rather time-consuming as they will probably be carried out in and out of the classroom and it cannot be forgotten that, as mentioned above, time constraints are usually a problem in the engineering classroom.

Another problem that we mentioned above is the attitude of students towards written assignments. In this respect, it is fundamental to raise the students’ awareness of the importance of developing their writing skills. One potentially effective tool to achieve this aim is the use of rubrics. In our experience, students tend to be more receptive to suggested courses of action when they have an impact (positive or negative) on their grades. Including descriptors in assessment rubrics that cover the areas in need of improvement may be very useful to give students a measure of the importance of those areas.

We turn now to a brief review of the literature on the use of rubrics.

1.2. Use of Rubrics

Rubrics are used systematically as a tool for the evaluation of many different skills and subjects. The reason for their popularity is simple: the rubric, when properly designed, is an optimal tool for an objective assessment, as it constitutes a punctuation guide with clearly defined criteria [29].

Rubrics can also be regarded as an effective teaching-learning resource, since they allow to identify the strengths and weaknesses of the items which are object of analysis in order to make a diagnostic approach to learning and evaluation [19,30]. They are also useful for the assessment of open educational resources [31].

The academic efficacy of rubrics has been described in the literature [29,30,32], among many others. Thus, it has been noted that the use of rubrics often translates into an improvement in the performance of students [33]. They may also have positive effects on the control of anxiety on the part of students, as they offer certainties concerning the evaluation [34]; and contribute to promoting the students’ self-knowledge and self-efficiency [35]. Similarly, the use of rubrics may also improve the efficiency of professors as the evaluation is less time-consuming, and they contribute to formative factors, such as transparency and feedback [36], as well as self-assessment [34,35,37].

Focusing specifically on writing skills, it can be said that rubrics have been used not only to improve the quality of argumentative texts [38,39], but also in the analysis of general linguistic competences [40], reflexive writing [41], interdisciplinary writing [42], in the evaluation of reports written by university students [14,43,44], and to evaluate the style of scientific papers [45].

However, the use of rubrics also has some limitations, as noted by Reference [46]. They simply give a numerical value to each item without taking into consideration information which might be important to enhance the teaching-learning process and for the diagnosis of formative needs. Similarly, some rubrics may be so specifically designed for an individual task that they fail to capture the essentials of the ability they try to measure or they may sometimes be inaccurate when it comes to describing different degrees in the qualitative differences in the responses.

Turning finally to rubrics specifically designed to evaluate the written production in the field of engineering, many authors underline the need to include, together with the purely linguistic ones, criteria specifically related to the contents of the subject being evaluated [2,47,48]. These criteria must comprise aspects, such as the characteristics of different genres, rhetorical structures, and argumentative formulations, rather than only the specific vocabulary of the subject or the scientific field.

Beyond their validity as evaluation tools, rubrics can also be used for diagnostic purposes, and, in this role, they can contribute greatly to the teaching-learning process. The use of rubrics will allow the instructor to identify gaps in the knowledge and abilities of students. This will, in turn, help in the design of future didactic interventions conceived to address the specific needs of the students.

Although with the limitations which are characteristic of any instrument of evaluation, we regard rubrics as extremely useful for the evaluation of the writing competence of engineering students.

Given that, in a previous study, problems of three different types were detected (mathematical, linguistic and rhetorical-organizational), we believe that an effective rubric should include all three areas. When the rubric is applied, students will see that their grade is calculated on the basis of several parameters rather than being exclusively based on the results of the technical questions. Professors, on their part, will have at their disposal a tool designed by engineering instructors which will facilitate an objective evaluation and the possibility of giving students valuable feedback. The rubric will likewise make it possible to evaluate the positive aspects (skills already mastered by the student) and to focus on those still in the process of acquisition.

The general aim of this research project is to develop a rubric with the potential to be used in different engineering courses and that serves the following purposes: firstly, to provide an efficient diagnostic tool that can be used to identify the strengths and weaknesses of engineering students regarding their writing skills; and secondly, to provide an objective measure of the achievements of students with respect to different parameters involved in the assessment of their writing abilities.

The specific goal of the pilot study reported in this paper was to test the first version of a rubric designed on the basis of the problems detected after systematically analyzing the written production of former students of the course Machine Design, with a view to testing the efficacy and to detecting areas of improvement of the rubric itself. Our hope is to be able to apply the improved version of the rubric in the assessment of the students’ written production in the following academic years.

2. Materials and Methods

The rubric that will be presented below is a first version, subject to future improvements based on the results obtained from its application to the evaluation of the writing performance of the students in the group described in the following sub-section. As has been mentioned in the introduction, the rubric is also conceived as a didactic tool.

2.1. Participants

The participants are Spanish-speaking students in their third year of the Mechanical Engineering Degree at the Faculty of Engineering in Vitoria (Spain), University of the Basque Country UPV/EHU (For Universidad del País Vasco/Euskal Herriko Unibertsitatea – official Spanish and Basque names of the University of the Basque Country). This group of students took part in the course Machine Design in the second semester of the academic year 2019/20. Their age range was between 21 and 23.

The course Machine Design is a 6-h/week course with a duration of 15 weeks. In the academic year 2019/20, some adjustments affecting both the instruction and the evaluation had to be made due to the lockdown decreed by the authorities in the context of the COVID-19 pandemic. As a result, 40% of the course had a face-to-face format, and 60% was taught remotely.

All the interactions in the classroom took place in Spanish, which is the language of instruction. It was, therefore, the language used to write the original rubric (which was translated into English for its presentation in this paper) and to give the students feedback. It was likewise the language used by the students to write their assignments.

For the purposes of this study, the 28 students enrolled in the course were divided into 7 groups of 4. Students were randomly allocated to groups by their professor in order to minimize the possibility of convergence of students with similar academic profiles in the same group. For the sake of anonymization, each group was assigned a letter from A to G.

2.2. Procedure and Corpus of Study

The writing tasks subject to analysis by means of the rubric presented here were assigned to each group following the chronology of the course. All the tasks counted towards the final grade. The evaluation of this course was structured as follows:

-

Individual final exam, 60% of the final grade: It was an online exam done at the end of the course.

-

11 (to 14) exercises, 10% of the final grade: They were completed by the groups along the first 10 weeks of the course. All the exercises will be treated as a single writing assignment throughout this paper. We will not refer to the individual exercises separately.

-

An essay worth 30% of the final grade submitted via Moodle by the end of the course: The students worked on this essay for 5 weeks at the end of the semester. Chronologically speaking, therefore, the submission of the essay took place after the students had received feedback on each of their exercises. The groups were required to complement this task with a 10- to 15-min oral presentation of the contents of the essay. This presentation was done on the platform Blackboard Collaborate.

Given that the rubric was intended to help students in their learning process, they were informed about its application to the assessment of the exercises and the essays. In order to make sure that the students understood the items contained in the rubric, it was presented and explained in class in the first week of the course and subsequently used to give students feedback. It was likewise included in the student guide, available for students on Moodle for the duration of the entire course.

As this was just a pilot study to test the rubric for the first time, the exercises were first assessed for correctness in the manner students were more familiar with. Later, the rubric was applied, and the students received the corresponding feedback, constructed on the basis of the categories and criteria contained in the rubric and intended to help them in subsequent exercises.

After using the rubric for ten weeks with the exercises, it was assumed that the students had developed a certain familiarity with the proceedings, and the essays were prepared and assessed by using the rubric.

As just mentioned, the tasks assigned to students were of two types. The first assignment consisted of exercises, related to all teaching units, and the second was an essay. As to the exercises, students were required to fully explain the development and result of each exercise. The word limit given for this explanation was set between 150 and 200 words and it had to be preceded by an introduction of between 50 and 100 words. The number of exercises that each group had to do varied from 11 to 14, as on specific occasions, one of the groups had to complete two exercises (instead of just one) to make up for the more complex one assigned to the other groups. This explains that not all groups did the same number of exercises. The degree of difficulty increased as the course progressed. The easiest ones, at the beginning of the course, were different for each group. Those associated with more complex subjects, with various design options, were identical. These were open exercises where the designer can choose different parameters, so that the design proposals of the different groups could be different. The students had one week’s time to complete each exercise and submit it as a Word document via the virtual platform Moodle.

Regarding the second assignment, the essay, students were asked to write an essay of approximately 10,000 words in which they had to provide a thorough description of the design of a part of a machine. Each group had to deal with a different one, assigned by drawing lots. The parts were (in parentheses, the letter-code assigned to each group): brakes (A), bearings (B), belts (C), clutches (D), bevel gears (E), springs and permanent unions (F), and screws and non-permanent unions (G).

The particulars concerning the contents and format required in this essay were explained to the students in one of the sessions of the course. A full description was also available in the student guide on Moodle. They received the instruction to write an original and systematic development, as well as practical applications of the different elements being analyzed. They should also explain various real cases of operation through proposed and solved practical exercises. They were likewise reminded of the need to avoid any form of plagiarism by correctly quoting the sources used. The following list shows the elements to be included in the essay, which are part of the description of the task in the student guide:

-

Title of the essay and name of the students in the group.

-

Table of contents.

-

Introduction.

-

Description of the element, operation, classification, applications. Attachment of videos of the item working is possible.

-

Calculations: propose 2 or 3 solved and explained examples of calculation and design. In the case of the simplest elements (such as bearings, screws, belts, springs), a greater number of exercises should be presented. The explanations should be phrased in a way, and contain the detail required, to be intelligible for self-study.

-

Conclusions.

-

Bibliography.

This procedure allowed us to collect a total of 79 exercises of the 84 which were expected, as some groups did not manage to submit their assignments on time (70% of the exercises were submitted respecting the deadline). This procedure allowed us to collect a total of 79 exercises of the 84 which were expected, as some groups did not manage to submit their assignments on time (70% of the exercises were submitted respecting the deadline). The mean number of words in the exercises was 400 ± 285, varying strongly from one group to another (the minimum being 65 ± 50 for group D and the maximum 778 ± 262 for group B).

Forty percent of the students requested online tutorials to clarify technical aspects. In relation to this task, the perception of the students was that their workload had increased exponentially due to the need to provide all the explanations in the exercises.

Regarding the essays, the average number of words was 8020 ± 1750, the longest being the essay submitted by group B with a total number of 10,930 words. Students declared during the online tutorials that they had been more motivated when writing the essay than when doing the exercises. The reason adduced was that they had had more freedom to develop the assignment. It must be noted that, due to the lockdown, students (not being able to go to the university library) only had access to electronic resources.

All these assignments were analyzed manually by careful examination of each exercise and each essay. A punctuation was given to each individual piece of writing using the objective scale from 1 to 4 described below in Section 2.2. Afterwards, the final scores were gathered on tables, and their mean values were calculated and plotted on the figures included in the discussions section of this paper.

Before moving on to the descriptions of the rubric, it must be noted that students were repeatedly reminded of the importance of their written expression.

2.3. The Rubric

As mentioned before, our intention is to design an analytic rubric that will assess interdisciplinary competences. This rubric incorporates different aspects related to the specific contents of the course, together with aspects related to written expression, and it considers the learning outcomes specified for this course in the student guide.

At this point, however, the rubric was mainly used for diagnostic purposes, that is, to identify the strengths and weaknesses of students in order to help them improve their writing skills. The rubric was applied to each exercise and feedback given to students so that they could address the aspects that had been problematic in one exercise, the final aim being that they could avoid repeating the same mistakes in the next exercise. The instructor worked closely with the different groups addressing the specific problems detected in each case. The use of the rubric also allowed the instructor to follow the progress of each group.

It had been previously detected that students tend to pay more attention to content and almost none to the linguistic correctness of the language they use (vocabulary and grammar) or the logical structure of the explanations they give. In this particular course, it is also important to pay attention to the use of graphic material, formulas, and so on.

In order to assess the level of achievement for each criterion, four levels of mastery were established: the lowest 1 is labeled NOVEL; level 2 APPRENTICE; level 3 ADVANCED; and the highest level 4 EXPERT.

The rubric includes 8 criteria, divided into 3 large areas: technical, linguistic, and rhetorical-organizational, as represented schematically in

Table 1. Overview of criteria in the rubric by area.

Area	Category
I Mathematical	1	Mathematical coherence: managing formulas and units.
	2	Technical description of operation and applications.
	3	Decision making.
II Linguistic	4	Sentence structure (syntax). Use of complex sentences (coordinated or subordinate) or simple ones. Use of discursive connectors.
	5	Word selection (Lexicon), empty words. Scientific correction
	6	Grammar, spelling and punctuation.
III Rhetorical-organizational	7	Clarity of reading: page layout, graphs.
	8	Rhetorical organization of the text (logical organization). Interpretation of results.

.

In what follows, details of the different areas are provided.

2.3.1. Mathematical Area

The first area comprises items related to technical criteria. In this area, three main questions are looked at: first, mathematical coherence, i.e., whether the formulas are well managed, correct units are used, etc.; second, the technical description of the operation and applications of the design that appears in the exercises/essay analyzed; third, whether the students carry out coherent and reasoned decision-making when they achieve the results of the exercise.

Table 2. Rubric for engineering students: mathematical criteria.

Mathematical Area	Expert 4	Advanced 3	Apprentice 2	Novel 1
1-Mathematical coherence: managing formulas and units	Appropriate interpretation of the formulas used.	Explanation of the terms of the formulas.	Introduction of formulas used.	No introduction of formulas used.
	Definition of units of all quantities and transformation of units, in a coherent way.	Definition of units and correct transformation.	Definition of units but incorrect transformation.	No definition of units.
	Obtaining experimental coefficients from tables or graphic information correctly explained.	Obtaining experimental coefficients from tables or graphic information with insufficient explanations.	Obtaining experimental coefficients from tables or graphic information without explanations.	No obtaining experimental coefficients from tables or figures.
2-Technical description of operation and applications	Reasoned interpretation of the design criteria used.	Reasoned explanation of the design criteria used.	Indication of design criteria used.	No indication of design criteria used.
	Definition of the physical limitations for the choice of the machine element.	Definition of variables to be determined in the design.	Definition of application of design criteria based on the type of material (brittle/ductile).	No definition of restrictions or design criteria.
	Indication of current technological uses and applications.	Indication of general and traditional technological uses and applications.	Indication of basic uses and applications.	No indication of uses or applications.
3-Decision-making	Decision-making associated with the maintenance and durability criteria, design, and material behavior.	Decision-making associated with the design and behavior of the material.	Decision-making associated with design: commercial availability (catalog).	No making of decisions based on design criteria.

lists each of the mathematical criteria (a total of 3) with a brief description of the levels of achievement in each of them.

2.3.2. Linguistic Area

In the second area, attention is paid to the problems students usually have with written expression from the point of view of language usage (sentence structure, syntactic complexity, use of discursive connectors), vocabulary choice (correctness and variety), spelling, and punctuation.

Table 3. Rubric for engineering students: linguistic criteria.

Linguistic Area	Expert 4	Advanced 3	Apprentice 2	Novel 1
4-Sentence structure (syntax). Use of complex sentences (coordinated or subordinate) or simple ones. Use of discursive connectors.	Use of subordinate, coordinated and simple sentences. Use of discursive connectors appropriately.	Use of subordinate, coordinated and simple sentences. Use of some discursive connectors but not always appropriately.	Use of coordinated and simple sentences, and Use of some discursive connectors but not always appropriately. Use of such complex sentences and elaborated discursive connectors that the text becomes unintelligible.	Use of simple sentences. No connectors.
5-Word selection (Lexicon), empty words. Scientific correction	Accuracy and precision in the choice of scientific terms.	Accuracy and precision in most scientific terms. (70%)	Use of many non-scientific and meaningless words. (50%)	Use of non-scientific words and imprecise and meaningless terms.
6-Grammar, spelling, and punctuation	No grammar or spelling mistakes.	Some spelling mistakes (accents) but no grammar mistakes.	Some spelling and grammar mistakes.	Repeated spelling and grammar mistakes prevent the optimal understanding of the text.

shows the linguistic criteria with a description of the levels of achievement for each.

2.3.3. Rhetorical-Organizational Area

In the third area, two main criteria have been used. One deals with the layout and format of the exercises and essays and includes questions, such as clarity of reading, arrangement of elements on the pages, graphs, etc. The second one has to do with the rhetorical organization of the text. Here, various aspects have been considered. We regarded it as necessary to make a distinction between the specific aspects related to the exercises and those related to the essays, as the two types of tasks have different characteristics. More details are provided below. Before that, we turn to

Table 4. Rubric for engineering students: rhetorical—organizational criteria.

Rhetorical-Organizational Area	Expert 4	Advanced 3	Apprentice 2	Novel 1
7-Clarity of reading: page layout, graphs	Visual order. Complete and color-coded representation, with reference systems, proportionality of the measures represented.	Complete and bicolor representation, with reference systems, proportionality of the represented measurements.	Basic and monochrome representation, without defining reference systems.	No visual schematic representation. If present, it is wrong.
	Appropriate management of space, clarity and order.	Clear and organized interpretation.	Clarity without systematic order, without crossings out.	No clarity or order.
	Numbered table of contents, page numbers.	Numbered table of contents, page numbers.	Page numbers, without a numbered table of contents.	No page numbers.
	Identification of each figure and table.	Identification of some figures with consistent format.	Identification of some figures, but no consistent format.	No identification of figures.
8-Rhetorical organization of the text (logical organization): Interpretation of results	Interpretation of everything.	Interpretation to discard solutions which are impossible or not compatible with restrictions.	Interpretation of negative dimensions.	No interpretation of results.

, which displays the rhetorical-organizational criteria with a description of the 4 levels of achievement for each of them.

Given the different formal and content characteristics of the two types of tasks under evaluation (exercises and essays), in this area, we had to propose some items that would be applied only to the exercises and some others which were specific to the essays. For the analysis of the exercises, the sub-criteria in 8.1 to 8.5 shown in

Table 5. Rubric for engineering students: sub-criteria for exercises.

	Expert 4	Advanced 3	Apprentice 2	Novel 1
8.1-Data collection. Identification of the type of exercise	Identification of the type of exercise, and collection of necessary data.	Identification of the type of exercise, but no collection of necessary data.	No identification of the type of exercise, although there is collection of necessary data.	No identification of the type of exercise and no collection of necessary data.
8.2-Make sure what the exercise is asking for/results	Declaration of what they ask explicitly.	Indication of what they ask for, but start by calculating other things.	Indication of what is normally requested, without relating it to this exercise	No comments and begin to do the exercise straight away.
8.3-Expression of the correct formulation and definition of the parameters present	Use of appropriate formulas and definition of parameters.	Definition of appropriate formulas, but no identification of parameters.	No definition of appropriate formulas, but identification of parameters.	No definition of the appropriate formulas, and no identification of parameters.
8.4-Explanation of the development and steps to follow	Initial explanation of the development and explanation of the steps as the exercise progresses.	No explanation at first, but then explanation of steps taken.	Initial incomplete explanation, but then no more descriptions.	No explanation of development or steps.
8.5-Underline the results attained	Underlining and interpretation of results.	No underlining of results, but interpretation given.	Results underlining, but not interpretation.	No underlining or interpretation of results.

below were considered relevant.

Turning to the essays,

Table 6. Rubric for engineering students: to sub-criteria for essays.

	Expert 4	Advanced 3	Apprentice 2	Novel 1
8.6-Introduction: motivation, aims, scope, applications, limitations	Complete introduction.	Introduction without limitations.	Introduction without applications.	Introduction without aim or scope.
8.7. Identification of the type of analysis: mechanical, static, dynamic, energetic. Description of hypothesis used, environment, technical unit system.	Identification of type of analysis.	No identification of type of analysis.	No identification of type of analysis.	No identification of type of analysis.
	Description of hypothesis used.	Description of hypothesis used.	No description of hypothesis used.	No description of hypothesis used.
	Correct technical unit system.	Correct technical unit system.	Correct technical unit system.	Incorrect technical unit system.
8.8. Description of components, assembly. Description of material used, layers, special components.	Description of components.	No description of components.	No description of components.	No description of components.
	Description of materials, and others.	Description of materials, and others.	Description of materials, but very generally and not specific to the components described.	No description of materials.
8.9. Practical case: description of the problem to be studied.	Complete description of the problem: type of analysis, identification of what is being asked, and interpretation of results.	Partial description of the problem: no type of analysis, but identification of what is being asked, and interpretation of results.	Partial description of the problem: no type of analysis, no identification of what is being asked, but interpretation of results.	No description of the problem: no type of analysis, no identification of what is being asked, no interpretation of results.
8.10. Expression of the correct formulation and definition of the parameters present.	Use of appropriate formulas and definition of parameters.	Definition of appropriate formulas, but no identification of parameters.	No definition of appropriate formulas, but identification of parameters.	No definition of the appropriate formulas, and no identification of parameters.
8.11. Explanation of the development and steps to follow.	Brief introduction and explanation of the steps as the exercise progresses.	No initial introduction, but then explanation of steps taken.	Initial incomplete explanation, but then no more descriptions.	No explanation of development or steps.
8.12. Underline the results attained.	Underlining and interpretation of results.	No underlining of results, but interpretation given.	Results underlining, but not interpretation.	No underlining or interpretation of results.
8.13. Conclusions.	Relevant conclusions, well explained.	Relevant conclusions.	Irrelevant conclusions.	No conclusions.
8.14. Bibliography.	Complete bibliography.	Complete bibliography. but with lack of details in some references.	Scarce bibliography and not complete.	No bibliography.

shows the sub-criteria relevant to this task.

2.3.4. On the Application of the Rubric

This rubric was used as follows. Each of the areas of the rubric were divided into three different categories or criteria (in the case of the Areas I and II) and two in the case of Area III. They are numbered from 1 to 8 in the rubric. Each of those criteria received a numerical value from 1 (NOVEL) to 4 (EXPERT). As a result, the highest mark that could be obtained in each category was 4. In order to obtain this mark, the exercise or essay submitted by the student should comply with all the descriptors of the category listed in the column EXPERT in the rubric. It must be noted that students could obtain a grade of 0 when the development was incomprehensible or obtained erroneous results without a coherent logical explanation and, of course, when they did not submit the exercise.

For a better understanding, we will provide some examples related to the different areas. In category 1 (Mathematical coherence, Area I), students should provide an appropriate interpretation of the formulas used, include all the intermediate steps, and define all the units and magnitudes coherently. Failure to comply with only one of these requirements automatically meant that the students’ exercise was demoted to a lower level. In the second area (linguistics), it was determined that, in category number 6 (grammar, spelling and punctuation), for instance, the exercises or essays should not present any grammar mistake, but it could present a reasonable number of spelling mistakes (mainly in the use of accents). Finally, in category 7 (Area III), essays in which the graphic representations were monochromatic, and no reference systems were provided, in which there was a certain clarity but no systematic order, and in which there were page numbers but no table of contents, would receive a mark of 2 (APPRENTICE). Exercises and essays receiving the lowest mark in the rubric (i.e., 1) presented serious deficiencies in the parameters of the category under evaluation. Grounds to an automatic 0 on any task were also: cases of plagiarism or copy, incomprehensible development, or non-ethical behavior.

Once the assessment of the different categories was completed, the mean was calculated for each of them. The evaluation process is closed with the calculation of the mean of all areas.

Once the rubric has been presented and its application described, it is time to turn to the discussion of the results.

3. Results and Discussions

This section presents the results of the analysis of the texts in the sample with the rubric developed for the purpose. This presentation is organized in three sub-sections corresponding to the areas of analysis that appeared in the rubric.

3.1. Mathematical and Technical Area

The average values of the mathematical and technical area are 2.64 ± 0.5 out of 4 for the submitted exercises and 3.3 ± 0.78 for the essays. Figure 1 below shows the mean values and standard deviation of the initial 3 categories in the exercises and essays of all the groups. In the exercises, the first category (Mathematical Coherence) reaches the highest value 2.8 ± 0.9 out of 4, the second (Technical Description) 2.6 ± 0.9, and the third, (Decision Making) 2.6 ± 0.6. In the essays, the values are, respectively, 3.6 ± 0.5, 3.6 ± 0.7, and 2.9 ± 0.8.

In the exercises, the highest score obtained in the area is 10.2 out of 12 (group F) and the lowest is 6 (group E). As mentioned in Section 2 above, in order to receive 12 points, the group should reach the EXPERT level, with 4 points, in all three categories. It must be mentioned that, in spite of the results, students had the perception that they had done well because they had obtained the right result for the exercise. In the essays, the highest score was 11 out of 12 and the lowest 6 (group E).

Considering that all the results are out of 4, all the groups did better in the first category both in exercises and essays (2.8 and 3.6) than in the others. They managed the formulas and units correctly though not optimally (category 1), the main problem being that they usually did not explain the parameters that appear in the formulas (category 2). This is more remarkable in the exercises that in the essays. They simply substituted values without indicating where they came from. Similarly, the conversion of units is often incomplete in that the results are not expressed in appropriate units (category 2).

On the other hand, justification in decision-making (category 3) is an aspect that must be improved. In general, both in the exercises (2.6) and in the essays (2.9), all the groups had difficulty in explaining the reasons associated with making design decisions. Problematic were also the use of reducing coefficients, experimental tables or unexplained formulas, although later the execution was correct.

The following example serves as illustration of the problems in this area. In several exercises, shafts and beams needed to be dimensioned under certain stress requirements. Some groups calculated the stress state due to normal and tangential solicitations. However, sometimes they did not localize the critical section within the shaft or beam that determines the optimal design dimensions. Some groups did not explain when the normal stresses due to normal forces and bending moments were added (or subtracted). In the same sense, they did not feel the need to explain different tangential stress-states due to torsional moments and shear forces. They simply calculated the highest numerical value and moved on.

Omissions of specific pieces of information and the mistakes just mentioned were made by students even though the rubric called their attention to these aspects and they knew their professors would be taking them into account in their assessment. Moreover, these mistakes appeared also in exercises that were submitted after receiving feedback on these areas of improvement.

It is true, however, that a certain improvement was observed in the three last exercises, which obtained a higher mean (3.1) than the other exercises (2.7). This progress is clearly seen in Figure 2, which shows the evolution of the average values of the 3 categories.

All in all, it seems that it is necessary to emphasize the need to integrate the theoretical knowledge acquired in the lectures into the explanation of their exercises. For some reason, students tend to keep theory and practice separate. A more detailed explanation of the rubric might also be necessary. Students in this course are not used to working with rubrics and that might also be part of the problem, as they tend to focus on the exercise without paying attention to the requirements concerning the type of information that they have to include in the explanations.

3.2. Linguistic Area

The average values of the linguistic area are 2.6 ± 0.6 out of 4 for the submitted exercises and 2.8 ± 0.6 for the essays. Figure 3 below shows the mean values and the standard deviation of the 3 categories associated with linguistic criteria. In the exercises, the 4th category (related to syntax) presents the lowest value 2.3 ± 0.9 out of 4, the 5th (related to the lexicon) 2.5 ± 0.6, and, finally, the 6th (related to grammar spelling and punctuation) 3.1 ± 0.8. In the essays the values are, respectively, 2.9 ± 0.6, 2.7 ± 0.4, and 2.9 ± 0.8

In general, it can be seen that the values are lower than those related to mathematical dexterity.

In the exercises, the lowest score (2.3) was obtained in category 4, which has to do with the syntactic correctness and complexity of the writing. In the essays, on the other hand, this score reaches 2.9 points. In the case of the 5th category, concerning the choice of vocabulary and the appropriate use of technical terms, the score is slightly higher than in the previous category in the case of the exercises (2.5) but lower in the case of the essays (2.7). Finally, in grammar, spelling, and punctuation, the exercises are better, obtaining the highest score in the area (3.1), whereas the essays obtain a slightly lower score (2.9). This is probably due to the difference in the number of words.

In the exercises, the highest score obtained is 9.3 out of 12 (group F), and the lowest is 5.2 (group D). In the essays, the highest score obtained is 10 out of 12 (groups B and F), and the lowest is 7 (group A and D).

Looking more closely at the results, it can be observed that, regarding syntax, students show a clear tendency to use simple and unstructured sentences. On certain occasions, students (specifically, Group D) chose to transcribe only mathematical formulas. This group obtains the following mean scores for the exercises: 1.55 (category 4), 1.45 in category 5, and 2.18 in category 6. As can be seen, the level of achievement of this group in particular is far from the general mean (calculated with the means of all the groups).

Given the characteristics of the two written tasks, it comes as no surprise that significant differences arise between the exercises and the essays. The former are shorter; therefore, fewer discursive connectors are used, and the sentences tend to be simpler. However, the length of the essay and the requirements of the task result in a greater elaboration of the sentences, which in turn translates into higher scores (2.9) than in the exercises. However, notice that this score is still far from the maximum 4.

The use of connectors and the use of more complex syntactic structures are related in the sense that both conjunctions and connectors contribute to the cohesion of the discourse. It is, therefore, not surprising that a correlation is found in the case of essays to the effect that those students who used more connectors also wrote more complex sentences. Among the types of connectors which were most frequently used are the causal ya que, puesto que ‘because, given that’, the final para que ‘so that, in order that’, the consecutive por lo tanto ‘therefore’, and sequence adverbials, such as primeramente ‘firstly’, luego ‘then, afterwards’, or por ultimo ‘finally, lastly’. They all correspond to the typology of an explanatory text.

Concerning the lexicon (category 5), variety in word choice and correctness in the use of scientific terms are both lower in the exercises than in the essays, as mentioned above. Students tend to use words typical of an informal register. Some examples follow: sacamos ‘lit. we get’ (instead of using the verb calculate (calcular), “cogemos” (meaning we take) or “pasamos de una unidad otra” (meaning we go from one unit to another). These verbs appear very frequently, as well as repetitions of words, which indicates that students do not have a wide range of vocabulary, and this includes technical and scientific terms. In general, their use of technical terminology was deficient.

Finally, in the third linguistic category being considered (no. 6) (grammar, spelling, and punctuation), students obtain quite generally better scores in the exercises than in the essays, contrary to what happened in the previous categories. This can be attributed to the length of the essays. In other words, the longer the piece of writing the more likely it is that mistakes will appear. Regarding grammar, agreement mistakes are frequent: Se pueden hallar la fuerza que ejerce ‘the force it applies can be calculated’. The Spanish sentence is ungrammatical because the verb is in the plural, while the subject is in the singular. In some other cases, the verb is simply not inflected or a sentence is left unfinished. Spelling, in general, is quite correct, probably due to the fact that students used a text-editing program with a spell checker. In spite of this, there are spelling mistakes mostly having to do with the lack of accents (ultimas instead of últimas ‘the last ones’) and with the confusion of homophonous words (haya, a form of the auxiliary verb ‘have’ and halla ‘finds’). Many deficiencies were also found in the use of capital letters and punctuation, especially in the use of commas.

Figure 4 shows a lot of fluctuation in the average values of the linguistic categories in the exercises, with two clear peaks on number 5 and number 11. At the end, the line plunges to timidly rise afterwards. We interpret this as a reflection of the problems that students have with the linguistic area, and the fluctuations indicate that the feedback does not seem to work. The intervention required to solve this problem has to be more prolonged in time and far-reaching.

The rubric in this case uncovers a problem that some students may be unaware of: their lack of command of the language of instruction. However, developing their vocabulary and grammar competences is not something that can be addressed in the classroom. The only aspect that can be dealt with in the course is the use of technical terms. What these results show is that university students reach this level of education with an important gap in their competence. A prolonged intervention will be required to solve this problem.

3.3. Rhetorical-Organizational Area

In this section, we turn to the discussion of the last area of study included in the rubric. It must be noted that, although only two parameters are taken into consideration here, the exercises and the essays have been evaluated using partially different rubrics. Sub-categories 8.6 to 8.14 are applied only to the essays, whereas sub-categories 8.1 to 8.5 are proposed specifically for the exercises.

The average values of the rhetorical-organizational area are 2.6 ± 0.6 out of 4 for the submitted exercises and 3.4 ± 0.6 for the essays. In the exercises, the 7th category (related to clarity layout, graphs) presents the highest value, 2.9 ± 0.8 out of 4, whereas the 8th category presents the lowest value 2.3 ± 1. In the essays, the values are, respectively, 3.7 ± 0.4 and 3.2 ± 0.7. Mean values and the standard deviation of the two parameters in the rhetorical-organizational area are displayed in Figure 5

Regarding the clarity of reading and the arrangement of objects on the page (category 7), students scored highly on this parameter, with almost the highest averages considering all categories in all areas, 2.9 for exercises and 3.7 for essays. These averages mean that the representation is in general complete, with two-color graphs, explicit reference systems, proportional or scaled figures, visual order, tables of contents and page numbers, etc.

However, the rhetorical-organization of the texts (category 8) was more problematic for students. The average obtained was lower, both in the exercises (2.3) and in the essays (3.2), when compared to category 7 above. The most significant problem, and the reason for the lower scores, was the general absence of an interpretation of the results. In the practical cases included in the essays (as well as the exercises), a clear tendency was observed to present an incomplete task, since the conclusions drawn were not appropriate for the requested textual genre. In the cases in which an interpretation of the results and conclusions appeared in the texts, they were of a very poor quality.

For completeness, the evolution of the achievement in the exercises is shown in Figure 6. As can be seen, the last three exercises obtained a higher mean score than the rest of exercises taken together, 2.7 as opposed to 2.6.

So far, we have presented the results concerning the two big categories in this area, but as just mentioned, category number 8 was sub-divided into 5 further categories for the exercises and 9 for the essays. Figure 7 and Figure 8 below show the detail.

The scores of all subcategories are very similar, with 8.2 standing out with 3.0 out of 4. The lowest mean score (2.6) is obtained in 8.3, involving correct formulation. However, this value is not significantly lower than that of categories 8.1, 8.4, and 8.5 with a mean score of 2.7.

In the 9 sub-categories for the essays, the values are again quite similar for 8.6 to 8.11 (in order, 3.0, 3.1, 3.3, 3.3, 3.0, 3.1). However, the scores obtained in 8.12 and, above all, in 8.13 and 8.14 are significantly lower; in order, they are 2.7, 1.9, and 2.0.

In sum, concerning categories 7 and 8, students paid attention to the aspects of essay writing which had to do with the visual presentation of different types of contents, a task which was probably facilitated by the use of technology (personal computers with editing utilities). They also generally structured their contents in an appropriate manner. However, on the debit side are the many problems detected in the citation of sources. Students are not aware of the importance of this aspect to avoid plagiarism. More work must be done in the classroom to raise their awareness and to show them how to acknowledge the sources used correctly.

4. Conclusions and Future Work

The following conclusions can be drawn from the application of the rubric to the written production of this specific group of engineering students.

In its role as a diagnostic tool, it helped us identify the gaps in the writing skills of the students in this course. In general, it was confirmed that they paid more attention to the technical and mathematical contents of the exercises and essays (Area I) than to the linguistic and rhetorical-organizational aspects (Areas II and III), crucial to the development of their writing skills. They seem to be unaware of the fact that communicating their results effectively may be as important as calculating them. They are 3rd-year students in a 4-year degree, which means that they are approaching their incorporation into the labor market, where they will probably need strong writing skills. It will be, therefore, necessary to insist on the importance of developing their linguistic and rhetorical competence.

Instructors profited from the use of the rubric inasmuch as it allowed them to draw a clear picture of the types of mistakes students made, which helped them to organize their feedback sessions and to adapt the contents of their classes to incorporate explanations that might solve the problems detected. Some of these explanations (for instance, the issues having to do with plagiarism) will be part of the program in future courses. Thanks to the application of the rubric, we were, therefore, able to make a precise diagnosis of the didactic strategies that must be implemented in order to improve the students’ writing skills, as, for instance, those leading to promote the use of the appropriate academic, technical vocabulary.

With respect to the second research question, we believe that the rubric facilitates an objective measure of the achievements of students, and, by including different areas (mathematical, linguistic and rhetorical), it also contributes to making students aware of the importance of these three areas in the development of their writing skills.

Finally, although a certain evolution could be seen in the achievements of students (they seemed to be applying the advice given in the feedback sessions), they were sometimes reluctant to use the rubric as an effective tool in the preparation of their writing assignments. This can be concluded from the fact that they sometimes repeated the same mistakes or their production did not comply with the requirements specified in the items described in the rubric. It is necessary to insist on the importance of rubrics as a source of information regarding the successful fulfilment of the task. In order to overcome the students’ resistance to use the rubric and to promote a correct and effective use, we are considering several courses of action: (i) working with the rubric in a more interactive way, for example, by discussing with students the categories and items that, in their opinion, should be considered in each area; or (ii) asking all the students in the group to separately assess their own exercises (or those of a different group) using the rubric, and compare the results obtained. The idea would be to make them participate, to a certain extent, in the elaboration of the rubric.

Needless to say, the use of rubrics may be particularly interesting in the case of courses divided in different groups, each taught by a different professor. A rubric may play a crucial role in this situation as it will guarantee equity in the assessment of all students.

Given that this was just a pilot study, we are conscious of the fact that it has some limitations, as, for instance, the number of participants and the potentially distorting factor that all the interactions with the students took place online from week 6 (60% of the course). It is, therefore, necessary to test the efficacy of the rubric in a face-to-face learning situation on a higher number of students.

These are all valuable conclusions that will be taken into consideration in the future stages of the project.