Evaluation of Technology Platforms for Use in Transdisciplinary Research

Abstract

Transdisciplinary research (TR) is a growing field in a myriad of subject domains ranging from sustainability to education. The idea is to solve complex problems through the collaborative participation of academic actors, community stakeholders, governments, and other leaders to jointly define the problem and find solutions. To accomplish such complex projects, one of the main tools is an efficient technology platform. The need for an efficient technology platform, in turn, leads to a need for an evaluation tool to determine which is the most suitable for a given transdisciplinary research situation, however, a literature review confirms that research on such evaluation tools is scarce. The goal of this paper is to improve the evaluation of technology platforms for individual TR projects, based on the characteristics of that specific TR. The research question is “Can TR characteristics be used to construct an evaluation tool for technology platforms?” The specific aims used to answer the question were: the following: (1) to define the characteristics (core activities and skills) of TR and (2) to construct a tool for evaluating technology platforms for use in TR. Methodology: For the first specific aim, a literature meta-analysis was used; the second specific aim was addressed following a newly developed methodology, including elements of the technology acceptance model and elements of a standard technology evaluation process. Results: A chart for defining the core activities and skills in TR was developed, and a tool was constructed for evaluating technology platforms that could be used in TR.

1. Introduction

Transdisciplinary approaches to research and education are critical for addressing complex problems [1]. Understanding and solving complex problems, in subject domains as varied as sustainability, healthcare, science and technology, and biological sciences, requires knowledge sharing and joint knowledge production. It is also contingent upon shared experiences across disciplines such as the technical, engineering, social and natural sciences, and the arts. The essence of a transdisciplinary approach is that it emphasizes a collaboration model of research across academic, industrial, military, and community settings. In such a context, there is a need for a common glossary of key terms in transdisciplinary collaborations [2]. Socio-ecological problems require transdisciplinary research (TR) teams, and the formation of such a team is known to be a challenging problem [3]. Transdisciplinary thinking skills are essential when working across disciplinary boundaries, and research has shown that successful creative thinkers tend to use a set of meta-level cognitive transdisciplinary skills [4]. TR, because of its importance for solving complex problems, also plays a role in educational outcomes by involving collaborative learning, critical thinking skills [2], and student engagement.

TR brings together academics, researchers, practitioners, government institutions, and community leaders who jointly solve complex problems. In this context, effective collaboration skills play a key role in the successful completion of projects. The importance of skills in TR is highlighted in [5], where the steps used for training transdisciplinary scholars in cancer prevention and control were discussed. The key aspects included were the importance of shared understanding and concepts, mentoring, didactic and skills training, communication skills such as writing and presentation, and preparing individual profiles. In [4], the seven skills presented were observing, patterning, abstracting, embodied thinking, modeling, playing, and synthesizing. Systems thinking, empathy, and meta-cognition as proxy skills were discussed in the context of transdisciplinary skills in engineering education [6]. As discussed in the paper, characteristics of transdisciplinarity are the ability to think in a complex manner, to consider knowledge integration, and to demonstrate empathy, open-mindedness, and other interpersonal skills. The skill of systems thinking enables dynamic assessments of the interrelationships among various knowledge domains and the process to construct an integrative system.

TR projects, involving collaboration groups, are comprised of a myriad collection of activities among the team members. For example, TR activities are reflected in the selection of the right tools and methods for participatory modeling (PM) which is a transdisciplinary exercise involving stakeholders [7]. PM emphasizes collaborative learning. The tools and methods used during PM projects are expected to promote system understanding and awareness for all stakeholders. Another example is the Transformative Knowledge Network (TKN) which is an international group of research organizations, collaborating to explore processes of social transformation and to share insights [8]. The paper analyzed the activities and processes of transdisciplinary collaboration for sustainability that took place between 2015–2019. They further analyzed how the projects are structured so that learning across disciplines, cultures, and contexts can occur.

There are considerable barriers to implementing transdisciplinary projects. While TR is growing, there is no common glossary, no focused technology platform for collaboration and communication, and no commonly shared research framework [9]. How to nurture transdisciplinary communities of practice is discussed in [10]. The nurturing techniques include opening the community to peripheral participants, shared knowledge, meetings with multiple academic and non-academic stakeholders, and promoting mutual learning. The outcome spaces framework to guide purposive TR is presented in [11]. Defining the desired outcomes influence how TR projects are conceived, designed, implemented, and evaluated.

2. Specific Aims

TR has recently been touted as vital to research that addresses complex environmental and societal problems [12]. The idea in this growing field is to solve complex problems through the collaborative participation of academic actors, community stakeholders, governments, and other leaders to jointly define the problem and find solutions. To solve such a complex problem, one of the main tools is an efficient technology platform. In the current paper, a technology platform was defined to include learning management systems such as Moodle and Google Classroom. The need for an efficient technology platform for TR [13], in turn, leads to a need for an evaluation tool to determine which is the most suitable technology platform for a given TR situation, however, a literature review confirms that research on such evaluation tools is scarce.

To tackle this problem, the goal of this paper is to contribute to improving the evaluation of technology platforms for individual TR projects by using identifiable characteristics of that specific TR. The characteristics of TR will provide insight into constructing a software tool for the evaluation of technology platforms. Thus, the research question is, “Can the TR characteristics be used to construct an evaluation tool for technology platforms?” Having the ability to quickly select the right technology platform, which is an essential tool, would enable research groups to focus on solving the complex problem at hand. Furthermore, an understanding of the definition of the characteristics of TR would enhance the organization of the team and its activities.

The first aim, i.e., Specific Aim 1, is to define the characteristics of TR using a well-established literature meta-analysis. This also includes an analysis of several international TR research centers that are intimately involved in researching complex societal and environmental problems. This part of the research yields a chart of characteristics (core activities and skills) of TR. These characteristics could be particularly useful for designing a framework for training transdisciplinary scholars [5]. The second aim, i.e., Specific Aim 2, is to construct a tool for evaluating technology platforms for use in TR, using a newly developed methodology, including elements of the technology acceptance model and elements of the standard technology evaluation process. This part of the research results in the construction of a software tool for evaluating technology platforms for use in TR.

In this paper, we define the characteristics (core activities and core skills) of TR. This new definition provides insights into TR and an effective tool for analyzing and evaluating any TR project. Furthermore, this definition can be used at the inception of TR as a planning tool. Lastly, these characteristics may be particularly useful for designing a framework or curriculum for training transdisciplinary scholars for careers in TR. More significantly, a software tool is developed to evaluate technology platforms for use in TR projects.

3. Methodology

3.1. Methodology for Determining Characteristics of Transdisciplinary Research (TR) (Specific Aim 1)

3.1.1. Background

PRISMA has become the standard for conducting systematic reviews [14]. Using PRISMA methodology, a systematic review of cloud computing tools for collaborative learning, as well as the opportunities and challenges it presents in the blended-learning environment, were discussed in [15]. Liao et al. followed the exact process of PRISMA to conduct a systematic literature review of the past, present, and future of Industry 4.0 [16]. The same methodology was used by Lytle and Sokol [17] who conducted a systematic review to investigate measures of the food environment. Finally, the PRISMA guidelines were used for the meta-analysis in the context of lung cancer treatment in Chinese patients [18].

3.1.2. Search Strategy

The search focused on the following three components: (1) articles that presented aspects of education, research, development, and collaboration, (2) TR centers, and (3) books on transdisciplinarity. For the articles, the period considered was January 2015 to 12 September 2020. For the books and other sources, the period considered was January 2013 to September 2020. The TR centers considered were based on the list, last updated on 31 July 2020. The search engines used included Google, Web of Science (WOS), Science Direct, Google Scholar, and JSTOR. Keywords and phrases included transdisciplinary education, TR, transdisciplinary collaboration, transdisciplinary skills, transdisciplinary activities, and were published in English. For book searches, the advanced features of the Google search mechanism provide options for narrowing down to “books only” results. The search focused on selecting a few books based on consensus.

3.1.3. Inclusion and Exclusion Criteria

These criteria were used to ensure a consistent approach in assessment with minimal subjective opinion and bias in the selection (see

Table 1. Inclusion and exclusion criteria.

Inclusion and Exclusion	Criteria	Details	Comments
Exclusion	Search engine results	No open access (NOA)	Cannot get access to the full paper (still had a count of 3 after EndNote consolidation)
		Non-related (NR)	Not academic per se (e.g., editorials, meetings, conference reviews, and marketing materials)
		Weakly related (WR)	The paper does not delve into the details of TR skills characteristics, technology tools other than a cursory mention
		Not language English (NLE)	Not language English
	EndNote consolidation	Total citation count (TCC) TCC_10	TCC <= 10
Inclusion	Search engine results	Closely related (CR)	Papers elaborate on transdisciplinary TR skills, activities, characteristics, and technology tools and process
		Partially related (PR)	Papers discuss the collaborative approach of transdisciplinary efforts where the focus is not so much on education and research but other aspects such as business, economy, and patient care.

). An objective review strategy that emphasized evidence-based data collection was adopted, where the papers were reviewed from the perspective of transdisciplinary skills, activities, and characteristics.

3.1.4. Study Selection

The selection was based on the initial screening of all the titles and abstracts of the articles by one member, and then these were reviewed by the other member. For books, the selection was based on consensus, driven by the thought leaders of the TR who wrote the books and the availability of the book for minimal cost. The consensus strategy was also adopted in the selection of the TR centers.

3.1.5. Data Collection

For each included article, three types of information were extracted and entered in an Excel workbook. The first type of data captured was as follows: (1) article title; (2) keywords; (3) database (where the article can be found); (4) source (e.g., journal name, and conference publication); and (5) other meta fields such as the publication year, volume, issue, and total citations. The second type of information captured for the articles was as follows: (1) focus of the article, for example, research, education, and development; (2) vertical domain, for example, medicine, engineering, and sustainability; (3) theoretical solution, identified TR research problems and proposed either a conceptual or a theoretical solution; and (4) practical solution, proposed an implemented or implementable solution to facilitate TR. The third type of information captured was specific data related to the following research sub-questions:

• What are the characteristics found in TR in different research domains?—data included for example characteristics, vertical domain, roles, and stakeholders
• What activities were devised to make the participants improve their transdisciplinary collaboration and development acumen? (data included activities, participants, tools, and technologies);
• What symbolized TR in different vertical domains? (data included for example characteristics, vertical domain, roles, and stakeholders);
• What were the skills learned and used in TR? (data included skills used, skills learned, tools, and technologies).

For books, the data captured included the meta data such as the book title, author(s), and publisher. The other relevant data captured included the characteristics, activities, skills, roles, stakeholders, tools, and technologies. For the TR centers, data captured included meta data such as the location of the center, vision, focus, and other data about their article publications between January 2105 and 14 September 2020, similar to what was mentioned in the data collection for articles.

3.2. Methodology for Constructing a Tool (Specific Aim 2)

3.2.1. Background on Technology Acceptance Model

Ever since the technology acceptance model (TAM) was first proposed [19], it has gained widespread acceptance for evaluating technology acceptance in many domains. It was first used to analyze the increasingly important roles that technologies were playing in companies and organizations. It relies on the following two constructs: (1) perceived usefulness and (2) perceived ease of use which reflects individual beliefs that are affected by external variables. These two constructs affect the attitude toward using technology and behavioral intention to use technology. This model hypothesizes a relationship between external variables and both perceived usefulness and perceived ease of use. Many extensions of the model have been developed to predict technology acceptance by considering different types of external variables that are relevant to the specific contexts in which technology innovation was introduced. For example, Konak et al. [20] discussed the model’s application to the usage of educational technologies such as web-based learning systems, gamification e-books, and computer technology education.

Application of the TAM model to evaluate the use of information technology was discussed in [21] where the TAM model was extended by entering the experiential variable as a moderation variable to see one’s intention in the use of technology especially internet banking. Adapting TAM with extensions to evaluate the potential of the e-learning system was discussed by Persico et al. [22]. Analysis of the use of social media in higher education institutions (HEIs) using the TAM was presented by Dumpit and Fernandez [23]. Their research adopted TAM but included subjective norm, perceived playfulness, Internet reliability, and speed as external constructs. In their paper, Weng et al. [24] discussed using TAM to explore the effects of the information technology (IT) environment on the perceived usefulness, perceived ease of use, and attitude towards using multimedia, and the relevance and influence of these attitudes on behavioral intention. Tsai explored TAM to study the effects of a Course Management System (CMS)-Assisted English as a foreign language writing instruction tool [25]. Kwok and Yang [26] demonstrated, using TAM, students’ intentions to use information and communication technologies tools.

3.2.2. Background on the Technology Evaluation Process

The standardized technology evaluation process (STEP) includes the following four-phase approach to technology evaluation: (1) scoping and test strategy; (2) test preparation; (3) testing, results, and final report; and a fourth, optional phase (4) integration and deployment [27]. Each phase has different objectives, actions, and associated document deliverables. In phase one, a primary activity for a multidisciplinary team is to identify key requirements through scoping. They may further identify a set of products for evaluation against these requirements. The focus of phase two is to develop a full set of evaluation criteria that the products are tested against, for any scenario tests that are performed. In phase three, the evaluation team tests and scores the products against all the test criteria. As the report stresses, the evaluation criteria must be solidified before starting hands-on product testing.

The new methodology developed to address Specific Aim 2 was based on combining elements of TAM and elements of STEP, as shown in Figure 1. TAM was extended with two external variables, skills, and activities that were attributed to TR. Phase one of STEP, which is focused on requirements scoping, formed the basis for the evaluation criteria. The resulting new methodology emphasized the identification of the roles of who would use the technology and understanding their needs (also referred to in this paper as requirements). In a nutshell, the methodology was comprised of the following steps: (1) consider two external variables relevant to TR, i.e., skills and activities; (2) select the following roles in the TR ecosystem that are the primary users of the technology platform, i.e., learner, educator (mentor), researcher, and community stakeholder; (3) for each role, consider the requirements (use cases); (4) model the requirements visually through use cases; (5) produce a set of aggregated requirements; and (6) map each of these requirements to one or more technologies. The mapping of roles, use cases, and technologies formed the basis for the evaluation criteria in the tool for evaluating technology platforms for use in TR. To assist TR teams in using the software tool resulting from the new methodology, the user input was a set of requirements and the tool output was the total score given to a technology platform under consideration. The score essentially represented how best the capabilities of a technology platform match the needs.

4. Results and Discussion

The two methodologies outlined in Section 3 yielded the following: (1) definitions of the TR characteristics (core activities and the core skills), (2) an approach for analyzing existing or new TR projects to identify and specify the evaluation criteria to be used for evaluation of technology platforms for use in TR, (3) identification of the various roles and use cases in TR, and (4) a software tool that enables the evaluation of technology platforms for use in TR. The software tool constructed has an algorithm that provides the backend logic of the software tool for evaluating the technology platforms. It also enables research teams to analyze and design their evaluation criteria driven by the activities. Lastly, the tool has an administrative interface to the software tool that allows other research teams to configure the activities pertinent to their research domains and design the evaluation criteria that can be used to evaluate technology platforms.

4.1. Results and Discussion of the Characteristics of TR (Specific Aim 1)

Following the methodology outlined in Section 3.1, the results of each phase of the systematic literature review are depicted in Figure 2. The search for relevant material consisted of the following three aspects: (1) searching the databases (1 January 2015 to 16 September 2020), (2) consulting the list of TR centers that are maintained (updated list as of 31 July 2020), and (3) reviewing the books available. The search strategy was based on an iterative algorithmic process of building queries using the technique of Venn diagrams. The search approach was as follows:

• The first search with the keyword “transdisciplinary” to obtain results, i.e., Set 1 (4876 records);
• Using the “AND” set combination search was conducted using the combination of “transdisciplinary” AND “education”. The resulting set is Set 2 (928 records);
• The search for “transdisciplinary” AND “research” yielded Set 3 (2806 records);
• The search for “transdisciplinary” AND (“skills” OR “activities” OR “characteristics”) yielded Set 4 (864 records).

Constructing the baseline sets like the one above, and then using the search combination features of WOS, resulted in a list of relevant records as a starter set. Applying the additional selection criteria available in WOS led to the selection of the final set of records for consideration. The other search databases were not identical to the capabilities provided in WOS. However, using the capabilities available, a set of records was obtained for consideration. Using the EndNote tool (plugin available in WOS), all the search results were imported for further consolidation, such as the removal of duplicates. Note that the duplicate removal was done twice, first when the Google Scholar results and the Science Direct search results were consolidated and duplicates removed, and again after all the search results were consolidated.

Searching the databases (Google Scholar, JSTOR, Science Direct, and WOS, timespan 1 January 2015 to 16 September 2020) comprised the following:

• From Google scholar, a total of 197 articles were selected based on advanced search criteria and selection of articles based on (1) access to the article and (2) no books and no editorials.
• From JSTOR, the preliminary list of 212 search results was filtered to 162 based on (1) selection of subject domains which had greater than 10 articles, and no editorials (selected biological sciences, Ecology & Evolutionary Biology, Environmental Science, Sociology, and General Science)
• From Science Direct, using the same search criteria as JSTOR, the initial list of 1123 articles was further filtered to 213 articles based on the timespan of 1 January 2015 to 16 September 2020, selection of only review or research articles, and open access articles.
• From WOS, starting from an initial list of 776 records, the results were filtered to 118 articles based on the following multiple criteria: (1) open access filter resulted in 378 records, (2) WOS core collection filter reduced the count to 343, (3) document type filter (article, or review) reduced the count to 327, (4) the filter Language = English further reduced the count to 285, and (5) the citation filter where average citations per year are greater than 5 finally resulted in 188 records.

All the results were imported into EndNote online which resulted in a total count of 678 articles. Using the “remove duplicates” feature of EndNote, 10 duplicates were removed, and then there were three articles that did not have full-text access. Finally, applying the filter of citations >= 10, downloaded a total of 97 articles for further consideration.

Next, the websites of TR centers were reviewed beginning with a list of TR centers that the authors maintained (updated as of 31 July 2020). A Google search resulted in selecting an additional eight TR centers (top five pages of Google search and selecting those that are not already on the list that is maintained). Each of these TR centers’ websites was reviewed, and a catalog was prepared that captured the following: (1) name of the center, (2) university/institution it belonged to, (3) school/college it belonged to in a university, (4) aim of the center, (5) website and contact information (name and/or e-mail).

Three books by experts in TR and education space (these books contained book chapters by other researchers) were reviewed and cataloged. The field capture includes the meta information related to the book title, authors/editors, publisher, international standard book number (ISBN), and date of publication. The TR center information was captured (two of the books have a TR center associated). From the perspective of the research thrust and aims, the additional information captured included the key ideas mentioned, i.e., stakeholders, skills, characteristics, and activities.

As indicated in Figure 2, artifacts included in the qualitative and quantitative analyses were 50 articles, 37 TR centers, and three books. This created the sample size n = 90. The complete list of reviewed artifacts in the sample is available in Appendix A.

Table 2. Characteristics (core activities and core skills) of transdisciplinary research (TR) mapped to roles.

Skills Requirements	Activity Requirements	Learner	Teacher/Mentor	Researcher	Chief information Officer	Community Stakeholder
Writing	Analyzing problems collaboratively	Collaborate with learners and teachers	Collaborate with learners and teachers	Collaborate with researchers internally	Facilitate security requirements	Provide domain expertise
Collaborative learning	Engage in group discussions	self-directed & on-demand learning	Facilitate class discussions	Collaborate with researchers externally	Facilitate integration requirements collaboratively	Collaborate with the researchers for joint problem statement
Critiquing	Share knowledge	Participate in-class group learning activities	Communicate with class and individual students	Create and participate in communities of practice	Assess Vendor from an Infosec viewpoint	Provide continuous feedback
Listening and engagement	Co-produce knowledge	Participate in project group discussions	Conduct online lectures	Engage in discussion forums
Curiosity	Involve community experts in problem definition and solution generation	Communicate with learners and teachers	Upload courses, tests, and assignments	Upload content
Group discussion	Participate in communities of practice	Take self-assessments	Facilitate activities to build skills	Mentor learners and other researchers
Reflection	Collaborative planning	Take graded assessments	Facilitate student reflections	Conduct online seminars
Interactive communication and collaboration with peers	Seek and give feedback	knowledge share with peers and others	Facilitate student evaluations	Facilitate mutual learning
Presentation	Produce shared language or vocabulary	Make presentations online	Mentor students	Co-produce knowledge
	Write critical reflection	Interact with experts and mentors	Conduct class surveys	Work with stakeholders for joint problem definition

captures the results obtained for Specific Aim 1 that focused on identifying the characteristics (core activities and skills) in TR. The table also depicts the activities from the perspectives of the key roles pertinent to the TR ecosystem.

To add granularity to the characteristics (skills and activities), they can be subdivided into requirements and sub-requirements, as shown in

Table 3. Characteristics (core activities and core skills) of TR classified in terms of requirements and sub-requirements.

Characteristic	Requirement	Sub-Requirements
Activities	Team formation	Announce and communicate research objectives	Collaborate with core team members in the recruitment process
		Search profiles of people	Accept member interests
	Assemble team	Create project groups	Engage in discussions
		Upload project documents
	Joint project definition	Create a study room for the project	Co-produce content
		Individual e-mail	Group e-mail
		Presentations like seminars	Schedule meetings
		Upload content	Asynchronous learning
		Create private collaboration rooms	Synchronous learning
		Share knowledge artifacts in project groups
	Team’s work integration	Create or take graded assessments	Create or take self-assessments
		Discussion and generation of multiple project solutions and alternatives	Facilitate integration of multiple perspectives
		Engage in critiquing
	Communications	Individual and group communications
	Assessments and surveys	Gather and respond to support inquiries	Evaluate project progress
		Create or Conduct surveys	Get or give feedback
Core Skills	Joint learning with peers	Study groups	Knowledge sharing
		Engagement through discussion forums	Self-assessment
		Co-production of knowledge	Joint problem solving
	Essential skills	Writing	Questioning
		Critiquing	Individual reflection
		Group reflection	Presentation
	Communication skills	Interaction with peers	Facilitation
		Conflict resolution

. The rationale is the following: In real-life TR projects, a small core team is usually formed. The members of the team brainstorm the issues surrounding a complex problem and eventually identify a starter set of research objectives. From a project management viewpoint, they typically announce the project along with the research objectives. The project is ready to be launched at this point, where a critical requirement is to identify the right people and assemble an extended TR team. Then, the team’s focus shifts to a joint problem definition. Once the problem is defined, the teams typically are further divided into sub-teams, each of which is responsible for studying the problem from the perspective of their team’s charter and responsibilities. When multiple teams produce results, it becomes important to manage the research work integration across the TR extended team. During the process outlined, there is communication that happens at different levels such as at the individual team level, multiple team levels, and the entire TR team level.

One of the important aspects during the TR project lifecycle is the ability to assess the project progress at different stages and it could include surveys as well. The classification of the activity requirements, along the lines of the team formation the team assembles, joint problem definition, teams’ work integration, communications, and assessments and surveys, provides a structured framework to identify the detailed sub-requirements. Similarly, when addressing the skills requirements, it has been noted that these are typically classified as essential skills and communication in most projects. In the case of TR projects, there is almost a universal mention of mutual learning and shared understanding of the problem. As a result, this research has created a specific skill required, called joint learning with peers, to signify its importance in TR.

An additional benefit of extracting the TR characteristics (core activities and core skills) and mapping them to roles, is the complementary insight it provides to the composition and structure of TR projects. As an example, consider a TR project discussed in reference [28]. In the project, the complex problem addressed was a sharp decline in the fish population of up to 60% in many Swiss lakes and rivers several years ago. This TR project was analyzed using the core activities and core skills and role mapping depicted in Table 2. Furthermore, an attempt was made to map some activities to common functions of a common technology platform. Once analyzed, the following observations were made:

• It is a complex problem that impacts society.
• A core team of participants came together from the fishery authorities, the federal government, and the research institutions.
• The team held a meeting to discuss the issues and establish a preliminary research goal.
• They agreed that it was unclear as to what the causes could be and determine to launch a TR project.
• A wide spectrum of participants from academia, industry, researchers, and other stakeholders were added to the project.
• Community stakeholders were added to the project.
• Joint meetings were held to establish the research goals and specific tasks.
• The transdisciplinary team worked on constructing a joint problem with an emphasis on a common understanding and description.
• Mutual learning facilitated the generation of the joint problem statement.
• Multiple perspectives were put forth by different participants.
• The perspectives were captured as hypotheses.
• Majority voting was used to select the hypotheses for further investigation.
• Multiple sub-projects were initiated to study the issues.
• Research results were presented at conferences.
• Seminars were held to share the research findings periodically.

Table 4. Analysis of a TR project [28] using the defined characteristics (core activities and core skills) of TR.

TR Project	Problem Description	Research Goal	Project Participants	Requirements/Process/Activities	Tool Activity/Sub-Activity Mapping
Decline of fish populations: Integration through the formulation of hypothesis with stakeholder participation	A sharp decline in fish catches. Revenue impacting both anglers and authorities who sell fishing licenses	Document size of fish stocks and their health Identify causes for the decline in fish population Propose counter measures to remedy the situation	Business representatives Federal government Research institutions and scientists from different disciplines, i.e., fisheries science, environmental chemistry, ecotoxicology, physiology, population biology, limnology, hydrology, and climate research Society stakeholders, i.e., fisherman, conservation groups, fishery authorities, operators of sewage treatment plants, and representatives from the Swiss Society of Chemical Industries	Joint meeting among participants to discuss their perspectives of why the fish catch declined. Different explanations were put forth by different disciplines. Experts from different disciplines put together possible causes of the fish catch decline in the form of hypothesis—documents/discussions The hypotheses were arrived at after mutual learning and discussions Majority voting determined if a hypothesis was accepted or not. Cause-effect model was created through mutual learning (discussions/whiteboarding) Seventy-seven sub-projects were created, and project leaders facilitated between teams. Results were exchanged between the teams and through conferences Continuous cooperation with stakeholders Project results published in various platforms including journals and conferences.	Team formation (communicate research goal/objectives, collaborate with core team members in the recruitment process), columns 3 and 4 provide insights about this need Assemble team (create project groups, engage in discussions, upload project documents), columns 4 and 5 provide insights about this need Joint project definition (synchronous/asynchronous learning, individual e-mail, presentations like seminars, share knowledge artifacts in project groups, co-produce content, upload content, create collaboration rooms, schedule meetings, chat, e-mail), column 5 provide insights about this need Research teams work integration (facilitating integration of multiple perspectives; discussion and generation of multiple projects solutions and alternatives; critiquing, reflection), column 5 provides insights about this need Communications (individual and group communication), column 5 provides insights about this need Assessments and surveys (project progress evaluations, feedback, group surveys (polling)), column 5E provides insights about this need

captures the results obtained from the analysis of a real-life project from the perspective of Specific Aim 1. The focus was to understand the societal problem description, the goals of such projects, the participants involved, and the requirements/process/activities that are typically associated with such initiatives. These results provide a complementary perspective and insights into the activities, skills, participants, and community stakeholders. In Table 4, column six captures the mapping of the understanding (represented in columns one through five) onto the activities/sub-activities that were architected and implemented in the software tool.

This section has cataloged the skills, activities, and characteristics that are prevalent in TR. Complex problems, joint-problem definitions, peer-to-peer interactions, knowledge sharing, integrative learning, mutual learning, critiquing, presentations, communication, and stakeholder involvement are recurring activity themes in almost all the areas of research in TR. Almost all the papers, in the study selection, mentioned that a transdisciplinary approach may be the right way to solve complex problems and people involved in such an effort must have or develop the competencies to solve these complex problems.

To define a complex problem needing a transdisciplinary approach, one has to distinguish between a well-defined and ill-defined problem [29]. Two attributes, problem definition, and the goal state help define complex problem solving. Well-defined problems have a clear set of means for reaching a precisely described goal state. For example, in a match-stick arithmetic problem, a person receives a false arithmetic expression constructed out of matchsticks (e.g., IV = III + III). According to the instructions, moving one of the matchsticks will make the equations true. Here, both the problem (find the appropriate stick to move) and the goal state (true arithmetic expression) are defined clearly. Ill-defined problems have no clear problem definition, their goal state is not defined clearly, and the means of moving towards the goal state are not clear.

An example of such a “complex problem” would be illicit drug use [12], which is characterized by contesting problem definitions, unresolvable unknowns, constraints ameliorating the problem, and the temporary nature of the solution to the problem. From the perspective of this research, this type of characterization of a complex problem at a meta-level (higher level of abstraction) is appropriate, but in a transdisciplinary approach, several other attributes would help clarify the challenges. The complex problem is viewed from the perspective of solving society problems involving various actors, users, and stakeholders, to begin with. The knowledge and skills required to solve these problems span multiple research domains and transdisciplinary teams are required to jointly define the problem to be solved. A key aspect of the joint problem domain is the value of collaboration with the community stakeholders who add knowledge from the perspective of practical implementation. It is in such a context that certain skills and activities become paramount.

Activities, such as joint-problem definition where the focus is on participatory research, problem analysis from multiple perspectives, peer-to-peer discussions, writing and presenting, and balancing the conflicting requirements of different stakeholders become critical. The collection of, in turn, leads to the co-production of knowledge, integrative and mutual learning, critiquing, and collaborative problem solving. A complex problem may be ill-defined with vague problem definition and a vague end-goal state, to begin with, but eventually, an end-goal state is reached where a solution is implemented that benefits society and community stakeholders.

In summary, it was shown that there are seven core activities (team formation, assemble team, joint project definition, research teamwork integration, communications, assessments, and surveys) and three core skills set (joint learning with peers, essential skills, and communication skills). These characteristics influence TR projects in terms of team formation, project management, mutual learning, co-production of knowledge, assessments, critical reflection, and communications. Furthermore, an approach for analysis of TR projects, using these characteristics, was shown to provide insights into the activities, skills, and resources used on the project. Such insights would in turn enable the teams to critically reflect on the challenges that they faced and discover the opportunities for improvements for subsequent TR projects. In the case of launching a new TR project, the approach can also be used to understand the applicable activities of the complex problem, assess the team formation requirements, resources needed, and stakeholders involved. Together, they are the drivers for the successful completion of the projects. These characteristics may be particularly useful for designing a framework or curriculum for training transdisciplinary scholars [5,30,31] for careers in TR.

4.2. Results and Discussion of the Tool (Specific Aim 2)

On the basis of the insight from the results of Specific Aim 1 (see Table 2), the previous methodology (Figure 1) can now be recast in terms of the phases approach, as depicted in Figure 3. This starts with the generation of the requirements driven by (1) the identity of the actors and their needs, (2) identifying the core activities and core skills, and finally (3) leads to a tool for evaluating technology platforms for use in TR.

The focus of phase one is to capture the requirements driven by users/actors, skills, and activities. In phase two, the requirements were analyzed and refined using role-use cases. The output of phase two is a set of aggregated requirements. Phase three focused on requirements grooming (some may call it analysis) modeled by representation, called role-use cases, as shown in Figure 4. The roles are the actor symbols, while the use cases are illustrated as ovals in the enclosed box. For any given problem statement, it is imperative to understand the requirements so that a solution can be designed and implemented. One of the modeling techniques available to development teams is the use-case modeling [32], depicted in Figure 4. This model describes the intended functionality of the new system. It is a tool for elicitation of the requirements. It depicts a graphical representation of the software system’s requirements. The key elements in a use case model are actors (roles), and the use cases themselves. A use case is a unit of functionality (a requirement), or a service, in the system. This figure depicts the actors/roles that are relevant in TR, i.e., student/learner, teacher/mentor, researcher, CIO office, and the community stakeholder. The roles are also listed in. In the context of constructing the tool, Figure 4 was effective in capturing the perspective of the various roles and what activities they perform.

The role-use cases were mapped to the technologies that would provide the functionality to realize the respective use cases (see Figure 5). The technology mapping was based on both the literature review, real-life project examples, and the team’s practical domain expertise. The mapping was coded into the backend system logic and served as a supplement to the evaluation criteria that were produced based on the core activities and core skills identified (see Table 2). The evaluation and scoring philosophy, that drove the tool to be constructed, followed a three-step process as follows:

• The technology platform “must” meet all the “must have” requirements. This is the gate to make the Go/No-Go decision. In the case of “must have” requirements, the evaluation engine does not even consider the weights of the requirements (which are mapped on the backend based on what was gathered from the literature). If it is a No-Go decision, the evaluation process stops. If the decision is a Go, then, the evaluation proceeds to step two.
• Execution of this step implies that all the “must have” requirements are present in the technology platform under consideration. This is the step where the “nice to have” and “don’t care” requirements are considered further. For the “nice to have” requirement, the technology platform either provides the capability or does not provide the capability. The “don’t care” requirement is essentially used to break ties in the case of multiple platforms scoring the same.
• Once the user fills out the form in the software tool and submits it, the back-end evaluation engine produces a score and additional insights to the user.

Using these three steps, the resulting evaluation criteria decision map is depicted in Figure 6. In this figure, the empty numbered boxes depict the structure. On the horizontal axis, the requirements are mapped to the boxes “must have” or “nice to have” or “don’t care”. On the vertical axis, the two values are “meets” or “does not meet”. The boxes are numbered 1 through 6 as shown. One can make the following observations: all “must have” requirements have to fall into Box 1. Otherwise, the process stops at the first step and the decision is a “No-Go”. Once step one is cleared, in the second step, the “nice to have” requirements are mapped either into Box 2 or Box 3. The “don’t care” requirements are mapped to either Box 5 or Box 6 accordingly.

Figure 7 depicts a screenshot (only showing the “activity requirements” part of the user interface) of the software tool. In the software tool, the activity requirements, as listed in Table 4, provide the basis for a user to select the specific requirements and mark them as a “must have” or “nice to have” or “not applicable. In addition, note the activities under the following headings: team Formation, assemble team, joint project definition, research teams work integration, communications, and assessments and surveys (also refer to Table 3). This classification reflects the interface of the tool that was architectured and implemented from the insights gained from Specific Aim 1 and from performing analyses of real-life projects (for example see Table 4).

To illustrate the effectiveness of the constructed tool, we used the TR project analyzed in Section 4.1 with results depicted in Table 4. From the perspective of the construction of evaluation criteria for the project (to select the right technology platform), the analysis of the above steps led to the observation that the following activities are relevant to this transdisciplinary project (refer to points 1 through 15, listed Section 4.1, a result of using Table 2 in the analysis of sharp decline in the fish population [28]): (1) team formation (points 2, 3, 5, and 6), (2) team assembly, (points 2, 5, 6, and 7), (3) joint project definition (steps 3 to 9), (4) research teams work integration (points 10, 11, 12, 13, and 15), (5) communication (all points), and (6) surveys (points 12 and 14). The activities mentioned correspond to the activities listed in the tool (see Figure 4). It should also be noted that mapping of the steps to the activity exercise was to provide an illustration and insights as to what aspects of the tool are applicable. The insight gained is that all the activity classifications of the tool apply to this project. The next step in the construction process was to map these to the tool activity/sub-activity which are used in the evaluation tool (column tool activity/sub-activity mapping in Table 4). The output of this step is a list of activity/sub-activities that were relevant to the project and whether each of these was a “must have” or “nice to have” in a technology platform to facilitate the TR for the team. At this point, the information was ready to be entered into the evaluation tool to evaluate the various platforms that could meet the needs of the project team. Figure 8 depicts the screenshot for entering information into the tool and the evaluation results (for illustration purpose only). In summary in the illustration, the five technology platforms, namely, Google Meet, Microsoft Teams, Google Classroom, Moodle, and Blackboard Learn Ultra, were evaluated for the TR project [28] analyzed in Table 4.

5. Conclusions

Transdisciplinary research (TR) is a growing field in many subject domains such as Sustainability, Health and Medicine, Natural Sciences, Social Sciences, Engineering, and Technology. The literature review provided insights into the possible research directions, one of which is technology enablement to facilitate TR. It was clear from the literature review that there is a need to increase the adaption of tools, such as technology platforms, to enhance and support TR. However, there was no clear evidence of what type of technology platforms are most appropriate for TR. Furthermore, it was not clear what type of process a team should adopt to systematically understand the needs of TR teams and document the requirements that the technology platform must support. In essence, this research undertaking concluding from the literature that there are no effective evaluation criteria for the selection of technology platforms for use in TR.

The research question posed was, “Can the TR characteristics be used to construct an evaluation tool for technology platforms?” The question captures the fundamental aspect of what goes into the selection of a technology platform. A technology platform, in any problem environment, facilitates the needs of the various participants. These needs are characterized by the activities and skills that apply to the problem domain. For example, a technology platform in an online education environment must support the faculty needs such as course upload, student assessments, and facilitating group discussions. Likewise, it should support student needs such as self-paced learning, peer-to-peer interaction, and learning communities. The research question led to the formulation of the following two specific aims: (1) define the characteristics of TR and (2) construct a tool for evaluating technology platforms for use in TR. The evaluation of the technology platforms for use in TR includes the criteria related to how best a platform meets the TR characteristics (core activities and core skills). In addition to the activities and skills, a problem domain is characterized by various people (in different roles) who participate in various activities and apply and/or learn new skills. In the process of executing the specific aims, the research captured these other characteristics as well.

From Specific Aim 1, it was shown that there are seven core activities (team formation, assemble team, joint project definition, research teamwork integration, communications, assessments, and surveys) and three core skills set (joint learning with peers, essential skills, and communication skills). The two sets of characteristics influence TR projects in terms of team formation, project management, mutual learning, co-production of knowledge, assessments, critical reflection, and communications. Furthermore, an approach for analysis of TR projects, using these characteristics, was shown to provide insights into the activities, skills, and resources used on the project. Such insights would, in turn, enable the teams to critically reflect on the challenges that they face and discover the opportunities for improvements for subsequent TR projects. In the case of launching a new TR project, the approach can also be used to understand the applicable activities of the complex problem, as well as assess the team formation requirements, resources needed, and stakeholders involved. Together, they are the drivers for successful completion of projects. The analysis of TR projects, in effect, reinforced the significance and relevance of Specific Aim 1. Furthermore, these characteristics may be particularly useful for designing a framework or curriculum for training transdisciplinary scholars for careers in TR.

From Specific Aim 2, a software tool was developed to evaluate technology platforms for use in TR projects. The software tool answered the research question posed and it demonstrated a practical approach for evaluating a technology platform for use in TR. In essence, this research resulted in a tool that could be used for selecting a technology platform for a conceptualized or planned TR project. Furthermore, the tool was designed and implemented in such a way that it could be extended to other research domains, as long as, using the methodology proposed in this paper, the activities are defined, technology mapping is performed, and other configurations are adjusted (such as the assignment of weights for activities) using the “admin function” of the tool. The contributions of this work, to the goal of improving the use of technology platform in TR, are the following: (1) the identification of the core activities and core skills; (2) a methodology to construct a tool to evaluate technology platforms for use in TR; (3) a process to analyze TR projects from the perspective of participants, requirements, and activities; and (4) a software tool for evaluating technology platforms for use in TR.