Open-Ended and Closed-Ended Measures of Religious/Spiritual Struggles: A Mixed-Methods Study

Abstract

Religious and spiritual struggles are typically assessed by self-report scales using closed-ended items, yet nascent research suggests that using open-ended interviews and prompts may complement and advance assessment and theories. In the current mixed-methods study, undergraduate participants (N = 976) completed open-ended descriptions of their religious and spiritual struggles, the Religious and Spiritual Struggles Scale (RSS), and a quantitative measure of religious belief salience. Qualitative description showed that the themes emerging from open-ended descriptions generally fell within the broad domains of the RSS though some descriptions reflected more contextualized struggles. Scores derived from the open-ended responses to assess RSS domains achieved evidence of reliability, and quantitative correlational analyses provided support for convergent and discriminant validity with the RSS. Correlations revealed a mix of similar and divergent associations between methods of assessing religious and spiritual struggles and religious belief salience. Open-ended descriptions of religious and spiritual struggles may yield reliable and valid information that is related to but distinct from assessments relying on closed-ended items.

1. Introduction

Religion and spirituality (r/s) are sources of comfort, support, and security for many individuals (Koenig 2012; Wilt et al. 2018); however, people also commonly experience r/s struggles, which are defined as tensions, concerns, and negative thoughts and emotions around r/s (Exline 2013; Exline and Rose 2013; Pargament and Exline forthcoming). R/s struggles may occur in several domains: (a) supernaturally-focused struggles focused on God, gods, or supernatural evil, (b) interpersonal r/s conflicts with individuals or institutions, and (c) intrapsychic turmoil around morality, doubts about one’s r/s beliefs, or questions about ultimate meaning or purpose in life (Exline and Rose 2013). Measurement of r/s struggles is typically done with self-report scales using closed-ended items and Likert-type scales designed to assess either specific domains of r/s struggles individually (Krause and Ellison 2009) or multiple aspects of r/s struggles simultaneously (Exline et al. 2000; Pargament et al. 2011).

Having a diverse array of reliable and valid self-report scales including closed-ended items is important for assessing the breadth and depth of r/s struggles efficiently and flexibly; yet there may be at least a few good reasons to complement this work by assessing r/s struggles with open-ended items as well. First, measures designed using closed-ended and open-ended items may overlap and/or diverge in terms of struggle domain content. Second, open-ended assessments might reveal richer, more idiosyncratic, and more nuanced descriptions of r/s struggles. Third, open-ended assessments may provide incremental validity beyond closed-ended assessments for predicting important outcomes.

The present project had three aims relevant to advancing the measurement and theory of r/s struggles within a mixed-methods design. First, we developed a qualitative coding system for open-ended descriptions of r/s struggles. Second, we derived quantitative scores from the qualitative codes and examined the convergent and discriminant validity of the emergent codes against the Religious and Spiritual Struggles Scale (RSS), which measures multiple domains of r/s struggles with closed-ended items (Exline et al. 2014). Third, we compared quantitative associations between open-and closed-ended assessments of r/s struggles to a measure of religiousness (Blaine and Crocker 1995).

1.1. Religious/Spiritual Struggles

As reviewed in Exline et al. (2014), most measures of r/s struggles (using closed-ended items) assess one struggle domain: for example, divine (Wood et al. 2010), doubt (Krause and Ellison 2009), or interpersonal (Ellison et al. 2009). The RSS measures multiple domains of r/s struggles (but not exhaustive of all domains) that appear commonly in the theoretical and empirical literature (Exline et al. 2014). The RSS assesses six domains with four or five items each: anger at God or feeling punished or abandoned by God (divine struggle), feeling tormented by evil spirits or the devil (demonic struggle), disagreements with other people about r/s or anger toward organized religion (interpersonal struggle), doubts and questions about one’s faith (doubt-related struggle), struggles to follow moral principles and guilt about perceived failures (moral struggle), and questions about a deeper, ultimate meaning in life (ultimate meaning struggle). Though each subscale is moderately, positively correlated with each other subscale, confirmatory factor analyses support distinguishing among the six subscales (Exline et al. 2014; Stauner et al. 2016b).

Quantitative research using self-report assessments with closed-ended items has been instrumental in establishing r/s struggles as an important topic of study. This work has shown robust links between higher levels of r/s struggles across domains and various indicators of poorer mental health and physical health, both concurrently and over time (for reviews, see Exline 2013; Exline and Rose 2013; Pargament and Exline, forthcoming; Stauner et al. 2016a; Wilt et al. 2018). Regarding the RSS in particular, in samples of college students and adults from the U.S., most subscales as well as a composite r/s struggles score are associated with higher levels of depressive and anxious symptoms, as well as lower levels of life satisfaction and meaning in life (Exline et al. 2014; Stauner et al. 2016b; Wilt et al. 2017a). When statistically controlling for overlap among domains, ultimate meaning struggles typically have the strongest predictive value for mental health (Exline et al. 2014; Stauner et al. 2016b). There is initial evidence that these associations largely replicate in samples of Israeli Jews (Abu-Raiya et al. 2016) and Israeli–Palestinian Muslims (Abu-Raiya et al. 2015). Furthermore, emerging evidence suggests that r/s struggles mediate the association between exposure to life stressors and negative adjustment (Evans et al. 2017; Pomerleau et al. 2019).

Though fraught with emotional pain, r/s struggles are thought of as a normal and natural part of r/s life (Pargament 1997, 2007). From this perspective, r/s struggles represent turning points or forks in the road during r/s development, capable of resulting in growth and/or decline. An emerging body of work supports this view, showing that person characteristics and adaptation strategies predict mental and spiritual trajectories in response to r/s struggles (for a review, see Stauner et al. 2020). For example, regarding person characteristics, religiousness has emerged as a particularly consistent predictor of positive mental and spiritual outcomes of r/s struggle (Wilt et al. 2017b; Wilt et al. 2016; Exline et al. 2017). Regarding adaptation strategies, people who responded to r/s struggles with higher levels of positive religious coping (i.e., collaborating with God, seeking support from others, finding meaning) showed higher levels of adjustment on indicators of mental and spiritual well-being (Wilt et al. 2019; Exline et al. 2017; Desai and Pargament 2015).

1.2. Assessing Religious/Spiritual Struggles with Open-Ended Items

1.2.1. Summary of Previous Qualitative Research

In contrast to the substantial and relatively rapid advances made using closed-ended items to assess r/s struggles, little research has relied on open-ended assessments of r/s struggles, with a few exceptions. The studies reviewed next provide a glimpse of the unique and valuable information that may be gleaned from using open-ended assessments derived from different qualitative research methods. The study presented in this manuscript builds on the findings of these studies.

In one mixed-methods study (Breuninger et al. 2019), a sample of 178 veterans gave brief, written descriptions of r/s struggles that were coded into either one of the six RSS domains, an “other r/s struggle” category that did not fit with the RSS domains (e.g., not having time to devote to r/s), or a “non-r/s struggle” (e.g., health struggle with no r/s connotation). The prevalence for each domain was: moral (16% of participants), interpersonal (12%), doubt (11%), and ultimate meaning (10%), divine struggles (8%), demonic (5%), other r/s struggle (10%), and non-r/s struggle (28%). It is possible that many Veterans perceived their non-r/s struggles (e.g., PTSD) as being relevant to r/s but did not state this connection explicitly. Veterans in this study also completed the RSS. Frequency was assessed on from 1 (not at all) to 5 (a great deal), allowing for comparisons of prevalence across the two methods. Similar to the open-ended assessments, moral struggles were endorsed at relatively high rates (M = 2.68), and the domains of interpersonal (M = 2.11), doubt (M = 2.08), and ultimate meaning (M = 2.27) were endorsed at moderate rates. Divine struggles were endorsed at relatively low rates on the RSS (M = 1.82), whereas demonic struggles were endorsed at relatively high rates (M = 2.45).

A subsample of 22 Veterans from the above study also completed more in-depth, semistructured interviews (lasting 45 min to one hour) about their r/s struggles (Fletcher et al. 2020), which were coded into RSS domains and subcategories within each domain. Divine struggles included questioning God’s choices and feeling abandoned. Demonic struggles comprised direct perceived experience with the devil and perceptions of battles between good and evil. Interpersonal struggles encompassed avoiding others, feeling misunderstood, experiences of discrimination, negative emotions toward others, and disturbing thoughts about relationships. Moral struggles fell into the categories of regretting past behavior, questioning the goodness of military actions, and gender/sexual identity orientations in conflict with r/s teachings. Struggles around ultimate meaning concerned general ultimate meaning and attempts to understand God’s plan for life. Finally, Doubt struggles involved general doubts, apathy about r/s, and questioning the existence of God.

Another study took a more in-depth look at interpersonal struggles (termed relational struggles) in a sample of 198 religious families including 476 participants in all (Dollahite et al. 2019). Semistructured interviews (lasting approximately two hours on average) conducted with multiple family members at the same time were aimed at exploring the challenges faced by r/s families. Team-based qualitative coding (Marks 2015) of interview transcripts identified four salient themes (the percentages listed next represent the proportion of respondents who listed the theme as a current concern). First, burdens (e.g., not living up to r/s responsibilities) were present in 23% of responses. Disunities (e.g., interpersonal conflicts ranging from arguments to divorce) were present in 10% of responses. Abuses (e.g., feeling attacked verbally) were present in 6% of responses. And offenses (e.g., misunderstandings) were present in 3% of responses.

Other studies, which focused on in-depth characterization of r/s experiences using small sample sizes, have employed qualitative coding procedures to identify themes relevant to r/s struggles. First, in a study of 10 Christian women who survived traumatic experiences (de Castella and Simmonds 2013), Interpretative Phenomenological Analysis (Eatough and Smith 2008) of semistructured interviews showed that doubting r/s beliefs and meaning-making through suffering were salient themes. In another study, 18 Christian emerging adults (Bailey et al. 2016) completed the Relational Spirituality Interview (Hall 2014), a two-hour semistructured interview exploring various r/s topics, such as conversion, relationships with others, perceived experiences with God. Grounded theory methodology (Corbin and Strauss 2014) revealed themes that reflected a variety of divine struggles: difficulties developing close relationships with God, feeling emotionally distant from God, negative feelings toward God, and emotional insecurity in relationships with God. Finally, a study of 13 participants used grounded theory to analyze semistructured interviews focusing on a spiritual emergency, a non-ordinary state of consciousness that centers on r/s themes and commonly involves sensory, perceptual, and emotional changes that can be profoundly transformative and distressing (Sedláková and Řiháček 2019). Results showed that some participants mitigated the meaning of the spiritual emergency, whereas others struggled to make meaning from their experiences and were eventually able to incorporate the spiritual emergency into their identities. This last example may deal more with coping than r/s struggle itself but still includes themes relevant to struggles with ultimate meaning.

1.2.2. Implications of Previous Research

The limited number of studies using open-ended assessments have enriched the study of r/s struggles in several ways. First, thematic content has been relevant to RSS domains, which provides further evidence that the RSS captures many salient domains of r/s struggles. Second, in Veterans, the relative prevalence of RSS domains across assessment methods may be similar for some domains (interpersonal, moral, doubt, and ultimate meaning) and different for others (divine, demonic), inviting research to explore potential explanations. Third, a few studies revealed more granularity within RSS domains. Finally, studies have elucidated context-specific content of all domains of RSS struggles in Veterans and for doubt and ultimate meaning struggles related to the experiences of trauma and spiritual emergency.

The advances made by using open-ended assessments to study r/s struggles are a microcosm of the general ways in which this method complements research using closed-ended assessments. Research using open-ended assessments may result in rich, descriptive findings that capture contextualized experiences and the phenomenology of those experiences (Neergaard et al. 2009). These findings may converge or depart from quantitative findings, as is commonly the case in mixed-methods research (Hanson et al. 2005). Quantitative and qualitative findings may inform each other over the course of theory development.

1.3. The Present Mixed-Methods Study

We conducted a large-scale, cross-sectional, mixed-methods study involving undergraduate participants who were currently experiencing an r/s struggle. Participants completed open-ended descriptions of the most important r/s struggle they were facing, and these descriptions were qualitatively coded into different r/s struggle categories. Closed-ended, quantitative assessments included the RSS and religiousness. These methods may facilitate the general advances yielded by open-ended assessments described above (e.g., providing rich descriptive data, informing quantitative findings, and contributing to refinement of theory) above while also building specifically on previous work using open-ended assessments in at least four ways.

First, previous studies typically involved small samples and focused on specific faiths (predominantly Christian), populations (e.g., Veterans), roles (e.g., families), or experiences (e.g., trauma, spiritual emergency) in which r/s struggles may emerge. The current study employed a large number (nearly 1000 participants) of undergraduates from somewhat diverse faith and demographic backgrounds and assessed experiences of any important r/s struggle; these features may increase the generalizability of our findings compared to previous research.

Second, in a purely qualitative aim, we attempted to create a relatively comprehensive coding manual for assessing themes arising from open-ended descriptions of r/s struggles. We used the RSS domains as a framework for identifying themes and also allowed for the identification of subthemes outside of RSS domains. This approach combined strengths of previous studies that have either coded for broad RSS domains explicitly or allowed for more specific r/s struggle themes to emerge organically without any guiding theoretical framework. By taking this hybrid approach, we hoped to reveal a more personal, granular description of r/s struggles within and potentially outside of the most inclusive framework (to date) for classifying struggles.

Third, we aimed to validate our coding system against the RSS itself, which is facilitated by using a mixed-methods approach. We anticipated being able to categorize open-ended descriptions of r/s struggles according to RSS domains (by assigning a quantitative score to each category), which would then allow for a quantitative analysis (e.g., computing correlations) of the associations between coding categories and quantitative scores on the RSS. This marks a point of integration between the purely qualitative descriptions, the quantitative scores derived from the descriptions, and quantitative associations among measures. Positive correlations between r/s struggles in the same domain across methods would constitute evidence of convergent validity, whereas a lack of correlations between r/s struggles across domains (or weaker correlations than observed within-domains) would constitute evidence of discriminant validity. No previous studies have used a mixed-methods approach to examine whether scores derived from open-ended codes achieve evidence of quantitative indices of validity.

Fourth, in a purely quantitative aim, we explored associations between overall religiousness to open- and closed-ended assessments of r/s struggles. Previous work has shown that religiousness relates positively to demonic and moral struggles but negatively to ultimate meaning struggles (Exline et al. 2014). Evidence of similar associations between religiousness to the RSS and our codes would constitute evidence supporting the criterion validity of our coding system. Additionally, codes may tap into unique information in terms of r/s struggle content and thus have different associations with religiousness; this pattern of findings could yield insight into what kinds of r/s struggles are not captured by the RSS.

1.4. Open Practices Statement

The study was not formally preregistered. The coding manual, de-identified data and data analysis scripts are posted at https://osf.io/a49gk/?view_only=5545d5beaba54e63b6c9253ccf84eb8a (see Supplementary Materials); access to the data and code are widely available.

2. Method

2.1. Research Design Overview

To meet our goals of developing and performing initial validation of an assessment of r/s struggles using open-ended items, we used a mixed-methods triangulation design: data transformation model (Creswell and Clark 2017). Following this design, (a) qualitative and quantitative data were collected simultaneously, (b) the qualitative data were transformed into quantitative data, (c) the two sets of quantitative data were compared, and (d) the quantitative and qualitative data were interpreted. Although we did not intentionally employ a formal qualitative coding method, our coding protocol included elements that were similar to qualitative descriptive methods (Sandelowski 2000, 2010). We coded open-ended descriptions of r/s struggles and then transformed these codes to quantitative measures. Quantitative indices were compared in the context of a cross-sectional, correlational study. Results from qualitative coding and quantitative analyses were interpreted in light of previous theory and research on r/s struggles.

2.2. Participants

We recruited participants who were enrolled in introductory psychology courses across three universities in the United States: a public university and private research university both in the Great Lakes region, and a private Christian university in the western United States. All methods for the study were approved by each university’s IRB. Participants received partial credit toward the class research participation requirement for participating; participants had the option to choose other research studies or complete a non-research option (e.g., writing a paper) to fulfill this requirement. A total of 3106 students enrolled, of which 976 (31.4%) reported a specific r/s struggle and were included in the current sample. Seventy percent of participants identified as women, and 30% identified as men. Mean age was 19.1 (SD = 2.1). Self-identified ethnicities included Caucasian (75%), Asian/Pacific Islander (15%), Latino/Hispanic (10%), Black/African American (6%), and other ethnicities (2%). The total percentage is greater than 100% because participants were able to choose more than one ethnicity. Self-reported religious affiliations were Christian (85%; 16% Catholic, 24% Protestant, 44% unspecified Christian), Jewish (1%), Hindu (1%), Muslim (1%), and Buddhist (0.5%). Non-religious identifications included agnostic (5%), “no affiliation” (4%), and atheist (2%). Other responses about r/s (or non-r/s) identification (“spiritual,” “other,” or “unsure”) comprised two percent of individuals.

2.3. Measures

2.3.1. Open-Ended Assessment and Coding of Religious and Spiritual Struggles

Participants were asked to “please describe a religious/spiritual struggle that you have experienced over the past few months…If possible, try to choose the struggle that you see as most important or serious. But even if you focus on a smaller struggle, that is OK.” Participants typed responses into a text box. The mean response was 15.3 words (SD = 13.9); most participants wrote one or two sentences, whereas some participants wrote as little as a few words or as much as a few paragraphs.

Developing a Coding Manual

Phase one of our coding was highly similar to qualitative description methodology (Sandelowski 2000, 2010). Qualitative description stays close to the data and involves low inference relative to other qualitative methods. Data are simply described without attachment to any particular theory, though some interpretation may be required to translate text to meaning units (i.e., codes).

In the first phase of our coding, two research assistants (RAs; the second and third authors, JTT and PJ) assigned a descriptive code to each response independently; RAs were instructed to simply describe the most salient r/s struggle themes, which fits well within the framework of qualitative description. Over the course of this initial coding, RAs discussed codes with the first author (JAW) in one-on-one and joint meetings, which occurred approximately once per week until this phase of coding was complete. In the initial meetings, relatively few participants (e.g., 30–50) were coded each week so that ample time was given for training coders and developing strategies and heuristics for coding. When coders became comfortable with the procedure, they coded relatively more participants (e.g., 100–200) per week.

Following the completion of coding, the first author then reviewed descriptive codes and developed the thematic coding manual independently (for the full coding guidelines, see https://osf.io/a49gk/?view_only=5545d5beaba54e63b6c9253ccf84eb8a). The first author assigned a label to each theme and wrote a short description of the coding criteria for the theme. When applicable, individual themes were grouped according to one of the six RSS struggle domains. For example, the theme of “concern about angering God” was grouped into divine struggle, whereas the theme of “meaninglessness of life” was grouped into ultimate meaning struggle. Though this step technically departed from pure qualitative description, it facilitated the mixed methods aim of deriving quantitative indices used for validation purposes. Indeed, one advantage of qualitative description is its ability to tie in with mixed methods research because it yields information useful for scale development (Neergaard et al. 2009). Furthermore, grouping individual codes according to RSS domains was informed by the quantitative and qualitative research described in the Introduction indicating that the RSS domains provide relatively comprehensive coverage of r/s struggle domains.

Most individual themes were grouped into RSS struggle domains. Because only about 1% of responses were classified as demonic struggles, the manual did not include this domain. One theme did not explicitly describe issues within an RSS domain (commitment to r/s activities), and three themes did not specifically reference r/s issues (personal suffering, bereavement, and struggles with other people that did not specifically reference r/s issues). Finally, the manual included a category for miscellaneous or nonsensical responses.

Quantitative Coding

Upon completion of the coding manual, RAs completed quantitative coding. Specifically, they scored participant responses for the absence (0) or presence (1) of each theme. Multiple themes could be assigned to each response. RAs met with the first author (in one-on-one and joint meetings, occurring approximately once per week) over the course of this coding process to discuss questions regarding application of codes. Again, initial weeks involved coding relatively few participants (to train coders) as compared to latter weeks of the process. Discrepancies across coders were discussed, but codes were not changed once assigned due to concerns about artificially inflating intercoder reliability coefficients (O’Connor and Joffe 2020).

We used individual theme scores to compute an index for the absence or presence of five RSS domains (excluding demonic) for each participant. Absence (0) at the domain level was assigned for participants who received an absence code for each specific theme in the respective domain. Presence (1) at the domain level was assigned for participants who received at least one presence code for any specific theme in the respective domain. Therefore, in each r/s struggle domain, each participant either received a 0 or 1 score for each coder. Coder ratings were averaged to compute scores for each participant; thus, each participant could receive a score of 0 (absent for both coders), 0.5 (present for one coder), or 1 (present for both coders). We used these scores to assess interrater reliability for each r/s struggle domain.

2.3.2. Closed-Ended Assessments

The Religious and Spiritual Struggles Scale

For the RSS (Exline et al. 2014), participants read, “Over the past few months I have…” followed by 26 closed-ended items that assess six types of struggle: divine (e.g., “felt angry at God”), demonic (e.g., “felt attacked by the devil or evil spirits”), interpersonal (e.g., “felt angry at organized religion”), moral (e.g., “felt torn between what I wanted and what I knew was morally right”), doubt (e.g., “felt confused about my religious/spiritual beliefs”), and ultimate meaning (e.g., “felt as though my life had no deeper meaning”). Participants responded to items from 1 (not at all) to 5 (a great deal). Scores on the RSS have shown high reliabilities in previous studies (Exline et al. 2014; Stauner et al. 2016b).

Religious Belief Salience

We assessed individual differences in the importance of r/s beliefs with the Religious Belief Salience Scale (Blaine and Crocker 1995). This scale includes four items (e.g., “Being a religious person is important to me”) on a scale from 0 (does not apply; I have no religious/spiritual beliefs) to 11 (strongly agree), we omitted one item from the original scale that assumed belief in God. Scores on the modified scale have shown high reliabilities in previous studies (Exline et al. 2014; Stauner et al. 2016b).

3. Results

3.1. Open-Ended Responses

3.1.1. Qualitative Themes

Table 1. Types of Religious/Spiritual Struggles Described in Open-Ended Responses.

R/S Struggle Domain and Specific Themes Emerging from Coding Procedures	Example Response	Item(s) Containing Similar Content on the Religious and Spiritual Struggles Scale
Divine Struggles
Conflict regarding God’s absence	“Not receiving God’s help through prayer”	Felt as though God had abandoned me; Felt as though God had let me down
Questioning God’s role in one’s life	“Trying to decipher what God is telling me”
Questioning God’s role in suffering	“Believing in God when bad things happen to good people”
Personal devotion to God	“I sometimes struggle to put God and religion first in my life”
Worthy of God’s love	“If I sin so much how could God love me?”	Questioned God’s love for me
Concern about angering God	“Feeling that God is angry with me”	Felt as though God was punishing me
Anger at God	“bitterness toward God because of the death of my father”	Felt angry at God
Interpersonal Struggles
General interpersonal conflict	“Arguing with my Catholic friend about what God thinks about homosexuals”	Had conflicts with other people about religious/spiritual matters; Felt hurt, mistreated, or offended by religious/spiritual people
General displeasure with religion	“Religious people feeling that their beliefs are superior to others”	Felt angry at organized religion
Religious intolerance	“Religious people’s views about abortion, LBGT rights, and other controversial matters”
Conflict regarding the morality of others’ actions/beliefs	“…the religious, such as nuns and priests, can be hypocrites”
Struggling with disparate beliefs	“I have heard many conflicting beliefs of people that are from the same religion as my own”
Feeling like an outsider	“Going to a Bible school where everyone seems to know a lot more than I do”	Felt rejected or misunderstood by religious/spiritual people; Felt as though others were looking down on me because of my religious/spiritual beliefs
Moral Struggles
Difficulty abstaining from immoral specific behavior/thoughts/feelings	“Temptation with Pornography”	Felt torn between what I wanted and what I knew was morally right
Difficulty with general transgressions	“I always feel horrible after doing even little things wrong”	Wrestled with attempts to follow my moral principles; Worried that my actions were morally wrong
Positive moral values	“Acting upon what I know to be right”	Felt guilty for not living up to my moral standards
Sexual orientation	“being pansexual”; “being attracted to men in addition to women”
Understanding morality	“the topic of homosexuality—Is it ok? Is it not? I’ve read arguments for both cases”
Doubt Struggles
Doubt about God’s existence	“Debating if God exists or not”
Questioning fate	“Does fate exist? Or do we make our own fate?”
Wondering about the nature of God	“whether God is a man in the sky versus a higher power of some sort”
Doubts about religious truth	“believing everything that occurs in the Bible”
General doubt / confusion	“Doubts in what to believe”	Felt confused about my religious/spiritual beliefs; Struggled to figure out what I really believe about religion/spirituality; Felt troubled by doubts or questions about religion or spirituality; Worried about whether my beliefs about religion/spirituality were correct
Struggles with Ultimate Meaning
Lack of deep meaning in life	“whether my life has any spiritual meaning or value”	Felt as though my life had no deeper meaning
Meaninglessness of life	“whether there is no meaning or value to anyone’s life”	Questioned whether life really matters; Had concerns about whether there is any ultimate purpose to life or existence
Finding purpose in life	“My life doesn’t seem to be headed in any direction”	Questioned whether my life will really make any difference in the world
Self-worth	“Not feeling beautiful in the eyes of people. Not important enough.”
Other Themes
Commitment/time	“I have struggled to get myself to start going to church again.”
Personal suffering	“I have felt alone over the past few months”
Loss/bereavement	“grandfather dying”
Interpersonal (no religious/spiritual connotation)	“I have a bad relationship with one of my friends”
Lack of r/s struggle or denying r/s struggle	“I’m sure I’ve had one, but I can’t think of it right now—clearly it wasn’t that important”

shows descriptive labels and example responses for each of the 32 themes in the coding manual. Most themes were grouped by RSS domain: divine (seven themes), interpersonal (six), moral (five), doubt (five), and ultimate meaning (four). Of the five themes that did not fit within an RSS domain, one (commitment/time) was explicitly concerned with r/s. Table 1 also lists items from the RSS that contain content similar to the specific themes (the matching of items to themes was discussed among all authors). For example, the RSS item, “Felt as though God had abandoned me,” was deemed to have similar content to the theme, “Conflict regarding God’s absence”. Of the 27 themes, 14 were thought to contain content similar to RSS items; all of the 22 RSS items assessing five RSS domains (excluding demonic) were matched to a specific theme. A few themes (e.g., general doubt/confusion) were thought to reflect content similar to multiple RSS items.

3.1.2. Quantitative Scores Derived from Themes: Descriptive Statistics for RSS Struggle Domains

Participants reported each RSS struggle domain commonly with the exception of ultimate meaning struggles (see

Table 2. Types of Religious/Spiritual Struggles Emerging from Open-Ended Responses: Descriptive Statistics and Reliabilities.

R/S Struggles in Open-Ended Responses Included in the Religious and Spiritual Struggles Scale	M	SD	Skew	Kurtosis	ICC3k
Divine	0.22	0.36	1.32	0.18	0.70
Interpersonal	0.13	0.31	2.18	3.12	0.83
Moral	0.20	0.37	1.49	0.46	0.84
Doubt	0.21	0.37	1.39	0.24	0.79
Ultimate Meaning	0.05	0.18	4.19	17.43	0.65
Other Struggle
Suffering	0.03	0.13	4.35	19.81	0.19
Loss	0.02	0.11	7.69	61.14	0.75
Commitment	0.05	0.18	4.21	17.52	0.67
Interpersonal (non r/s)	0.02	0.11	6.37	44.16	0.42

ICC3k = Intraclass correlation reliability for the average of fixed coders. For all struggles, Min = 0 and Max = 1.

). Each RSS domain showed substantial between-person variation and had acceptable to high levels of reliability. Responses falling outside of RSS domains were relatively uncommon. Loss and Commitment had acceptable levels of reliability, whereas Suffering and Interpersonal (non-r/s) did not. Skews and kurtoses indicated non-normal distributions, with departures from normality being more severe for more uncommon struggles.

3.2. Quantitative Scores Calculated from Closed-Ended Assessments: Descriptive Statistics

Table 3. Descriptive Statistics for Closed-Ended Assessments.

	M	SD	Skew	Kurtosis	α
Religious and Spiritual Struggles Scale
Divine	1.62	0.81	1.79	3.12	0.90
Interpersonal	1.92	0.86	1.10	0.72	0.84
Moral	2.86	1.07	0.15	−0.85	0.88
Doubt	2.13	1.04	0.88	−0.02	0.90
Ultimate Meaning	2.10	1.07	1.02	0.18	0.88
Religious Involvement
Religious Belief Salience Scale	8.34	3.21	−1.17	0.23	0.97

For all struggles, Min = 1 and Max = 5. For Religious Belief Salience, Min = 1 and Max = 11.

shows descriptive statistics for all measures using closed-ended items. Participants reported moderate levels of moral struggles and modest levels of other r/s struggles. On average, participants reported high levels of religious belief salience and participating in r/s activities a few times per week.

3.3. Quantitative Associations between Open-Ended and Closed-Ended Assessments of Religious and Spiritual Struggles

As described previously, participants were assigned a score of 0, 0.5, or 1 for each RSS domain code. We assumed only that these scores were rank-ordered (i.e., higher scores correspond to higher rank) and therefore completed Spearman rank-order correlations between the coding domains and the RSS scale domains (see

Table 4. Spearman Correlations between Religious and Spiritual Struggle Scale Domains and Religious/Spiritual Struggle Domain Codes.

	Religious/Spiritual Struggle Domain Codes (Open-Ended Measure)
	Divine	Interpersonal	Moral	Doubt	Ultimate Meaning
Religious and Spiritual Struggles Scale Domains (Closed-Ended Measure)
Divine	0.16	−0.12	−0.11	0.03	0.05
Interpersonal	−0.09	0.12	−0.02	0.03	−0.03
Moral	−0.02	−0.14	0.26	−0.06	−0.05
Doubt	−0.07	−0.01	−0.10	0.30	−0.01
Ultimate Meaning	−0.04	−0.05	−0.15	0.20	0.10

Note: Correlations > |0.10| have p < 0.01.

). All within-domain correlations were positive, whereas the only positive cross-domain correlation was between the Doubt code and the Ultimate Meaning subscale of the RSS. We also calculated partial correlations within domains (controlling for other RSS scales) to examine unique associations between RSS subscales and domain codes. For example, the partial correlation between RSS-divine and the divine code (controlling for other RSS subscales) was r = 0.19, p < 0.001. Other within-domain partial correlations were: interpersonal (r = 0.15, p < 0.001), moral (r = 0.30, p < 0.001), doubt (r = 0.30, p < 0.001), and ultimate meaning (r = 0.11, p < 0.001).

3.4. Associations between Open-Ended and Closed-Ended Assessments of Religious and Spiritual Struggles with Religious Belief Salience

Table 5. Spearman Correlations Relating Closed-Ended and Open-Ended Measures of Religious and Spiritual Struggles to Religious Belief Salience.

Religious and Spiritual Struggles Scale Domains (Closed-Ended Measure)	r	Religious/Spiritual Struggle Domain Codes (Open-Ended Measure)	r
Divine	−0.05	Divine	0.16
Interpersonal	−0.04	Interpersonal	−0.17
Moral	0.18	Moral	0.15
Doubt	−0.25	Doubt	−0.25
Ultimate Meaning	−0.34	Ultimate Meaning	0.04

Note: Correlations > |0.10| have p < 0.01. Open-ended measures are signified by “Code,” and Religious and Spiritual Struggles subscales are signified by “RSS”.

shows Spearman correlations between measures of r/s struggles and religious belief salience. For both r/s struggle measures, moral struggles related positively to religiousness, whereas doubt struggles related negatively to religiousness. The open-ended codes for divine struggles related positively to religiousness, and the open-ended code for interpersonal struggles related negatively to religiousness. The ultimate meaning RSS subscale related negatively to religiousness. No other associations were reliable.

4. Discussion

Previous research attests to the value of closed-ended assessments of r/s struggles, and results from the current study begin to demonstrate the complementary utility of open-ended assessments of r/s struggles. First, qualitative coding of open-ended descriptions of r/s struggles revealed rich conceptual content and various specific r/s struggle themes that were not captured explicitly by closed-ended items on the RSS. Second, scores derived from codes of open-ended descriptions showed evidence of convergent and discriminant validity when compared to the RSS. Third, these scores related to religious belief salience at magnitudes similar to the RSS subscales, which constitutes evidence of criterion validity, and the mix of similar and divergent associations to religious belief salience across methods for assessing r/s struggles suggests that each method may be tapping into unique information about r/s struggles.

4.1. The Varieties of Religious/Spiritual Struggles

Dividing r/s struggles according to RSS domains captures broad distinctions across r/s struggle types, yet qualitative results from this study suggest the possibility of finer distinctions within these domains. Though some themes emerging from open-ended descriptions were general, abstract, and similar to RSS domains (e.g., General Interpersonal Conflict, Difficulty with General Transgressions, General Doubt/Confusion), most themes were more specific and/or contextualized. The more specific themes may be suggestive of a lower-order structure to r/s struggle domains and may be thought of similarly to to facets (i.e., relatively narrower constructs that together make up the broad domain). Many of the specific themes identified were similar conceptually to items appearing on the RSS, suggesting (not surprisingly) that the RSS already assesses narrower kinds r/s struggles at the item level. Because other specific themes were not conceptually similar to RSS items, coding of open-ended descriptions may have identified r/s struggle content outside of the RSS. Indeed, one r/s struggle theme (Commitment/Time) fell outside of the RSS domains. This struggle reflected feeling upset or guilty about being unable to prioritize r/s activities highly in daily life. These findings are in line with the intent for the RSS to be a relatively comprehensive but not exhaustive measure of r/s struggles (Exline et al. 2014). Revisions to the RSS may include a wider scope of conceptual content, for instance, items may be written to assess struggles with commitment/time (e.g., “I am having trouble finding enough time to attend r/s services.”). Revisions to the RSS may also assess r/s struggles at both the domain and facet levels.

In addition to categorizing themes that emerged from open-ended responses qualitatively, we computed reliable quantitative indices of RSS domains based on these themes. Four of the five RSS domains we measured (divine, interpersonal, moral, doubt) were quite common, whereas ultimate meaning struggles were uncommon. Among RSS domains, ultimate meaning struggles typically have the strongest associations with poorer mental health (Exline et al. 2014; Stauner et al. 2016b), and we speculate that the low prevalence of ultimate meaning struggles in this study is because relatively few participants in our sample—a typical sample of university students—were experiencing severe levels of distress. Ultimate meaning struggles may be more commonly identified as salient r/s struggles in samples with lower levels of mental health. For example, among Veterans who were recruited from VA hospitals, 10% reported ultimate meaning struggles as being most salient (Breuninger et al. 2019). Another reason for low prevalence in this sample is that questions of ultimate meaning may not be as relevant an issue for college students developmentally as issues around identity, explortation, and planning for the future (Arnett 2000). We did not develop codes for demonic struggles because a very low proportion of responses (around 1%) seemed to touch on this domain. We previously noted a variety of reasons why demonic struggles may be uncommon in open-ended responses: Compared to other r/s struggle domains, demonic struggles may be less tangible, less amenable to logical understanding, and more stigmatized (Breuninger et al. 2019). Future work attempting to assess open-ended descriptions of demonic struggles may need to prompt participants to report demonic struggle content.

A total of 88% of r/s struggles in the current sample were categorized according to an RSS domain. This contrasts with the results from a sample of Veterans described in the introduction in which only 62% of r/s struggles were categorized according to an RSS domain (Breuninger et al. 2019). This discrepancy is likely because a high proportion of Veterans described a mental health difficulty (without any mention of r/s issues explicitly) as an important r/s struggle, decreasing the percentage of struggles captured by an RSS domain. We expect that higher proportions of open-ended descriptions of r/s struggles will fall within RSS domains in samples with relatively higher levels of mental health. This finding also implies that it will be important for future qualitative research on r/s struggles to sample from populations with lower levels of mental health, such as psychotherapy clients.

4.2. Validity of Scores Derived from Open-Ended Responses

4.2.1. Convergent and Discriminant Validity

All zero-order and partial correlations between RSS scales and codes within the same domain were positive, and the only positive cross-domain zero-order correlation was between the Ultimate Meaning subscale of the RSS and the doubt code. These results largely provide good evidence of convergent and discriminant validity. The finding relating RSS Ultimate Meaning to the doubt code suggests that endorsing struggles around meaning on closed-ended items could reflect struggles with the specific themes included in the composite doubt code (doubts about God’s existence, questioning fate, wondering about the nature of God, doubts about religious truth, and general doubt/confusion). It is plausible that any of the specific themes comprised by the Doubt composite could be related to more general concerns about whether life has meaning.

4.2.2. Criterion Validity

Turning to the correlations between religious belief salience and r/s struggles, we found a mix of similar and divergent associations across methods for assessing r/s struggles. Religious belief salience related positively to moral struggles and negatively to doubt struggle across methods, which is similar to previously documented associations (Exline et al. 2014; Stauner et al. 2016b) and suggests that people who are higher in religiousness tend to be more concerned with perceived transgressions and less about whether their r/s beliefs are correct. The finding that ultimate meaning struggles related negatively to religious belief salience for the RSS is consistent with previous findings (Exline et al. 2014; Stauner et al. 2016b) and may indicate that people with higher levels of religiousness have fewer concerns about whether there is an ultimate, transcendent purpose in life. The null association for the ultimate meaning code may be due to a floor effect for the composite ultimate meaning codes; in other words the low mean level and variability may have limited our ability to detect associations. The findings for divine and interpersonal struggles, discussed next, diverged across methods.

The positive association between religious belief salience and the divine struggles code (but not the Divine subscale of the RSS) suggests that people with higher levels of religiousness may be likely to struggle with God around issues that are not necessarily reflected by the RSS. Divine struggles that are not reflected by the RSS (e.g., questioning God’s role in one’s life, questioning God’s role in suffering, personal devotion to God) seem somewhat less extreme than those reflected by the RSS (e.g., conflict regarding God’s absence, worthiness of God’s love, concern about angering God, anger at God). We speculate that grappling with issues around the degree to which God is involved in the world and one’s own life might be characteristic of an engaged faith. More extreme divine struggles may be equally likely for those at low levels of religiousness (e.g., those considering exiting their faith tradition or worldview) and those at high levels of religiousness (e.g., those who are comfortable expressing negativity toward God).

The negative association between religiousness and the interpersonal code (but not the Interpersonal subscale of the RSS) suggests that people with higher levels of religiousness may be less likely to struggle with interpersonal issues not reflected by the RSS. Interpersonal struggles that are not reflected by the RSS (e.g., religious intolerance, conflict regarding the morality of others’ actions/beliefs, and struggling with disparate beliefs) center on more specific grievances than those reflected by the RSS (e.g., general interpersonal conflict, general displeasure with religion, feeling like an outsider). Conflict around more specific negative characteristics of r/s institutions and/or individuals may be more common among less religious individuals, whereas more general interpersonal struggles may be equally likely across levels of religiousness.

Overall, the number and magnitude of reliable associations with religiousness was similar across methods for assessing r/s struggles. These findings attest to the criterion validity of each measure of r/s struggles. The differences in associations across some domains suggests that each r/s struggles measure may contain unique information.

4.3. Advancing Qualitative and Quantitative Assessment of Religoius and Spiritual Struggles in Research and Applied Settings

The results from this study suggest that responses to open-ended questions about types of r/s struggles may be coded reliably and that scores derived from our codes have initial evidence of different types of validity (convergent, discriminant, criterion). One potentially fruitful way to extend this work may be to enrich the descriptions of the different types of r/s struggles by subsequently prompting for details such as perceived precursors and consequences, as well as how people make meaning from the r/s struggle. Doing so may generate descriptions of r/s struggle phenomenology that would be amenable to more sophisticated qualitative coding techniques (Creswell and Poth 2016). Additionally, adding more to the open-ended probes may be needed to help participants zero in on spiritual struggles in greater detail in their responses, as opposed to phenomena that may be indirectly related to r/s struggles (e.g., bereavement, isolation, mental and emotional suffering). Quantitative research may seek to extend the RSS by generating additional items that assess the categories revealed by qualitative coding. Psychometric analyses (e.g., exploratory and confirmatory analyses, item-response theory) may then be used to refine the scale in an iterative fashion across several studies, as studies incorporating both qualitative and quantitative components (i.e., mixed-methods research) may carry great, and largely untapped, potential to advance understanding (Hanson et al. 2005).

Open-ended assessments of r/s struggles may also have utility in practical settings. Providers working with people who are experiencing r/s struggles (e.g., clinicians, counselors, clergy) may ask for short descriptions of the r/s struggle and then determine whether the description aligns with any of the codes identified in this study. Open-ended assessment may be used in conjunction with standardized scales (e.g., the RSS) or may substitute for standardized scale administration depending on clinical circumstances. For instance, if a provider is hesitant about using standardized measures because the client might perceive them as impersonal, asking open-ended questions when discussing r/s matters in therapy may be one way to foster openness in the therapeutic relationship (Post and Wade 2009; Pargament 2007). Such openness may be especially important because clients have more trust in providers and feel more understood when able to discuss r/s issues openly and honestly (Mayers et al. 2007).

4.4. Limitations

Our qualitative results should be interpreted in the context of researchers bringing their own perspectives and biases to the process of generating coding categories and identifying themes (Creswell and Poth 2016). Though we attempted to develop objective criteria for codes, other researchers may have developed a different set of guidelines or could interpret our coding protocol differently. One salient bias of our team may be the preference to couch our themes within the RSS framework, and therefore we encourage other researchers to explore open-ended descriptions of r/s struggles from other theoretical and empirical perspectives. Another potential bias in our protocol was the decision to code responses that did not explicitly include r/s content into non-r/s categories (e.g., suffering, loss). Because our prompt asked explicitly for r/s struggles, such responses may still have been perceived by participants as including r/s themes, which may have become evident upon further prompting.

A limitation of our qualitative coding protocol is that it did not adhere to formal cross-validation procedures, such as using a small calibration sample, coding one half of the remainder of the sample, and then cross-validating results using the other half of the remaining sample. Our procedures leave open the possibility that some themes were not well-represented throughout the sample. Though we acknowledge the importance of cross-validation for hypothesis testing and theory generation, in this initial work we primarily aimed to explore and describe themes and to generate a coding manual including a highly comprehensive database of themes.

The generalizability of qualitative and quantitative findings may be limited by our reliance on a sample of predominantly Christian undergraduates from a Western, educated, industrialized, rich, and democratic (WEIRD) society (Henrich et al. 2010). Though initial work has validated the RSS framework for studying r/s struggles in more diverse samples (Abu-Raiya et al. 2015; Abu-Raiya et al. 2016), culture’s profound effects on r/s in general (Johnson and Cohen 2014; Saroglou 2017) lead us to expect that culture will influence r/s struggles as well. The current results may be viewed conservatively as advancing our understanding of assessment, conceptualization, and validity of the RSS framework among undergraduates in a particular sociopolitical, religious, and geographical context.

5. Conclusions

R/s struggles are a common and impactful part of r/s life, and therefore it is important to assess r/s struggles accurately and efficiently. Research has progressed toward these goals over the past few decades, with the development of standardized measures of different domains of r/s struggles using closed-ended items. In line with the traditions of qualitative and mixed-methods research, which have seldom been applied to r/s struggles, the current study sought to develop a reliable and valid coding system for open-ended descriptions of r/s struggles.

We found that themes characterizing open-ended descriptions of r/s struggles tended to fall within the domains covered by the RSS, however, individual codes reflected both the broad r/s struggle domains assessed by the RSS and more contextualized r/s struggles that are not assessed by the RSS. Scores derived from the open-ended coding to reflect RSS domains showed initial evidence of convergent and discriminant validity with the RSS. When predicting religiousness, we found similar and divergent associations across methods for assessing r/s struggles, with both methods showing evidence of criterion validity. In all, these findings represent a promising start for assessing r/s struggles with open-ended questions and attest to the potential utility of open-ended assessments to complement standardized measures of r/s struggles in research and applied settings.