NSF INCLUDES Shared Measures

The Hub's Collaborative Infrastructure (CI) Survey is designed to collect information about the progress that NSF INCLUDES-funded projects are making addressing specific components of the design elements of collaborative infrastructure. A primary purpose of the survey is to highlight the progress that NSF INCLUDES-funded projects are making in developing a cohesive collaborative infrastructure as they operationalize the five design elements of collaborative infrastructure. In addition, the individual projects that participated in the survey are using their own data to assess their progress and inform their decision making. Going forward, the findings will serve as a resource for partnership projects that are seeking information about what constitutes a cohesive infrastructure (and how to measure progress toward having a cohesive infrastructure).

Six of the eight Alliances participated in the initial administration of this web-based survey (the other two opted out because they already administer similar surveys but provided the Hub with their survey data). Each of the six Alliances identified key stakeholders (e.g., PIs, Co-PIs, staff, partners, researchers, and evaluators) who were in a position to assess progress regarding the design elements. The response rate for the survey was 70 percent (representing 88 of the 126 individuals who were asked to complete the survey).

The NSF INCLUDES Coordination Hub's Collaborative Infrastructure Survey is designed to document respondents' assessment of their project's progress addressing specific components of each design element. The graphics on the site portray survey findings using respondents as the unit of analysis. While this perspective is useful, it fails to consider whether respondents within a given project agreed about the level of progress around the design elements. In addition, there is considerable benefit in considering projects as the unit of analysis (e.g., to assess which design elements projects appear to be addressing in a given year of NSF INCLUDES funding). To address this, we used item-response theory and confirmatory factor analysis to generate project-level composite scores for each survey item.¹ This approach allowed us to assess the extent to which Alliances have operationalized the design element of collaborative infrastructure at a given point in time. Learn more about the methodology used to generate project-level findings.

Project-level survey responses regarding Shared Vision were high, with an overall score of 82.5 (and a range of 77.5 to 87.2 on a scale of 1 to 100). Additionally:

• At the item level, Alliance-level responses for Shared Vision were highest for two statements: “Our project's goals are informed by an assessment of the participant population's needs” (88.1) and “All of our core partners are involved in the process of developing our project's goals” (87.5).
• Alliance-level responses for Shared Vision were lowest for the third statement pertaining to shared vision: “Our project has a plan that addresses systemic barriers to broadening participation in STEM” (72.7).
• There were no noteworthy differences regarding Shared Vision across Alliances by year of NSF INCLUDES funding.^2,3

Project-Level scores for Shared Vision

Note: The score for a given survey item represents the overall standardized scale score obtained from the item-response theory and confirmatory factor analysis. Each score has a range of 1 to 100, with 100 representing the highest possible score—i.e., all respondents within a project answered the highest response category (either “achieved” or “strongly agree”) for a given survey item. In addition, we provide the minimum and maximum project-level standardized scale score response (in italics) for a given survey item.

¹ These approaches are designed to assess the relationship between the latent construct and observed items to test the reliability and validity of the measurement and quantify the attributes of interest.

² Because the survey was administered for the first time in spring 2021, we presently have no data on respondents' perceptions of progress at the end of the first year of NSF INLCUDES funding. (Going forward, we expect to obtain Year 1 data from NSF INCLUDES Planning Grants and Cohort 3 Alliances.) As a result, we are currently unable to provide information about the relative progress that respondents would have reported for their initial year.

³ In theory, one would expect that Alliances with more years of NSF INCLUDES funding would report more progress around the operationalization of a given design element. However, we are somewhat cautious when making such comparisons, because it is possible that the characteristics of Alliances funded in a given cohort differ (e.g., in terms of the maturity and complexity of their partnership structure, the range of barriers they are designed to address, the characteristics of their participant population, and the complexity of their approach). In addition, respondents' perspectives concerning their accomplishments (or the progress they still need to make) around a given design element may shift as they recognize the complexity of a given issue—with respondents realizing more work is needed as they begin to delve more deeply into a particular task.

The NSF INCLUDES Coordination Hub's Collaborative Infrastructure Survey is designed to document respondents' assessment of their project's progress addressing specific components of each design element. The graphics on the site portray survey findings using respondents as the unit of analysis. While this perspective is useful, it fails to consider whether respondents within a given project agreed about the level of progress around the design elements. In addition, there is considerable benefit in considering projects as the unit of analysis (e.g., to assess which design elements projects appear to be addressing in a given year of NSF INCLUDES funding). To address this, we used item-response theory and confirmatory factor analysis to generate project-level composite scores for each survey item.⁴ This approach allowed us to assess the extent to which Alliances have operationalized the design element of collaborative infrastructure at a given point in time. Learn more about the methodology used to generate project-level findings.

Project-level survey responses regarding Partnerships were largely positive, with an overall score of 75.3 (and a range of 64.3 to 82.1 on a scale of 1 to 100). Additionally:

• At the item level, Alliance-level responses for Partnerships were highest for the following two statements: “The sum of our core and supporting partners represent the range of institutions needed to achieve our project's goals” (83.3) and “Our project has a plan that clearly specifies each partner's role” (80.1).
• Alliance-level responses for Partnerships were lowest for the following statement: “Our project adds new partners to address a given need” (66.5).
• There were no differences for Partnerships across Alliances by year of NSF INCLUDES funding.^5,6

Project-Level scores for Partnerships

Note: The score for a given survey item represents the overall standardized scale score obtained from the item-response theory and confirmatory factor analysis. Each score has a range of 1 to 100, with 100 representing the highest possible score—i.e., all respondents within a project answered the highest response category (either “achieved” or “strongly agree”) for a given survey item. In addition, we provide the minimum and maximum project-level standardized scale score response (in italics) for a given survey item.

⁴ These approaches are designed to assess the relationship between the latent construct and observed items to test the reliability and validity of the measurement and quantify the attributes of interest.

⁵ Because the survey was administered for the first time in spring 2021, we presently have no data on respondents' perceptions of progress at the end of the first year of NSF INLCUDES funding. (Going forward, we expect to obtain Year 1 data from NSF INCLUDES Planning Grants and Cohort 3 Alliances.) As a result, we are currently unable to provide information about the relative progress that respondents would have reported for their initial year.

⁶ In theory, one would expect that Alliances with more years of NSF INCLUDES funding would report more progress around the operationalization of a given design element. However, we are somewhat cautious when making such comparisons, because it is possible that the characteristics of Alliances funded in a given cohort differ (e.g., in terms of the maturity and complexity of their partnership structure, the range of barriers they are designed to address, the characteristics of their participant population, and the complexity of their approach). In addition, respondents' perspectives concerning their accomplishments (or the progress they still need to make) around a given design element may shift as they recognize the complexity of a given issue—with respondents realizing more work is needed as they begin to delve more deeply into a particular task.

The NSF INCLUDES Coordination Hub's Collaborative Infrastructure Survey is designed to document respondents' assessment of their project's progress addressing specific components of each design element. The graphics on the site portray survey findings using respondents as the unit of analysis. While this perspective is useful, it fails to consider whether respondents within a given project agreed about the level of progress around the design elements. In addition, there is considerable benefit in considering projects as the unit of analysis (e.g., to assess which design elements projects appear to be addressing in a given year of NSF INCLUDES funding). To address this, we used item-response theory and confirmatory factor analysis to generate project-level composite scores for each survey item.⁷ This approach allowed us to assess the extent to which Alliances have operationalized the design element of collaborative infrastructure at a given point in time. Learn more about the methodology used to generate project-level findings.

Project-level survey responses regarding Goals & Metrics were largely positive, with an overall score of 74.7 (and a range of 65.9 to 82.1 on a scale of 1 to 100). Additionally:

• At the item level, Alliance-level responses for Goals & Metrics were highest for the following statement: “All of our core partners are involved in the process of making sense of findings that emerge from the project's analysis of shared measurement data” (81.2).
• Alliance-level responses for Goals & Metrics were lowest for the following statement: “Our project uses data to make regular improvements” (68.3).
• There were some noteworthy differences for Goals & Metrics across Alliances by year of NSF INCLUDES funding. Alliances with 3 years of NSF INCLUDES funding were slightly more likely to have higher composite scores for the following two statements: “Our project has participatory processes to refine its measures, indicators, metrics, and/or data collection methods” (77.3, compared to 71.2 for Alliances with 2 years of NSF INCLUDES funding) and “Our project has the capacity to track progress across all partners” (72.1, compared to 66.8 for Alliances with 2 years of NSF INCLUDES funding).^8,9

Project-Level score for Goals & Metrics

Note: The score for a given survey item represents the overall standardized scale score obtained from the item-response theory and confirmatory factor analysis. Each score has a range of 1 to 100, with 100 representing the highest possible score—i.e., all respondents within a project answered the highest response category (either “achieved” or “strongly agree”) for a given survey item. In addition, we provide the minimum and maximum project-level standardized scale score response (in italics) for a given survey item.

⁷ These approaches are designed to assess the relationship between the latent construct and observed items to test the reliability and validity of the measurement and quantify the attributes of interest.

⁸ Because the survey was administered for the first time in spring 2021, we presently have no data on respondents' perceptions of progress at the end of the first year of NSF INLCUDES funding. (Going forward, we expect to obtain Year 1 data from NSF INCLUDES Planning Grants and Cohort 3 Alliances.) As a result, we are currently unable to provide information about the relative progress that respondents would have reported for their initial year.

⁹ In theory, one would expect that Alliances with more years of NSF INCLUDES funding would report more progress around the operationalization of a given design element. However, we are somewhat cautious when making such comparisons, because it is possible that the characteristics of Alliances funded in a given cohort differ (e.g., in terms of the maturity and complexity of their partnership structure, the range of barriers they are designed to address, the characteristics of their participant population, and the complexity of their approach). In addition, respondents' perspectives concerning their accomplishments (or the progress they still need to make) around a given design element may shift as they recognize the complexity of a given issue—with respondents realizing more work is needed as they begin to delve more deeply into a particular task.

The NSF INCLUDES Coordination Hub's Collaborative Infrastructure Survey is designed to document respondents' assessment of their project's progress addressing specific components of each design element. The graphics on the site portray survey findings using respondents as the unit of analysis. While this perspective is useful, it fails to consider whether respondents within a given project agreed about the level of progress around the design elements. In addition, there is considerable benefit in considering projects as the unit of analysis (e.g., to assess which design elements projects appear to be addressing in a given year of NSF INCLUDES funding). To address this, we used item-response theory and confirmatory factor analysis to generate project-level composite scores for each survey item.¹⁰ This approach allowed us to assess the extent to which Alliances have operationalized the design element of collaborative infrastructure at a given point in time. Learn more about the methodology used to generate project-level findings.

Project-level survey responses regarding Leadership & Communication were overwhelmingly positive (compared to the other design elements), with an overall score of 85.6 (and a range of 67.2 to 94.2 on a scale of 1 to 100). Additionally:

• Alliance-level responses for were 81.0 or higher across all survey items pertaining to Leadership & Communication—with the highest score for the following statement: “Our project's leadership structure leverages the collective knowledge of partners and other stakeholders” (90.1).
• Alliance-level responses for Leadership & Communication were lowest for the following two statements (although the Alliance-level composite scores were still quite high): “Our project has internal procedures that minimize power imbalances among partners” (81.3) and “Our project's decisions are informed by input from our participant population” (81.0).
• There were some noteworthy differences across Alliances by year of NSF INCLUDES funding. Alliances with 2 years of NSF INCLUDES funding were more likely to have higher composite scores for all of the statements pertaining to Leadership & Communication.^11,12

Project-Level score for Leadership & Communication

Note: The score for a given survey item represents the overall standardized scale score obtained from the item-response theory and confirmatory factor analysis. Each score has a range of 1 to 100, with 100 representing the highest possible score—i.e., all respondents within a project answered the highest response category (either “achieved” or “strongly agree”) for a given survey item. In addition, we provide the minimum and maximum project-level standardized scale score response (in italics) for a given survey item.

¹⁰ These approaches are designed to assess the relationship between the latent construct and observed items to test the reliability and validity of the measurement and quantify the attributes of interest.

¹¹ Because the survey was administered for the first time in spring 2021, we presently have no data on respondents' perceptions of progress at the end of the first year of NSF INLCUDES funding. (Going forward, we expect to obtain Year 1 data from NSF INCLUDES Planning Grants and Cohort 3 Alliances.) As a result, we are currently unable to provide information about the relative progress that respondents would have reported for their initial year.

¹² In theory, one would expect that Alliances with more years of NSF INCLUDES funding would report more progress around the operationalization of a given design element. However, we are somewhat cautious when making such comparisons, because it is possible that the characteristics of Alliances funded in a given cohort differ (e.g., in terms of the maturity and complexity of their partnership structure, the range of barriers they are designed to address, the characteristics of their participant population, and the complexity of their approach). In addition, respondents' perspectives concerning their accomplishments (or the progress they still need to make) around a given design element may shift as they recognize the complexity of a given issue—with respondents realizing more work is needed as they begin to delve more deeply into a particular task.

The NSF INCLUDES Coordination Hub's Collaborative Infrastructure Survey is designed to document respondents' assessment of their project's progress addressing specific components of each design element. The graphics on the site portray survey findings using respondents as the unit of analysis. While this perspective is useful, it fails to consider whether respondents within a given project agreed about the level of progress around the design elements. In addition, there is considerable benefit in considering projects as the unit of analysis (e.g., to assess which design elements projects appear to be addressing in a given year of NSF INCLUDES funding). To address this, we used item-response theory and confirmatory factor analysis to generate project-level composite scores for each survey item.¹³ This approach allowed us to assess the extent to which Alliances have operationalized the design element of collaborative infrastructure at a given point in time. Learn more about the methodology used to generate project-level findings.

Project-level survey responses regarding Expansion, Sustainability & Scale were relatively low (compared to the other design elements), with an overall score of 62.6 (and a range of 48.0 to 70.6 on a scale of 1 to 100). Additionally:

• At the item level, Alliance-level responses were highest for the following two statements: “Our project contributes to the field's knowledge base about effective strategies for broadening participation in STEM” (72.7) and “Our project has a strategic vision of what activities will be sustained beyond the current award period” (72.7).
• Alliance-level responses were lowest for the following statement: “Our project has secured funding beyond the current award period” (42.8).
• There were some noteworthy differences for Expansion, Sustainability & Scale across Alliances by year of NSF INCLUDES funding.^14,15 Specifically:
• Alliances with 3 years of NSF INCLUDES funding were more likely to have higher composite scores for the following two statements: “Our project contributes to the field's knowledge base about effective strategies for broadening participation in STEM” (76.8, compared to 68.6 for Alliances with 2 years of NSF INCLUDES funding) and “Our project has secured funding beyond the current award period” (47.8, compared to 37.9 for Alliances with 2 years of NSF INCLUDES funding).
• Alliances with 2 years of NSF INCLUDES funding were more likely to have higher composite scores for the following statement: “Our project has a strategic vision of what activities will be sustained beyond the current award period” (76.3, compared to 69.0 for Alliances with 3 years of NSF INCLUDES funding).

Project-Level score for Expansion, Sustainability & Scale

Note: The score for a given survey item represents the overall standardized scale score obtained from the item-response theory and confirmatory factor analysis. Each score has a range of 1 to 100, with 100 representing the highest possible score—i.e., all respondents within a project answered the highest response category (either “achieved” or “strongly agree”) for a given survey item. In addition, we provide the minimum and maximum project-level standardized scale score response (in italics) for a given survey item.

¹³ These approaches are designed to assess the relationship between the latent construct and observed items to test the reliability and validity of the measurement and quantify the attributes of interest.

¹⁴ Because the survey was administered for the first time in spring 2021, we presently have no data on respondents' perceptions of progress at the end of the first year of NSF INLCUDES funding. (Going forward, we expect to obtain Year 1 data from NSF INCLUDES Planning Grants and Cohort 3 Alliances.) As a result, we are currently unable to provide information about the relative progress that respondents would have reported for their initial year.

¹⁵ In theory, one would expect that Alliances with more years of NSF INCLUDES funding would report more progress around the operationalization of a given design element. However, we are somewhat cautious when making such comparisons, because it is possible that the characteristics of Alliances funded in a given cohort differ (e.g., in terms of the maturity and complexity of their partnership structure, the range of barriers they are designed to address, the characteristics of their participant population, and the complexity of their approach). In addition, respondents' perspectives concerning their accomplishments (or the progress they still need to make) around a given design element may shift as they recognize the complexity of a given issue—with respondents realizing more work is needed as they begin to delve more deeply into a particular task.