Published 5/3/2024

The Roux Institute at Northeastern University
Dan Koloski, Professor of the Practice and Head of Learning
Zach Blattner, Director of Partner Products and Programs
Berkeley Almand-Hunter, Technical Lead for Custom Learning Programs

Introducing the Analytical LEAP framework

The Analytical LEAP framework is a new framework developed at the Roux Institute at Northeastern University, designed to help organizations target investment and improvement in the areas of workforce upskilling and organizational learning culture to maximize business value in the artificial intelligence (AI) economy.  The LEAP framework is designed to map an organization’s enabling capabilities against the specific skill needs of various members of its data community. This approach will yield targeted investment recommendations for upskilling and learning culture improvement. The framework differs from common approaches that are more technology-centric or focused on organizational strategy and instead focuses attention on the specifics of enabling experiential learning in the flow of work.

Why does experiential learning in the flow of work matter?

At Northeastern University, we strongly believe in the value of experiential learning – specifically, learning-by-doing.  Our commitment to experiential learning is deep and longstanding, becoming more important and recognized in recent years. In February 2024, the The Wall Street Journal shared findings from new research by the Burning Glass Institute and nonprofit Strada Education Foundation that emphasized the importance of applied and experiential learning in securing employment after graduation (Fuhrmans & Ellis, 2024). The article juxtaposed Northeastern, where 91% of graduates “report having jobs related to their major” within six months, to the broader post-graduate pool, where only 55% report having jobs requiring a degree five years after leaving college.

But learning-by-doing extends far beyond the university; indeed, applying skills in context of work is how most learning happens in the workplace context. Learning-by-doing is particularly true in data literacy, because while adoption typically starts in technology-focused teams, the professional skillsets associated with data literacy are required of all organizational roles (Sabar, 2021).

Synchronous vs. asynchronous learning in the workplace

While we recognize the value of asynchronous learning in providing on-demand, scalable offerings, we’ve intentionally decided to focus the workplace learning we design at the Roux Institute on live delivery, both virtual synchronous and in-person. Our rationale is that effective experiential learning, if it is to replicate the actual workplace, must include cohort-based social interaction, conversation, and feedback, both between instructor and participant as well as amongst participants themselves.

Why does the world need another framework?

We asked the same question. We certainly did not set out to build “yet another framework.”  And yet…

Our mission at the Roux Institute is to provide economic opportunity for Maine and northern New England by building a new tech and life sciences economy. Part of that mission requires helping partner organizations of all kinds adapt to and generate value from the data economy. After all, achieving our mission of creating economic activity requires many growing, thriving organizations who employ many tech-forward workers.

In working with these partners, we heard the same refrains repeatedly:

• Our workforce is not ready for the AI revolution
• We’re not sure how to prepare our workforce
• Existing models of upskilling are not moving the needle
• We have limited examples of good partnerships with higher ed
• We don’t know how to assess the results of data analytics training

There is wide agreement on the problems, such as a lack of readiness, lack of understanding of how to address, lack of confidence in traditional approaches, and an assumption that mainstream academia is unwilling or unable to engage in workplace-focused learning. However, there is also an almost universal feeling of paralysis in terms of how to address these problems. Couple this paralysis with the astonishing rate of capability growth and penetration of generative AI, and it’s clear that guidance would help.

Most frameworks widely used in organizations today fall into four classes: technology-centric frameworks (ex., Snyder, 2024; MITRE, 2023; Gartner, 2019, and many vendor-provided frameworks); organizational-strategy-centric frameworks (ex., Bartlett, 2013; Davenport & Harris, 2017); subject-agnostic design or post-training evaluative frameworks (Roy, Pollock, et al., 2015; Kirkpatrick, 2016), or are focused narrowly around specific personas (data citizens or data practitioners, for example). While all offer value in identifying organizational gaps, they fail to provide actionable recommendations around specific near-term workforce learning activities that can be undertaken by specific segments of an organization’s workforce.

Enter Northeastern University’s Analytical LEAP Framework, called “LEAP” for short.  Analytical LEAP proposes a simple explanation of the organizational underpinnings required to quickly adapt to the data and AI revolution by benchmarking four areas against ratings of “launch,” “explore,” “accelerate,” and “perform:”

• Learning Culture: What is the evidence of continuous learning throughout the organization?
• Ecosystem: Is the organization’s data strategy infused throughout all levels of the organization?
• Analytical Architecture: Are there strong practices and associated technology that enable data usage throughout the enterprise?
• People: Do teams and individuals have the requisite knowledge and skills to accelerate organizational progress using data, analytics, and AI?

It is our contention that the first three categories (Learning Culture, Ecosystem, Analytical Architecture) are enablers of an organization’s adaptation to the data economy, but the value-unlocking opportunity is in the fourth category – People. LEAP further drills into the People dimension to focus on requisite skillsets for the major categories of data-centric roles in an enterprise…

• Senior Leadership (executives)
• Consumers (e.g. line-of-business stakeholders, HR)
• Curators (e.g. data engineers, IT)
• Practitioners (e.g. analysts, data scientists)
• Data Citizens (everyone!)

… and then helps organizations assess baseline skillsets for representative personas and target upskilling investments.

While we recognize that not every role neatly fits into one of the above categories nor directly aligns with every organization’s structure, we believe the broad categories contain the critical skill sets for analytics and AI success. We further posit that the idea of consumers, curators, and practitioners will hold true across industry, even when the titles of specific roles themselves might look quite different. For instance, a hospital system has clinicians, researchers, and lab technicians, all titles that are unlikely within an insurance company. However, both organizations have analytical positions who must consume, curate, and analyze the available data.

The approach

We begin by engaging in a full workforce assessment to understand organizational placement on the LEAP framework and how that maps to the actual skills and knowledge across crucial data roles. To do this comprehensively, we analyze an organization’s workforce against our model, using a range of methods including interviews, job descriptions, skill assessments, self-reporting, and performance reviews, incorporating support from large language models to process data.

For the key data-centric roles within the People dimension, such as data analysts, non-technical LOB, and leadership, we’ve identified 5 proficiency levels ranging from “Emerging” to “Expert” and the corresponding skills and knowledge that we’d attribute to each. For some organizations, the skill level of any individual group may be homogeneous. For instance, all HR business partners could be at the “Proficient” level, while others may have more diversity within a given role or team, e.g., mostly “Developing” analysts along with a few who are in the “Advanced” category. Either way, the model helps to locate and name these users, which is essential when considering the next steps for their professional growth.

To move beyond a theoretical or simply consultative approach and make LEAP actionable, we structured our foundational course catalog with offerings that align to both the roles and skill levels discussed above. This through-line enables organizations to not only identify key roles for upskilling, but also to determine a specific learning pathway based on a given group’s skill set and the company’s long-term AI and analytics strategy. The intent is to build a comprehensive approach to job-based learning, one that sequences thoughtfully from one course to the next, rather than a piecemeal experience that risks a disconnect from organizational priorities or individual needs.

As we move into more detail, we’ve developed a Scope and Sequence for each course that establishes a range of options with respect to time, performance objectives, experiential learning opportunities, and focus content topics. By structuring courses like Data Driven Decision Making or Introduction to Machine Learning in this manner, we’re able to better customize the experience to the skill gaps identified in the audience as well as remain mindful of practical constraints like attention and time that any employee faces when engaging in professional learning. Provided they aren’t essential prerequisite skills, we also remove any subtopics that are irrelevant to a given audience to make room for increased discussion and attention towards critical information. The Scope and Sequence also intentionally identifies how we plan to integrate an organization’s context into the course. This enables us to build experiential opportunities grounded in an organization’s tech stack, use cases, and industry, which are essential for habit-changing learning.

Create momentum via a framework-enabled forcing function

Companies operate in an environment where the scarcest resource is time.  Investing in employee development, previously seen as a nice-to-have, is required for organizational transformation in the age of AI. But those with responsibility for specifying, designing, and approving these initiatives also suffer from the scarcity of time! Analytical LEAP serves as a roadmap and guidepost to quickly target the highest-impact investments in a way that is contextualized for the organization in question. In this way, it can serve as a unifying set of nomenclature and a rallying point for organizational initiatives in and around AI transformation while also pragmatically helping to achieve results quickly.

Analytical LEAP in action

We are using the LEAP model to design a custom AI and analytics learning program for a regional bank.

Our approach to mapping the bank’s people to LEAP will consist of three strategies for assessment: artifact collection and evaluation, interviews, and individual assessments.

The first strategy, collecting and evaluating artifacts, consists of acquiring company resources, including job descriptions, performance reviews, career progression frameworks, training records, and records of digital badges or certifications. Then, the artifacts can be summarized and evaluated with the assistance of large language models. Artifact collection is the first step in our process for several reasons.

Artifacts will enable us to evaluate whether the financial services company is making hiring and promotion decisions based on analytics and AI skills and if these skills are required for all employees or only for those in technical roles. It will also give us insight into whether the skills listed by the financial services company match those required for their desired LEAP level. For example, if an organization lists “proficient in Microsoft Excel” as their only technical requirement for a data analyst, their analysts will be at a “Pre-Launch” level. If their organizational goal is to become “Proficient,” they need to update their job descriptions and career progression frameworks for data analysts to include “proficient in BI tools and starting to learn SQL and a scripting language,” in addition to offering the relevant training to their employees.

Another advantage of processing artifacts first is time. With the assistance of AI-enabled diagnostics, processing artifacts is faster than conducting individual assessments or interviews. The downside of processing artifacts is that certain artifacts rely on self-assessment. What an individual claims verbally or on their resume is not always an accurate reflection of their knowledge and skills. People are also frequently hired without meeting all the requirements in a job description. Reviews by managers, training records, and records of digital badges or certifications may be more reliable, but still don’t necessarily indicate skill mastery.

The next step is interviewing a variety of key leaders who report directly to the C-suite. Speaking with these leaders will help us assess learning culture, ecosystem, and analytical architecture, in addition to the skills of individuals. Interviewees will be able to provide the following insights:

• Strength of their learning culture (e.g. “Are you and your team given time during work hours to learn about AI and Analytics? Does the organization offer courses to help you learn?”)
• Depth of Analytics and AI ecosystem (e.g. “Is data-driven decision making widespread or siloed within certain teams? Do you have a collaborative relationship with a data practitioner who can help you with analyses?”)
• Sophistication of Analytical Architecture (e.g. “Can you access the data you need? Do you trust it?”)
• Skills of Individual Team Members can be assessed by walking through the Part 2: People and Skills table in LEAP with their managers

After reviewing artifacts and talking with key leaders, we will have an estimate of where the organization falls in LEAP and an idea of team members’ skills. Still, managers might not be able to accurately assess their employee’s level in every skill, particularly if it is a skill they don’t have. This brings us to individual assessments.

Data literacy assessments for individuals typically fall into two categories: self-assessment and objective assessment (Kim et al., 2023). The questions asked in self-assessments are based on personal assessment of comfort and skills in various areas of data literacy (e.g. “Select the statement that best describes your comfort with data visualization”). Objective assessments, on the other hand, ask questions that confirm the test-taker’s knowledge of a subject (e.g. “Which of the following visualizations most effectively communicates change in revenue over time?”).

We will use a combination of self-assessments and objective assessments. Self-assessments indicate how a person feels about their data skills, which is important because skills are meaningless without the confidence to apply them. The downside of self-assessments is that they may not be accurate. Objective assessments will confirm whether people really have the skills that they claim they have!

For data consumers like tellers, HR team members, and the sales team, assessments will lean more toward self-assessment, with a few objective questions. They will be brief, since working with data isn’t the primary function of a data consumer’s job. Assessments for data curators and practitioners like analysts will be more thorough and primarily objective, with a few self-assessment questions.

The three-step process of artifact collection and evaluation, interviews, and individual assessments will provide us with a thorough understanding of both organizational maturity and the skills of the organization’s people. We will map this understanding to tables 1 and 2 in LEAP (Org-wide Systems and Technology and People and Skills). We can then use the third part of LEAP, the “Custom Learning Catalog – Recommended Courses by Role and Level” table, to determine which courses are the best fit for every member of the organization. We will them use the combination of the organizational maturity assessment, recommended courses, and project budget to create a learning plan that maximizes ROI for our partner. A final comprehensive recommendation might look like the one below:

Analytical LEAP AI Framework | Custom Learning Catalog: Recommended Courses (example)

Analytical LEAP AI Framework | Part 1: Organizational Assessment (example)

Analytical LEAP AI Framework | Part 2: Role-Specific Skills (example)