Member Engagement Member Scoring

November 1, 2019

Member Engagement Score for Associations

At the core of every association, there is a clear mandate to understand their members and keep them engaged. Engagement is the result of an association that understands the needs of their members and offers them information, education, support and other services that fulfill those needs. Because of that, measuring engagement becomes the equivalent of measuring the health pulse of an association.

I previously reported on the challenges and significance of member engagement for associations based on the results of a research survey conducted by Wicket. In that report, I argue that ultimately, associations want to collect valuable and actionable qualitative information directly from individual members while being able to observe the member’s behaviour and infer meaning from the underlying data trail of those interactions, without having to ask members about it.

The results of the survey have given us valuable insight and inspiration to define a framework for calculating a member engagement score that leverages the unique nature of the data that Wicket is able to collect about members.

The building blocks for member engagement scoring

In order to calculate a score of member engagement, we require data. And the data that an association collects about their members can be of two types:

Member feedback

This is a written or verbal communication from the member to the association in a particular context or about a certain topic. This is qualitative information that represents the perception and opinions that a member has about the association.

This data is usually collected via questionnaires, forms, surveys or interviews and it usually is processed manually by reading the answers and organizing responses into categories, placing members into segments or assessing the sentiment of the feedback provided.

Associations may typically initiate this type of data collection via onboarding forms, targeted campaigns with surveys or via focus groups and other interview techniques. Sources for this type of information can be found in member-initiated interactions as well, such as customer support requests, comments on the association website, or posts on community and social media platforms.

Member activity

A trail of data points that represent member actions. This is quantitative information that represents all the points of interaction (touchpoints) that the member has with the association.

This type of data is collected by some form of automated tracking system and is typically processed through a combination of manual and automated quantitative analysis or business intelligence reporting. The nature of such touchpoints can be considered of two types:

Transactional

These are goal-driven activities that are required of the member at specific times and are typically associated with some form of a financial transaction. Examples of such touchpoints may include onboarding, renewal, event registration, donation, course completion, etc.

Behavioural

These are typically ongoing and/or peripheral to the transactional activity. They represent member interactions with people and systems that are part of the association and also include a quantifiable representation of member feedback activity. Examples of these touchpoints include page views on a website, resource downloads, click interactions, opening and clicking an email, commenting, posting, sharing, etc.

Some examples of types of systems or services that may act as a source of member activity include:

Web Analytics
(e.g. Google Analytics, Optimizely, etc.)

Product Tracking Analytics
(e.g. Mixpanel, Intercom, etc.)

Email Marketing Automation Tracking
(e.g. Mailchimp, Rasa.io, etc.)

Customer Data Platforms / CRM
(e.g. Blueshift, Segment, SalesForce, Hubspot, etc.)

Social Media Platforms
(e.g. Twitter, Facebook, LinkedIn, YouTube, Instagram, etc)

The approach to calculating a member score using Wicket’s Member Data Platform currently uses quantitative data (transactional and behavioural) that comes from systems that are integrated into the platform.

The definition of member engagement

Not all data is created equal. And this is particularly true when we talk about member engagement data. One of the things we learned from our research survey is that the definition of member engagement is a moving target and very unique to each association. Because of this fact, not all member data collected by an association would represent engagement at a given point in time.

As a starting point for the calculation of a member engagement score, we need the association to be able to express what engagement looks like for them. In order to do this, associations should think of what we call Key Engagement Indicators (KEI).

Key Engagement Indicators are areas or categories of member activity (and perhaps the specific actions) that the association deems important and an accurate representation of what they think member engagement is. Some examples of these may include event registration, learning management achievements, community participation etc.

The KEIs are akin to what KPIs are for business planning, however, its definition is more generalized and less focused on the achievement of a particular performance goal. KEIs aim at providing the context for the calculation of member engagement with a focus on what is important for the association.

Furthermore, associations should consider the relative importance of the KEIs among themselves. Some may be more or less indicative of member engagement, and this should be expressed by weighting them. This weight can then be used in the score calculation.

Wicket’s member engagement scoring framework is built on the notion of a hierarchy of Key Engagement Indicators (KEIs) as parents and the corresponding set of touchpoints that represent each KEI as the children.

For example, if one of the KEIs is “Events”, some of the touchpoints that represent Events Engagement could be:

Member registered for the annual conference
Member registered for a chapter meetup
Member attended a conference or meetup
Member purchased additional training during the event

Wicket MDP member engagement score

We define a member engagement measure that considers multiple key engagement indicators and their corresponding weights to calculate a composite score based on the underlying touchpoint information in the Wicket Member Data Platform (MDP).

Key engagement indicators are mirrored by the different software categories of Wicket integrations (i.e. Event Management, LMS, CMS, Community Platforms etc.). Wicket tracks a certain number of touchpoints within each software category and for each specific software tool (e.g. Eventbrite, Cvent etc).

Weights are initially defined for each engagement indicator in a way that they always tally up to 100. Furthermore, weights can alternatively be assigned to individual touchpoints within each engagement indicator (i.e. software tool integration). If no weights are indicated for individual touchpoints, then an equal weight is assumed among them for that engagement indicator.

High-level engagement score algorithm

Associations need only to define their desired weights and Wicket will calculate the engagement of individual members based on the activity it is able to track across multiple integration points. The algorithm to do this follows these high-level steps:

Tally up the #total number of touchpoints for each KEI
Normalize to a value between 0 and 1 (by dividing the #total by the #max in that KEI)
Multiply the normalized value by the weight to get the KEI Score
Add all KEI scores and multiply by 100 to get the member engagement score (represented as a value between 0 and 100)

Where:

#total: Total number of touchpoints that the member in question has for a particular Key Engagement Indicator

#max: The number of touchpoints of the member with the most touchpoints in that Key Engagement Indicator

This algorithm removes the need for associations to define or know beforehand the values that defines a good engagement indicator. The normalization happens against the best performer in the group (i.e. the person that had the highest number of touchpoints in a given category or KEI)

An example of score calculation

In the following example we present the calculations for a situation where the association defines three Key Engagement Indicators (KEIs): Events, Learning and Community.

Input Data:

KEIs Weights:

Events: 50%
Learning: 30%
Community: 20%

Max # of touchpoints within each KEI (across all members):

Events: 20
Learning: 400
Community: 1500

Calculation:

Tally a member’s total # of touchpoints:

Events: 15
Learning: 200
Community: 500

Normalization (#total / #max):

Events: 15 / 20 = 0.75
Learning: 200 / 400 = 0.5
Community: 500 / 1500 = 0.33

Category Score (Normalized * Weight):

Events: 0.75 * 0.50 = 0.375
Learning: 0.5 * 0.30 = 0.15
Community: 0.33 * 0.20 = 0.066

Final Score (Cat1 + Cat2 + … CatN) * 100:
(0.375 + 0.15 + 0.066) * 100 = 59.1%

In this case, equal weight is assumed for the touchpoints within each category. However, the algorithm could apply the same approach to the individual touchpoints by multiplying by the weights and then use the normalized touchpoint score as the input for the higher level calculation at the KEI level.

In an alternative approach, normalization can be controlled by setting a max threshold for the number of touchpoints in each KEI to be used for the calculation. When the true MAX is beyond that threshold, then the threshold value itself will be used in the calculation instead. This alternative normalization method avoids bias to outliers which could be skewing the score for other members when there are noisy or exceptionally large MAX data points.

Organizational Engagement Score

The scoring algorithm can also be used to calculate an organizational level of engagement. This includes the association itself or another organization that has membership representation in the association.

Organizational Score can be calculated by two methods: One is to identify indicators that represent pure organizational engagement and apply the same score computation using weights for those organizational KEIs. The second is to average the scores of members that belong to a particular organization. Each of these methods addresses a particular scenario and alternatively, they can be combined.

Scores for specific segments of the population (e.g. by member tier, demographics, etc) can also be calculated by having a different set of KEIs and corresponding weights that are relevant to each segment. Wicket would then take segmentation parameters along with the weights in order to calculate a score that applies only for members within that segment.

Summary

Based on our previous research survey of member engagement, we have identified the need for associations to define and continuously refine their own definition of engagement and to track how individual members measure up to such definition.

Through Wicket’s Member Data Platform, we created a framework to define and track member engagement as a measure of Key Engagement Indicators (KEIs) and related member activity touchpoints. The framework allows for the inclusion of weights that can be applied to KEIs and to individual touchpoints to give the relative importance in the calculation of a composite member engagement score.

Wicket’s algorithm can automate the process of calculating the engagement score based on touchpoints (activity data) stored across multiple software integrations and incorporating engagement indicator weighting into its heuristics. The algorithm also offers options for normalization in a way that makes the values coming from different systems stand in an even grounding so they are comparable.

In future work, we will explore the idea of extending the member engagement scoring algorithm to incorporate qualitative data as well.

November 30, 2020|admin