Litmus

This summer, I interned at Litmus, the leading email marketing platform used by marketers, designers and developers at more than 250,000 companies, including Google, Amazon, Salesforce, Uber and Airbnb. Litmus stood out to me as an incredibly design oriented company- the platform was cofounded by a designer for designers. Because of this, my internship at Litmus sharpened my use of design processes and experience working with UX designers, product managers and growth marketers. During my time at Litmus, I led a user research project for the ESP Syncing feature, explored new directions for the home dashboard and redesigned the Spam Test feature. Learn about my research project below. 

Esp Syncing Research Project

OVERVIEW & RESEARCH GOALS

The Litmus Growth and Product teams have uncovered substantial drop-off in the ESP Sync set-up process in Builder, suggesting that there could be improvements to the ESP Sync experience. ESP Syncing in Builder is a feature that allows users to automatically transfer their code from Litmus' coding platform, Builder, to their Email Service Provider. This feature is an integration between Litmus and Salesforce Marketing Cloud, IBM Watson Campaign Automation, Oracle's Eloqua, Marketo and MailChimp. As a study focused on a live 3rd-party integration, this user research project was relatively unprecedented as extensive testing is primarily employed in the prototyping process for new features. Because of this, clear definition of goals and methodologies were key to deliver useful insights from data analysis and user interviews on the issue of ESP Syncing drop-off. Through conversations with the Product team, I primarily identified four goals for this study: 

  1. Determine whether the ESP Sync feature is/could be integral to users’ email workflows.

  2. Uncover usability issues with ESP Sync.

  3. Uncover efficiency issues with ESP Sync.

  4. Assess the efficacy of the feature across platforms (i.e. find bugs that may occur on the ESP side of the Sync)

METHODOLOGY I: AUDIENCE & PROFILE

  • Designers & Developers: primary user persona is a designer/developer who wishes to automatically upload HTML to their ESP, in order to avoid the inefficient and error-prone copy-pasting process.

  • Marketers: a marketer can avoid mistakes handling HTML, as her email is preloaded to their ESP by this feature.

Participant Group should include: 

  • US-Adjacent Time Zones (PST, EST & GMT)

  • Enterprise customers preferred

  • Designers/developers who have clicked on ESP Sync but did not sync

  • Designers/developers who have clicked on ESP Sync and synced their email

  • Users who began the ESP Sync funnel but did not complete set-up

METHODOLOGY II: ISOLATING USER DEMOGRAPHICS 

Using Heap Analytics, I was able to isolate users who clicked on ESP Sync but did not sync between April 20th, 2018 and July 20th, 2018. I was also able to find the converted users (those who clicked on ESP Sync and successfully synced their email) in the same time range. While the number of drop off users was incredibly large (~2000), after filtering for users in other Litmus research studies and users that were on Enterprise plans (rather than Pro, Plus or Premium), we narrowed the user testing demographic to 120 enterprise customers. Using this methodology for the converted users, we narrowed the second user testing demographic to 29 (however, we decided to include premium customers due to the small number). We decided that we would use the limited user lists for our first batch and based on the response rate to our feedback request form, we would send the survey request to another batch. 

METHODOLOGY III: SOLIDIFYING THE INTERVIEW EXPERIENCE

The Product team and I quickly realized that ESP Syncing was an incredibly difficult feature to test due to constraints on the users' login credentials (the user would need login credentials for all ESPs, as well as, admin credentials for Salesforce and Marketo. Specifically, we were challenged by the question of when to ask users if they had access to login credentials, afraid that if we asked them too early, we would deter them from participating in the user research study. Simultaneously, I was inquiring ways for Litmus to set up tester accounts in the respective ESPs, to avoid any concerns about sharing sensitive client information (specifically when interviewing subjects who work at marketing agencies). In order to demonstrate the different iterations of the interview flow, I created a LucidChart to help pitch to the team why we should send a survey inquiring about the login credentials of potential user interview participants before confirming an appointment slot, rather than allowing them to book their appointment prior to requesting login credentials. 

USER INTERVIEW FLOW

After I presented these potential outcomes to key stakeholders in the project (Lead UX designer and Senior Product Manger of Enterprise), the team decided to move forward with flow four which entailed sending out a survey about login credentials prior to scheduling a user interview. However, while our response rate (more or less) aligned with our predicted response rate of 10-15%, many users did not have access to their ESP. Because of this, we had to pursue the contingency plan articulated in the fourth flow, which involved setting up a tester MailChimp account for our participants to use. 

FINDINGS

In this user research project, we explored the low usage of ESP Syncing in Builder. After encountering data indicating the feature is underutilized, we decided to collect a group of users who both have used the feature and those who dropped off in the ESP Syncing process. After conducting six interviews with the aforementioned demographics, we encountered two primary conceptual issues that result in low usage from our customers. First, we found that many users did not trust ESP syncing due to data security concerns, such as fears of problematic overriding of code and syncing inconsistencies. This especially seemed true in the case of users from digital marketing agencies who were working with tight deadlines and sensitive client information. Second, we found that some users did not find copy and pasting code to be a major pain point in their email workflow. Thus, users appear to drop-off, due to both distrust of the feature and perception of its limited value. However, other users articulated ESP Syncing to be a major feature in their email workflow, indicating that alleviating confusion and reluctance surrounding the feature may substantially improve Litmus’ user experience. We encountered two primary types of issues that perpetuated user distrust and lack of engagement with the feature:

  • Learnability issues

  • Utility Issues

Alongside both these issues, a substantial number of bugs were uncovered during this study, which are similarly reported below.

LEARNABILITY ISSUES

Overall Learnability of the Feature

Throughout the user interviews, especially with those who dropped off in the ESP Syncing Flow, users reported that there was insufficient information about the feature throughout the set-up process for ESP Syncing. Many users cited that they were unsure what the feature did and that upon clicking “Sync ESP,” there was not enough information to convey its function.

Potential Solutions: Users suggested that a more expanded description of the feature would be helpful to better understand how it adds value and generally functions. This could both convey to the user why ESP Syncing can truly expedite their email workflow, as well as, why their data is safe when they use ESP Syncing in Litmus.

1. Templates vs. Campaigns Option does not provide sufficient information to users: Some users suggested that Litmus elaborate on the two options for syncing, especially for users that had never synced before. Those who had synced generally reported that the two options made sense to them, but that they rarely sync to templates.

2. Redirect link to project in ESP is confusing for new users: Users reported that they would prefer some description of the link’s destination accompanied with its url. Another potential change is to include the full link to allow users to copy and paste into another browser. In the case of a participant using Eloqua, he had to use another web browser when accessing his ESP, and thus may find the ability to copy and paste the link to be helpful. One user also reported that the log-in redirect to be misleading and that she expected to be led directly to her synced campaign upon clicked Sync.

Potential Solutions: One user suggested that she would prefer to be prompted to log in before syncing her campaign. Thus, a potential solution is to prompt people to login in the ESP Syncing side panel.

  Figure 1.  Log-In Prompt in ESP Syncing.

3. The default configuration does not adequately represent the likely preferences of the user: This is compounded by the lack of discoverability of ESP syncing preferences. These default settings challenge users by:

  • Turning email tracking off when synced

  • Changing the bounce back email

  • Overwriting the subject line with the Builder project title

  • Importing files into the “All Files” folder in ESPs, rather than prompting the user to specify a folder or group.

Potential Solutions

One user suggested that Litmus make these settings both customizable and more accessible to the user. To reflect this, I believe that each ESP listed should be accompanied with a settings wheel to provide the user an easy way to change these default preferences. When the user clicks the setting wheel, a modal pops up that allows the user to input the campaign into a given folder, enable auto-sync, enable tracking and choose which data to transfer over as the subject line.

Figure 2. ESP Syncing Settings

4. Users are unsure about the re-syncing process: In multiple user interviews, the user was unable to properly resync his or her campaign within the first attempt. It appeared that users were often unaware that they must save the campaign prior to resyncing. Many users also assumed that they ought to press the button “Sync to Your ESP” again for Litmus to resync their campaign. This demonstrates that many users are unaware that syncing is automatic, rather than manual. When asked about whether they would prefer manual versus automatic syncing, some users actually indicated that they would prefer for syncing to be manual to avoid unintentional overwrites.

Potential Solutions: Given this feedback, there are several directions the product team can take to alleviate any uncertainty about the re-syncing process. First, the product team can transition the feature from automatic to manual. I suggest that the user should be able to resync solely by clicking “Sync to Your ESP,” rather than having to access the “Sync to Campaigns” button in the ESP Syncing menu. Alternatively, we can preserve the current functionality, but focus on better educating the user about the ESP Syncing process. In this scenario, I suggest that Litmus introduces a “Last Saved” indicator into Builder, to keep the user informed as to which version has overwritten her code in the ESP. This could both prompt the user to save her code, in order to reflect changes in her ESP, as well as, inform the user of overwrites. In addition, this “Last Saved” feature could be expanded to allow the user to access previous versions of her email in Builder, helping her use the project in Litmus as “the source of truth” across multiple versions in different platforms.

5. Users are unsure how to disconnect ESP Sync: Users were often confused by how to properly remove ESPs from ESP Syncing. Some users opted to remove an ESP via “Manage ESPs” in Settings, rather than from the ESP Syncing feature in Builder. This is primarily because of confusing iconography in the feature. In order to remove an ESP, a user must click the green check mark, which then turns into a trash can. Thus, many users are unaware of how to access the trash can, as the green check mark does not connote “delete.” This issue with iconography is echoed in the “stop syncing” option which is represented by a cloud, during the syncing process (which becomes the green check mark).

6. Users are uninformed about how ESP Syncing transfers their assets: Some users indicated that they avoided the ESP Syncing feature because they often are unwilling to try new feature (especially with client data, in the case of digital agencies). Because they are unaware of how code and assets will be transferred over to the ESP, they prefer copy and pasting, to avoid potential version inconsistencies and time wasted.

UTILITY ISSUES

Overall Utility of the Feature

1. Teammates can keep making changes to Litmus that will override users’ edits in Eloqua (this especially negates Eloqua work-arounds for syncing inconsistencies): As seen throughout this report, a common theme among participants in the ESP Syncing study is fear surrounding version control across platforms. Some users expressed that they are wary of syncing their ESP because teammates can override changes made in their ESP. This concern surfaced in our interview with an Eloqua user, whose work-around for syncing inconsistencies is depicted by the whimsical chart, linked above. The user expressed fear of sending an email without the components secured by his workarounds in Eloqua, due to a last-minute overwrite by a teammate which he was hypothetical unaware of.

 2. Even with ESP Syncing, users have to re-enter data that is not properly carried over to their ESP (ex. Subject Line): His workarounds in Eloqua draw upon a larger utility issue-- the existence of numerous syncing inconsistencies which result in users re-entering data. Multiple users expressed that the pain-point of copy and pasting is not sufficient addressed by ESP Syncing, because they still must re-enter data, such as the subject line into their ESP.  Moreover, the presence of multiple bugs similarly contribute to the decreased value of the feature in users’ workflows.

Figure 3. Uncovered Bugs

PREDICTED DROP-OFF DUE TO ESP SYNCING LOADING BUG

In attempts to quantify the effect of this common issue facing UX research participants, I examined the relationship between user interaction with the “Syncing” Loading Icon and drop-off in the ESP Syncing flow. I found that many users actually pressed on the “Syncing” Loading Icon, presumably to attain confirmation and links to their synced campaigns. Throughout user interviews, this was a common response to the bug. To illustrate the user drop-off in the ESP Syncing funnel, I documented the number of users who completed each step in this flow between July 13th and August 13th.Using this information, I calculated the percent change in ESP Syncing conversion rates to illustrate how users who clicked on the “Syncing” Loading Icon (presumably in order to speed up the delayed syncing process) were less likely to successfully sync their campaign.

GENERAL IMPROVEMENTS TO BUILDER & PROOF

Many users had more general feedback about the Litmus product, specifically focusing on two of Litmus' primary features, Builder and Proof. Below, I reported user feedback to suggest potential directions for product improvements.