Over the past six years, as I’ve built a UX team from a solo practice to a team of 11, I’ve seen how design research can greatly influence product development. Early on, we had little time for customer interviews or usability tests. We were mostly shooting from the hip, listening to customer support, and revising on the fly.
We now do scores of usability tests, user interviews, and competitive analysis, and we create detailed reports summarizing our findings. But this brought us to a new problem: without a way to preserve and combine our results, our insights quickly slipped into the hazy distance as documents got lost on a hard drive, or ignored by someone in a different department.
We ended up living in a Groundhog Day research loop, asking the same questions and rarely building upon what we already knew.
Now we need connections—a way to pull together disparate data points, qualitative and quantitative data, and long histories of research into a central clearinghouse that can be shared, searched, and maintained by different teams. After years on a research treadmill, that’s exactly what we’ve started doing at MailChimp—and far from being just a data solution, open access to this information has strengthened the connections between teams, and supported a general culture of inquiry.
It all started with a personal crisis.
A moment of crisis#section2
Customer feedback streams into my inbox in spades from a form on the MailChimp website. Hundreds of emails offer ideas for new features or ways to make things better. I love reading them, but last summer I started to feel overwhelmed. I was reading hundreds of emails daily, many of which had useful feedback, but weren’t worthy of adding to our roadmap. Maybe down the road an issue would reach critical mass, but until then they sat in limbo.
It was choking my productivity, and making my head spin. A friend of mine who’s helped many people tame their inbox and prioritize their work life recommended I simply nuke all of the emails and shut down the form. “If you can’t process the information, then stop wasting your time!” But my gut told me there was value in the feedback; I just wasn’t sure how to use it.
In Gmail I starred the feedback that was worthy of consideration later, and set up a script to forward it to an email address associated with an Evernote account. It was now curated and preserved in a searchable database, which cleared my inbox—but I still had no plan for how I’d use the data.
Months went by. While studying trends in email automation to inform some new designs, Ben Chestnut, co-founder and CEO of MailChimp, sent me an email late one afternoon asking what we knew about how customers use RSS-to-Email, a related feature. We were making plans for our roadmap, and needed insights that would help us rethink both features to better serve customers. With little time for a detailed study, I turned to my bucket of feedback to see if a search would reveal anything.
Simply searching for “RSS to Email” returned 45 extremely helpful pieces of feedback on the topic, each with an email address provided when users filled out the feedback form. Normally, when conducting a study, recruiting users for interviews was like fishing with a big net. We’d post something on Twitter, or maybe even use a recruiting tool to find users that meet our criteria. It can be very time consuming. But with a database of feedback with email addresses, recruiting was like spear fishing. We found just the right people to speak to within seconds.
Patterns emerged in hours, not weeks. I could see what people struggled with, and how we could make simple changes to improve usability. I followed up by email with a few customers to learn more, and was quickly able to create a plan for how we could make this part of our app much better.
It was a lightbulb moment that left me wondering what patterns might emerge if we had more data to search.
More data please#section3
I shared my story with my colleagues Gregg Bernstein, Jenn Downs, and Fernando Godina, each of whom had data to contribute. Gregg moved user interview transcripts and notes into Evernote. Jenn added usability test findings, and Fernando, God bless him, read through more than 10,000 account closing surveys to find the ones that could tell us the most about why people close a MailChimp account. We were amassing a nice little collection of data of various types.
Ben once again sent us an email asking for insights about a feature. “It seems like more people are asking for an easier way to embed YouTube videos in an email. Do you guys know if there’s a trend here?”
Normally a question like this would be left unanswered, because it’s not significant enough to warrant deep investigation—but now we could find quantitative answers in seconds. Gregg searched our big bucket of data, and sure enough there was a small trend emerging. “11,” Gregg told Ben. “11?” Ben replied. “Yup, that’s how many people have mentioned YouTube video embedding in feedback, account closings, usability tests, and customer interviews.”
We were already blown away by this success. But we knew our data pool still wasn’t showing the whole picture. What pockets of data were out there in other corners of the company?
Data avalanche#section4
We talked to our support team, engineers, data scientists, analytics folks, social media people, and the email delivery team, all of whom have lots of valuable data. Soon large amounts of diverse data streamed in, and the ownership of the data pool shifted from the UX team to the entire company. The support team shared patterns they’d seen in emails and live chats. Engineers wrote scripts that grabbed aggregate data about popular pathways in the app and industry demographics of our users, and emailed them into the database weekly. Because Google Analytics lets you schedule emails with custom reports, we were able to stream in mobile device usage data and completion rates of important workflow funnels. Tweets, Facebook, and blog comments streamed in with even more customer feedback. Survey data, email delivery stats, industry research, and notes from each of our app releases—we added everything we could to the data pool to gain an even broader perspective.
Bringing teams together#section5
As we’ve opened our data up between teams, interesting things have happened. People who wouldn’t normally have occasion for conversation are meeting regularly to compare notes and share what they’ve learned. Our “data nerds” now get together for lunch, and share stories of the projects they’re working on. Data sharing is leading to new collaborations we never would have imagined.
Design researchers collaborated with the DesignLab to turn a months-long user study into a series of beautiful posters illustrating MailChimp’s customer archetypes. After a recent major redesign of the app, support and design research teams collected feedback from customers, printed it out, and tacked the notes underneath each persona poster—helping us triage issues from different customers and devise solutions quickly.
Each persona has a name and a detailed series of traits. When we do customer interviews, we store the transcripts and notes in our big database, and tag each with the name of a persona so we can see patterns in user types. What does “Andre” have to say about coding HTML email templates? How is “Ada” reacting to the redesign? Now we can answer questions like this by searching with both tags and keywords.
Though we’re not a company that struggles with political drama, close collaboration between teams certainly helps bring our people together and builds respect for the work we’re all doing.
We’ve found that when people are given the opportunity and the platform to share their data or do something new with existing data, they feel pride knowing their work is valuable to others. It feels good to see different areas of the company benefiting from the work you’re doing. Everyone wants their work to be valued and appreciated.
Everyone is a researcher#section6
Sharing the Evernote account with everyone in the company who wants access has encouraged ambient learning scenarios, in which people in all departments are browsing through our research and stumbling upon insights they never knew existed. As he was working on a new iOS interface, Stephen Martin, a designer in our MobileLab team, was curious which stats are most important to customers looking at campaign reports. He did a quick search in Evernote, and stumbled upon a chart from our 2013 survey sent to thousands of customers. He could see a clear ranking of the stats that customers use the most. He used this data to create a poster showing the hierarchy, which helped him make smarter design decisions driven by user research.
Now that everyone has access to the data, everyone is a researcher.
Regular insight outputs#section7
Even though the power of Connected UX is amazing, it’s easy for insights to languish in obscurity unless you’re regularly pulling them out and sharing them with your team.
Rather than wait for a project to provide this motivation, every other week we compose an email that goes to the entire company sharing interesting stats and broad trends. We rotate authorship to make sure many perspectives are represented. Each email concludes with an invitation to start contributing data—or simply browse out of curiosity.
It’s not the tool that matters#section8
We tried a lot of different solutions before landing on Evernote as our data hub. Wikis and custom databases always seemed too technical, and were bound to alienate some people who would love to contribute or just lurk. Your organization might find a simple Drupal install or a custom database works best. The storage tool really doesn’t matter, so long as it helps you adhere to these basic principles:
Easy in, easy out#section9
Any hindrance, no matter how small, preventing anyone from contributing or browsing data will kill the process. People shouldn’t have to learn new systems to be involved. Contributing data via email is perfect, because it requires no additional learning. Using a consumer software solution is also advantageous because many people will have experience with it. Eliminate all barriers to participation to get lots of people involved.
Ubiquitous access#section10
We all know that mobile devices are outselling PCs and extending the desktop experience into every part of our lives. By making your data accessible across multiple devices, you’ll find insights will happen more routinely—in the line at the grocery store, in meetings, or on the couch in the evening. Ubiquity of access makes ambient learning easier.
Data for everyone and everyone’s data#section11
Give everyone in your company access to the data and diligently invite contributions. It’s important that the data is open and shared so teams are encouraged to collaborate. From this collaboration you’ll find the most mind-boggling insights you would’ve otherwise never discovered.
Silo busting#section12
By connecting disparate data, you’ll discover trends in seemingly disconnected things. That’s exactly what has us so excited. We’re finding patterns between departments, and among customers. We’re breaking down the silos that separate data streams and the teams that manage them.
From the insights we’re gleaning come new, research-driven strategies for our company. That’s new for us, and it’s completely changed the way we work in just a few short months. It’s made us a smarter company and has helped us create more informed design strategies. We no longer lose research, and we’re all more aware of the collective knowledge we possess. What we’re building is more than just connected data—it’s a connected company.
The experience has left me wondering, are other companies also taking a connected approach to research? What insights are you discovering, and what tools are you using? Do you have a story to share? I’d love to hear from you.
This seems like an exceptional way to engender a user-centered culture across all of MailChimp’s departments.
Aarron, it sounds like this was a gradual, bit-by-bit process to get the whole company using Evernote and contributing data regularly. How long was that transition?
Also, did you run into any surprises with how you organized the “research pile?” For example, you realized you should’ve been using Tags for X and Notebooks for Y, but didn’t start out that way.
We guessed and shot from the hip for years, then started to get the people and time to do proper research. Making the transition from lots of fragmented information to a connected system happened over the course of a year. It’s not that the process takes a long time (there is some overhead in finding the data you have on hand and getting teams on board), but we just didn’t really know what we had until we’d experimented a little.
We’d put a little data into a central database, someone would ask a hard question, and we’d pull out an answer in minutes. Each time we did this we grew ever more excited about the potential. As we poured in more data and asked more questions, the principles of the experiment and why it worked became clear.
Your milage may vary in your organization. If you have lots of fragmentation in your teams, it could take longer to get people to contribute. The best way to get people excited is to show everyone what you’ve learned and tell them how you did it. Start small, shock and awe, then watch it grow.
Aarron, great article! Couldn’t help nodding my head in agreement, especially about busting silos. Thank you for sharing.
Love the article, Aaron! Very insightful.
Would it be accurate to say that most things are better being open source (with reasonable limitations, of course – e.g. open source only within the company)? It seems to me one of the greatest hurdles in making this work is finding the perfect balance between big, open source data and innovative organization of that data. It looks like you guys over at MailChimp UX have really gotten the recipe right here. Bravo!
Great article. I especially agree with the points stated in “It’s not the tool that matters”. If anything, I would probably add some thoughts about reputation points or some other social mechanism that encourage sharing. Motivating everyone is the key here.
Our company is facing a similar challenge right now, however in a slightly different environment (UX agency). We have lots of quality content, tools and templates, which are scattered across the company.
Evernote example is inspiring, but our plan is to set up a closed WordPress platform with some smart plugins to enhance posting, subscribing and reputation building. I believe it can be flexible and yet simple enough for people to use it. Has anyone here tried that?
@Brad: When data is open in companies, it moves, gets reused, gets questions and generates questions. It creates a culture where collaboration is natural and leads to outcomes that can’t be anticipated. You can see the ethos of the open source movement in this story, but to me it’s just about remaining connected to one another and creating a culture of sharing.
@Wojtek: The UK Cabinet office has done something similar with WordPress and had some success. I’d love to hear how your experiments shake out.
The problem with tests is that they are not sales. There is a real but subtle difference in terms of testing and actually selling a product.
Testing is hard, but compared to selling it’s easy. That is why designers love to test, we can avoid the real issue of sales.
The real magic of Mailchimp is the way it sells you it’s product and how design has influence over the resulting purchasing decision.
Hopefully we can talk and think more of how testing helps to sell better (if not any other product or service then to sell design services itself)
It is great article!
Thank you for sharing.
自動車査定
@Wojtek: We built something similar using WordPress (for a slightly different scenario but similar insofar as finding/sharing information across the organization that can serve as a single data source for multiple purposes).
One of the main drivers for going the WP route was Aaron’s first principle of “Easy in, easy out” – a lot of folks in our organization have experience with WordPress as do many of our new hires.
We took a standard WordPress install, added a custom post type with some unique fields as well as custom taxonomies, and added in some functionality to filter, search, and subscribe to submissions.
We still have a ways to go to realize our bigger vision for the tool, but to speak to Aaron’s comment to Brad, our initial release certainly has helped shift that mindset in the organization to think more about connecting, sharing, and collaborating. Culture shifts take time and every little step helps 🙂
Great article, Aaron! Thanks.
Hello there, just became alert to your blog through Google, and found that it is truly informative. I’m going to watch out for brussels. I’ll Thankful if you continue this in future. Many people will be benefited from your writing. Cheers!
Inspiring read! I love the idea of going with Evernote versus some obscure CMS.
Any tips on getting buy-in? For instance, if we start dumping customer data into Evernote, how can we demonstrate the value of it to executives?
Really great practical insights here. Have been using Google Docs in a similar fashion, but the way Evernote (or other similar tools) can tag and reorganize things seems like an really powerful incentive to shift over and make connections from disparate pieces of data.
Thanks!
@Jake: The best way to get buy in is to start small, feed in a bit of data you already have on hand and experiment. When tackle your next research project, turn to your database of insights first to see if patterns are present. If you have feedback in your database with email addresses attached, try recruiting users for interviews with this data.
Start small and if you can get away with it, don’t ask for permission. Evernote is free, other similar tools can be used for free too. Just feed in the data and start experimenting. When you have a couple of cool insights, share it with colleagues. They’ll wonder how the hell you found that info, that’s when you tell them what you’ve been up to.
Aaron, this was a great article, thanks! As soon as I finished reading I started to dumb data into Microsoft OneNote but I found a limitation that also exists in Evernote. How do you pull your data out without complex boolean operations? It just seems to be a limitation that will be frustrating to work around…
@Holly: So far we’ve not really needed complex search operators, though some of us have used them for special circumstances. We’ve found that tagging regularly and triaging into specific notebooks helps us keep things organized. We search inside one notebook to get specific or across all when we have a broad question.
I’m curious as to what sort of queries you’re trying to run.
I often use the WP blog, your article very helpful to me. I took a standard WordPress install, added a custom post type with some unique fields as well as custom taxonomies, and added in some functionality to filter, search, and subscribe to submissions.
My Blog
Great post! I’m lucky to work for an organization that has been working hard recently to break down these silos, and I think we have an advantage in that our user research team doesn’t belong to any product group, so they tend to be able to see these connections pretty readily.
what Lois said I’m surprised that some people able to make $4808 in one month on the internet. visit this web-site http://BAZ4.COM
Great article! User data what ever might be the source user research or analytics or customer support are useful only when they are not hoarded…shared across the silos in an open environment and there is an attitude use it for drive improvement.
Thanks for the fantastic article. Question: when data is unstructured (like yours), does that make it hard to do a search in Evernote and find everything related to a certain topic? For example, there may be a lot of customers complaining about the design of your dashboard, but they all might use different terms (dashboard, account page, main page, home, my account, control panel, etc.). To overcome that, do you need to tag each piece of data? Thanks Aarron!
We often hear about data collection and interpretation but I realize organizing it (with the right tools) would help a lot.
I’m currently using Olark for a personal project and the information flow stops at me reading the emails in the inbox. Next step: integrating Email+Evernote and understanding how to share that info across a 3-person team.
I’d be interested in more details of how you make that Evernote hub work so well within your team.
Thanks again for the great article.
Great workflow – looks amazing.
Aarron how do you handle your data over time with keeping consistent with your updated product?
I mean you have feedback/bugs/insights/etc., you iterate, your product evolve and then few month/years later all those useful insights don’t match your current product and becomes obsolete.
Do you clean your database? do you tag it? or just keep the info and then back to square one with a huge sets of data with only some relevant.
How do you deal with those issues?
Though not directly relevant, I must take this opportunity to say that I really dislike the mailchimp rss email template. I have now integrated it a handful of times, and every time I find it frustratingly limiting. As a matter of fact, I have now dumpted using it all together due to the styling limitations that it imposes, and instead opt to have the web pages designed, and point mailchimp at the url where the page is hosted instead.
@marc d So far stale feedback hasn’t been a big issue for us. As we do searches in our database and come across feedback about things that we’ve already addressed, it’s almost always very clear to us and we just delete the things that could lead us astray. This may be more of an issue for gigantic organizations working on monolithic apps, but for us it’s been quite manageable.
I really enjoyed this article, and advocated for using Confluence as our DB. I never saw people going to that well to find insights, and I’m convinced that’s just because of the nature of our business i.e. consulting vs product development. Regardless, I’m convinced this can be a highly valuable tool for cross functional teams.
My biggest “worry” was/is the one that @Michael W raised above: if people don’t tag or add other useful meta data to help with structure and venacular, how do you know search’s will get you useful information and understanding how quantitative it really is? Would a simple tagging scheme or an optional form that people fill in when they contribute data help ensure that the results returned from search’s is complete, useful and meaningful? My gut is that such form’s would make people less likely to contribute and just get in the way, but I’m curious what people here think.
I like that Evernote has the concept of related articles. That just seems like a brilliant concept. It seems like a little tool or survey that allowed you to somehow train the algorithm for that would be quite useful.
love it. great to read things that affirm what I am doing!
Me too Joel Me too My Website
I really liked it, thanks Aaron. I´ve got a question:
From time to time, a very low computer affinity gives some feedback, but they never purchase. This means this is not a kind of customer to address.
Would you create a low computer affinity persona in order to tag notes that you won´t solve in the short term?
Thanks in advance!