Exploring How AI is Changing UX Through Product Analytics
Product Design (UX, UI, UXR), Product Management, Product Marketing
Overview
This personal project originated from my curiosity about human-AI interaction and how AI could accelerate non-technical learning of technical concepts. What began as a small idea evolved into designing a full design system after multiple product pivots.
The project has provided me with substantial growth opportunities, allowing me to design without technology constraints. It deepened my interests in AI and product analytics and allowed me to explore career paths outside product design to incorporate product management, and marketing. While some may view pursuits across these disciplines as stretching myself thin, I thrive at the intersection and complexity of these disciplines.
This project explores the use of generative AI in a product analytics platform. I wanted to understand the potential of human-AI interaction and how this groundbreaking technology can help non-technical folks speed up their learning curve when it comes to understanding technical stuff.
The goal of this project is to learn two main things:
How people engage with AI and adopt it into their lives.
How AI can help product leaders make sense of product analytics.
This project was treated as a real product and a real company because I want the experience of identifying a product-market fit and deliver solutions to immediate pain points.
🧑🎨
Product Designers
Product designers require efficient data analysis tools to extract high-impact insights to support their design efforts and strategy validation. Their goal is to discover opportunities for design improvements.
👩🏫
Product Managers
Product managers need product data insights to allocate company resources effectively to ensure their product strategies’ success. Their goal is to identify and monitor key product metrics.
👨🏫
Product Marketing Managers
Product managers need product data insights to effectively communicate their product positioning in the crowded market. Their goal is to identify opportunities for product adoption and growth.
What started as a smallscale project to demonstrate my product design skills and explore product management and product marketing, has evolved into something much larger and more complex than originally planned. Below is the evolution of the project.
🤔
Initial Scope
This project began as a personal challenge: to showcase my product design skills by designing a user flow for interacting with AI for data exploration. The initial focus was on a single flow – users asking data questions using AI.
🤯
Surprise! A Feature Hit
Through user interviews, I discovered a major opportunity beyond the initial scope: product designers found my concept of AI-generated hypotheses to be a value prop for them, stating that they’d “totally use this if this is a real product.” This unexpected insight led me to iterate and create a high-fidelity design that incorporated this functionality.
🔀
Shifting Gears (It Happens!)
Further user testing with product designers revealed a shift in priorities. While AI-powered data exploration was interesting, their core need was data insight consumption – receiving reports generated by product managers. This valuable feedback allowed me to pivot and refocus the project on the needs of product managers.
🍱
Introducing a Design System
With a clear target audience, I saw the opportunity to create a scalable design system alongside the product itself. This system would ensure design consistency and facilitate future growth for the product. Besides, I can use take this design system with me to any company that needs help with their design system.
This project evolution has been a transformative experience for me, offering a glimpse into a first-hand understanding of the iterative process in bringing an innovative product vision to life based on real user needs. This experience realigned my desire for a role that provides the creative freedom to explore bold, cutting-edge product ideas without being constrained by engineering limitations from the outset.
Since this is an ongoing project, I’ve come up with a few versions for each round of iteration. This section aims to give you a quick overview of how this product concept has evolved.
Version 3B
This version is an extension of 3A. The UI has been updated to demonstrate my design system based on Google Material Design 3.
Additionally, this version highlights how I leverage data visualization to present useful product analytics information, particularly with the HEART framework that focuses on UX metrics.
There are three scenarios so far. Click on each of them to go through an interactive user flow!
Version 3A
This revamped version shifts its focus towards product managers. Based on recent interviews with product designers, it’s likely that many would act as passive consumers of product analytics. They expect the PMs to handle the data, analytics, and metric-setting curation.
Drawing from my past experiences working with and as a partial product manager, I recognize that finding the time to develop meaningful dashboards can be challenging. The workflow would be greatly enhanced if an AI could generate a preliminary draft to jumpstart the process.
This concept still needs to be tested, so stay tuned!
Version 2
This iteration aimed to gauge the potential utility of AI-generated dashboards and reports as an integral aspect of product designers’ workflows. As a result, I learned that designers:
Have zero interest in generating dashboards. However, they find that receiving a report from PMs would be immensely helpful in helping them understand the metrics that they should be striving for.
Want AI to provide tailored suggestions on how they could improve the metrics set by PMs.
Version 1
This version aimed to understand how non-technical users naturally search for data so that I could make the data experience as intuitive as possible. In this case, I asked product designers to pick the type of data they’re interested in and walk me through where they’d click to gather more information. As a result, I learned that designers:
Love the hypothesis generation feature where the AI forms hypotheses based on data trends on the users’ behalf. This would significantly accelerate their workflow by reducing the time needed to locate and assemble the required data. As a result, users can dedicate more time to thoroughly examining and exploring the hypotheses, enhancing their overall productivity.
Find the AARRR framework that product managers like to use to be valuable in guiding them on what data trends to look for.
Below were the challenges I faced as I started this project.
😵💫
Limited Knowledge of AI and Its Rapid Evolution
It was a race to keep up with the evolving landscape.
😕
Limited Knowledge in Product Analytics
My understanding of product analytics was minimal, posing a challenge in effectively measuring and interpreting data.
🧭
Defining Project Scope And Direction
Determining the appropriate scope of the project was a challenge, as I debated with whether to aim for a larger or more focused project.
🤔
Primary User Selection
I was unsure whether to prioritize product designers or product managers as the primary audience for the project.
🎯
Product-Market Fit
How will I know if I found product-market fit?
To address the challenges, I adopted a proactive and structured approach, focusing on continuous learning and user-centric design.
🤓
Limited Knowledge of AI and Its Rapid Evolution
I attended AI and design events to gain insights into the evolving landscape and interacted with product designers, managers, and AI enthusiasts to understand their data usage and AI expectations.
👩💻
Limited Knowledge in Product Analytics
To improve my knowledge, I enrolled in data courses on DataCamp and attended webinars on product analytics.
📝
Defining Project Scope And Direction
I utilized my templated project kickoff questions, which helped me narrow down my focus and establish clear objectives.
🧑🎨
Primary User Selection
I started by targeting product designers, as I could leverage my own experience and insights, while keeping product managers as a stretch goal.
📍
Product-Market Fit
My goal was to design a product that addresses real user problems. I researched competitors to identify market gaps and strengths, and conducted concept tests with the target audience before finalizing the UI design to address user pain points effectively.
Continue reading the rest for my approach to Version 1 of the project! (As you can see, I'm now on Version 3B. This section details how I developed Version 1. For a summary of later versions, click here.)
This case study will highlight the messy reality of designing a product in an emerging industry. If you're ready to dive into the journey, read on!
Market Discovery
Goal: Focused on understanding the problem space and users’ needs.
I once read an article that went somewhere along the lines of: “I think my startup failed because I had a solution in search of a problem.”
This resonated with me, as I had previously approached product design in a similar way. To change this, I shifted my focus to understanding the problem I was aiming to solve rather than jumping straight to a solution.
So, I attended design conferences and AI-related events to learn how data can guide design decisions. Through these experiences, I discovered:
Senior and generalist designers, as well as managers, want to use data but often need help from data analysts.
Quantitative data is valuable for validation but should be paired with qualitative data.
Less experienced designers focus more on improving their skills than on data analysis.
Many people don't have access to their company's data but wish they did.
Product Design Brief
Goal: To define the scope of this personal project.
To initiate this personal project, I started with a product design brief - a tool I rely on to define project scope and align teams.
This brief serves as a consistent starting point for me and the team in any new project. After outlining the scope of this personal project—which seemed straightforward in hindsight—I struggled with where to begin.
I decided to start from my perspective as a product designer. As I considered potential questions, I realized I lacked in-depth experience in this area. I could formulate some product management and business development questions, but I was unsure if they were the most effective.
Industry Research
Goal: To understand the market and competitors in order to identify useful data.
Product Management Questions
Realizing that if I, as a designer with a product management and business development mindset, needed guidance on what types of questions to ask, then other designers with limited PM and Biz Dev mindset would need similar guidance. Since I couldn’t get access to talk to PMs right away, I was pondering about where to start. Even if I did have access, I wouldn’t know what questions to start asking them.
Around that time, I came across Interview Queries, a learning platform offering take-home product analytics challenges. While it provided useful questions and themes, I wanted more examples.
This was when I had the idea to ask ChatGPT for help.
To get the ball rolling, I gave ChatGPT the prompt:
“What are some saas product usage analysis that product managers want to know.”
While asking ChatGPT a series of follow-up questions, I started to notice some trends in the questions. I wanted to identify the trends, so I started a doc to try to identify them.
Competitor Research
Before I jumped into what a solution could look like, I thought that I should scope out what’s already available in this product usage analytics landscape that I can use as a benchmark.
I checked out “competitors” of this fictional company, such as Mixpanel and Amplitude, and documented what their onboarding experience was like.
I also looked at adjacent competitors for inspiration, such as Thoughtspot, Akkio, and Generative BI.
Things that stood out to me that I would bring to this AI analytics platform were:
Provide demo data so that potential users can have something to play around with
Better onboarding educational content, such as why some metrics matter
Access to SQL queries (for those who want to double-check AI’s calculations)
Provide benchmark metrics to help users understand the importance of what their business is tracking
See if there’s a way to distinctly differentiate product lifecycle and customer lifecycle
User Journey Map
Goal: To visualize a product designer workflow throughout all stages of the company.
Creating a user journey map has been a key part of my design process since the start of my career. It provides a high-level overview of the user experience, enabling my team to take a holistic approach to developing an MVP. This visual representation helps illustrate user inputs and outputs, guiding the next steps in the journey.
For this project, even though I'm working alone, I'm following the same approach. I created an initial user journey map to outline the potential experiences of a product designer throughout the product and feature lifecycle.
Main User Stories
Goal: To identify which user flows to focus on first.
Ideation
I needed to get some ideas out of my head and onto something that didn't look official, so I decided to start with pen on paper. I started with the two target audiences that I was already contemplating: product designers and product managers.
Expansion
After the initial ideas were written down, I wanted to expand on them and also needed a way to keep track of them in the future. So, I jumped on Coda to start documenting, planning, and potentially prioritizing.
Wireframes – Round 1
Goal: Create low-fi sketches so that people could have something to react to.
Hand Sketches
I began sketching my ideas while writing down user stories. To avoid becoming too attached to my own solutions, I used sketching to explore different user flows rather than defining them as set paths. Although idealistic ideas may seem wasteful, this process helped me explore various options and visualize design opportunities and trade-offs.
Digital Sketches
There were many angles which I could start designing from. I took a look at my list of user stories and user journeys to see if they would give me any ideas on where to start, but my own creations were starting to overwhelm me.
To simplify the process, I set aside the user stories and focused on the core of the product: asking AI a question. This became the starting point for my wireframe.
I refined several concept variations, but with my first user interview approaching, I stopped iterating and went with the latest iteration.
Concept Testing – Round 1
Goal: To explore if the low-fi wireframe is the design direction that I should take the product.
Participants
I interviewed three product designers with various backgrounds:
PD with limited PM & BD experience. This designer works in an enterprise setting with other designers and a more established organizational hierarchy.
PD with some PM & BD experience. This designer works at a startup and specializes in building 0→1 products. They’re also taking courses to learn more about product management and business development. And they’re working on their own side product.
PD with extensive PM & BD experience. This designer works at a startup and has extensive experience in product management and business development.
Research Questions
I took the usability test approach to this exploratory UX interview. The framework below guided my questions, which were all aimed at uncovering directions for the core of the product: asking an AI an initial question, asking follow-up questions, and incorporating insights into their workflows.
Context into what product initiative they would want to start
How they expected to navigate through the screens
How they would ask follow-up questions
What types of follow-up questions they would ask
What their next steps would be
Research Findings
My initial assumption was that they would take the chat-based approach to ask questions. To my surprise, they wanted both chat-based and interactive chart capabilities.
Here were some more surprising observations:
Participants with data analytics experience expected familiar design features like clickable charts for detailed info.
Initially using AI as a chat tool, participants later wanted AI to complete tasks beyond calculations, like generating charts from filtered tables.
Some findings confirmed my assumptions:
Participants wanted to grasp their data landscape before formulating additional questions.
They also wanted AI as an educational tool to explain metrics and their significance.
Wireframes – Round 2
Goal: To update the wireframe based on the concept test findings.
During this time, I learned about the AARRR framework (Acquisition, Activation, Retention, Referral, Revenue), which outlines the stages of the customer journey. It helped me link marketing, product, and sales data, so I included it on the first screen for users upon login.
I also added the “hypothesis-generation” capability to see if the designers would find the AI-generated hypothesis helpful in sorting through data noise and understanding data more quickly.
Concept Testing – Round 2
Goal: To conduct a follow-up concept test on the updated wireframe to determine whether the new design approach could result in a shorter learning curve.
I conducted the concept test with the same three product designers from the first round. Again, I took the usability test approach to understand the participants’ trains of thought.
Research Findings
The updated wireframe was a success. All three product designers found the AARRR framework helpful in guiding their data analysis. The AI-generated hypothesis pleasantly surprised them, with all three expressing interest in using it if the product were real.
UI Design / Interactive Prototype
Goal: To translate the wireframe into a UI to share progress.
To save time, I searched for existing design systems that matched the simple UI I wanted, including graphs and dashboards. After testing several, I chose Lookscout’s design system. If I need to scale in the future, they offer add-on templates for purchase.
With a design system selected, I started playing around with the UI and created several iterations.
What I Learned
This personal project provided invaluable lessons that spanned multiple product disciplines. Let me break down some key takeaways from the lens of a product designer, product manager, and product marketer.
Product Design Lessons
One key lesson I learned is the importance of clearly defining my hypothesis before engaging with potential users. While I had rough hypotheses in mind, not clearly articulating them upfront led to some confusion during early exploratory research sessions. Moving forward, I'll make it a point to thoughtfully capture assumptions that can then be pressure-tested through user feedback.
Product Management Takeaways
From a product management perspective, this experience sharpened my ability to identify market gaps, gather meaningful user insights, and adapt strategy accordingly. The pivots and evolutions this project underwent demonstrated skills like vision, user-centricity, strategic thinking, and leadership - all vital for successfully driving innovative product development.
Product Marketing Perspectives
I learned how to differentiate my product in the competitive product analytics space. When discussing the idea with other professionals, I often received questions about how my idea stood out from existing products. I shared my perspective as a product designer with management and marketing skills, highlighting the gaps I noticed and the opportunities to empower user groups I resonated with. The frequent question, "Would you turn this into a real company?" was a positive indicator that my message resonated and effectively positioned the product.
Transferable Skills for Success
This project has helped me hone my ability to:
Craft a clear vision: Articulating a product's direction through a well-defined hypothesis.
Embrace user-centricity: Prioritizing user feedback throughout the design process.
Adapt with agility: Pivoting strategies based on user insights.
Communicate effectively: Explaining product value propositions in a competitive landscape.
Overall, this project taught me invaluable lessons that will make me an even more well-rounded, efficient, and impactful product professional going forward. The ability to adeptly wear different hats while maintaining a unified product vision is a superpower I aim to continue developing.
What’s Next?
This project has been immensely fulfilling, and the journey continues! While I'll keep tinkering and adding features whenever inspiration hits, my immediate focus has shifted to growing a robust design system. Get in touch if you want to learn more about where I’m at!