A Data Science AI Management Software
Role: UX Designer and Researcher
How might we help data scientists in better tracking their work, tracing decisions and simplifying team communication?
Vectice is a data science AI management software that helps in Driving knowledge, Collaboration, and Traceability to increase ROI and lower risk of AI initiatives. Vectice automatically connects the apps you use everyday for you and your team.
Vectice allows Data Scientists and Data Science Managers to achieve their potential goals of “reproducibility”, “replicability” and “referencing”.
As the sole product designer, I helped Vectice through the product release by conducting research and navigating through the design thinking process. I also mentored a group of visual designers on the team. I designed and prototyped major parts of the application, performed usability testing, and ensured that the experience was intuitive, cohesive, and accessible. I also wrote the script for user interviews, usability tests and conducted them with data scientists and managers from companies such as Apple, Google, Facebook, Cisco, SAP Concur, Lyft.
To identify our users and to better understand how our users currently navigate, track projects, maintain documentation and the associated pain points, I conducted user interviews with the help of my manager. We interviewed 30 Data Scientists, Data Science Managers, Product managers, ai scientists ranging from smaller companies to larger firms like Google, Apple, Lyft, SAP, Facebook, Microsoft.
The main research questions I had going into the interviews were:
1. What products do they currently use to perform their tasks
2. Do they document their progress of work? If yes what platforms or tools do they use for documentation?
3. What tools or resources do you generally use to collaborate with your manager/ teammates/ stakeholders?
4. What are the biggest challenges preventing them from doing their job efficiently?
During our research phase I also looked into other products which our users currently use as a source of inspiration. For product inspirations we looked at kaggle, benchling, notion, mlflow, Jira, Sagemaker etc.
Insights & Takeaways
1. Tracking of assets such as projects, progress, performance, experiments, is the most essential
2. Contextual documentation is missing right now
3. Need for Searchable assets which can help in reproducibilityNeed for data lineage to understand what goes into models
4. A more organized way to differentiate experiments, quoting a participant “Keeping experiments today in spreadsheets - can be more organized”
5. Quoting a participant "When I left 90% of what I did, died with me, even though it would be very helpful for the group" - knowledge transfer is. difficult when everything is all over the place
6. Data versions and documentation in context to the dataset doesn’t exist
The type of assets that matters to data science teams are:
1. Derived datasets used for modeling or data analysis.
2. Experiments that were tried and worked/did not work
3. Documentation and learnings from projects
4. Code and notebooks tied to data analysis, data transformation or experimentation
5. Online models and their performance
From our research, we created 3 personas: Data Scientist, a busy individual who wants to track projects, metrics, rather than business metrics. A Data Science Manager who wants to showcase, track team progress , and business Analyst/ Prod Manager, who are interested in Identifying high value data science projects and providing business context to Data Scientists. Distilling our research into personas led us to effective, user centric solutions. For example, we designed the project dashboard, and main dashboard to incorporate the Data Science Manager’s needs to track project and team progress.
Designing the product
Product Vision and Definition
Vectice is the first data science management platform that allows data science teams to:
1. Catalog their existing assets to make them more discoverable and reusable
2. Capture the knowledge on top of those assets and the context where they are used
3. Enable the team to capture their project progress and documentation, collaborate
The product is the most appealing to enterprises with 15+ data scientists where those challenges are most acute.
To succeed, the product should:
1. Have a low barrier to entry for data scientist to use it
2. Not disturb DS team current work and be compatible with the tools they already have
3. Provide visibility to managers and allow them to take control of their DS processes
4. Be as automated as possible to catalog and understand the relationship between assets
5. Support enterprise-scale, be secure, offer a flexible deployment model.
To better understand the set of tasks the users go through to achieve their goal I created a user flow. This formed as a basis for me to start designing the application. This was followed with various brainstorming sessions with my manager and the team. I iterated over the design quite a few times with feedback from within vectice as well as outside vectice before I arrived at something that was implemented.
We conducted usability tests on the initial prototype. Since this was an enterprise software we conducted the usability test by sending our participants a tutorial. The tutorial gave the participants an overview of the product since providing a test directly to a newbie would produce skewed results. The goal of the tests was to identify breakdowns in the prototype and to understand our users’ general perceptions of the designed service.
From our usability tests, we got some positive feedback and discovered the following usability issues. We then brainstormed about possible solutions to address the usability issues:
1. Makes cataloging much easier - content more discoverable and reusable
2. Does a good job at capturing the knowledge on top of those assets and the context where they are used
3. Enables team collaboration between data scientists, managers
4. Should there be a segregation between workspaces or should it be accessible to everyone in an organization
5. Adding a approval based system for projects
6. Adding more granularity to contextual documentation
Vectice tackles a completely new area and product category. The AI landscape is rapidly changing with cloud vendors pushing for wall-to-wall solutions (Sagemaker, Google) and competing with strong incumbents like Databricks or DataRobot is very challenging. As a product we cannot afford to support each and every ecosystem on the market and hence some places might not use the underlying tools that Vectice relies on for tracking of experiments, datasets etc.
"Mrunmayee had a tremendous impact on the quality of the product at Vectice during her UX internship. She joined us as the product was in the early stages, quickly understood product requirements, and related to the users in the field of data science. Within a span of 6 months, Mrunmayee designed and prototyped big parts of the application, performed usability testing, and ensured that the experience was intuitive, cohesive, and accessible. She eventually led a remote design team of contractors to scale her impact and oversee all the application design. The level of maturity Mrunmayee has shown during her internship is something I would expect from more senior designers with multiple years of experience, especially for a B2B application." - Gregory Haardt, CTO Vectice
Made by Mrunmayee Patil