IBM watsonx.data
Revolutionizing the open data lakehouse for AI and analytics
Are you one of the VIP’s that attended IBM’s 2024 THINK Conference in Orlando? Whether you attended virtually or in-person, you cannot miss IBM’s biggest announcement: watsonx.
Watsonx takes the technology behind Watson (yes, the Watson that won Jeopardy) to the next level, with even more advanced machine learning algorithms and data processing capabilities. From being able to process mounds of unstructured data, fluent in natural language processing, scalable, and user-friendly, WatsonX is taking lead in helping businesses making more accurate and informed decisions.
There are three primary capabilities within watsonx: watsonx.ai, watson,x.data, and watsonx.governance. Each capability offers a tailored view into generating enterprise-ready AI models, generative AI for machine learning, and improved governed data access.
Stakeholders
IBM Data Engineers Data Scientists
Skills
Software Engineering UX Design
Domain
Technology
Date
October 2024
IBM Data & AI team conducting a brainstorming session in the Silicon Valley Lab, California.
Overview
Overcoming expensive disparate data sourcesThe volume of data across industries is exploding. The aggregate volume of data store is targeted to grow over 250% over the next 5 years. Many Data Scientists and AI/ML Engineers have data stored in multiple locations and silos in complex forms and in poor quality.
Watsonx.data makes it possible for enterprises to scale analytics and AI with a fit-for-purpose data store, built on an open lakehouse architecture, supported by querying, governance and open data formats to access and share data.
We began initial concepts for Lakehouse in 2021, and started implementing designs in 2022. In this case study, I will focus on the process in co-designing the watsonx SQL Editor.
Figure 1: In order to design an impactful platform, I knew I would have to put myself in the shoes of a data scientist. As a Master's student at Harvard, I committed myself to learning SQL through Harvard's CS50 curriculum.
Step 1: Learn
Now that we’ve unveiled the power of IBM’s AI data platform, I’m here to give you an inside look on what it’s been like as a designer on the watsonx.data team (Hint: it’s been amazing! 😄) I would like to have said this process was linear, but some of the most profound design insights came later in the process.
Our timeline followed IBM’s Enterprise Design Thinking Framework with a focus on user outcomes, relentless innovation, and diverse empowered teams. IBM’s loop of observing, reflecting, and making helped us understand the present and envision the future of data Lakehouse.
Figure 2: Industry Standard for SQL client applications. Kinda scary… right?
Structured query language (SQL): is a programming language that enables users to create, read, update, and delation relational data.
Every application that you use (let's take Instagram, Tik Tok, Twitter for example) use relational databases to store information at scale. These databases store data in rows and columns in structures called tables. This data is stored, processed, and secured in engines.Design Development
Who uses SQL?
Data Scientists, Data engineers, and AI/ML Engineers, are some of the primary people that use SQL everyday to query their company’s data and drive valuable insights.
To design the most optimum experience for our users, I was commitment becoming an expert in the field. I took Harvard's SQL Computer Science course to become proficient in running and writing queries
Figure #: This mockup identifies key SQL components in the viewport such as query history, saved queries, editor capabilities
Step 2: Design
One of my favorite parts of the process was looking at other IBM Products, like Db2 and Watson Query, as a foundation for our SQL Editor. These products served as valuable building blocks to form insights on how we can enhance the existing SQL experience.
Our early drafts consisted of mockups in Mural. We found that this was the best way to visualize the information architecture of the SQL Editor, and a clear way for our PM’s and Engineers to collaborate with us.
After prioritizing different features and the overall information architecture, it was time to move to Figma! As UX Designers at IBM, we depend on the Carbon Design System.
One of my favorite parts of the design process is converting our low-fi designs to Carbon components. With our robust library, there are patterns for every user need.
Figure 3: This mockup visualize features with the Carbon Design System UI. We shared these concepts with PM's and developers.
Step 4: Implement
IBM announced watsonx at the May, 2023 Think conference with extreme excitement. As a designer, I was proud to work on a product that provides build-in governance across data ecosystem, with institute ways to open data and table formats to analyze data sets. Additionally, 60% of workloads are 2.2x faster than our strongest competitors.
Designing watsonx.data has been one of the most formative design experiences I have had. While we were moving fast to make design process, we had Data Scientists, AI/ML Engineers, and Database administrators at the forefront of every design decision. I cannot wait to see the enterprise improvements that are made from watsonx.
Key design decision breakdown:
👓 Accessibility: Access all data through a single point of entry with a shared metadata layer across clouds and on-premises environments.
💰 Price optimization:Optimize costly data warehouse workloads across multiple query engines and storage tiers, pairing the right workload with the right engine.
🖥️ Query workloads with AI: Watsonx.data leverages watsonx.ai foundation models to simplify and accelerate the way users interact with data. Use natural language to explore, augment, and enrich data from a conversational user interface.
⏰ Launch in minutes: Connect to storage and analytics environments in minutes and enhance trust in data with built-in governance, security, and automation.
Figure 4: Final Demo with watsonx.data
Conclusion
This was an extremely transformative project for me. Our users remained at the forefront of ever stage of our design process. Reflecting on the Watsonx.data project, it's evident how my coding skills significantly enhanced the communication and collaboration with developers. By acquiring proficiency in SQL and understanding the intricacies of relational databases, I was able to actively participate in technical discussions and contribute meaningful insights to the SQL Editor's design. This technical fluency enabled me to bridge the gap between UX design concepts and their practical implementation, ensuring that the final product not only met but exceeded user expectations while facilitating seamless developer interaction.