May 17, 2021

5 Minutes

End User Perspective in Data Management

When talking with data management experts around the world, there is one thing that has started to bother many – even though they have invested in new technology, they don’t seem to have the ability to keep business people satisfied.

Business folks claim they just can’t access the data they need.

Currently, all organizations have a lot of technology and tools for managing data.

There have been enormous advancements in technology in the past ten years: computing power has increased enormously, and the cloud is now a perfectly feasible solution for most data platforms.

Then there are the highly developed software, tools, and platforms for all areas from data storage to visualization of beautiful graphs and diagrams.

We have all the technological ingredients for building a data-driven organization, so what is the problem? Why do so many companies struggle with becoming data-driven?

The problem is that the actual data users and consumers have been left out of the development.

Building a water supply system

Building a data platform is like building a water supply system.

There’s a use case (thirsty people) and a user-facing tool (bottle or tab) for drinking water. Then there’s a water pipeline system that delivers the water from a well to the bottles, all the way to retailer distribution.

Now imagine a situation where all of this would be new to humankind.

Nobody has thought that water could be consumed from bottles. Everyone has always just gone to the well themselves, but now somebody figures that hey, this process is not cost-efficient.

Let’s automate all of this, the people’s council decides.

If the current data platform approach were applied, this work would start from engineering a water pump and a supply pipeline; in other words, a system that supplies water from the well to a massive silo.

This would take a lot of time and effort, and while it’s under construction, consumers would see no significant improvement on their end – going to the well would still be the way to go.

Soon thirsty consumers would start complaining about the system.

Even though an investment to build the system was made, people still couldn’t access the water resource better than before, and they’d still need to walk all the way to the well.

Now, they are told to walk to the massive water silo to get their water. But from a consumer perspective, it’s not a significant improvement over walking to a well.

How to solve this problem? The solution is to ask the consumer how water should be consumed.

They would probably say that some kind of a small container of water that would also enable drinking from the container would be convenient.

Suddenly, the people who built the massive silo and the pipelines realize that they have forgotten at least half of the whole process!

They have just concentrated on the water pump and supply system and neglected the consumer.

They realize that a “bottle” needs to be introduced, which has to be designed, not to mention the whole process of bottling the water.

Basics of web application development

Let’s face it; the modern data industry is very immature and young. When building a data-driven organization or a data platform, very few people know what they are doing!

Some may claim they have a great idea, but nobody has a silver bullet or best practice that would fit all organizations in all honesty.

At this point, no one can claim to have that one-size-fits-all silver bullet solution because all the new Azure, AWS, GCP, and Snowflake data platforms have been around only for a few years. Simply not enough time to converge into a single design!

On the other hand, “normal” application and IT system development has been around for quite some time already, so those people can claim they have come up with some beneficial generic practices. These could be applied in data management as well.

Let’s say you’d like to build a web or mobile application; how would you start that?

You could ask a bunch of coders to “build me this”, and Most of them, even the experienced programmers, would immediately ask if you have a UI/UX designer in the team!

Why is such a person needed? How come the coders can’t just build it if I show how it should work?

The reason is that the experienced developer knows that all applications should be developed by listening to the users’ needs. They know that “just building it” without considering the actual users makes for a crappy app, and they will get the blame.

How would you like to consume your water?

The importance of a UI/UX

The UI/UX people are experts in understanding the user. They can transform user needs into requirements for the existing software.

All of those user-friendly apps on your desktop and cell phone are based on this type of UI/UX design process.

When Uber changed their UI a few years back, it was big news in Silicon Valley, and investors were freaking out as they feared losing billions.

They thought they lost their game to Lyft for good because the new UI update was considered unsuccessful.

Bad UI/UX can cost you billions of dollars; it’s hugely important.

The data world is now going in that direction as well. The technical data pipeline is just one part of the entire process – the other is data consumption.

Today, many people in data management have realized the same thing that application developers realized years earlier – the whole thing needs to be flipped upside down and start from the user.

The old way doesn’t work.

Back to data management and how this currently works.

The traditional approach to deploying Snowflake or any other data platform does not work ideally from a consumer perspective because they’re not involved in the construction phase.

Many people still consider setting up data platforms to be the IT department’s and data engineer’s job.

If we are to analyze, say, sales data, we know that data is often produced in CRM and other sales and marketing tools.

When a company purchases a CRM, the sales director is heavily involved in the process. Nobody would expect that the IT department alone would manage CRM deployments without anyone from the sales team.

But when an organization builds a data platform for sales data, the sales director and her team are often not an integral part of the process.

This leads to an inferior data platform from a consumer perspective.

It can be still good from an engineering perspective, meaning that it processes enormous amounts of data very efficiently. There are integrations to dozens of source systems, or maybe it has been automatized to the extent that there’s a minimal amount of traditional database administration tasks left.

But none of this helps the “data-thirsty” business people.

At the end of the day, the only real purpose of a data platform is to deliver business value. That’s the only real metric or KPI for a “good” data solution.

Some tips on how to work this out

Here are some of the tips we’ve realized to be helpful in this—these eight exquisite pieces of advice work.

(Note: we use here an example of building a sales-related data platform, but it could just as well be HR, production, finance, or whatever!)

People in data management positions should learn how to sell internally. For example, what are the three main benefits of a new data platform for the sales director?
When involving the users, such as the sales director, in the process, apply UX design methods to find out requirements – for example, service design thinking.
Compose an Ellie concept model with your business experts, like the sales director. Basing your design on the actual business information content is how you make sure your data platform is scalable, durable, and benefits the business.
Show some of the available sales data very quickly to the sales director. They are usually amazed and happy about that. Many times they see this data for the first time!
Now, as you have caught the interest of the business domain expert, you can start building an MVP or a proof-of-concept of your Azure/Snowflake/AWS or Databricks solutions – but don’t scale yet! Build small end-to-end slices to deliver value.
Make the sales director your internal champion. Pitch your data platform to the CFO, and ask for a reference from a sales director. Or even better, ask them to do the pitch!
If your POC works and people are getting excited, now it’s time to scale and start building up the infrastructure.
Keep the feedback loop short; ask for feedback from your users constantly!

The point here is that the sales director (or whoever your business champion is) wants to, e.g. improve their department’s performance, meaning that, for the sales director, she probably wants her company to sell more.

She needs data on things like how many sales calls lead to customer purchases, what’s the profit margin of a specific product category? How much will we sell this last quarter? And so on. These are all vital business questions that make a huge difference in how the entire company is performing.

Viewed in this light, a data platform that gives her answers to those questions is not some secondary “use case” of data that you could do “on the side”, something that is perhaps nice to have but not crucial for the sales team, something that you’d let IT handle (it has to do with computers, right?)

It’s, in fact, the essence of all business development.

This is what becoming a data-driven organization means. Ellie, the SaaS-based architecture tool, can help you engage the business user and bring them into the core of a data platform project.