I can’t believe it’s already been a year since we covered some great gift ideas for data people! We’re back with some more last minute ideas, some may look familiar albeit bigger (or smaller) and better while others are new arrivals. Whatever you’re looking for, we hope at least one of these ideas will help you find something that really excites the techie / data lover in your life this holiday season!
By now, the idea of agile development and a Minimum Viable Product or MVP is prevalent. The problem is, while most people have the minimum part down, people often haven't mastered the viable…. especially when it comes to analytics.
To quickly recap,a Minimum Viable Product is an approach, where you’re focusing on creating a product with a sufficient level of features to be able to solve a particular problem. This first iteration is used to collect user feedback and develop the complete set of features for the final product to be delivered.
That’s all nice and well, but you may be wondering what the benefits to this approach are as it concerns analytics projects...
Learning, and learning quickly
Is your solution actually delivering the value that you are trying to create? In a typical project, you may be months down the road before what you’re building is actually in front of users. This makes it difficult to determine its viability for solving the business case. The whole point is to prove or disprove your initial assumptions sooner.
What part of your users current process is really frustrating them?
Are the analytics we designed actually guiding them through their workflow and making their life better?
By getting a usable set of features in front of user’s earlier in the process, we can collect feedback and determine if we are in fact on the right track.
Every great data insight or data visualization starts with good, clean data. Whether you want to understand lifetime value, design upsell and cross-sell strategies, define personas, or develop sophisticated data models, having clean and consolidated data will enable better analytics, improve the performance of your marketing campaigns and maximize your marketing ROI.
Clean data is particularly crucial for CRM, ERP, sales and IT systems with customer data. For example, proper planning and cleansing of your customer data from the beginning will keep you from falling behind on your CRM implementation. Your data needs to be reviewed, filtered and cleaned to ensure that bogus data is not transferred. The the cost to the business of processing errors can be evaluated from the time spent on manual troubleshooting, forced ETL re-runs and at worst, representing incorrect or invalid data to the customers or employees to drive their business decisions.
How do you ensure your data is not wrong or incomplete when you digest data from various third-party sources, especially sources like FTP and AWS S3 which (unlike an API) do not have given structure all the time?
How do you successfully migrate data from an old system to new one?
It is safe to say that the majority of data flows have set of expected data types defined and very often the value range as well.
One option is to use SQL or Python transformations but such hard coded configuration or approach can be very time-consuming and it is lacking of the flexibility or simplicity to be reused. Additionally, it would not be obvious which rows and columns include rogue values until these transformations run into an error (or you would have to design a specific workflow to off-load them.)
Another option is to describe the data and set up value and type conditions for it in the form of rules. Once that’s done, all you need to do is make sure data flows include rules that check every time you run the orchestration (ETL process). KBC Data Health App has been designed to help you automate this data check process.
Typical use cases:
Ensuring data quality from systems with data collected by users (internal IT systems, CRM, user forms, etc.)
Migrating data from legacy systems - data migration assumptions vs. reality check
Validating crucial fields for report buildup
Validating location data (store locations and customer locations) to drive contextual marketing (How healthy is your location data?)
KBC Data Health Application
Data Health Application is an app designed to aid users to produce a clean data file. To boost user productivity, it provides users a simple and convenient solution to cleanse or filter data instead of creating multiple long queries in transformation to obtain the same results. Some primary features include:
Filtering data based on user configured rules to match business needs
Can be triggered to run on a scheduled basis
Generate a report with descriptions and reasons why rows are rejected
As many users did not have any prior knowledge in SQL, this application is capable of creating basic SQL functionalities through simple user interface inputs. The application does not have any pre-configured rules. It allows users to have the freedom to create rules tailored to their needs and wants. With the combination of KBC orchestration, this application can be triggered on a daily/weekly basis depending on user’s business requirements. With that being said, users will have an automated progress that generates “clean” data to conduct any in depth analysis without worrying about handling corrupted data or outliers.
Numeric Comparison (Value comparison)
Regular Expression (Regex)
Column Type (Applicable value type)
User wants anything within the “Western Europe” Region
User is only interested in countries placed within the top 10 happiness rank
Happiness score cannot be empty
If you’re already a KBC user you can find the Data Health app alongside the rest of our data applications. Not yet a user and want to learn more? Contact us to discuss.
Embedded analytics, data products, data monetization, big data….there are plenty of buzz words we can use to “categorize” the idea.
IDC reports that the big data and business analytics market growing at a rate of over 11% in 2016 and at a compound annual growth rate of 11.7% through to 2020. This rapidly growing area of investment can’t be for naught….can it?
Let’s look beyond the hype at some specific approaches for extracting additional value (and ultimately dollars) from your data.
According to Gartner, Data Monetization refers to using data for quantifiable economic benefit.
The first thing that may come to mind is outright selling of data (via a data broker or independently.) Although a potentially viable option, with increased data privacy policies and the sheer amount of data needed to be successful with this approach, it can be quite limiting.
There are many other approaches to monetizing your data, such as:
We’ve been assisting people with data and automation for a while now, helping them become “data-driven.” Several months ago, we had an exciting opportunity to automate within Keboola itself. After we lost our two main finance wizards to the joys of childcare, we decided to automate our internal business processes.
There was a lot of work ahead of us. For years, the two of them had been issuing invoices one by one. Manually. Hating unnecessary manual tasks, we were eager to put the power of our platform — Keboola Connection into work and eliminate the manual invoicing.
We expected approximately 2-3 mandays per month to be cut down. We also wanted to get much better data about what’s going on.
As our sales activities have been taking off around the globe, we would need to automate this process anyway. Otherwise soon we would have to hire a new employee just for invoicing and that is a no-go for us. Plus we didn’t want to overload Tereza, our new colleague, with this tedious work and take away her weekends from her.
When it comes to data, we often preach the agile methodology: Start small, build quick, fail fast and have the results in production from day one - slow Kaizen style improvement. This is exactly what we did with our invoicing automation project. We didn’t want to have someone write a custom app for us. We wanted to hack our current systems, prototype, fail fast and see where it would lead us. We wanted to save Tereza’s time but didn’t want to waste it 10x in the development of the system. :-)
Our “budget” was 3-4 mandays max!
Step 1 — Looking for a tool to use for the invoicing
We were looking for a tool which can handle all the basic things we need: different currencies (it’s Europe!), different bank accounts, with or without tax, paid or unpaid, and a handful of other features. Last but not least, the tool HAD to have a nice RESTful API. After some trials we opted for a Czech system – Fakturoid. They have great support, by the way. That’s a big plus.
Step 2 — Getting data about customers from Fakturoid into Keboola Connection
First, Padak took all clients we already had in Flexibee, our accounting tool, and exported them to Fakturoid. Then we added all the necessary info to the contacts.
Great. Now we had all the customers’ records ready and needed to get the data into Keboola Connection. It was time to set up our Generic Extractor. It literally took me half an hour to do it! Check it out here:
Keboola Generic extractor config for getting clients’ info from Fakturoid into Keboola Connection
Step 3 — Creating two tables with invoices and their items for uploading into Fakturoid
There was only one more thing to know. Who is supposed to pay for what and when? We store this info in our Google Spreadsheet. It contains basic info about our clients, the services they use, the price they pay for them, the invoicing period (yearly, quarterly, monthly), and the time period for which the info is valid (when their contract is valid; new contract/change = new row). To be able to pair the tables easily, we added a new column with the Fakturoid client ID.
Finally, we set up our Google Drive Extractor and loaded the data into Keboola Connection. Once we had all the data there, we used SQL to create a Transformation that took everything necessary from the tables (who we bill this month, how much, if out of country = don’t put VAT, add info about current exchange rate, etc.) and created clean output tables.
Part of the SQL transformation which creates an output table with items to pay for Fakturoid.
Step 4 — Sending the tables into Fakturoid and letting it create the invoices
This step was not as easy as exporting data from Fakturoid. We couldn’t use any preprogrammed services. Thankfully, Keboola Connection is an open environment and any developer can augment it and add new code to extend its functionality. Just wrap it up in Docker container. We asked Vlado to write a new writer for Fakturoid which would take the output tables from our Transformation (see Step 3) and create invoices in Fakturoid from the data in those tables.
It took Vlado only 2 hours to have the writer up and running!
Now when the writer is completed, Keboola Connection has one more component which is available to all its users.
Step 5 — Automating the whole process
It was the easiest part. We used our Orchestration services inside Keboola Connection and created an orchestration which automatically starts on the first day of each month. Five minutes later, all the invoices are done and sent out. #easypeasy
It is not a complicated solution. No rocket science. We believe in splitting big problems into smaller pieces, solving the small parts and putting them back together just like Lego bricks. The process should be easy, fast, open and put together from self-contained components. So when you have a problem in one part, it doesn’t affect the whole solution and you can easily fix it.
Saving Tereza’s time, this is the springboard for automating other parts of her job. We want her to spend more time doing more interesting things. And the process scales as we grow.
It took us:
- 4 hours to analyse and understand the problem and how things are connected,
- 1 hour to export clients from the accounting system,
- 1/2 hour to write a Generic Extractor from Fakturoid,
- 2 hours to write a Transformation preparing clean output data,
- 2 hours to develop a new Writer for Fakturoid, and
- 1-2 hours to do everything else related to the whole proces.
Total = circa 11 hours
Spoiler alert: I’m already working on further articles from the automation series. Look forward to reading how we implemented automatic distribution of invoices to clients and the accounting company, or how we let the systems handle our invoicing for implementation work.
Think about all the social media platforms out there, which ones do you use the most (and why)? I’m not talking about giving your LinkedIn profile a face lift before you put in a job application or searching for a long lost friend on Facebook; which of these apps are actually driving user engagement? For me, it’s Instagram; the interface is easy to navigate and more than once I’ve found myself re-opening it after I’ve just closed it. Have you thought about why many of these platforms have exploded in user engagement with many people posting to their Twitter or Facebook accounts multiple times per day? According to a recent Gartner blog, adoption rate for some of the BI tools in their Magic Quadrant are at a low but not too surprising 21%. Are people sick and tired of “all that data” or is there something more sinister at work…
We’ve thought a lot about social media platforms (and other apps) that seem to drive such high user engagement and put together a few thoughts on how you can do the same within your data product to ensure you keep users coming back for more. Before we reveal the secret sauce for building engagement in your data products, let’s take a quick look at how many analytics teams approach the problem.
Too often, teams building an analytics product for this customer’s approach the project in the wrong way, the story is oh so familiar. As we covered in a recent blog, this meant taking the reports existing in an Excel spreadsheet and web-ifying them in a cloud BI tool. It’s essentially surfacing the exact same information as before, but now with shiny new charts and graphs, more color choices, and some interactivity. After the initial excitement over the new toy in the room, the latter solution isn’t doing any better than the former at driving engagement; let alone delivering “insights” or creating a new revenue stream.
One of the big reasons customer aren’t lining up to write a check for the latest, greatest data product a vendor has rolled out is that the analytics team failed to make it engaging. Simply put, product teams need to let users know “hey—check this out,” “hey—we’ve got some important information for you”, and “hey—you should come back and see us.” Most teams do the second part, the “we’ve got insights” piece, but they fail to inform users why they need to keep coming back for more. These are essential elements of establishing engagement; not building these in is like skipping the foundation of a new skyscraper. "It's like when you see a skyscraper; you're impressed by the height, but nobody is impressed by the foundation. But make no mistake, it's important," said Akshay Tandon, Head of Strategy & Analytics at LendingTree.
Want to avoid the killer mistakes of failing to build engagement into your data product? Here’s how:
Intelligent Narratives are the data-driven stories of the enterprise. They are automated, insightful communications packed with the information that matters most to you—specific to your role or industry—written in conversational language, and at machine scale. By giving your employees and your customers a richer, more nuanced understanding of your business, they can make more informed decisions and realize their greatest potential.
Narrative Science is the leader in advanced natural language generation (Advanced NLG) for the enterprise. Quill™, its Advanced NLG platform, learns and writes like a human, automatically transforming data into Intelligent Narratives—insightful, conversational communications packed with audience-relevant information that provide complete transparency into how analytic decisions are made.
As we all know, one of the biggest barriers to successful data projects is having the right data in the right place; that's why Narrative Science and Keboola have partnered to bring the next generation of analytics to you faster. Automate data workflows, reduce time and complexity of implementations and start gaining new insights now! Leverage this app, powered by Narrative Science, to produce machine-generated narratives of data ingested by Keboola.
Video is one of the hottest trends in digital marketing. YouTube, which has expanded more than 40 percent since last year, reaches more 18-49 year-old viewers than any of the cable networks and has a billion users watching hundreds of millions of hours every day.
Freethink, a modern media publisher, uses online video to tell the stories of passionate innovators who are solving some of humanity’s biggest challenges by thinking differently. While telling important stories is their primary focus, data underlies all of their decisions. As a publisher, they need to understand how well each piece of content performs, as well as how that content performs across platforms (they currently publish videos on their website, YouTube and Facebook.)
Prior to working with Keboola, collecting and combining data for cross-channel video analysis was a time consuming, manual effort (particularly because Facebook has separate APIs to track page content and promoted content.) In addition, this process made performing time-over-time analyses a real challenge.
The goal was to provide a dashboard solution for the team to have better visibility into their data. Keboola Connection (KBC) was able to overcome this by leveraging existing API connections to get data from Facebook and YouTube. In addition, Keboola utilized its partnership with Quintly (social media analytics) in order to pick up cleaned and verified data from their API. All this data is combined additional data sources including Google Sheets to provide additional metadata for advanced reporting and segmentation. This blended data enables universal reporting across platforms to get a 360-degree picture of each piece of content.
Freethink now has all their data populated in Redshift, where Chartio is able to connect to create beautiful dashboards for reporting. They are able to go into the Keboola platform and manually adjust and run configurations to get exactly the data they need. The biggest gains have been in time saved, being able to show change over time and freeing the team up to focus on more complicated analyses. This also opened up data access to the broader team, promoting collaboration and data driven decision making.
"Keboola really helped simplify and automate the process of collecting and combining data. Working together, Chartio and Keboola Connection deliver a full stack solution for modern analytics, taking full advantage of the cloud. I’m able to give my team better insights into our performance and make better decisions, quicker."
For anyone creating an analytics product, the pressures of engaging customers and generating revenue while protecting your core product and brand can be overwhelming, especially when aiming to hit so many goals on the horizon:
Does it target users effectively?
Will it guide users to a solution to their business problem?
Can it scale to many customers?
Will it deliver real results that customers are willing to pay for??
Fortunately, we've been there, done that, and understand what it takes to build a great data product. That's why we've created a map to help you navigate your way to success, built on the experience of countless voyagers who have sailed the same seas before you; the Data Product Readiness Assessment.
In recent years, a term started appearing across the technology world: “data monetization,” turn your data into dollars.. (as we mentioned in a previous, post, you can Find Gold in Your Data!) Businesses reacted to the hype, started spending on every solution under the sun and then… Nothing. Nada. Zilch. In many cases the revenues never materialized, buyers became frustrated with the lack of results and blamed the whole concept of data monetization. The problem is, you’ve got to avoid certain mistakes... and they’re silent killers.
In truth, data products are a great opportunity for most businesses to engage customers and create new streams of revenue. Untapped, dormant data can, when refined properly, become a crucial resource for your company. Fortunately, we’ve worked on many analytics projects ourselves, have seen these mistakes made and have put together a guide to help you avoid making them yourself.
To provide some quick insight, we thought we’d share one of the tips we’ve found most helpful when starting to create an analytics product.