New Start, Building a Web Analytics Time Machine

Last modified date

Comments: 0

Today is my last day at IAG, the place I worked in the last 5 years. Initially, I was planning to work there for 6 months as a contractor to help them enhance the web analytics capability, and then to start to do my own thing. But it turns out I stayed there for 5 years, contracting for almost 2.5 years and then I became a permanent employee for a bit more than 2.5 years. It was a good place to work with a lot of nice people. I left because I have been building a web analytics time machine and I have to focus on it now.

A couple years ago I started to think about how to make web analytics implementation easier and faster so that we as analysts can focus on insights and generating true value for businesses. That’s how I started to create the concept of “web analytics time machine”. The idea is to build a system that can re-create the moment visitors use the website, so analysts can do analysis like travelling back in time to investigate and identify the barriers to conversion. This time machine won’t let you change history, but you can use it to set the future.

As I started to explore the idea in more details, I realised although there are a lot of different technologies are used in building websites, after all visitors are looking at an UI built with HTML which is kind of a standard. So what if both user behaviours and HTML can be captured and be stored for re-creating the scenario of a visitor using a web page? That’s the way to go for building a web analytics time machine!

It took me more than a year to research on how feasible this idea is. Although there are session replay tools in the market that can replay individual visitor using the website in video style, most of them do not capture the HTML code so when the page is personalised, or the page has changed, it will break the replay. One of the goals I set at the very beginning was to be able to resolve this problem so that the time machine can truly re-create the scenarios as it happened. That means it has to fully archive web pages and all of the changes in the DOM when visitors are browsing.

Another issue I wanted to resolve was in data processing. Currently, web analytics tools capture data and process data with business logic and write the result to the database. That means if the data is incorrect or missing, or there is a defect in the business logic, it would be impossible or very difficult to correct it. However, if the data is the raw data of what happened, business logic can be built on the raw data and when it needs to be changed, there will be no problem. Just re-run the revised business logic through the raw data. This goal also means the data has to be stored in the database and be queryable, which is very different from session replay tools in the market. This approach actually resolves the biggest problem of session replay tools which is there are too many random footages to watch. Now it is possible to query and find the exact footage that matters and gain insights effectively.

After about 2 years of long nights and working on weekends, finally, this time machine is ready to be launched. It is called Insightech! Starting from tomorrow, I will be working full-time on it. It is very exciting and challenging at the same time. Insightech will move to Fishburners Wynyard office on 22nd Jan. I am looking forward to meeting the Sydney start-up communities and learning how to build a successful start-up.

I want to thank all the friends who spent their time to see my demos and provided me with feedback. I can’t go this far without their support.

Spread the love

Chaoming

Leave a Reply

Your email address will not be published. Required fields are marked *

Post comment