Saturday morning after meeting our liaisons for the first time, we reconvened at the d.school to brainstorm solutions to the task that Karthik had laid at our feet. Once again, Santi led us by moderating.
We quickly came to the conclusion that we wanted to work with unstructured log data. We felt that this helped keep our product more "data-agnostic" as simple logging techniques are both ubiquitous and information-rich. We began brainstorming about methods to extract insightful information from these logs. We came up with a variety of solutions from easy querying and visualizations to summary statistics to alerts and report generation. We felt like we had hit a jackpot of great ideas and started to dive into more detail about each one.
At some point, Splunk came up. We knew Splunk was a large company with amazing technology to access your "machine log data." So we googled them. We found that all the sticky notes we had on our board were already on Splunk's sales page. We had spent the last hour reinventing Splunk.
This was both saddening and encouraging. On one hand, our "new innovative ideas" were already implemented, and were surely to be better in basically every way than something we would make in 6 months. On the other hand, we realized there was value in what we were talking about. In about an hour we had thought up the core products to a billion dollar multinational company. We knew we were onto something, but we had to change direction. We had to distinguish ourselves from companies like Splunk.
Differentiating from Splunk ended up being great for our brainstorming productivity. All of a sudden, we had a flurry of ideas. Synthesizing and compiling them, we found an ambitious product with a catchy, buzzword-filled tagline "Turning Big Data into Small Data." Inspired partially by the Stanford course, we postulated that with proper unsupervised modeling techniques we could turn the large, intractable data companies had on their servers into smaller representative sets that users could interact with in their browser or mobile app. We envisioned that with smaller scale data we could allow for sweet visualization and interactivity without sacrificing the integrity of the overall data. Our main users would be executives and managers (decision-makers) who wanted to make data-driven decisions quickly without a lot of technical expertise. Additionally, we felt our product would be useful for employees that had to directly deal with the data at hand, such as IT Admins or Data Scientists, as they could get a summary of the data before using more technical tools like Splunk for deep-dives. Our envisioned product could give you an overview of your data, a "feel" for your data in some sense, with the potential to shed light on problematic or interesting trends automatically and without waiting for external reports.
