The Dublin Diaries

On the morning of March 18, Andrew Jon, and Hussain took flight for Dublin, Ireland in order to to present DVation's project idea to the people at SAP’s Service and Support Center.  We arrived in Dublin on March 19th and spent 4 days roaming the city.  From the surrounding suburbs to the city center we explored parks like Phoenix Park and St. Stephen’s Green, tasty brunch spots like Brother Hubbard, and pubs like Temple Bar and Fitzsimons.  We also sipped some coffee from a few cozy cafes and did some reading and work.

After the few days of exploring Dublin, we were ready to get down to business and present to the folks at SAP.  Our two hosts, Chris and Manish, kindly put together a great agenda for the day.  First, Chris met us around 9:45am, took us to coffee, and prepped us for the day ahead.  Then we headed to our conference room for the day where we gave our presentation at 10:15am.  The presentation went extremely well and by the end we had the folks at SAP jokingly making offers about investing in our company.  Additionally, we received some great questions about the product that we had yet to consider.  For example, one person asked “how does our product work handle the case of a globalized company? Will it cater to companies that extend across time zones and work in multiple languages.”  Another person alerted us to the fact that “displaying a worker score may have negative effects if the score is visible to all, but the score would is extremely useful from a managers perspective.”  We definitely plan to keep the things we learned in this meeting in mind as we continue to grow our product throughout the next quarter.

For part 2 of the day, Chris and Manish, had set up a lunch for us and meetings with folks who work with trouble tickets daily.  These meetings really helped us understand the lifecycle of a ticket.  Because of the NDA we signed, we cannot really expose any of the details specific to SAP that were mentioned but we can say these meetings led to multiple insights that will be extremely valuable in shaping our product.

As our time in Dublin came to a close, we had had a great time and left with some big takeaways.  First, the reaction following our presentation affirmed the great value in our product.  Second, there was a lot of insights to be garnered from truly understanding how a company handles trouble tickets.  Lastly, Dublin is a fantastic place to do business and some team bonding.

DVation greatly appreciates the help of Jay Borenstein for setting us up with the visit to SAP and Chis, Manish, and the other people at SAP that met with us and showed us amazing hospitality.

Today was a good day

On Tuesday March 10, the team headed back to VMware’s campus to present our project idea and the work we had accomplished throughout the quarter.  With Sanjay Poonen, General Manager of End-User Computing, and other big names at VMware all in attendance, this was by far the biggest stage we had seen to date.

The slides to the presentation can be found here.  To summarize our presentation, we discussed the IT Service Management (ITSM) space, the opportunities that we found within the space, our 3 value propositions that fill these needs, and lastly we showed a demo of our product.  At the end of the presentation we were asked some questions about our product.  All of this went really smoothly and the people at VMware were extremely excited about our idea.  Sanjay was impressed enough to tweet about us after and even discussed an opportunity to present in front of the CEO of VMware.  This was a huge moment for the team: we had spent a good portion of the quarter playing catch-up after pivoting ideas fairly late in the game, and this moment was when we realized we had finally caught back up and landed at a product with tons of value.

With the affirmation of the value in our product idea and more helpful contacts at VMware, team DVation is well set for a productive quarter of transforming the idea into quality software. Who knows, we may be able to sell it someday!

DVation would like to thank Jay Borenstein for his help throughout the quarter and setting up this meeting, and Sanjay Poonen and the other folks at VMware who took the time to listen to our idea and provide some great feedback.

Mini-Hackathon

Thursday, March 5th at 7:00pm Andrew, Hussain, and Jon met in the loft for an informal hackathon. The goal of this hackathon was to further familiarize ourself with our technical stack and to begin implementing some of the key features necessary for our product to run. In some sense, we wanted a skeleton of our product built. 

In a previous meeting, we had decided upon a MEAN (MongoDB, ExpressJS, AngularJS, NodeJS) stack. This decision was made on behalf of the entire team for a variety of reasons. First of all, there is a lot of hype around full-javascript stacks, and we wanted to see what that was about and build something for ourselves with it. Secondly, these frameworks are known to work well together and have plenty of documentation and tutorials online. This meant that although our experience with these frameworks was limited, we could learn from the resources available to us online. Lastly, the flexibility, ease of development, and scalability of these frameworks appealed to our team. 

So at 7 we began to code. Andrew took on the task of a login system via PassportJS, and Hussain and Jon took to getting Mongo to be connected to our Node application via MongooseJS. 

After about 4.5 hours of coding, reading tutorials, and tinkering we found ourselves with a login functionality waiting for database integration, a database schema, and a ticket creation form capable of storing tickets within our database. Our progress was slower than we wished form but this was partially because we were learning how to build this out ourselves. We figured putting in the work early would pay off later. 

After consolidating our work, we discussed our strategy for incorporating our machine learning skills into the application. After some brainstorming and discussion, we came to the conclusion that we faced two main ML tasks that could be approached in two different manners:

• Ticket Tag Suggestion 
  • Accomplishable via NLP and Supervised Learning
  • Train N classifiers where N is the number of active tags
    • If a classifier has a score above some threshold, suggest that tag
    • Alternatively, suggest the top k tags
• Similar Ticket Resurfacing
  • Accomplishable via Unsupervised Learning 
  • Use techniques such as K-Means or K-Nearest Neighbors to determine similarity between tickets
    • The feature vector we use these techniques on would be a mix of keyword extraction, metadata, and tags. As such, our first ML task augments our second. 
    • Once we have the most similar resolved tickets, we retrieve the highest rated links, templates, and comments from these tickets and surface them to the admin. 

We were very happy with these preliminary ML ideas, as they seemed to strike a great balance between usefulness and accomplishability. The last comment we had regarding the machine learning discussion we had was that although these ideas seemed great, data science is very empirical, and the only way we could validate these concepts was to implement them and test them rigorously on our data. This motivated us to focus on ML development next time we code and to acquire more data. 

Leaving the loft past midnight, we were content with our progress and optimistic for future software development. It's an exciting time, as we have spent the last 10 or so weeks brainstorming, needfinding, and prototyping but it seems now we are finally approaching the building stage. 

Pink Bagel ends

At the end of our nine day Pink Bagel exercise we found we had meaningful data from our two ad services indicating the perceived value of our three value propositions to our target demographic. That data is summarized below: 

Facebook Ad Services

Value Proposition	Impressions	Clicks	Click-Through Rate
Assisted Resolution	26546	60	.22%
Integrated Search	23612	44	.18%
Automatic Documentation	23502	40	.17%

Google AdWords

Value Proposition	Impressions	Clicks	Click-Through Rate
Assisted Resolution	10790	19	.18%
Integrated Search	8290	11	.13%
Automatic Documentation	3168	6	.19%

With this real-world data from our customers, we were able to determine that Assisted Resolution was consistently the most desired and clicked on feature. This did not surprise us, as we felt it was the biggest disruptor within the IT Service Management space. We were glad to see that this was validated with real data. 

We were also lucky enough to see the work of our colleagues as well as present our conclusions in an LGM. 

Pink Bagel begins

While Jon, Santi, and Jai were meeting with the Stanford Remedy team, Hussain and Andrew began the Pink Bagel exercise. We decided that we were going to use Facebook ads to get statistics regarding our product and vision to help refine it. Confident in our demographic, we decided to use Facebook Ads as a form of A/B testing, evaluating each of our three value propositions based on advertising performance. 

Running this as an experiment, we decided to run three simultaneous ad campaigns, one for each value prop, keeping all things constant besides the value prop itself. The constant factors included demographics, the link posted, the lifetime of the ad. These are listed below

• Demographics (according to Facebook this demo targeted ~490k individuals who use Facebook daily)
  • Location: United States
  • Age: 22-60
  • Language: English
  • Interests: System administrator, Information technology, Software engineering, or Technical support
  • Behavior: Professional or Technical 
• Posted by DVation linking to our website 
• Duration: 9 Days (02/23/2015 - 03/05/2015)

For images, we took a picture of Jon, added a speech bubble, and filled in different text for each value proposition. 

We look forward to the results of these experiments. 

Stanford Remedy Demo

On Monday, February 23, Santi, Jon, and Jai met with Tracy Neil, the Remedy ticketing system administrator, Chris Lundin, Director of the IT Service and Operations Center, and Adam, an employee at the Tier 2 IT Service Desk.  The purpose of this meeting was to further discuss the Remedy ticketing system as well as to present and gain feedback on another iteration of prototypes.  As a double bonus, we also got a brief walkthrough of the Remedy ticketing system and Chris offered to send some sample ticket data our way!

Probably the single biggest annoyance that Adam faces daily is user error when filling out and submitting a ticket.  This user error could be improper categorization of the issue which leads to a misrouted tickets.  Additionally, deficient user-written descriptions can make it nearly impossible for IT admins to determine what the issue is.  Even though issue reporting falls outside of our intended scope (ticket resolution), this observation gave us an idea.  A chat feature or at least an fast and easy way to contact the user reporting the issue would be convenient for the problem solver to find out more about the problem.

Another problem that occurs when using Remedy is that users forget to check the audit log, a log that keeps track of all the departments a ticket has been routed to, before re-routing a ticket.  Because of this, tickets sometimes get re-routed to the same department twice.  This made us think of providing a pop-up feature that warns when a ticket is about to re-routed to the same department.

When walking through our prototype, Chris and Adam provided some good feedback.  One important point was that routing a ticket to a specific person is usually not desired as it is against “etiquette”.  Routing a ticket to a group and then letting the group decide where the ticket should go is generally the desired behavior.  Additionally, many admins at Stanford use canned templates they send back to the user to help the user resolve an issue, so including this could be a helpful feature.  They also were not fans of our use of stars to denote priority, which we agreed with.  The last huge insight from the prototypes was that in order to get admins to post useful information and links for solving an issue, there needs to be some tangible benefit.  Or at least, they need to know that their work is helping others.  Chris suggested creating a good citizen score that would based on the number of people that found information an admin posted as helpful. We liked this idea a lot.

We left this meeting feeling good about our next iteration of prototypes while also ready to actually start some of the software development.  Oh and finally, we were going to have our hands on some data.

Prototype Presentation

After our meeting at VMWare we were able to meet up in the auditorium with the rest of the class to present our prototyping progress. We were extremely impressed by our peers, especially since it seemed many of them had digital prototypes made out whereas all we had was whiteboard sketches and POPs. We were able to present our deck as well, and we hope the Teaching Team was able to see our iterative creative process and product refinement. 

Prototype V3, V4 Demo

On Wednesday, February 18th the whole team headed to VMWare to meet with TJ, a Principal Architect associated very closely with the sales side, with the intent to show him our newest prototypes and get his feedback. Although in his position didn't put him close to the IT Admins we aimed to serve, we were able to learn a bit about the enterprise sales cycle. TJ also let us know that he thought out 'Integrated Search' feature was very useful and lets us know of some of the troubles the clients he had contact with had dealt with in the past. We walked away from the meeting not changing our prototype much but knowing we added value with this product and met a well-connected individual at VMWare. 

The next day, we returned to VMWare to meet with Vivian, Amit, and Denis again. This was the first meeting since our pivot, so we wanted to not only inform them of our new vision, but get some feedback on the prototypes we had developed. We focused many of our questions towards Denis, as he used to be an active IT Admin who would deal with ticketing systems.

The feedback we got was mixed. We felt that we had a hard time explaining to them our audience- we wanted to focus on those who have to deal with end-users who are often not technical submitting IT help tickets while it seemed that they believed we wanted to make a bug reporting system. This led into a large digression into Bugzilla and Helpzilla, which they walked us through. However, we did find that some features were definitely useful. Denis especially liked our Integrated Search feature which would allow for searching of internal documentation and Google simultaneously, as well as the Predictive Resolution, which would help to suggest ways to solve tickets. Amit, on the other hand, seemed very skeptical and asked questions about our user acquisition, infrastructure concerns, and other business aspects which we were not yet even thinking about. Amit later reflected his doubts about our direction in an email. 

This interaction left us with the conclusion that our application had the potential for great improvements for the IT ticketing space, but perhaps we were not presenting our value propositions clearly and concisely enough. 

Prototype V2 Demo

On Tuesday, February 17 at way too early in the morning (9:00 am) we met with Stephen Wong, the Director of IT at the Graduate School of Education.  Despite the team’s collective sleep deprivation, this was an extremely fruitful meeting as we learned about the ticketing system in place at Stanford (Remedy) and its annoyances, we discovered that there is a need for a smart ticketing system like the one we were prototyping, and we were given another contact within Stanford to discuss our ideas with further.

After introductions, Stephen briefed us on IT in academia and its difficulties by touching on the many different layers as well as the discrepancies in IT departments due to budgetary differences.  After this, Stephen told us about Stanford’s ticketing system — Remedy.  There were many important takeaways from Stephen’s discussion of Remedy.  First, the most common problem in issuing tickets is the routing of tickets to the wrong group.  Once assigned to a group, tickets are hand assigned to a particular person.  During our earlier brainstorming, the team had come up with the idea of auto-allocating tickets.  Stephen pointed out that this is an extremely difficult problem as there are hidden personal and political factors that go into assigning a ticket.  For example, tickets from certain faculty members who are known to be difficult to work with are assigned to IT personnel who are very patient.  Another idea that we thought of during brainstorming was automatically assigning a priority to a ticket based on learned features extracted from the ticket.  Stephen grounded us once more by mentioning the difficulty in this task.  With ticket prioritization political factors heavily influence the order in which a ticket is resolved.  For example, if the person who submitted the ticket is the dean or someone known to complain a lot, the ticket is handled quicker.

Despite finding out that two features we were excited about after brainstorming would be too hard to implement, we were not discouraged.  After elaborating more on our ideas and beginning to show the prototypes that Santi drew up, Stephen mentioned that to his knowledge, a system that learns from previous tickets does not exist and it would be pretty awesome for a system to automatically suggest resolutions.  This was extremely good news to hear!

After walking Stephen further through the prototypes, he pointed out some key insights.  First, many IT workers rely heavily on Google when troubleshooting so providing the ability to search Google and retain helpful links would be a great feature.  Additionally, Stephen mentioned that issues fall broadly into two categories: problems that are quick fixes taking less than 5 minutes to resolve and problems that the IT worker doesn’t immediately know how to fix and require more than 5 minutes.  If a problem fell under the 5 minute or less category, Stephen was a firm believer that the ticketing system should be as unobtrusive as possible allowing the user to dedicate his time to solving the problem and not navigating some feature-filled pages.  For problems requiring more than 5 minutes to solve, Stephen believed more information could be extremely helpful out streamlining the resolution process.

At the end of the meeting, Stephen put us in contact with Tracy Neil, the Remedy Administrator, another excellent person we could pick the brain of to advance our project.

Prototype V1 Demo

On Friday, February 13, Hussain and Jon woke up bright and early for the first meeting with VMware since we pivoted ideas.  We met with Terry, a user endpoint support guru, and Nate, an executive support employee.  The goal of the meeting was to learn more about the troubleshooting process from their perspective and to get some initial feedback on our first prototype.

The first thing we discussed was their day-to-day in order to better understand the work they performed.  Both Terry and Nate solved the issues assigned to them by directly working with users to solve the problem.  Nate specialized in working with executives while Terry focused on the average user.  One key insight we learned was that that 30-40% of the issues they encounter are quick fixes.

Next, we asked some questions to test some of the underlying assumptions to our idea of a smart ticketing system.  From this, we learned that many of the issues are repetitive – meaning that support is asked to resolve the same issues multiple times.  Additionally, under the current system used at VMware an article is written for repetitive issues to be added to the knowledge base.  However, Nate and Terry mentioned that these articles are frequently not used as they tend to be poorly written.  We also asked about the usefulness of logs when diagnosing a problem.  Nate and Terry said they do sometimes use logs to assist their work, but that the logs come from many sources, such as from logs from iOS devices, desktop computers, servers, etc.  The many sources of logs present a difficulty if we wanted to create software that could manage these logs and then diagnose problems from them.

Lastly we showed them a very rudimentary POP (Prototype on Paper).  Although the prototype was quite simple, it helped Terry and Nature better understand our idea.  From this, they provided a very helpful suggestion — filling out a report to resolve an issue needed to be as unobtrusive for the person resolving the error.  If this process was too cumbersome and there was no apparent benefit to completing it, they were confident workers would quickly disregard filling out reports.

At the conclusion of the meeting we had some good information that would be helpful in creating the next iteration of prototype.