Under the hood: How does Unfare work?

Ever since the launch of the app – Unfare, I have been asked this question; how does it work? So to answer that, I decided to write a post about it.

Everyday morning, Unfare gets a price feed for all the routes and dates that is in scope and stores it into a database. It then automatically runs a few processes on this data which updates information like the “fair price”, price list of last 7 days, etc. It also updates the “training data” with the necessary predictor variables for the algorithm. Once all these processes are complete, it will then hand it over to the algorithm.

After completing a few sanity checks, the algorithm updates itself with the data from the flights that had taken off the previous day. This ensures that any mistake in its previous prediction is taken into account. Once updated, the algorithm runs in stages. Each stage taking in new training data compared to the previous set. This ensures that the algorithm improves its accuracy and at the same time is generic to predict on new data.

Once the algorithm is ready to predict, it is fed with that day’s flight and price information. The prediction along with the confidence levels are stored in the database which are then made available to anyone who uses the Unfare app.

The beauty of this workflow is that it is 100% automated and requires no manual intervention from my side. Even if the workflow is disrupted at any point, it will auto-restart from the same point of disruption.

A few technical details:

  • Cloud computing service: Amazon EC2
  • OS: Linux
  • Server: Apache
  • Database: MySQL
  • Statistical tool: R project

I will do another post on the logic or the hack that Unfare exploits in the coming days. Stay tuned!

Common terminologies in Unfare app

Here are the common terms used in the Unfare app and the explanations for the same:

  1. Overall prediction – BUY: When you see an overall prediction as “BUY” in the Unfare app against a given search, it means the following. The lowest price for this route and flight date is expected to go up in the next few days.
  2. Overall prediction – WAIT: When you see an overall prediction as “WAIT” in the Unfare app against a given search, it means the following. The lowest price for this route and flight date is expected to come down in the next few days. However, this does not mean that the prices will go up the next day. It only means that before the departure of the flight, you will find a better lowest fare for the route. However, we will not be able to predict the exact flight which will come with the lower fare. For example, you intend to start from Bangalore to Delhi on a Friday evening. The overall prediction says wait. A week later, just as the prediction, you may find a lower fare, however, that fare maybe for a flight on Friday morning, which may not be convenient to you. Hence if you have specific flight preference, use the individual flight predictions.
  3. Prediction – BUY: When you see a prediction as “BUY” in the Unfare app against a given flight, it means the following. The price for this flight is expected to go up during the next few days. So if this flight is what you want to book, you should book it right away.
  4. Prediction – WAIT: When you see a prediction as “WAIT” in the Unfare app against a given flight, it means the following. The price for this flight is expected to come down during the next few days. So if this flight is what you want to book, you could probably wait for the prices to come down. However, do note that this does not mean that the next price change will be lower than this price. For example, you searched for a flight and the current prediction for this flight is “WAIT”. The current price is Rs. 6000/-. It could very well happen that the next day the price goes up to Rs. 7500/-. Remember that the prediction – “WAIT”, in this example, only means that you will find a price lower than Rs. 6000/- before the flight departs. So, even though the price went up to Rs. 7500/-, maybe after a week, you will find the price at 4500/- thus saving Rs. 1500/- for you.
  5. Fair price: Fair price is the average observed price for the given flight for the given number of days to departure. For example, suppose there are 10 observed prices for a given flight, say Indigo 7am flight from Delhi to Mumbai, each taken exactly 3 days before its departure, the fair price will be an average of these 10 price observations. One important thing to note here is that this does not take into account high demand day prices like that on a holiday/ Monday mornings/ etc. So this parameter should not be looked at in isolation.
  6. Last 7 day prices: This graph shows the last 7 price observations that were observed on each of the last 7 days.

Understand the beta limitations of unfare app

logo200

Unfare app is currently in Beta. While it is always good to bring out the “perfect” app, it’s not really a practical thing to do in app development. We believe that it is your feedback that will make it perfect. And so, we decided to put out a limited functionality application as a public beta to understand how the users are using the application and to iteratively add features into it. So, here are some of the key  known limitations of the Unfare beta version as of today:

  1. Only the flights between the top 10 busiest airports in India are in scope. These are:
    • New Delhi
    • Mumbai
    • Bangalore
    • Chennai
    • Kolkata
    • Hyderabad
    • Pune
    • Goa
    • Ahmedabad
    • Kochi
  2. The prices are updated only once a day. Hence any intra-day price fluctuations may not be displayed.
  3. Only prices within the last 90 days before the flight departure are available.
  4. Round trip fares are taken as sum of onward flight and return flight. While this generally works in the Indian context, in some cases, the airlines offer round trip discounts, which will not be shown in the Unfare app.

Please do note that these are not technical limitations and will be addressed in a phased manner.

Unfare: The story so far

applescriptIt was back in August ’15 when I got curious about the movement of airfares in India. I knew the price fluctuates for any given flight before the day of departure but I wanted to find a pattern around it. But to start my research, I needed the price data. So I wrote a simple Applescript that would search the flights in a browser, copy the relevant search results and then parse it into a comma separated text file. Pretty crude, but hey, it worked! But the big downside to this method was that it required me to wake up every morning, open up my laptop and run the script manually. The screen would flash for 30min to get 30day fares for 6 routes.

I started looking for patterns. At first, I thought maybe the flight price fluctuation was limited to a few airlines. But the data suggested otherwise. There were price drops and increases across every airline. So it had to be more complicated than that. Then I thought the price patterns were just historical. Well, that wasn’t the case either. I then started studying the airline pricing models. Though it was complicated, it did help a great deal. Based on the hundreds of hours of research, I started creating a pricing model that could predict close to the actual price movements of the flights. And the best part of it was that it was created with machine learning algorithm. So once it goes wrong, it will learn from that mistake and re-model for the next prediction. And that’s how unfare was born. In the coming days I will post about how it’s actually done.

The idea of airfare prediction is not unique or innovative. Almost everyone wants to know the future price movements. Be it stock market price, commodities price, airfares. The only difficulty is to build a reliable model. Even in the airfare prediction market, there are several players but but none of them actually cater to the Indian domestic market. And given the price sensitivity of the customers in India, I thought it would be a great idea to put together an app for it.

So ladies & gentlemen, presenting, the airfare prediction app in India, unfare.

Let’s unfare it!