Dataset Prediction Model

To download the Yelp Dataset you first have to register to join their dataset challenge on their site linked here. This allows you to choose from the businesses, check-ins, users, tips, and photos datasources, each of which is stored in JSON format. For this project we needed to obtain the businesses and check-ins files to get the amount of checkins associated with each businesses throughout the times provided, and correlate that to the business rating and review count that is specified in the businesses file.

Conceptually what we want to do is first parse through all of the check-in data provided by Yelp, and for each business get the total amount of check-ins that occur throughout the entire history of the business, and then finally map them to the business id in the object format {business_id: total_checkins}. From there we can parse through the businesses data and at each value map the business id provided to the newly parsed check-ins file to get total check-ins and create an object for each business in the format {business_id: [rating, review_count, total_checkins]}. This process can also be modified to only include certain cities by only choosing to log businesses that have a matching 'city' parameter in their object definition. Now that we have a JSON object that represents all important data for us it has to be brought into a useable MATLAB format to run K-means clustering algorithms on for analysis. This is achieved using the JSONlab plugin for MATLAB that converts the JSON sting into a structure format representing all the data. Finally this structure has to be parsed one at a time and account for null data entries (when there are no recorded check-ins) and then create a usable a matrix format of just relevant data.

/*Create a usable data format of {businessid: total checkins}*/
var fs = require('fs');

var checkIns = {};
var checkInRaw = fs.readFileSync('../data/yelp_academic_dataset_checkin.json', 'utf8').toString().split("\n");
for (var j = 0; j < checkInRaw.length -1; j++){
  checkInRaw[j] = JSON.parse(checkInRaw[j]);
  var totalCheckIns = 0;
  var keys = Object.keys(checkInRaw[j].checkin_info);
  var businessKey = checkInRaw[j].business_id;
  for(var i = 0; i < keys.length; i++){
    totalCheckIns += checkInRaw[j].checkin_info[keys[i]];
  }
  checkIns[businessKey] = totalCheckIns;
}

fs.writeFile('../data/yelp_checkins.json', JSON.stringify(checkIns, null, 2), function(err){
  if(err){
    return console.log(err);
  }
});

/*Create a usable data format of {businessid: [rating, review count, checkin count]}*/
var fs = require('fs');

var businesses = {};
var businessesRaw = fs.readFileSync('../data/yelp_academic_dataset_business.json', 'utf8').toString().split("\n");
var checkins = JSON.parse(fs.readFileSync('../data/yelp_checkins.json', 'utf8'));
for (var j = 0; j < businessesRaw.length - 1; j++){
  businessesRaw[j] = JSON.parse(businessesRaw[j]);
  businesses[businessesRaw[j].business_id] = [businessesRaw[j].stars, businessesRaw[j].review_count, checkins[businessesRaw[j].business_id]];
}

fs.writeFile('../data/yelp_businesses_all.json', JSON.stringify(businesses, null, 2), function(err){
  if(err){
    return console.log(err);
  }
});

Data split by cities script here.

function return_mat = RemoveNulls(input)
  result_cells = struct2cell(input(1,1));
  return_mat = [];
  for i=1:length(result_cells)
    try
      cell = cell2mat(result_cells(i));
    catch
      result_cells{i,1}{3} = 0;
      cell = cell2mat(result_cells{i});
    end
    return_mat = [return_mat; cell];
  end
end

One thing that we wanted to explore was how different population densities would affect the results in our clustering. To analyze this futher we sifted through our data to see large concentrations of businesses in certain cities that we could define as being large metropolitan areas, and areas a lower but still large enough data space to analyze as what we can define as a suburban area (~50-100 businesses).

Once we aggregated all of our data we ran the K-means clustering algorithm in MATLAB with our parsed data to determine any cluster trends across specific permutations of paired data sets. We determined this through what seemed to be most likely to give an obvious trend through observations of the paired sets. The data is displayed as follows for the overall and sub-sectioned aggregates.

K-Means Plot of the ratings of each of the business versus the number of reviews for the entire dataset.

1st cluster - 3.8602, 968.0172, K = 2
2nd cluster - 3.6937 27.9724, K = 23

For the first cluster of data there seems to be a slight increase in the average rating, given a large amount of user reviews. This seems to signify that a higher volume of user ratings may typically imply popularity, which could influence users to give higher ratings. However it is possible that this change could just be due to random variation, but given the large sample size of the entire data set, it can be safe to say that this slight change is relatively accurate without completely resampling the data set.

K-Means Plot of the ratings of each of the business versus the number of checkins for the entire dataset.

1st cluster - 3.6618, 13,304, K = 2
2nd cluster - 3.6949 98.7103, K = 28

For the second cluster of data we found that rating and the number of Check-ins have no significant impact on each other. Both data clusters gave a similar rating average despite the second cluster having a significant amount of check-ins per business.It can definately be said that one can conclude nothing from the clustering due to the apparent lack of variation among the data fields.

K-Means Plot of the number of reviews versus the number of checkins for the entire dataset.

1st cluster - 1,940.7, 13,303, K = 2
2nd cluster - 32.8421, 98.7113, K = 30

For the third data set it can be seen from the first cluster that a large amount of reviews result in significantly large amounts of check-ins and from the second cluster a small amount of reviews result in smaller amounts of check ins. This can be easily explained as typically popular business are promoted to incentivize their customers to check-in for some added benefit. Another aspect that this clustering uncovers is the disparity between popular and unpopular businesses, that the difference is vast enough to be extremely polarizing.

K-Means Plot of the ratings of each of the business versus the number of reviews for the suburban areas.

1st cluster - 3.7222, 75.1111, K = 2
2nd cluster - 3.6789, 9.5963, K = 5

For the first clusters of data it can be seen that the averages are much lower compared to the previous two data sets. This is probably indicative that smaller populations have smaller numbers of active users for yelp. The ratings for each cluster suggest that there is a higher concentration of similar ratings in smaller cities. This could be because smaller populations propagate opinions faster, leading to a growing similarity between user ratings of businesses.

K-Means Plot of the ratings of each of the business versus the number of checkins for the suburban areas.

1st cluster - 3.6875, 15.6964, K = 2
2nd cluster - 3.6429, 184.9524, K = 8

For the second clusters of data, like all of the other respective cluster sets, the values in each cluster show little variation and are therefore no particular conclusions can be drawn from the specific set of clusters. But it can be seen from the similarity that Yelp activity is proportionally the same for users as concluded in the large cities cluster set.

K-Means Plot of the number of reviews versus the number of checkins for the suburban areas.

1st cluster - 69.6818, 180.8636, K = 2
2nd cluster - 11.6009, 15.3408, K = 7

For the third clusters of data, again, population seems to be affecting values of each cluster, the averages for the smaller data set is significantly smaller indicating that in smaller cities there are less users and possibly less restaurants available for review.

Plot of the ratings of each of the business versus the number of checkins for Belmont, North Carolina.

Plot of the ratings of each of the business versus the number of reviews for Belmont, North Carolina.

Plot of the ratings of each of the business versus the number of checkins for Carnegie, Pennsylvania.

Plot of the ratings of each of the business versus the number of reviews for Carnegie, Pennsylvania.

Plot of the ratings of each of the business versus the number of checkins for San Tan Valley, Arizona.

Plot of the ratings of each of the business versus the number of reviews for San Tan Valley, Arizona.

K-Means Plot of the ratings of each of the business versus the number of reviews for the metropolitan areas.

1st cluster - 3.8218, 1,224.98, K = 2
2nd cluster - 3.6737, 35.3996, K = 29

For the first clusters of data it is apparent that it resembles the values of the overall data set with slight variation, the values for each cluster seem to be slightly lower, especially the rating values. Each cluster seems to have a higher value of number of reviews, this is expected due to the large populations found in metropolitan cities.

K-Means Plot of the ratings of each of the business versus the number of checkins for the metropolitan areas.

1st cluster - 3.6786, 14,568, K = 2
2nd cluster - 3.6750, 134.8531, K = 17

For the second clusters of data we found that the values are once again, similar to its overall counterpart. Ratings compared to number of check-ins seem to be similar additionally with the small cities data set. This seems to suggest that population numbers don’t necessarily affect yelp activity for users.

K-Means Plot of the number of reviews versus the number of checkins for the metropolitan areas.

1st cluster - 42.8709, 134.6624, K = 2
2nd cluster - 2117.4, 14,438, K = 17

For the third clusters of data it can be seen once again that due to a higher population found in cities, that all values in each cluster are higher than the overall data set. The same trends noticed in the overall set are apparent in this set of clusters as well.

Plot of the ratings of each of the business versus the number of checkins for Charlotte, North Carolina.

Plot of the ratings of each of the business versus the number of reviews for Charlotte, North Carolina.

Plot of the ratings of each of the business versus the number of checkins for Phoenix, Arizona.

Plot of the ratings of each of the business versus the number of reviews for Phoenix, Arizona.

Plot of the ratings of each of the business versus the number of checkins for Las Vegas, Nevada.

Plot of the ratings of each of the business versus the number of reviews for Las Vegas, Nevada.

Fitting a linear plane equation to the entire dataset to find a rating based on the input of review count and checkin count. This process was replicated for both Suburban and Metropolitan areas as well to get unique equations for each.