Automatidata Project
A multi-faceted data science project on NYC Taxi data to drive revenue insights through EDA, machine learning, and statistical analysis.
Exploratory Data Analysis (EDA)
The initial phase involved cleaning the dataset, handling outliers, and performing EDA to understand variable distributions and uncover initial insights into the taxi trip data.
Rides per Day of the Week
Payment Method Distribution
Key Findings:
- Trip distances are heavily right-skewed, with most trips being under 2 miles.
- Ride counts are highest from Wednesday through Saturday.
- Credit card is the most common payment method, followed by cash.
- Tip amounts do not vary significantly based on the number of passengers.
Classification: Predicting Generous Tippers
A tree-based classification model was built to predict whether a customer would be a "generous tipper" (tipping 20% or more). This section covers the ethical considerations, feature engineering, and model results.
Ethical Considerations & Objective Modification:
Initially, the objective was to predict non-tippers, which raised concerns about rider accessibility. The objective was modified to predict generous tippers (≥20%), a positive framing that aims to increase driver earnings without excluding customers.
Feature Engineering:
New features were created to improve model performance, such as trip duration categories, mean distance per location ID, and ratios related to fare amounts.
Key Predictors of a Generous Tip
Regression: Predicting Fare Amount
A multiple linear regression model was constructed to estimate the fare amount based on several independent variables like trip distance and duration. This model helps understand the factors that drive trip costs.
Fare Amount vs. Trip Distance
Model Performance & Insights:
The model demonstrated a strong linear relationship between trip distance and fare amount. After handling outliers and imputing values, the model achieved a high R² score, indicating it explains a significant portion of the variance in fare amounts.
Data Leakage Note:
During development, it was noted that calculating mean distance and duration from the entire dataset before splitting caused data leakage. While this improved the score in this specific case, it's not a recommended practice for production models.
Statistical Analysis: A/B Test
A two-sample t-test was conducted to determine if there is a statistically significant difference in the average fare amount between customers who pay with credit cards versus cash.
Hypothesis:
Null (H₀): There is no difference in the average fare amount between credit card and cash payments.
Alternative (Hₐ): There is a difference in the average fare amount between the two payment types.
Result:
The t-test yielded a p-value of approximately 6.80e-12. As this is significantly less than the 5% significance level, we reject the null hypothesis.
Business Recommendation:
The analysis shows that customers paying by credit card have a statistically significant higher average fare. Encouraging credit card payments could lead to increased revenue for drivers.