General
 |
Business Analytics Special Project can be seen as a capstone-like experience in which the student would demonstrate a broad knowledge of Business Analytics by undertaking hands-on projects with realistic data. |
Weekly outline
Project Charter and Data Dictionary (Deliverable #1)
Description:
The project charter is a formal document that describes the project in its entirety, guiding project development.
In order to define all the essential components for the successful development of the project, a Project Charter document must be developed. You should make sure that all relevant details regarding the scope, requirements and risks of the project are well documented and the final version of this document should also be validated by the client. Your Project Charter should include the following sections:
- Identification of the team and main stakeholders involved
- Business challenge context and key numbers
- Identification of the project objective(s)
- Actions that will be informed by the final result (to be taken by end users)
- Summary of data to be used
- Main risks, assumptions and constraints & mitigation plan
- Project schedule (timeline & milestones)
- Communication tools, meetings and frequency
- Ownership of data, ethics and GDPR
The data dictionary is a document that allows anyone to easily understand a dataset. In this case, this dataset will be the one used in your PBL project. The document must present a detailed description of the information contained in the shared database, identifying variables/ features, their description, their source, among other. In case the database consists of different tables, you should also detail how they are related.
• Variable name • Type • Description • Example • # of distinct values (if applicable) • Source • Confidentiality • Min. value (if applicable) • Max. value (if applicable) • Mean (if applicable) • Volume (# of entries) • % missing values •
Exploratory Data Analysis (Deliverable #2)
Description:
The exploratory data analysis helps guide data validation and allows the project teams to have a better understanding of the data and the problem, by discovering patterns, spotting anomalies, and check assumptions with the help of summary statistics and graphical representations.
The exploratory data analysis helps to understand all the variables, their formats and examples, and contextualize them within the business and through time. This step also allows the team to validate that the dataset selected for analysis meets the established objective, as well as other data quality requirements. The exploratory data analysis should contain univariate and multivariate analysis:
- Visualizations and summary statistics that describe the data and allow to find patterns and/or to detect outliers that may exist within it (e.g., Histograms, Box plots, bar and line charts, …)
- Correlation plots between variables and other analyses that allow for the identification of trends and the detection of anomalous behaviour (e.g., Heat maps, Scatter plots, Cross-tabulation, …)
Modeling Roadmap including the baseline model (Deliverable #3)
Description:
The modeling roadmap will specify the modelling dataset, the models to be trained, and the models' validation and evaluation.
The modelling roadmap serves as an agreement on how the modelling will be developed. It should detail and justify all technical aspects related to modelling, such as:
- What is the model goal
- What is the target variable (for supervised machine learning models)
- How the modelling dataset will be built (i.e., what does a line in the dataset represent)
- What variable groups will be used to train the model
- What will be the modelling baseline
- What algorithms will be applied, and the minimum set of hyperparameters to be tested, for each algorithm
- Which evaluation metric will be used to select the model
- How will the model be validated (e.g., train-test split, cross-validation...)
- Which additional metrics will be used to evaluate the models, if any.
To say that a machine learning model is good or bad, we need to compare it with the existing practice. The baseline model represents the ”business-as-usual” of the task. If there was no machine learning model implemented, how would the task be performed? With what performance? It is important to establish a baseline in order to understand how much your model would improve the performance of the task.
A baseline model adapted to the problem in hand should be defined, justified, and implemented, evaluating it using the selected metric for model performance.
Modeling Report and Plan for Testing (Deliverable #4)
Description:
The modelling report details the results obtained in the modelling stage, including results from a bias audit, if applicable, to exploring potential unintended consequences of deploying the model.- Different combinations of parameters should be experimented to develop the best machine learning model possible, combining, for example, different algorithms, hyperparameters or sets of variables.
Two types of models should be presented:- At least one simple, interpretable model, for instance, a shallow decision tree, where the model output is interpretable. The goal of this model is to verify that the pipeline is working as expected, and the modelling results make sense.
- More complex models, such as ensemble models, which should have a better performance than the simple model.
The presentation of the modeling results should include:- Description of the different algorithms, hyperparameters and features you experimented with
- Model evaluation visualizations, comparing different models
- Selection of the best model, justifying your choice
- Definition of the groups that may be differently affected by your model, and a comparison of the best model performance for the different groups
- Future work (e.g., what would you like to had developed if you had the time).
Your report should include a description of the plan for testing, namely:- Who are the end-users of your model?
- How would you design the interface for deploying your model?
- Which study design would you use to evaluate the adoption of your system by its end-users?
- Which measures would you use to evaluate the adoption of your system by its end-users?
- Who would participate in your pilot study and why?
- How would you prepare your pilot implementation (e.g., is training needed?)
- How many data points/ for how long should the pilot be running?
Pipeline (Deliverable #5)
Description:
The pipeline consists of a sequence of scripts that perform all the tasks needed for modelling and evaluation of model performance.A code repository that is well-organized, clean (e.g., all branches merged) and well-documented (i.e., has a readme, has instructions to run the code, functions are described…) should be delivered. The repository should have no data or passwords committed to it. The pipeline should allow the user to automatically train one or more models, starting from the raw data. Then, it should allow the user to visualize the results of the trained model(s). The more automated and flexible the pipeline is (e.g., allowing the user to select and run multiple algorithms with different hyperparameters, and then to compare their results; allowing the user to pick different train and test sets, or different sets of features, etc.,), the better.
Your project’s pipeline should include scripts that perform the following tasks:
- Data collection
- Data cleaning
- Data transformation for modeling (including ways to deal with invalid and null values, if any)
- Data splits for model validation
- Model training
- Performance Evaluation of models
- Visualization of model results
Output:
Github repository
Peer Assessment
Please assess the work of your colleagues by using the following criteria. Your feedback will be considered in assigning the individual grades. Please try to be as honest and fair as possible in your assessment.