Lesson 17: Graphical Exploratory Data Analysis (Note: meet in HSB 120)

Professor Huff needs HSB 117 for a lab exam so we will meet in HSB 120, today only.

Review:

• Exam 2 Results
• Exam Score % (top) > HW % (middle) > Overall % (below)
• Review Exam Questions

Presentation:

• Grades
  • Posted to Bb … finally caved, then learned to hate it all over again
  • Submit late homework for partial credit
    • DataCamp
    • Dice Game Simulation
  • Exam 2 Redo Option (not required)
    • Replaces lower exam score, cannot hurt your grade

• Graphical Exploratory Data Analysis
  • Instructor of “Statistical Thinking” DataCamp course
  • Always “look before you leap” (to conclusions)
  • Seaborn

Assignment:

• Begin new DataCamp course: Statistical Thinking Part 1 
  • Ch. 1, Graphical Exploratory Data Analysis

Exam 2 Results

Lesson 16: Review for Exam 2

Review:

• Dice Game Simulation
• Exam 2 on Wed Mar 13

Presentation:

• Review Topics for Exam 2
  • Intro to Python for Data Science
    • Basics
    • Lists
    • Functions and Packages
    • Numpy
  • Intermediate Python for Data Science
    • Matplotlib
    • Dictionaries & Pandas
    • Logic, Control Flow and Filtering
    • Loops
  • Daily Practice

Assignment:

• Complete Dice Game Simulation
• Study for Exam 2

Lesson 15: Python Loops for Monte Carlo Simulation

Review:

• Importing CSV
• Pandas and DataFrames
• Exam 2 on Wed, Mar 14

Presentation:

• Loops
  • for loops
  • while loops
  • Remember the Selling Price Estimates program? It has a “for” loop.
  • Here’s an example of a nested “while” loop: Sample Code
• Monte Carlo Simulation
  • Same as “Random Walk”
  • Enables “probabilistic” forecasts/estimates/etc

Assignment:

• Complete Intermediate Python for Data Science course on DataCamp
  • Complete all chapters before beginning this programming assignment
• Create a Python program on repl.it simulating a simple Dice Game:
  • Roll 7, 11 or doubles to win
  • Other rolls (not 7, 11 or doubles) result in a loss
  • Start with $100
    • bet $1 per roll
    • win pays $3 (a losing roll means you lose your $1 bet)
    • roll 100 times and find ending balance
    • simulate 100 outcomes (10,000 total rolls of the dice)
  • Report the mean and standard deviation of your final balances
  • Produce a histogram illustrating the distribution of results
  • Submit your assignment
    • Email me a link to your repl.it program
    • You’ll be scored on the program content + timeliness
      • Speed counts, i.e., no due date, earlier submission = better score
      • Submit your own program before helping others

Lesson 14: Python Importing CSV files with Pandas

Review:

• Graphs with Matplotlib
• DataCamp

Presentation:

• Lists
• Dictionaries
• Pandas and DataFrames
• Importing csv files
• Checkout my ImportCSVbaseball program on repl.it

Assignment:

• Write a program in repl.it that does the following:
  • Imports real estate csv data – use the same data set (with last digit of PID) you used for Exam 1
  • Calculate the mean and standard deviation for Bedrooms, Bathrooms and SqFt
• DataCamp: Intermediate Python for Data Science
  • Complete Ch. 4 Loops
  • Complete Ch. 5 Case Study: Hacker Statistics
  • Due Wed

Lesson 13: Python Data Graphics with Matplotlib

Review:

• repl.it
• matplotlib chapter

Presentation:

• How to generate plots in repl.it
• Check out my SimpleGraphs Program
  • Note: in repl.it click on the “New File” icon below “Settings” to enable “savefig()” functionality
  • If you don’t heed the note above, your program won’t work
  • Even if you do heed the note above, you may need to refresh to see the new files

Assignment:

• Use repl.it to produce a Python program that creates a scatter plot
  • Produce a scatter plot comparing Actual to Estimated Selling Prices (like Lesson 8)
  • Use *your* data!
  • Add a title to your scatter plot that includes your name and section (11:15 or 1:00)
  • Add axis labels for the x and y axis
• DataCamp
  • Complete Ch. 2 “Dictionaries & Pandas” in Intermediate Python for Data Science
  • Complete Ch. 3 “Logic, Control Flow and Filtering” in Intermediate Python for Data Science
  • Both chapters should be completed before class on Monday

Lesson 12: Using Python to Calculate Regression Estimates

Review:

• DataCamp
  • Variables
  • Lists
  • Functions
  • NumPy
• Exam 1 Remodels

Presentation:

• Basic Python Programming
  • We need a “sandbox” where we can write programs
    • Let’s try this repl.it
  • Write a basic “Hello World” program
    • Check out my HelloWorld program
  • Write a program to calculate a y^ equation
    • Use Lists and basic functions
    • Check out my SellingPriceEstimates program

• Preview Matplotlib
  • Line plots
  • Scatter plots
  • Histograms

Assignment:

• repl.it
  • First, join my classroom on repl.it
  • Write a “Hello World” program
  • Write a program to calculate y^ for the eight Exam 1 properties
• DataCamp
  • Complete Ch. 1 “Matplotlib” in Intermediate Python for Data Science

Lesson 11: Python Basics, Lists, Functions and Packages

Review:

• Intro to Python for Data Science
• DataCamp

Presentation:

• Review Chapters 1-2
  •  Variables
    • Initializing and Modifying
    • Types
      • Numeric: float, integer
      • Non-numeric: string, boolean (true/false)
      • Conversion
  • Lists
    • Like arrays or spreadsheets (think rows and columns as indices)
    • Store and process data with attributes
    • methods like append(), remove() to modify lists
• Preview Chapters 3-4
  • Functions
    • print(), type()
    • str(), int(), float(), bool()
    • many, many others
  • Packages
    • NumPy
      • import statement (e.g., import numpy as np)
      • numpy functions (e.g., mean, median, std, corrcoef; use by calling np.mean(name of array)
    • thousands of others

Assignment:

• Intro to Python for Data Science
  • Ch. 3: Functions and Packages (due today)
  • Ch. 4: NumPy (due tomorrow)

Lesson 10: Intro to Python for Data Science

Review:

• Multiple Regression Variable Selection

Presentation:

• Data Science
  • Combination of statistics + programming with big data
  • Hot like computer software development in the 1990’s
  • Salaries are high 
    

    

•  Python
  • Programmers and companies are moving to Python
  • Google uses Python extensively
  • Python is gaining market share rapidly

  • 

• DataCamp
  • Video + interactive tutorials
  • Classroom version for teachers/professors
  • Free to you and me

Assignment:

• Sign up on DataCamp using the email invitation you should have received (check your csupueblo.edu mail)
• Begin Intro to Python for Data Science
  • Ch. 1 Python Basics due today
  • Ch. 2 Python Lists due tomorrow

Lesson 9: Multiple Regression Variable Selection Procedures

Review:

• Measuring Forecast Error
• Exam 1 Remodel

Presentation:

• Variable Selection Procedures
  • Automated Selection
  • Forward Selection
  • Backward Selection
  • Stepwise Selection
  • Backward then Forward Selection
• Modeling Objective:
  • Achieve highest R^2 possible
    • For some data this may be below 0.50
    • Don’t sacrifice too much significance for small gains in R^2
  • Achieve high F-statistic
    • Fewer variables = generally preferable (keep it simple)
    • Significance of F (p-value) should be <0.05, lower is better
    • Individual variables should have p-value < 0.05
    • Occasionally it’s appropriate to make exceptions

Assignment:

• Produce a model to estimate MLB winning percentage
  • Use 2013 MLB Team statistics
  • Test all variables using one or more of the selection procedures above
  • Produce a Final Model, aiming to maximize R^2 and F-Statistic
  • Submit 1-page model summary in class on Monday, Feb 19