Store Sales Prediction With Linear Regression.¶

Import Library¶

#Basic python library which need to import
import pandas as pd
import numpy as np

#Date stuff
from datetime import datetime
from datetime import timedelta

#Library for Nice graphing
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.formula.api as sn
%matplotlib inline

#Library for statistics operation
import scipy.stats as stats

# Date Time library
from datetime import datetime

#Machine learning Library
import statsmodels.api as sm
from sklearn import metrics
from sklearn.cross_validation import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.svm import SVC, LinearSVC
from sklearn.metrics import mean_squared_error as mse
from sklearn.metrics import mean_absolute_error, mean_squared_error

# Ignore warnings
import warnings
warnings.filterwarnings('ignore')

# Settings
pd.set_option('display.max_columns', None)
np.set_printoptions(threshold=np.nan)
np.set_printoptions(precision=3)
sns.set(style="darkgrid")
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12

C:\ProgramData\Anaconda3\lib\site-packages\sklearn\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
C:\ProgramData\Anaconda3\lib\site-packages\sklearn\ensemble\weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d

Import Data from Csv files¶

# Importing training data set
train = pd.read_csv("train.csv")

#Import Test Data
test=pd.read_csv("test.csv")

# Import Store data set
stores = pd.read_csv("stores.csv")

# Now import features data set
feature = pd.read_csv("features.csv")

Merge the data sets:¶

  -(train+Store+Feature)
  -(test+Store+Feature)

# For Train data set
train_bt = pd.merge(train,stores) 
train = pd.merge(train_bt,feature)

#For test data set
test_bt = pd.merge(test,stores)
test= pd.merge(test_bt,feature)

train.head(2)

test.head(2)

print (train.info())
print ("*****************************************")
print (test.info())

<class 'pandas.core.frame.DataFrame'>
Int64Index: 421570 entries, 0 to 421569
Data columns (total 16 columns):
Store           421570 non-null int64
Dept            421570 non-null int64
Date            421570 non-null object
Weekly_Sales    421570 non-null float64
IsHoliday       421570 non-null bool
Type            421570 non-null object
Size            421570 non-null int64
Temperature     421570 non-null float64
Fuel_Price      421570 non-null float64
MarkDown1       150681 non-null float64
MarkDown2       111248 non-null float64
MarkDown3       137091 non-null float64
MarkDown4       134967 non-null float64
MarkDown5       151432 non-null float64
CPI             421570 non-null float64
Unemployment    421570 non-null float64
dtypes: bool(1), float64(10), int64(3), object(2)
memory usage: 51.9+ MB
None
*****************************************
<class 'pandas.core.frame.DataFrame'>
Int64Index: 115064 entries, 0 to 115063
Data columns (total 15 columns):
Store           115064 non-null int64
Dept            115064 non-null int64
Date            115064 non-null object
IsHoliday       115064 non-null bool
Type            115064 non-null object
Size            115064 non-null int64
Temperature     115064 non-null float64
Fuel_Price      115064 non-null float64
MarkDown1       114915 non-null float64
MarkDown2       86437 non-null float64
MarkDown3       105235 non-null float64
MarkDown4       102176 non-null float64
MarkDown5       115064 non-null float64
CPI             76902 non-null float64
Unemployment    76902 non-null float64
dtypes: bool(1), float64(9), int64(3), object(2)
memory usage: 13.3+ MB
None

Select only positive weekly sales¶

# tale only those values whose sales is positive. 
train = train[train['Weekly_Sales']>0]

Data Description:¶

1. Training Data¶

numeric_var_train=[key for key in dict(train.dtypes) if dict(train.dtypes)[key] in ['float64', 'int64', 'float32', 'int32']]
cat_var_train=[key for key in dict(train.dtypes) if dict(train.dtypes)[key] in ['object']]

# Train Numerical Data
train_num=train[numeric_var_train]

# Train Categorical Data
train_cat=train[cat_var_train]

print (numeric_var_train)
print (cat_var_train)

['Store', 'Dept', 'Weekly_Sales', 'Size', 'Temperature', 'Fuel_Price', 'MarkDown1', 'MarkDown2', 'MarkDown3', 'MarkDown4', 'MarkDown5', 'CPI', 'Unemployment']
['Date', 'Type']

# Use a general function that returns multiple values
def var_summary(x):
    return pd.Series([x.count(), x.isnull().sum(), x.sum(), x.mean(), x.median(),  x.std(), x.var(), x.min(), x.dropna().quantile(0.01), x.dropna().quantile(0.05),x.dropna().quantile(0.10),x.dropna().quantile(0.25),x.dropna().quantile(0.50),x.dropna().quantile(0.75), x.dropna().quantile(0.90),x.dropna().quantile(0.95), x.dropna().quantile(0.99),x.max()], 
                  index=['N', 'NMISS', 'SUM', 'MEAN','MEDIAN', 'STD', 'VAR', 'MIN', 'P1' , 'P5' ,'P10' ,'P25' ,'P50' ,'P75' ,'P90' ,'P95' ,'P99' ,'MAX'])

num_summary=train_num.apply(lambda x: var_summary(x)).T
num_summary

def cat_summary(x):
    return pd.Series([x.count(), x.isnull().sum(), x.value_counts()], 
                  index=['N', 'NMISS', 'ColumnsNames'])

cat_summary=train_cat.apply(lambda x: cat_summary(x))
cat_summary

2. Testing Data¶

numeric_var_test=[key for key in dict(test.dtypes) if dict(test.dtypes)[key] in ['float64', 'int64', 'float32', 'int32']]
cat_var_test=[key for key in dict(test.dtypes) if dict(test.dtypes)[key] in ['object']]

# Train Numerical Data
test_num=test[numeric_var_test]

# Train Categorical Data
test_cat=test[cat_var_test]

print (numeric_var_test)
print (cat_var_test)

['Store', 'Dept', 'Size', 'Temperature', 'Fuel_Price', 'MarkDown1', 'MarkDown2', 'MarkDown3', 'MarkDown4', 'MarkDown5', 'CPI', 'Unemployment']
['Date', 'Type']

# Numerical data summary report
num_summary=test_num.apply(lambda x: var_summary(x)).T

num_summary.head()

# categorical data summary report
def cat_summary(x):
    return pd.Series([x.count(), x.isnull().sum(), x.value_counts()], 
                  index=['N', 'NMISS', 'ColumnsNames'])

cat_summary=test_cat.apply(lambda x: cat_summary(x))
cat_summary

# Run Pandas profilingto see the over all report
import pandas_profiling
pandas_profiling.ProfileReport(train)

pandas_profiling.ProfileReport(test)

	Store	Dept	Date	Weekly_Sales	IsHoliday	Type	Size	Temperature	Fuel_Price	MarkDown1	MarkDown2	MarkDown3	MarkDown4	MarkDown5	CPI	Unemployment
0	1	1	2010-02-05	24924.50	False	A	151315	42.31	2.572	NaN	NaN	NaN	NaN	NaN	211.096358	8.106
1	1	2	2010-02-05	50605.27	False	A	151315	42.31	2.572	NaN	NaN	NaN	NaN	NaN	211.096358	8.106

	Store	Dept	Date	IsHoliday	Type	Size	Temperature	Fuel_Price	MarkDown1	MarkDown2	MarkDown3	MarkDown4	MarkDown5	CPI	Unemployment
0	1	1	2012-11-02	False	A	151315	55.32	3.386	6766.44	5147.7	50.82	3639.9	2737.42	223.462779	6.573
1	1	2	2012-11-02	False	A	151315	55.32	3.386	6766.44	5147.7	50.82	3639.9	2737.42	223.462779	6.573

	N	SUM	MEAN	MEDIAN	STD	VAR	MIN	P1	P5	P10	P25	P50	P75	P90	P95	P99	MAX
Store	115064.0	2.558817e+06	22.238207	22.000	12.809930	1.640943e+02	1.000	1.000	3.000	5.000	11.000	22.000	33.000	40.000	43.000	45.000	45.000
Dept	115064.0	5.101883e+06	44.339524	37.000	30.656410	9.398155e+02	1.000	1.000	4.000	7.000	18.000	37.000	74.000	92.000	95.000	98.000	99.000
Size	115064.0	1.570597e+10	136497.688921	140167.000	61106.926438	3.734056e+09	34875.000	34875.000	39690.000	39910.000	93638.000	140167.000	202505.000	204184.000	206302.000	219622.000	219622.000
Temperature	115064.0	6.206760e+06	53.941804	54.470	18.724153	3.505939e+02	-7.290	11.440	23.980	29.970	39.820	54.470	67.350	79.480	83.820	92.140	101.950
Fuel_Price	115064.0	4.121070e+05	3.581546	3.606	0.239442	5.733244e-02	2.872	2.957	3.161	3.227	3.431	3.606	3.766	3.866	3.951	4.079	4.125

Number of variables	16
Number of observations	421570
Total Missing (%)	21.1%
Total size in memory	51.9 MiB
Average record size in memory	129.0 B

Distinct count	2145
Unique (%)	0.5%
Missing (%)	0.0%
Missing (n)	0
Infinite (%)	0.0%
Infinite (n)	0

Store Sales Prediction With Linear Regression.¶

Import Library¶

Import Data from Csv files¶

Merge the data sets:¶

Select only positive weekly sales¶

Data Description:¶

1. Training Data¶

2. Testing Data¶

Overview

Variables

Correlations

Sample

Overview

Variables

	Date	Type
N	420212	420212
NMISS	0	0
ColumnsNames	2011-12-23 3018 2011-11-25 3016 2011-12-...	A 214961 B 162787 C 42464 Name: Type...

	Date	Type
N	115064	115064
NMISS	0	0
ColumnsNames	2012-12-21 3002 2012-12-07 2989 2012-12-...	A 58713 B 44500 C 11851 Name: Type, d...

Numeric	13
Categorical	2
Boolean	1
Date	0
Text (Unique)	0
Rejected	0
Unsupported	0

Minimum	126.06
5-th percentile	126.5
Q1	132.02
Median	182.32
Q3	212.42
95-th percentile	221.94
Maximum	227.23
Range	101.17
Interquartile range	80.394

Standard deviation	39.159
Coef of variation	0.22873
Kurtosis	-1.8297
Mean	171.2
MAD	38.066
Skewness	0.085219
Sum	72174000
Variance	1533.4
Memory size	6.4 MiB

Value	Count	Frequency (%)
129.8555333	711	0.2%
131.1083333	708	0.2%
129.84596670000002	707	0.2%
130.38490320000002	706	0.2%
130.683	706	0.2%
131.0756667	706	0.2%
130.6457931	706	0.2%
130.7196333	705	0.2%
130.4546207	705	0.2%
129.98454840000002	704	0.2%
Other values (2135)	414506	98.3%

2011-12-23	3027
2011-11-25	3021
2011-12-16	3013
Other values (140)	412509

Standard deviation	30.492
Coef of variation	0.68893
Kurtosis	-1.2156
Mean	44.26
MAD	26.537
Skewness	0.35822
Sum	18658822
Variance	929.77
Memory size	6.4 MiB

Minimum	2.472
5-th percentile	2.653
Q1	2.933
Median	3.452
Q3	3.738
95-th percentile	4.029
Maximum	4.468
Range	1.996
Interquartile range	0.805

Standard deviation	0.45851
Coef of variation	0.13642
Kurtosis	-1.1854
Mean	3.361
MAD	0.4032
Skewness	-0.1049
Sum	1416900
Variance	0.21024
Memory size	6.4 MiB

Value	Count	Frequency (%)
126.064	678	0.2%
126.0766452	679	0.2%
126.08545159999998	675	0.2%
126.08929029999999	682	0.2%
126.1019355	686	0.2%

Value	Count	Frequency (%)
2011-12-23	3027	0.7%
2011-11-25	3021	0.7%
2011-12-16	3013	0.7%
2011-12-09	3010	0.7%
2012-02-17	3007	0.7%
2011-12-30	3003	0.7%
2012-02-10	3001	0.7%
2011-12-02	2994	0.7%
2012-03-02	2990	0.7%
2012-10-12	2990	0.7%
Other values (133)	391514	92.9%

Value	Count	Frequency (%)
1	6435	1.5%
10	6435	1.5%
38	6435	1.5%
21	6435	1.5%
67	6435	1.5%
16	6435	1.5%
14	6435	1.5%
13	6435	1.5%
79	6435	1.5%
81	6435	1.5%
Other values (71)	357220	84.7%

Value	Count	Frequency (%)
1	6435	1.5%
2	6435	1.5%
3	6435	1.5%
4	6435	1.5%
5	6347	1.5%

Distinct count	2278
Unique (%)	0.5%
Missing (%)	64.3%
Missing (n)	270889
Infinite (%)	0.0%
Infinite (n)	0

Minimum	0.27
5-th percentile	149.19
Q1	2240.3
Median	5347.4
Q3	9210.9
95-th percentile	21801
Maximum	88647
Range	88646
Interquartile range	6970.6

Standard deviation	8291.2
Coef of variation	1.1442
Kurtosis	17.606
Mean	7246.4
MAD	5262.8
Skewness	3.3418
Sum	1091900000
Variance	68744000
Memory size	6.4 MiB

Distinct count	1500
Unique (%)	0.4%
Missing (%)	73.6%
Missing (n)	310322
Infinite (%)	0.0%
Infinite (n)	0

Minimum	-265.76
5-th percentile	1.95
Q1	41.6
Median	192
Q3	1926.9
95-th percentile	16497
Maximum	104520
Range	104790
Interquartile range	1885.3

Standard deviation	9475.4
Coef of variation	2.8415
Kurtosis	37.59
Mean	3334.6
MAD	4690.4
Skewness	5.4413
Sum	370970000
Variance	89782000
Memory size	6.4 MiB

Distinct count	1663
Unique (%)	0.4%
Missing (%)	67.5%
Missing (n)	284479
Infinite (%)	0.0%
Infinite (n)	0

Minimum	-29.1
5-th percentile	0.65
Q1	5.08
Median	24.6
Q3	103.99
95-th percentile	1059.9
Maximum	141630
Range	141660
Interquartile range	98.91

Standard deviation	9623.1
Coef of variation	6.6854
Kurtosis	77.688
Mean	1439.4
MAD	2578.1
Skewness	8.3995
Sum	197330000
Variance	92604000
Memory size	6.4 MiB

Distinct count	1945
Unique (%)	0.5%
Missing (%)	68.0%
Missing (n)	286603
Infinite (%)	0.0%
Infinite (n)	0

Minimum	0.22
5-th percentile	28.76
Q1	504.22
Median	1481.3
Q3	3595
95-th percentile	12646
Maximum	67475
Range	67475
Interquartile range	3090.8

Standard deviation	6292.4
Coef of variation	1.8599
Kurtosis	29.997
Mean	3383.2
MAD	3329.7
Skewness	4.8475
Sum	456620000
Variance	39594000
Memory size	6.4 MiB

Distinct count	2294
Unique (%)	0.5%
Missing (%)	64.1%
Missing (n)	270138
Infinite (%)	0.0%
Infinite (n)	0

Minimum	135.16
5-th percentile	715.52
Q1	1878.4
Median	3359.4
Q3	5563.8
95-th percentile	11269
Maximum	108520
Range	108380
Interquartile range	3685.4

Standard deviation	5962.9
Coef of variation	1.2882
Kurtosis	107.85
Mean	4629
MAD	2989.8
Skewness	8.1699
Sum	700970000
Variance	35556000
Memory size	6.4 MiB

Minimum	34875
5-th percentile	39690
Q1	93638
Median	140170
Q3	202500
95-th percentile	206300
Maximum	219622
Range	184747
Interquartile range	108870

Standard deviation	60981
Coef of variation	0.446
Kurtosis	-1.2063
Mean	136730
MAD	52517
Skewness	-0.32585
Sum	57640387438
Variance	3718600000
Memory size	6.4 MiB

Standard deviation	12.785
Coef of variation	0.5759
Kurtosis	-1.1465
Mean	22.201
MAD	10.996
Skewness	0.077763
Sum	9359084
Variance	163.46
Memory size	6.4 MiB

Distinct count	3528
Unique (%)	0.8%
Missing (%)	0.0%
Missing (n)	0
Infinite (%)	0.0%
Infinite (n)	0

Minimum	-2.06
5-th percentile	27.31
Q1	46.68
Median	62.09
Q3	74.28
95-th percentile	87.27
Maximum	100.14
Range	102.2
Interquartile range	27.6