The first dataset I'm going to explore is from the FBI's National Instant Criminal Background Check System (NICS). NICS is used to check eligibility for the purchase and possession of firearms and explosives. More information about NICS can be found at the official site: https://www.fbi.gov/services/cjis/nics. The dataset I'll be using is composed of the number of background checks conducted by month and state for various kinds of transactions, such as obtaining a permit or buying one or more firearms. The dataset comes with an important disclaimer from the FBI that the data is for the number of background checks initiated, not the number of firearms sold (https://www.fbi.gov/file-repository/nics_firearm_checks_-_month_year_by_state_type.pdf/view):
It is important to note that the statistics within this chart represent the number of firearm background checks initiated through the NICS. They do not represent the number of firearms sold. Based on varying state laws and purchase scenarios, a one-to-one correlation cannot be made between a firearm background check and a firearm sale [emphasis added].
To supplement the data from NICS, I will also be exploring US Census data. This dataset contains census data for each state in 2010 and estimations for 2016, as well as various demographic information during that timespan. I plan to use the census data to calculate the number of NICS background checks conducted per capita for each state, then investigate the relationships between background checks per capita and several educational and economic factors.
Specifically, my variables and research questions are as follows:
Dependent Variables:
Independent Variables:
Research Questions:
This project was created with python 3.7.7 and makes use of the NumPy, Pandas, Matplotlib, and Seaborn libraries.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%pylab inline
%matplotlib inline
I will start by importing the raw CSV files using the pandas read_csv function. While inspecting the CSV files in excel prior to starting the project, I noticed the census data had several missing values indicated by letters under "Value Flags." I will replace these letters with NANs using the na_values parameter.
raw_nics_data = pd.read_csv('gun-data.csv')
raw_cen_data = pd.read_csv('u.s.-census-data.csv', na_values=['D', 'F', 'FN', 'S', 'X', 'Z'])
NANs are classified as floats in pandas, so I will convert the raw census dataframe to string data types for easier manipulation later.
raw_cen_data = raw_cen_data.applymap(str)
Next, I will check the raw NICS data for structure and cleanliness. I notice that the range of dates in 'month' is from November 1998 to September 2017. From previewing the census data in Excel, I know I am primarily interested in the years 2010 and 2016, so I will need to aggregate the NICS data by year later on.
raw_nics_data.head()
raw_nics_data.tail()
raw_nics_data.info()
I just need the columns for month, state, and totals, so I will make a new dataframe with these columns and convert the 'month' column to datetimes.
nics_data = raw_nics_data[["month", "state", "totals"]].copy()
nics_data["month"] = pd.to_datetime(nics_data["month"])
nics_data.info()
I'm primarily interested in NICS data from the years 2010 and 2016, so I will create two dataframes for totals from each year, then merge them together.
year_2010 = nics_data['month'].dt.year == 2010
nics_data_2010 = nics_data[year_2010]
nics_data_2010 = nics_data_2010.groupby('state').sum()
nics_data_2010.rename(columns={"totals": "2010 NICS totals"}, inplace=True)
nics_data_2010.head()
year_2016 = nics_data['month'].dt.year == int(2016)
nics_data_2016 = nics_data[year_2016]
nics_data_2016 = nics_data_2016.groupby('state').sum()
nics_data_2016.rename(columns={"totals": "2016 NICS totals"}, inplace=True)
nics_data_2016.head()
nics_data = nics_data_2010.merge(nics_data_2016, left_index=True, right_index=True)
nics_data.index.name = None
nics_data.head()
Later, I will merge this NICS data dataframe with a cleansed subset of the census data.
Now I will check the raw census data for structure and cleanliness. Right away, I notice the dataframe needs to be transposed to put the states as the row index in order to match the NICS data. I'll also set the column headers to the proper descriptions in the first row and drop the first two rows.
raw_cen_data.head()
trans_cen_data = raw_cen_data.transpose()
trans_cen_data.columns = trans_cen_data.iloc[0]
trans_cen_data.drop(['Fact', 'Fact Note'], inplace=True)
trans_cen_data.columns.name = None
trans_cen_data.head()
I will trim this dataframe down to only the columns I need, which include those related to population data for 2010 and 2016, as well as the education and economic related factors.
cen_data = trans_cen_data[["Population estimates, July 1, 2016, (V2016)",
"Population, Census, April 1, 2010",
"Land area in square miles, 2010",
"Population per square mile, 2010",
"High school graduate or higher, percent of persons age 25 years+, 2011-2015",
"Bachelor's degree or higher, percent of persons age 25 years+, 2011-2015",
"Median household income (in 2015 dollars), 2011-2015",
"Per capita income in past 12 months (in 2015 dollars), 2011-2015",
"Persons in poverty, percent"]].copy()
cen_data.info()
When I initially attempted to convert the datatypes from objects to floats, I discovered that some states (such as New Jersey) input whole percents while others (such as New Mexico) input decimals. I also need to strip the commas and dollar signs from several columns before doing the conversion.
cen_data.iloc[27:33, :]
I will use applymap to remove the percents, commas, and dollar signs and convert each value to a float.
def convert_to_float(cell):
cell = cell.replace(',', '')
cell = cell.replace('$', '')
if '%' in cell:
cell = round(float(cell.replace('%', ''))/100, 3)
cell = float(cell)
return cell
cen_data = cen_data.applymap(convert_to_float)
cen_data.iloc[27:33, :]
cen_data.info()
The census data includes data for 50 states, however, the NICS data has data from 50 states plus 5 additional territories. I will make a set of these additional territories and drop their corresponding rows from the NICS dataframe.
print ('Number of states:', len(nics_data.index.unique()))
nics_data.index.unique()
not_in_cen = set()
for i in nics_data.index:
if i not in cen_data.index:
not_in_cen.add(i)
not_in_cen
nics_data = nics_data[~nics_data.index.isin(not_in_cen)]
print ('Number of states:', len(nics_data.index.unique()), '\n')
print (nics_data.index.unique(), '\n')
Finally, I will merge the census and NICS data together, as well as add columns to calculate the totals per capita in each year.
merged_data = cen_data.merge(nics_data, left_index=True, right_index=True)
merged_data['2010 NICS per capita'] = round(merged_data['2010 NICS totals']/merged_data['Population, Census, April 1, 2010'], 3)
merged_data['2016 NICS per capita'] = round(merged_data['2016 NICS totals']/merged_data['Population estimates, July 1, 2016, (V2016)'], 3)
merged_data.info()
merged_data.head()
Here are my research questions:
I will use the raw NICS data to answer this question. The first line plot shows a steady increase in background checks from 1998 to 2017. Noticing the yearly cyclical pattern, I decided to create a second plot showing the mean number of background checks grouped by month. December is the busiest month of the year, March is the second busiest month of the year, and the summer months of May, June, and July are the least busy months of the year.
#Sets style and font sizes for plots
plt.style.use("seaborn")
SMALL_SIZE = 12
MEDIUM_SIZE = 14
BIG_SIZE = 18
plt.rc('font', size=SMALL_SIZE) # controls default text sizes
plt.rc('axes', titlesize=MEDIUM_SIZE) # fontsize of the axes title
plt.rc('axes', labelsize=MEDIUM_SIZE) # fontsize of the x and y labels
plt.rc('xtick', labelsize=SMALL_SIZE) # fontsize of the tick labels
plt.rc('ytick', labelsize=SMALL_SIZE) # fontsize of the tick labels
plt.rc('legend', fontsize=MEDIUM_SIZE) # legend fontsize
plt.rc('figure', titlesize=BIG_SIZE) # fontsize of the figure title
nics_x = sorted(raw_nics_data['month'].unique())
nics_y = raw_nics_data.groupby('month')['totals'].sum()
plt.figure("Total # of Background Checks by Year", figsize=(12, 4))
plt.suptitle("Total # of Background Checks Over Time")
plt.ticklabel_format(style='plain')
plt.plot(nics_x, nics_y)
plt.xticks(nics_x[2::12], rotation=45)
plt.xlabel('Year')
plt.ylabel('Total # of Background Checks')
plt.grid(True)
#convert datatype and extract month
import calendar
month_as_dtmonth = raw_nics_data.month.astype('datetime64').dt.month
nics_x = sorted(month_as_dtmonth.unique())
nics_y = raw_nics_data.groupby(month_as_dtmonth)['totals'].mean()
#convert to month names
month_names_x = []
for month in nics_x:
month_names_x.append(calendar.month_name[month])
#plot data
plt.figure(figsize=(10, 3.5))
plt.suptitle("Average # of Background Checks by Month")
plt.ticklabel_format(style='plain')
plt.plot(month_names_x, nics_y)
plt.xticks(month_names_x[::1], rotation=45)
plt.xlabel('Month')
plt.ylabel('Avg # of Background Checks')
plt.grid(True)
I will start by taking the average of the NICS per capita columns from 2010 and 2016, sort the series, and print the top and bottom 5 ranked states.
Note that Kentucky is significantly higher than the rest of the states because Kentucky is the only state that conducts monthly background checks on people with concealed-carry permits. This phenomenon will be explored further in question 3.
avg_bgc_per_capita = round((merged_data['2010 NICS per capita'] + merged_data['2016 NICS per capita'])/2, 3)
print('The top 5 states with the highest number of background checks per capita are:')
print (avg_bgc_per_capita.sort_values(ascending=False).head(),'\n')
print('The bottom 5 states with the lowest number of background checks per capita are:')
print(avg_bgc_per_capita.sort_values().head())
avg_bgc_per_capita.sort_values(ascending=False).plot.bar(figsize=(12, 3),
ylim=(0,1),
title='Background Checks per capita by State');
I will start by finding the difference in background checks per capita between 2010 and 2016 using the merged dataset. Then I will calculate the national average, the top 5 states with the greatest difference, and the bottom 5 states with the least difference.
difference_per_capita = merged_data['2016 NICS per capita'] - merged_data['2010 NICS per capita']
difference_per_capita.head()
national_avg = round(difference_per_capita.mean(), 3)
print ('The national average increase of background checks per capita for all states between 2010 and 2016 is '+str(national_avg)+'.')
print ('The top 10 states with the greatest increase are:\n')
print (difference_per_capita.sort_values(ascending=False).head())
print ('The bottom 10 states with the lowest increase are:')
print ('(Note that Utah is the only state to experience a decrease)\n')
print(difference_per_capita.sort_values().head())
The next plot compares the historical trend of total NICS checks over time for the top 3 states (Kentucky, Illinois, Indiana) and the only state to experience a decrease (Utah) compared with the national average.
#create new dataframes for the national average and individual states
raw_avg = raw_nics_data.groupby('month')['totals'].mean()
raw_kentucky = raw_nics_data[raw_nics_data.state=='Kentucky'].sort_values(by='month')
raw_indiana = raw_nics_data[raw_nics_data.state=='Indiana'].sort_values(by='month')
raw_illinois = raw_nics_data[raw_nics_data.state=='Illinois'].sort_values(by='month')
raw_utah = raw_nics_data[raw_nics_data.state=='Utah'].sort_values(by='month')
#create subplot variables for x and y axis
avg_x = raw_avg.index
avg_y = raw_avg
kentucky_x = raw_kentucky['month']
kentucky_y = raw_kentucky['totals']
illinois_x = raw_illinois['month']
illinois_y = raw_illinois['totals']
indiana_x = raw_indiana['month']
indiana_y = raw_indiana['totals']
utah_x = raw_utah['month']
utah_y = raw_utah['totals']
#plot data
plt.figure(figsize=(12, 5))
plt.suptitle("Total # of Background Checks Over Time for Specific States")
plt.ticklabel_format(style='plain')
plt.plot(avg_x, raw_avg, label='Average')
plt.plot(kentucky_x, kentucky_y, label='Kentucky', alpha=.8)
plt.plot(indiana_x, indiana_y, label='Indiana', alpha=.8)
plt.plot(illinois_x, illinois_y, label='Illinois', alpha=.8)
plt.plot(utah_x, utah_y, label='Utah', alpha=.8, color='gold')
plt.axvspan('2010-01', '2016-12', color='brown', alpha=0.2) #creates shaded background for 2010-01 to 2016-12
plt.legend(loc='best')
plt.xticks(avg_x[2::12], rotation=45)
plt.xlabel('Year')
plt.ylabel('Total # of Background Checks')
plt.grid(True)
Above, the shaded part represents the time period between January 2010 and December 2016.
Kentucky(green) saw a notable increase around 2006, which was when the state started requiring monthly background checks for concealed-carry permits according to Jacob Ryan(2015) from 89.3 WFPL (https://wfpl.org/kentucky-background-checks-stand-out/):
But outside of higher-than-average gun sales in the state, one reason Kentucky reports such a high number of checks annually — more than double the number of Texas and California combined so far this year — is a policy that requires automatic monthly background checks on every holder of concealed-carry permits in the commonwealth.
Kentucky appears to be the only state with such a policy.
Utah(yellow) saw a period of notable increase during 2010 and 2011 before returning to more stable levels. I was unable to find a reason for this spike in background checks, but its existence in the data explains why Utah experienced a decrease in background checks when comparing 2010 and 2016.
After considering this information, both Kentucky and Utah appear to be anomalies that don't represent the general behavior of other states.
To answer questions 4 and 5, I will first create a series of the average difference in background checks per capita between 2010 and 2016 to use as the dependent variable (This is the same calculation and variable used for question 2). I chose to take the average because several of the dependent variables from the census data are reported for the timespan of 2011-2015. However, due to the anomalies discovered in question 3, I will create a new dataframe of 48 states without Kentucky and Utah.
avg_bgc_per_capita = (merged_data['2016 NICS per capita'] + merged_data['2010 NICS per capita'])/2
avg_bgc_per_capita.sort_values(ascending=False).head()
#new dataframe without Kentucky and Utah
merged_data_48 = merged_data.drop(['Kentucky', 'Utah'])
#recalculate average difference in background checks per capita between 2010 and 2016
avg_bgc_per_capita = (merged_data_48['2016 NICS per capita'] + merged_data_48['2010 NICS per capita'])/2
avg_bgc_per_capita.sort_values(ascending=False).head()
Next, I will create a function to calculate the Pearson's r correlation coefficient and create a corresponding scatterplot for two variables. Then, I will call this function to compare each independent variable with the dependent variable of background checks per capita.
def pearsons_r(corr_x, corr_y, x_label, y_label):
output = round(numpy.corrcoef(corr_x, corr_y)[0,1], 3)
print("The Pearson's r correlation coefficient between\n{}\nand\n{}\nis:".format(x_label, y_label), output)
sns.regplot(corr_x,corr_y).set(xlabel=x_label,
ylabel=y_label);
I'm now ready to investigate the correlation between educational factors and background checks per capita.
First, I will look at the column for the percent of high school graduates or higher persons age 25 years and older between 2011-2015. The correlation coefficient is 0.168, meaning there's a very slight tendency for the higher percentage of high school graduates, the higher number of background checks per capita in that state. However, 0.168 is so small that the relationship is considered to be insignificant or non-existent.
high_school_x = merged_data_48["High school graduate or higher, percent of persons age 25 years+, 2011-2015"]
high_school_y = avg_bgc_per_capita
pearsons_r(high_school_x, high_school_y, "% of High School Graduate or Higher", "Background Checks per capita")
Next, I will look at the column for the percent of Bachelor's degree or higher for persons age 25 years and older between 2011-2015. The correlation coefficient is -0.416, meaning there's a moderate tendency of the higher percent of four-year college graduates, the lower number of background checks per capita in that state. -0.416 is considered to be a relationship of moderate strength and certainly more significant than the % of high school graduates.
bachelor_x = merged_data_48["Bachelor's degree or higher, percent of persons age 25 years+, 2011-2015"]
bachelor_y = avg_bgc_per_capita
pearsons_r(bachelor_x, bachelor_y, "% of Bachelor's Degree or Higher", "Background Checks per capita")
I will continue the same process for analyzing economic factors as I've done with educational factors.
First, I will look at the column for median household income between 2011 and 2015. The correlation coefficient is -0.350, meaning there's a moderate tendency of the higher median household income, the lower number of background checks per capita in that state. -0.350 is considered to be a relationship of moderate strength, though slightly weaker than seen for the percentage of four-year college graduates in a state.
median_income_x = merged_data_48["Median household income (in 2015 dollars), 2011-2015"]
median_income_y = avg_bgc_per_capita
pearsons_r(median_income_x, median_income_y, "Median Household Income", "Background Checks per capita")
Next, I will look at the column for per capita income in the past 12 months between 2011 and 2015. The correlation coefficient is -0.329, meaning there's a moderate tendency of the higher per capita income, the lower number of background checks per capita in that state. It's not surprising to see a similar strength relationship between median household income(-0.350) and per capita income(-0.329).
per_capita_income_x = merged_data_48["Per capita income in past 12 months (in 2015 dollars), 2011-2015"]
per_capita_income_y = avg_bgc_per_capita
pearsons_r(per_capita_income_x, per_capita_income_y, "Per capita Income", "Background Checks per capita")
Finally, I will look at the column for percent of persons in poverty. The correlation coefficient is only -0.140, meaning there's a very slight tendency of the more people in poverty, the more background checks per capita in that state. -0.140 is the weakest relationship of those I investigated and is considered to be insignificant or non-existent.
poverty_x = merged_data_48["Persons in poverty, percent"]
poverty_y = avg_bgc_per_capita
pearsons_r(poverty_x, poverty_y, "% of Persons in Poverty", "Background Checks per capita")
Combining the FBI's NICS data with the US Census data allowed me to not only analyze the total number of background checks conducted, but also the number of background checks conducted per capita for each state. Furthermore, I was able to explore correlations between background checks per capita and several demographic factors, specifically statistics on each state population's educational and economic situation.
One limitation of this investigation was that different states have different laws and requirements for conducting background checks and gun sales. This was particularly evident in the case of Kentucky, where checks are conducted monthly for people with concealed-carry permits. Furthermore, background checks are conducted for a variety of reasons and purchasing situations, not just a single gun sale, so a one-to-one relationship between NICS background checks and firearm sales cannot be made.
Nevertheless, some interesting trends and did emerge from this investigation. First, the data showed a clear upward trend in the total number of background checks being conducted throughout the US from 1998 to 2017. During that timespan, the busiest month for background checks was December, presumably due to the annual shopping spree leading into the winter holidays. Furthermore, when comparing the average number of background checks conducted per capita in 2010 and 2016, all states (except for the anomaly of Utah) experienced an increase as well.
The educational factor with the strongest relationship to background checks per capita was the percentage of four-year college graduates in a state. The Pearson's r correlation coefficient was -0.416, moderately indicating that states with higher percentages of college graduates tend to conduct a lower number of NICS background checks per capita.
The economic factor with the strongest relationship to background checks per capita was the median household income in a state. The Pearson's r correlation coefficient was -0.350, moderately indicating that states with higher median household incomes tend to conduct a lower number of NICS background checks per capita.