I was inspired by this thread (https://twitter.com/jon_mellon/status/899663861976113152) - expanded on in this paper (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3022169) - to look at the ways in which Authoritarianism isn't so much a political value itself but a reflection of psychological differences which mediate the connection between an individual's personal Circumstances and their Attitudes to related issues.

I've constructed a series of charts each looking at the interaction between the Circumstances of the respondents (e.g. Gross Household Income) and some related political Attitudes (e.g. Redistribution) - plotting Circumstances on the x-axis and the respective (weighted, error-barred) means of those Attitudes on the y-axis.

Strictly speaking, each of those charts is 5 charts, each built using the fifths of the British Election Study (Wave 17) dataset who answered both questions.

Those fifths reflect the population of the dataset ordered by where they are on the Social Liberal - Authoritarian axis[1].

The interactive slider lets you see the how the chart changes for different fifths. That's it - no other controls.

Where the x-axis variables are ordered, I've added a regression line/slope-label - just to help the eye.

[1] using a homebrew mix incorporating all al variables across the dataset, hence why I have even fifths

In [74]:

%%time
dataset_name = "W19_comb"
df_list = [ "BES_Panel" ]

%matplotlib inline 
%run BES_header.py {dataset_name} {df_list}

if "id" in BES_Panel.columns:
    BES_Panel = BES_Panel.set_index("id").sort_index()

(var_type, cat_dictionary, new_old_col_names, old_new_col_names) = get_small_files(data_subfolder, encoding)    
    
# get full set of inferred "cross wave" auth-lib/left-right values and ages
pan_dataset_allr_values = pd.read_csv(BES_small_data_files + "pan_dataset_allr_valuesW19"+".csv")
pan_dataset_ages = pd.read_pickle(BES_small_data_files + "pan_dataset_ages"+".zip", compression='zip')

var_type (7911, 14)
Wall time: 28.1 s

In [2]:

BES_Panel = pd.read_pickle("..\\BES_analysis_data\\"+"W19_comb"+os.sep+"BES_Panelv02",compression='zip')
BES_Panel = BES_Panel.set_index("id")
BES_Panel = BES_Panel.sort_index()

pan_dataset_allr_values = pan_dataset_allr_values.set_index("id").loc[BES_Panel.index]

In [3]:

def weighted_mean(x):
    if x.empty:
        return np.nan    
    val, weight = map(np.asarray, zip(*x))
    val, weight = val[~np.isnan(val)],weight[~np.isnan(val)]
    return (val * weight).sum() / weight.sum()

import numba

def weighted_mean_bstrap(x,size):
    if x.empty:
        return np.nan
    val, weight = map(np.asarray, zip(*x))
    return wt_mean(val,weight,size)

@numba.jit
def wt_mean(val, weight,size):
    val, weight = val[~np.isnan(val)],weight[~np.isnan(val)]

    bs_reps = np.empty(size)

    # Draw replicates
    n = len(val)
    for i in range(size):
        ind = np.random.choice(range(0,n), size=n)
        bs_reps[i] = (val[ind] * weight[ind]).sum() / weight[ind].sum()
        
    return bs_reps

def wt_err(x,size,conf_int = [2.5,97.5]):
    lower,upper = np.percentile( weighted_mean_bstrap(x,size) , conf_int)
    return (lower,upper)

In [4]:

import numpy as np
import holoviews as hv
hv.extension('matplotlib')

In [5]:

lr_col = pan_dataset_allr_values.columns[2]
al_col = pan_dataset_allr_values.columns[3]

from holoviews import dim
from scipy import stats

In [44]:

def hol_plot_var_by_lr_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var, al_var_name="al",
                             zlabel=None,q=5,x_cat_list=None,non_ord_indices=[],reg_plot=True,title=None):
    if title is None:
        title = '{label} {group}\n{dimensions}'
    else:
        title = title+"\n"+'{label} {group}\n{dimensions}'
    
    if zlabel is None:
        zlabel = "Social Liberal (0) - Authoritarian ("+str(q-1)+")"

    df = BES_Panel[[x_var_name,wt_var]]
    if x_cat_list is None:
        x_cat_list = BES_Panel[x_var_name].cat.categories
    
    hv_dict = {}

    mask = BES_Panel[wt_var].notnull() & BES_Panel[x_var_name].notnull() & pan_dataset_allr_values[[lr_col,al_col]].notnull().all(axis=1)
    lr = weighted_qcut(pan_dataset_allr_values[lr_col][mask],BES_Panel[wt_var][mask],q )
    al = weighted_qcut(pan_dataset_allr_values[al_col][mask],BES_Panel[wt_var][mask],q )

    df = df[mask]
    df[al_var_name] = al
    df[y_var_name] = lr

    df[y_var_name] = df[y_var_name].replace("Don't know",np.nan).cat.codes.replace(-1,np.nan).astype('float32')
    max_y_value=df[y_var_name].max()
    df[al_var_name] = df[al_var_name].cat.codes.replace(-1,np.nan).astype('float32')
    df[y_var_name+"_wts"] = list(zip(df[y_var_name],df[wt_var]))


    for al_var in range(0,q):
        mask = df[al_var_name]==al_var
        means = df[mask].groupby(x_var_name)[y_var_name+"_wts"].agg(weighted_mean)
        wt_mean_errors = df[mask].groupby(x_var_name)[y_var_name+"_wts"].agg(wt_err, size=1000)
        lower = wt_mean_errors.apply(lambda x: x[0])
        upper = wt_mean_errors.apply(lambda x: x[1])
        count = df[mask].groupby(x_var_name)[y_var_name+"_wts"].count()

        chart_df = pd.DataFrame()
        chart_df["means"]=means
        chart_df["lower_error"]=lower
        chart_df["upper_error"]=upper
        chart_df["N"]=count
    #     chart_df

        chart_df = chart_df.reset_index()
        chart_df[x_var_name] = chart_df[x_var_name].cat.codes
        chart_df[["lower_error","upper_error"]] = chart_df[["lower_error","upper_error"]].apply(lambda x: x-chart_df["means"])
        chart_df["lower_error"] = chart_df["lower_error"].abs()

        chart = hv.Curve(chart_df) * hv.ErrorBars(chart_df, vdims=['means', 'lower_error', 'upper_error'])


        non_ord_mask = chart_df[x_var_name].apply(lambda x: x not in non_ord_indices)
        xs = chart_df[x_var_name][non_ord_mask]
        ys = chart_df["means"][non_ord_mask]

        slope, intercep, rval, pval, std = stats.linregress(xs, ys)
        # rho, pval = stats.spearmanr(xs,ys)
    #     xs = np.linspace(0,14,2)
        if reg_plot:
            reg = slope*xs+intercep
            chart = chart * hv.Curve((xs, reg)).relabel('r2: %.3f' % (slope) )    
        chart.opts(ylim=(0,max_y_value),xlim=(-.1,len(x_cat_list)-.9),
                  xticks= [(x,x_cat_list[x]) for x in range(0,len(x_cat_list))],
                  fig_size=300,aspect=2,    
                  xlabel =xlabel, ylabel=ylabel,
                  title = title)
    #xrotation=45,
        hv_dict[al_var] = chart
    return hv_dict

In [48]:

import textwrap

def hol_plot_var1_by_var2_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var, al_var_name="al",
                             zlabel=None,q=5,x_cat_list=None,non_ord_indices=[],reg_plot=True,
                             title=None,x_cat_text_width=13):
    
    if title is None:
        title = '{label} {group}\n{dimensions}'
    else:
        title = title+"\n"+'{label} {group}\n{dimensions}'
    
    if zlabel is None:
        zlabel = "Social Liberal (0) - Authoritarian ("+str(q-1)+")"

    df = BES_Panel[[x_var_name,wt_var]]
    if x_cat_list is None:
        x_cat_list = BES_Panel[x_var_name].cat.categories

        wrapper = textwrap.TextWrapper(width=x_cat_text_width) 
        x_cat_list = [wrapper.fill(text=x)  for x in x_cat_list]          
    
    hv_dict = {}

    mask =  BES_Panel[wt_var].notnull() &\
            BES_Panel[x_var_name].notnull() &\
            pan_dataset_allr_values[al_col].notnull()&\
            BES_Panel[y_var_name].replace("Don't know",np.nan).notnull()

    # lr = weighted_qcut(pan_dataset_allr_values[lr_col][mask],BES_Panel[wt_var][mask],q )
    al = weighted_qcut(pan_dataset_allr_values[al_col][mask],BES_Panel[wt_var][mask],q )

    df = BES_Panel[[x_var_name,wt_var]][mask]
    df[al_var_name] = al
    df[y_var_name] = BES_Panel[y_var_name]

    df[y_var_name] = df[y_var_name].replace("Don't know",np.nan).cat.codes.replace(-1,np.nan).astype('float32')
    max_y_value=df[y_var_name].max()
    df[al_var_name] = df[al_var_name].cat.codes.replace(-1,np.nan).astype('float32')
    df[y_var_name+"_wts"] = list(zip(df[y_var_name],df[wt_var]))


    for al_var in range(0,q):
        mask = df[al_var_name]==al_var
        means = df[mask].groupby(x_var_name)[y_var_name+"_wts"].agg(weighted_mean)
        wt_mean_errors = df[mask].groupby(x_var_name)[y_var_name+"_wts"].agg(wt_err, size=1000)
        lower = wt_mean_errors.apply(lambda x: x[0])
        upper = wt_mean_errors.apply(lambda x: x[1])
        count = df[mask].groupby(x_var_name)[y_var_name+"_wts"].count()

        chart_df = pd.DataFrame()
        chart_df["means"]=means
        chart_df["lower_error"]=lower
        chart_df["upper_error"]=upper
        chart_df["N"]=count
    #     chart_df

        chart_df = chart_df.reset_index()
        chart_df[x_var_name] = chart_df[x_var_name].cat.codes
        chart_df[["lower_error","upper_error"]] = chart_df[["lower_error","upper_error"]].apply(lambda x: x-chart_df["means"])
        chart_df["lower_error"] = chart_df["lower_error"].abs()

        chart = hv.Curve(chart_df) * hv.ErrorBars(chart_df, vdims=['means', 'lower_error', 'upper_error'])


        non_ord_mask = chart_df[x_var_name].apply(lambda x: x not in non_ord_indices)
        xs = chart_df[x_var_name][non_ord_mask]
        ys = chart_df["means"][non_ord_mask]

        slope, intercep, rval, pval, std = stats.linregress(xs, ys)
        # rho, pval = stats.spearmanr(xs,ys)
    #     xs = np.linspace(0,14,2)
        if reg_plot:
            reg = slope*xs+intercep
            chart = chart * hv.Curve((xs, reg)).relabel('r2: %.3f' % (slope) )    
        chart.opts(ylim=(0,max_y_value),xlim=(-.1,len(x_cat_list)-.9),
                  xticks= [(x,x_cat_list[x]) for x in range(0,len(x_cat_list))],
                  fig_size=300,aspect=2,    
                  xlabel =xlabel, ylabel=ylabel,
                  title = title)
    #xrotation=45,
        hv_dict[al_var] = chart
    return hv_dict

In [53]:

y_var_name = "lr"

title = "\n".join(["Economic Left-Right Value Position",
                   "(BES Wave 17, September 2019, fully weighted)"])
ylabel = "Economic Left (0) - Economic Right ("+str(q-1)+")"
wt_var = "wt_new_W17"

x_var_name = "p_gross_householdW17"
xlabel = "Gross Household Income"

x_cat_list = ["-£5k","£5-\n10k","£10-\n15k","£15-\n20k","£20-\n25k","£25-\n30k","£30-\n35k","£35-\n40k","£40-\n45k","£45-\n50k",
"£50-\n60k","£60-\n70k","£70-\n100k","£100-\n150k","£150k+","Don't\nknow","Pref not\nto answer"]
non_ord_indices = [15,16]

hv_dict = hol_plot_var_by_lr_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var,
                                   q=5,x_cat_list=x_cat_list,non_ord_indices=non_ord_indices,title=title)

hv.output(backend='matplotlib',fig='png', holomap='widgets')    
hm = hv.HoloMap(hv_dict, kdims=[zlabel])
hm

Out[53]:

In [60]:

%%time
          
y_var_name = "lr1W17"
title = "\n".join(["Government should redistribute income from the better off to those who are less well off",
                   "(BES Wave 17, September 2019, weighted)"])
ylabel = "Strongly Disagree (0) - Strongly Agree (4)"

wt_var = "wt_new_W17"

x_var_name = "p_gross_householdW17"
xlabel = "Gross Household Income"

x_cat_list = ["-£5k","£5-\n10k","£10-\n15k","£15-\n20k","£20-\n25k","£25-\n30k","£30-\n35k","£35-\n40k","£40-\n45k","£45-\n50k",
"£50-\n60k","£60-\n70k","£70-\n100k","£100-\n150k","£150k+","Don't\nknow","Pref not\nto answer"]
non_ord_indices = [15,16]

hv_dict = hol_plot_var1_by_var2_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var,
                                   q=5,x_cat_list=x_cat_list,non_ord_indices=non_ord_indices,title=title)

hv.output(backend='matplotlib',fig='png', holomap='widgets')    
hm = hv.HoloMap(hv_dict, kdims=[zlabel])
hm

Wall time: 12.8 s

Out[60]:

In [68]:

# hv.output(hm, holomap='gif', fps=1)
# hv.save(hm, '..//BES_analysis_publications//Authoritarianism as Mediator//'+'Income_by_Redist_by_Auth.gif', fps=1)

In [55]:

y_var_name = "lr1W17"
title = "\n".join(["Government should redistribute income from the better off to those who are less well off",
                   "(BES Wave 17, September 2019, weighted)"])

ylabel = "Strongly Disagree (0) - Strongly Agree (4)"
    
wt_var = "wt_new_W17"

x_var_name = "p_socgradeW17"
xlabel = "NRS Social Grade"
non_ord_indices = [6]

hv_dict = hol_plot_var1_by_var2_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var,
                                      non_ord_indices=non_ord_indices,title=title)

hv.output(backend='matplotlib',fig='png', holomap='widgets')    
hm = hv.HoloMap(hv_dict, kdims=[zlabel])
hm

Out[55]:

In [56]:

y_var_name = "lr1W17"
title = "\n".join(["Government should redistribute income from the better off to those who are less well off",
                   "(BES Wave 17, September 2019, weighted)"])

ylabel = "Strongly Disagree (0) - Strongly Agree (4)"
    
wt_var = "wt_new_W17"

x_var_name = "p_edlevelW17"
xlabel = "Educational Level"
non_ord_indices = []

hv_dict = hol_plot_var1_by_var2_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var,
                                      non_ord_indices=non_ord_indices,title=title,
                                      x_cat_text_width=16)

hv.output(backend='matplotlib',fig='png', holomap='widgets')    
hm = hv.HoloMap(hv_dict, kdims=[zlabel])
hm

Out[56]:

In [57]:

y_var_name = "lr1W17"
# title ="Government should redistribute income from the better off to those who are less well off\n"
title = "\n".join(["Government should redistribute income from the better off to those who are less well off",
                   "(BES Wave 17, September 2019, weighted)"])

ylabel = "Strongly Disagree (0) - Strongly Agree (4)"
    
wt_var = "wt_new_W17"

x_var_name = "ns_sec_analyticW16W17W18"
xlabel = "Goldthorpe Class Schema"
non_ord_indices = []

hv_dict = hol_plot_var1_by_var2_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var,
                                      non_ord_indices=non_ord_indices,title=title)

hv.output(backend='matplotlib',fig='png', holomap='widgets')    
hm = hv.HoloMap(hv_dict, kdims=[zlabel])
hm

Out[57]:

In [58]:

y_var_name = "lr1W17"
title = "\n".join(["How happy or sad would you be if Scotland left the United Kingdom?",
                   "(BES Wave 17, September 2019, weighted)"])

y_var_name = "happyScotIndepResultW17"
ylabel = "\n".join(["Extremely Sad (0) - Extremely Happy (10)"])
    
wt_var = "wt_new_W17"

x_var_name = "gorW17"
xlabel = "British Regions"
non_ord_indices = []

hv_dict = hol_plot_var1_by_var2_by_al(x_var_name,xlabel,y_var_name,ylabel,wt_var,
                                      non_ord_indices=non_ord_indices,title=title,
                                      reg_plot=False,x_cat_text_width=10)

hv.output(backend='matplotlib',fig='png', holomap='widgets')    
hm = hv.HoloMap(hv_dict, kdims=[zlabel])
hm

Out[58]:

In [ ]:

How Authoritarianism Mediates Link Between Circumstances and Attitudes

Comments