Reducing a linear regression (OLS) model by dropping non-significant coefficients

Question

Would it be proper for me to reduce a model by iterating though the coefficients and dropping the ones with high p-values and then refitting and doing this again until all coefficients are significant? The algorithm I have in mind is (starting with the full model):

1. fit regression on current set of regressors
2. find coefficient with highest p-value
3. drop regressor variable from model
4. go to step 1

I select the "highest p-value" because the null hypothesis in my software package (python statsmodels) is that the coefficient is 0 so we only keep ones with low p-values. I can potentially then perform a stepAIC after running the above step. For example, here is a model I ran just now which I'd like to reduce.

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                  c0000   R-squared:                       0.183
Model:                            OLS   Adj. R-squared:                  0.105
Method:                 Least Squares   F-statistic:                     2.337
Date:                Fri, 14 May 2021   Prob (F-statistic):           3.53e-28
Time:                        13:14:07   Log-Likelihood:                 23470.
No. Observations:                3285   AIC:                        -4.636e+04
Df Residuals:                    2997   BIC:                        -4.461e+04
Df Model:                         287                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
Intercept  -1.323e-05   3.86e-06     -3.428      0.001   -2.08e-05   -5.66e-06
c1620         -0.0211      0.014     -1.541      0.123      -0.048       0.006
c1655          0.0021      0.015      0.137      0.891      -0.027       0.032
c0705          0.0300      0.010      2.890      0.004       0.010       0.050
c1210          0.0095      0.014      0.671      0.503      -0.018       0.037
c1900         -0.0082      0.013     -0.634      0.526      -0.034       0.017
c0905          0.0034      0.012      0.280      0.780      -0.021       0.027
c0650          0.0064      0.015      0.424      0.672      -0.023       0.036
c1130          0.0368      0.015      2.400      0.016       0.007       0.067
c1015          0.0477      0.014      3.426      0.001       0.020       0.075
c0300          0.0313      0.023      1.346      0.178      -0.014       0.077
c1535          0.0084      0.013      0.671      0.502      -0.016       0.033
c1650          0.0301      0.016      1.862      0.063      -0.002       0.062
c0235         -0.0446      0.024     -1.848      0.065      -0.092       0.003
c0545          0.0030      0.025      0.120      0.904      -0.046       0.052
c1515          0.0264      0.011      2.331      0.020       0.004       0.049
c2240         -0.0799      0.029     -2.763      0.006      -0.137      -0.023
c1255         -0.0166      0.010     -1.612      0.107      -0.037       0.004
c1820          0.0142      0.017      0.843      0.399      -0.019       0.047
c1215          0.0297      0.012      2.573      0.010       0.007       0.052
c2340         -0.0208      0.017     -1.200      0.230      -0.055       0.013
c2155         -0.0734      0.029     -2.491      0.013      -0.131      -0.016
c2145          0.0401      0.033      1.224      0.221      -0.024       0.104
c0920          0.0165      0.014      1.165      0.244      -0.011       0.044
c0215          0.0178      0.024      0.749      0.454      -0.029       0.064
c1855         -0.0314      0.018     -1.736      0.083      -0.067       0.004
c0535          0.0373      0.025      1.502      0.133      -0.011       0.086
c1310          0.0127      0.011      1.160      0.246      -0.009       0.034
c1420         -0.0342      0.010     -3.307      0.001      -0.054      -0.014
c1705         -0.0188      0.014     -1.318      0.188      -0.047       0.009
c1005         -0.0167      0.014     -1.212      0.225      -0.044       0.010
c0810          0.0198      0.011      1.760      0.079      -0.002       0.042
c0620          0.0100      0.018      0.568      0.570      -0.024       0.044
c0255         -0.0397      0.025     -1.617      0.106      -0.088       0.008
c2320          0.0530      0.027      1.959      0.050   -4.49e-05       0.106
c2200         -0.0014      0.009     -0.159      0.874      -0.019       0.016
c0520          0.0447      0.026      1.696      0.090      -0.007       0.096
c0835          0.0020      0.012      0.170      0.865      -0.021       0.025
c0155         -0.0242      0.023     -1.055      0.292      -0.069       0.021
c1000          0.0033      0.012      0.270      0.787      -0.021       0.027
c2040         -0.0329      0.026     -1.270      0.204      -0.084       0.018
c0530         -0.0024      0.026     -0.094      0.925      -0.053       0.049
c0250         -0.0072      0.024     -0.296      0.767      -0.054       0.040
c1240         -0.0170      0.008     -2.055      0.040      -0.033      -0.001
c1430         -0.0120      0.011     -1.102      0.270      -0.033       0.009
c0610          0.0213      0.015      1.373      0.170      -0.009       0.052
c1050          0.0436      0.016      2.762      0.006       0.013       0.075
c1320         -0.0146      0.012     -1.255      0.210      -0.037       0.008
c1710         -0.0257      0.016     -1.600      0.110      -0.057       0.006
c0855          0.0050      0.014      0.354      0.723      -0.023       0.033
c0130         -0.0459      0.022     -2.123      0.034      -0.088      -0.004
c0350          0.0087      0.028      0.316      0.752      -0.045       0.063
c0210          0.0229      0.025      0.909      0.364      -0.026       0.072
c1020         -0.0027      0.015     -0.184      0.854      -0.032       0.026
c0625         -0.0370      0.017     -2.219      0.027      -0.070      -0.004
c1915         -0.0137      0.019     -0.729      0.466      -0.051       0.023
c0600          0.0142      0.017      0.832      0.406      -0.019       0.048
c2135         -0.0091      0.031     -0.292      0.770      -0.070       0.052
c0305         -0.0187      0.024     -0.789      0.430      -0.065       0.028
c1330          0.0188      0.008      2.503      0.012       0.004       0.034
c1545          0.0085      0.012      0.692      0.489      -0.016       0.033
c2330          0.0163      0.028      0.582      0.560      -0.038       0.071
c1530         -0.0189      0.012     -1.603      0.109      -0.042       0.004
c1640         -0.0386      0.015     -2.512      0.012      -0.069      -0.008
c1450         -0.0291      0.011     -2.697      0.007      -0.050      -0.008
c0200         -0.0389      0.022     -1.795      0.073      -0.081       0.004
c0310          0.0906      0.026      3.464      0.001       0.039       0.142
c1815          0.0201      0.015      1.367      0.172      -0.009       0.049
c1315          0.0050      0.010      0.478      0.633      -0.015       0.025
c0120         -0.0145      0.021     -0.700      0.484      -0.055       0.026
c0605          0.0068      0.015      0.455      0.649      -0.022       0.036
c1630          0.0136      0.014      0.953      0.341      -0.014       0.042
c0040          0.0090      0.023      0.387      0.699      -0.037       0.055
c0435         -0.0921      0.029     -3.172      0.002      -0.149      -0.035
c0755         -0.0116      0.013     -0.886      0.376      -0.037       0.014
c0555          0.0050      0.023      0.221      0.825      -0.039       0.049
c2125          0.0148      0.031      0.483      0.629      -0.045       0.075
c0345         -0.0166      0.030     -0.551      0.581      -0.075       0.042
c0800          0.0244      0.011      2.208      0.027       0.003       0.046
c1610          0.0076      0.013      0.592      0.554      -0.018       0.033
c0455          0.0385      0.032      1.211      0.226      -0.024       0.101
c0410         -0.0141      0.024     -0.583      0.560      -0.062       0.033
c1910          0.0206      0.016      1.255      0.210      -0.012       0.053
c0025         -0.0155      0.022     -0.705      0.481      -0.059       0.028
c0735          0.0283      0.013      2.240      0.025       0.004       0.053
c0510          0.0655      0.026      2.545      0.011       0.015       0.116
c0335         -0.0153      0.027     -0.559      0.576      -0.069       0.038
c0240         -0.0022      0.025     -0.088      0.930      -0.051       0.046
c1200         -0.0295      0.013     -2.328      0.020      -0.054      -0.005
c1335          0.0090      0.007      1.274      0.203      -0.005       0.023
c0840         -0.0142      0.013     -1.108      0.268      -0.039       0.011
c0645         -0.0089      0.017     -0.531      0.595      -0.042       0.024
c0330          0.0298      0.030      0.993      0.321      -0.029       0.089
c1925          0.0145      0.019      0.746      0.456      -0.024       0.052
c1930         -0.0003      0.020     -0.017      0.986      -0.039       0.038
c0830         -0.0073      0.012     -0.604      0.546      -0.031       0.016
c0205          0.0291      0.023      1.266      0.206      -0.016       0.074
c1600         -0.0064      0.010     -0.649      0.517      -0.026       0.013
c0740          0.0228      0.013      1.768      0.077      -0.002       0.048
c1250          0.0183      0.010      1.910      0.056      -0.000       0.037
c2100      -4.595e-05      0.007     -0.007      0.994      -0.013       0.013
c1030          0.0035      0.015      0.228      0.819      -0.026       0.033
c1730         -0.0291      0.017     -1.716      0.086      -0.062       0.004
c2255          0.0331      0.027      1.204      0.229      -0.021       0.087
c2105          0.0025      0.009      0.278      0.781      -0.015       0.020
c1225          0.0117      0.014      0.826      0.409      -0.016       0.040
c1145         -0.0005      0.013     -0.039      0.969      -0.026       0.025
c1745         -0.0355      0.017     -2.116      0.034      -0.068      -0.003
c1750          0.0074      0.017      0.424      0.672      -0.027       0.042
c1055          0.0154      0.016      0.965      0.335      -0.016       0.047
c2220         -0.0140      0.030     -0.469      0.639      -0.073       0.045
c1635         -0.0013      0.015     -0.085      0.933      -0.031       0.028
c1135         -0.0317      0.015     -2.093      0.036      -0.061      -0.002
c1440         -0.0157      0.011     -1.408      0.159      -0.038       0.006
c0640         -0.0008      0.016     -0.052      0.959      -0.033       0.031
c2315         -0.0387      0.027     -1.432      0.152      -0.092       0.014
c1345          0.0115      0.009      1.290      0.197      -0.006       0.029
c0825          0.0360      0.013      2.762      0.006       0.010       0.062
c0420         -0.0588      0.033     -1.792      0.073      -0.123       0.006
c1615         -0.0406      0.014     -2.948      0.003      -0.068      -0.014
c2115          0.0190      0.028      0.686      0.493      -0.035       0.073
c2335          0.0278      0.027      1.034      0.301      -0.025       0.080
c0100          0.0343      0.022      1.560      0.119      -0.009       0.077
c1700          0.0195      0.014      1.376      0.169      -0.008       0.047
c0750          0.0329      0.014      2.426      0.015       0.006       0.060
c0925         -0.0223      0.014     -1.541      0.123      -0.051       0.006
c1555         -0.0116      0.013     -0.916      0.360      -0.036       0.013
c1040          0.0288      0.015      1.885      0.060      -0.001       0.059
c1455         -0.0111      0.011     -1.053      0.292      -0.032       0.010
c0930         -0.0047      0.014     -0.334      0.738      -0.032       0.023
c1120          0.0065      0.014      0.448      0.654      -0.022       0.035
c2015         -0.0532      0.024     -2.177      0.030      -0.101      -0.005
c1150          0.0031      0.012      0.267      0.789      -0.020       0.026
c2110          0.0138      0.026      0.543      0.588      -0.036       0.064
c2010          0.0134      0.023      0.580      0.562      -0.032       0.059
c0245         -0.0055      0.026     -0.217      0.828      -0.056       0.044
c0230          0.0355      0.025      1.407      0.160      -0.014       0.085
c0700          0.0058      0.012      0.492      0.623      -0.017       0.029
c0505          0.0117      0.025      0.459      0.646      -0.038       0.062
c1010         -0.0173      0.015     -1.181      0.238      -0.046       0.011
c1845          0.0503      0.020      2.579      0.010       0.012       0.089
c1940          0.0031      0.020      0.152      0.880      -0.037       0.043
c1725          0.0270      0.016      1.639      0.101      -0.005       0.059
c1140         -0.0145      0.015     -0.964      0.335      -0.044       0.015
c2225          0.0234      0.028      0.838      0.402      -0.031       0.078
c1245          0.0043      0.009      0.504      0.615      -0.012       0.021
c1850         -0.0157      0.020     -0.796      0.426      -0.054       0.023
c0745          0.0273      0.013      2.117      0.034       0.002       0.053
c1950          0.0076      0.021      0.358      0.720      -0.034       0.049
c1445         -0.0271      0.011     -2.570      0.010      -0.048      -0.006
c0050          0.0164      0.023      0.709      0.478      -0.029       0.062
c2350         -0.0219      0.026     -0.858      0.391      -0.072       0.028
c0030          0.0427      0.024      1.781      0.075      -0.004       0.090
c0440          0.0468      0.030      1.557      0.120      -0.012       0.106
c0710         -0.0056      0.011     -0.492      0.622      -0.028       0.017
c2230         -0.0200      0.031     -0.652      0.515      -0.080       0.040
c2345          0.0007      0.024      0.030      0.976      -0.047       0.049
c0515          0.0410      0.028      1.461      0.144      -0.014       0.096
c2310         -0.0475      0.030     -1.590      0.112      -0.106       0.011
c1840          0.0240      0.016      1.465      0.143      -0.008       0.056
c0935          0.0352      0.013      2.806      0.005       0.011       0.060
c0425         -0.0794      0.033     -2.395      0.017      -0.144      -0.014
c0320         -0.0263      0.028     -0.943      0.346      -0.081       0.028
c0055         -0.0163      0.019     -0.858      0.391      -0.053       0.021
c2250         -0.0565      0.029     -1.975      0.048      -0.113      -0.000
c0945          0.0030      0.014      0.209      0.835      -0.025       0.031
c0525         -0.0607      0.028     -2.184      0.029      -0.115      -0.006
c0140         -0.0044      0.022     -0.206      0.837      -0.047       0.038
c1715         -0.0266      0.015     -1.735      0.083      -0.057       0.003
c2005         -0.0076      0.017     -0.447      0.655      -0.041       0.026
c1415          0.0072      0.010      0.732      0.464      -0.012       0.027
c1735         -0.0173      0.015     -1.185      0.236      -0.046       0.011
c2025          0.0092      0.026      0.355      0.723      -0.042       0.060
c1830          0.0310      0.018      1.713      0.087      -0.004       0.066
c2235          0.0037      0.030      0.126      0.900      -0.054       0.062
c0010         -0.0042      0.019     -0.219      0.827      -0.042       0.034
c0635         -0.0338      0.016     -2.105      0.035      -0.065      -0.002
c1520         -0.0172      0.012     -1.391      0.164      -0.041       0.007
c1350         -0.0142      0.010     -1.464      0.143      -0.033       0.005
c0035         -0.0155      0.021     -0.727      0.467      -0.057       0.026
c0445          0.0249      0.032      0.777      0.437      -0.038       0.088
c0225         -0.0239      0.024     -0.995      0.320      -0.071       0.023
c0910         -0.0190      0.013     -1.442      0.149      -0.045       0.007
c1755          0.0225      0.018      1.260      0.208      -0.013       0.058
c1435         -0.0201      0.010     -1.930      0.054      -0.041       0.000
c0340         -0.0885      0.026     -3.371      0.001      -0.140      -0.037
c0655          0.0246      0.015      1.648      0.099      -0.005       0.054
c0315         -0.0327      0.029     -1.116      0.265      -0.090       0.025
c0815         -0.0066      0.012     -0.531      0.596      -0.031       0.018
c1100          0.0208      0.014      1.464      0.143      -0.007       0.049
c0955         -0.0159      0.015     -1.050      0.294      -0.045       0.014
c2035          0.0469      0.026      1.825      0.068      -0.003       0.097
c0500          0.0040      0.027      0.149      0.881      -0.048       0.056
c0115         -0.0138      0.021     -0.653      0.514      -0.055       0.028
c1550          0.0007      0.013      0.056      0.955      -0.025       0.027
c1115         -0.0108      0.015     -0.717      0.473      -0.040       0.019
c1305          0.0070      0.010      0.706      0.480      -0.012       0.026
c0015          0.0275      0.021      1.335      0.182      -0.013       0.068
c0045         -0.0014      0.022     -0.063      0.950      -0.044       0.041
c0430         -0.1043      0.033     -3.150      0.002      -0.169      -0.039
c1105         -0.0387      0.015     -2.555      0.011      -0.068      -0.009
c0020         -0.0524      0.023     -2.314      0.021      -0.097      -0.008
c1155          0.0168      0.015      1.152      0.249      -0.012       0.045
c0400         -0.0263      0.028     -0.938      0.348      -0.081       0.029
c1800         -0.0096      0.010     -0.963      0.335      -0.029       0.010
c0105         -0.0097      0.020     -0.479      0.632      -0.050       0.030
c2055         -0.0240      0.025     -0.955      0.340      -0.073       0.025
c0450         -0.0150      0.031     -0.481      0.630      -0.076       0.046
c1825         -0.0108      0.018     -0.605      0.545      -0.046       0.024
c1300         -0.0006      0.010     -0.054      0.957      -0.021       0.020
c0540         -0.0524      0.026     -2.018      0.044      -0.103      -0.001
c1205         -0.0312      0.013     -2.357      0.019      -0.057      -0.005
c0725         -0.0408      0.013     -3.100      0.002      -0.067      -0.015
c0145          0.0541      0.022      2.482      0.013       0.011       0.097
c1125         -0.0061      0.016     -0.385      0.700      -0.037       0.025
c1540          0.0123      0.012      1.042      0.297      -0.011       0.036
c1220          0.0113      0.012      0.958      0.338      -0.012       0.034
c1230         -0.0026      0.006     -0.433      0.665      -0.014       0.009
c1510          0.0094      0.010      0.902      0.367      -0.011       0.030
c0415          0.0412      0.030      1.371      0.170      -0.018       0.100
c0355         -0.0218      0.030     -0.731      0.465      -0.080       0.037
c2150         -0.0052      0.030     -0.174      0.862      -0.063       0.053
c1400         -0.0023      0.008     -0.274      0.784      -0.019       0.014
c1805         -0.0149      0.011     -1.369      0.171      -0.036       0.006
c1945         -0.0205      0.020     -1.033      0.302      -0.059       0.018
c2045          0.0530      0.024      2.201      0.028       0.006       0.100
c1235         -0.0117      0.008     -1.415      0.157      -0.028       0.005
c0615          0.0133      0.017      0.778      0.437      -0.020       0.047
c2030         -0.0444      0.024     -1.822      0.069      -0.092       0.003
c1920          0.0375      0.019      2.000      0.046       0.001       0.074
c2050          0.0954      0.025      3.849      0.000       0.047       0.144
c0005          0.0059      0.015      0.399      0.690      -0.023       0.035
c2130          0.0307      0.030      1.029      0.304      -0.028       0.089
c0940          0.0242      0.014      1.679      0.093      -0.004       0.052
c2205          0.0187      0.021      0.870      0.384      -0.023       0.061
c1405         -0.0193      0.009     -2.254      0.024      -0.036      -0.003
c2305          0.0333      0.024      1.364      0.173      -0.015       0.081
c1935          0.0058      0.019      0.301      0.764      -0.032       0.044
c0325         -0.0593      0.030     -1.969      0.049      -0.118      -0.000
c1810         -0.0213      0.014     -1.497      0.134      -0.049       0.007
c1835          0.0327      0.015      2.130      0.033       0.003       0.063
c0900          0.0037      0.011      0.325      0.745      -0.019       0.026
c1425         -0.0215      0.011     -1.965      0.049      -0.043   -5.06e-05
c0125         -0.0408      0.022     -1.891      0.059      -0.083       0.002
c0850         -0.0030      0.013     -0.236      0.814      -0.028       0.022
c1325         -0.0477      0.012     -3.888      0.000      -0.072      -0.024
c1525          0.0098      0.012      0.821      0.412      -0.014       0.033
c0135          0.0353      0.020      1.731      0.084      -0.005       0.075
c1505          0.0205      0.010      2.140      0.032       0.002       0.039
c1740          0.0008      0.018      0.044      0.965      -0.034       0.035
c0805          0.0098      0.011      0.924      0.355      -0.011       0.031
c2020         -0.0237      0.024     -0.981      0.327      -0.071       0.024
c2120         -0.0276      0.028     -1.002      0.316      -0.082       0.026
c0950         -0.0076      0.015     -0.517      0.605      -0.037       0.021
c1605         -0.0037      0.013     -0.288      0.773      -0.029       0.021
c1025         -0.0109      0.014     -0.755      0.450      -0.039       0.017
c1355          0.0063      0.011      0.588      0.556      -0.015       0.027
c0845          0.0061      0.013      0.469      0.639      -0.019       0.032
c1905         -0.0246      0.014     -1.774      0.076      -0.052       0.003
c2000         -0.0022      0.018     -0.122      0.903      -0.038       0.033
c1955         -0.0034      0.021     -0.160      0.873      -0.045       0.038
c0150         -0.0024      0.022     -0.110      0.912      -0.045       0.040
c0110         -0.0321      0.021     -1.541      0.124      -0.073       0.009
c2210         -0.0170      0.024     -0.724      0.469      -0.063       0.029
c1340         -0.0003      0.009     -0.029      0.977      -0.018       0.017
c0220          0.0204      0.026      0.789      0.430      -0.030       0.071
c1110          0.0089      0.015      0.609      0.542      -0.020       0.038
c0405         -0.0499      0.030     -1.662      0.097      -0.109       0.009
c0915         -0.0351      0.013     -2.605      0.009      -0.062      -0.009
c2140          0.0210      0.030      0.690      0.490      -0.039       0.081
c0630          0.0278      0.016      1.718      0.086      -0.004       0.059
c2355          0.0119      0.023      0.512      0.609      -0.034       0.057
c0720          0.0248      0.013      1.939      0.053      -0.000       0.050
c1410          0.0022      0.009      0.237      0.813      -0.016       0.021
c1645         -0.0095      0.016     -0.604      0.546      -0.040       0.021
c1500          0.0441      0.009      5.129      0.000       0.027       0.061
c2325          0.0680      0.029      2.338      0.019       0.011       0.125
c0730          0.0118      0.011      1.060      0.289      -0.010       0.034
c0715         -0.0039      0.012     -0.317      0.751      -0.028       0.020
c1045         -0.0221      0.016     -1.411      0.158      -0.053       0.009
c1625         -0.0135      0.014     -0.952      0.341      -0.041       0.014
c0820          0.0259      0.012      2.146      0.032       0.002       0.050
c1720          0.0090      0.016      0.553      0.580      -0.023       0.041
c2245         -0.0404      0.030     -1.346      0.178      -0.099       0.018
c2300          0.0118      0.024      0.486      0.627      -0.036       0.060
c2215          0.0115      0.026      0.435      0.664      -0.040       0.063
c1035         -0.0063      0.015     -0.423      0.672      -0.036       0.023
c0550          0.0230      0.024      0.942      0.346      -0.025       0.071
==============================================================================
Omnibus:                     4043.713   Durbin-Watson:                   1.954
Prob(Omnibus):                  0.000   Jarque-Bera (JB):          4176392.615
Skew:                          -5.781   Prob(JB):                         0.00
Kurtosis:                     177.295   Cond. No.                     1.12e+04
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.12e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

MSr: 0.0003056669174634134

MSe: 0.00019993423813406298

Answer 1

The short answer is "No".

• Stepwise regression is popular, but problematic. Frank Harrell lists several reasons why stepwise is problematic in his book Regression Modelling Strategies.

• If your goal is interpretability, the procedure is not guaranteed to pick important variables (so you can't say for instance that a variable with p>0.05 is unimportant). Moreover, the p value tells you nothing about the effect of the variable. It is completely possible that you have a variable with a large effect but also large uncertainty. I've demonstrated that here and elsewhere.

• If your goal is prediction, variable selection can often hurt performance as I show here.

• If your goal is inference, you're out of luck again because the p values lose all meaning since they do not condition on the selection procedure.

Can you tell us what the goal of your analysis is? That can help us choose a better way of moving forward.