Loans - Delinquency (Details) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2017 USD ($) loan	Dec. 31, 2016 USD ($) loan
Loan delinquency information
Number of loans held for investment | loan	3,312	3,640
Loans, Held-for-investment net of specific allowance for loan losses	$ 1,587,842	$ 1,587,099
General allowance for loan losses	1,685	2,011
Total	1,586,157	1,585,088
30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	25,805	32,568
90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 34,229	$ 44,563
Fixed-rate loans held for investment
Loan delinquency information
Number of loans held for investment | loan	624	692
Loans, Held-for-investment net of specific allowance for loan losses	$ 635,026	$ 679,174
Fixed-rate loans held for investment | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	10,609	11,761
Fixed-rate loans held for investment | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 9,134	$ 13,701
Fixed-rate loans held for investment | 0-500k
Loan delinquency information
Number of loans held for investment | loan	359	401
Loans, Held-for-investment net of specific allowance for loan losses	$ 49,888	$ 56,602
Fixed-rate loans held for investment | 0-500k | Minimum
Loan delinquency information
Interest rate	0.00%	0.00%
Fixed-rate loans held for investment | 0-500k | Maximum
Loan delinquency information
Interest rate	24.00%	24.00%
Fixed-rate loans held for investment | 0-500k | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 1,547	$ 2,383
Fixed-rate loans held for investment | 0-500k | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 5,676	$ 8,676
Fixed-rate loans held for investment | 500k - 1mm
Loan delinquency information
Number of loans held for investment | loan	82	88
Loans, Held-for-investment net of specific allowance for loan losses	$ 59,296	$ 62,246
Fixed-rate loans held for investment | 500k - 1mm | Minimum
Loan delinquency information
Interest rate	4.00%	4.00%
Fixed-rate loans held for investment | 500k - 1mm | Maximum
Loan delinquency information
Interest rate	9.90%	9.90%
Fixed-rate loans held for investment | 500k - 1mm | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 1,253	$ 499
Fixed-rate loans held for investment | 500k - 1mm | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 2,125	$ 2,199
Fixed-rate loans held for investment | 1mm - 1.5mm
Loan delinquency information
Number of loans held for investment | loan	43	50
Loans, Held-for-investment net of specific allowance for loan losses	$ 53,300	$ 61,167
Fixed-rate loans held for investment | 1mm - 1.5mm | Minimum
Loan delinquency information
Interest rate	4.76%	4.50%
Fixed-rate loans held for investment | 1mm - 1.5mm | Maximum
Loan delinquency information
Interest rate	10.25%	11.00%
Fixed-rate loans held for investment | 1mm - 1.5mm | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 1,333	$ 1,854
Fixed-rate loans held for investment | 1.5mm - 2mm
Loan delinquency information
Number of loans held for investment | loan	42	48
Loans, Held-for-investment net of specific allowance for loan losses	$ 75,618	$ 85,555
Fixed-rate loans held for investment | 1.5mm - 2mm | Minimum
Loan delinquency information
Interest rate	4.56%	4.56%
Fixed-rate loans held for investment | 1.5mm - 2mm | Maximum
Loan delinquency information
Interest rate	7.52%	7.52%
Fixed-rate loans held for investment | 1.5mm - 2mm | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses		$ 972
Fixed-rate loans held for investment | 2mm - 2.5mm
Loan delinquency information
Number of loans held for investment | loan	27	32
Loans, Held-for-investment net of specific allowance for loan losses	$ 61,468	$ 72,760
Fixed-rate loans held for investment | 2mm - 2.5mm | Minimum
Loan delinquency information
Interest rate	4.66%	4.91%
Fixed-rate loans held for investment | 2mm - 2.5mm | Maximum
Loan delinquency information
Interest rate	7.52%	7.52%
Fixed-rate loans held for investment | 2mm - 2.5mm | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses		$ 2,282
Fixed-rate loans held for investment | >2.5mm
Loan delinquency information
Number of loans held for investment | loan	71	73
Loans, Held-for-investment net of specific allowance for loan losses	$ 335,456	$ 340,844
Fixed-rate loans held for investment | >2.5mm | Minimum
Loan delinquency information
Interest rate	4.50%	4.50%
Fixed-rate loans held for investment | >2.5mm | Maximum
Loan delinquency information
Interest rate	11.00%	11.00%
Fixed-rate loans held for investment | >2.5mm | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 7,809	$ 6,597
Adjustable rate loans held for investment
Loan delinquency information
Number of loans held for investment | loan	2,688	2,948
Loans, Held-for-investment net of specific allowance for loan losses	$ 952,816	$ 907,925
Adjustable rate loans held for investment | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	15,196	20,807
Adjustable rate loans held for investment | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 25,095	$ 30,862
Adjustable rate loans held for investment | 0-500k
Loan delinquency information
Number of loans held for investment | loan	2,160	2,369
Loans, Held-for-investment net of specific allowance for loan losses	$ 227,617	$ 245,631
Adjustable rate loans held for investment | 0-500k | Minimum
Loan delinquency information
Interest rate	0.00%	0.00%
Adjustable rate loans held for investment | 0-500k | Maximum
Loan delinquency information
Interest rate	9.75%	9.75%
Adjustable rate loans held for investment | 0-500k | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 6,338	$ 12,525
Adjustable rate loans held for investment | 0-500k | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 7,179	$ 8,725
Adjustable rate loans held for investment | 500k - 1mm
Loan delinquency information
Number of loans held for investment | loan	318	345
Loans, Held-for-investment net of specific allowance for loan losses	$ 201,516	$ 218,479
Adjustable rate loans held for investment | 500k - 1mm | Minimum
Loan delinquency information
Interest rate	2.66%	2.66%
Adjustable rate loans held for investment | 500k - 1mm | Maximum
Loan delinquency information
Interest rate	8.63%	8.63%
Adjustable rate loans held for investment | 500k - 1mm | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 2,201	$ 5,079
Adjustable rate loans held for investment | 500k - 1mm | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 3,236	$ 4,792
Adjustable rate loans held for investment | 1mm - 1.5mm
Loan delinquency information
Number of loans held for investment | loan	121	136
Loans, Held-for-investment net of specific allowance for loan losses	$ 135,693	$ 151,415
Adjustable rate loans held for investment | 1mm - 1.5mm | Minimum
Loan delinquency information
Interest rate	3.50%	3.50%
Adjustable rate loans held for investment | 1mm - 1.5mm | Maximum
Loan delinquency information
Interest rate	8.20%	8.21%
Adjustable rate loans held for investment | 1mm - 1.5mm | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 3,204	$ 2,166
Adjustable rate loans held for investment | 1mm - 1.5mm | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 4,938	$ 6,971
Adjustable rate loans held for investment | 1.5mm - 2mm
Loan delinquency information
Number of loans held for investment | loan	34	48
Loans, Held-for-investment net of specific allowance for loan losses	$ 50,565	$ 70,676
Adjustable rate loans held for investment | 1.5mm - 2mm | Minimum
Loan delinquency information
Interest rate	4.00%	2.62%
Adjustable rate loans held for investment | 1.5mm - 2mm | Maximum
Loan delinquency information
Interest rate	6.75%	6.25%
Adjustable rate loans held for investment | 1.5mm - 2mm | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 1,200	$ 1,037
Adjustable rate loans held for investment | 1.5mm - 2mm | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 2,076	$ 2,278
Adjustable rate loans held for investment | 2mm - 2.5mm
Loan delinquency information
Number of loans held for investment | loan	4	8
Loans, Held-for-investment net of specific allowance for loan losses	$ 8,736	$ 16,412
Adjustable rate loans held for investment | 2mm - 2.5mm | Minimum
Loan delinquency information
Interest rate	4.25%	3.85%
Adjustable rate loans held for investment | 2mm - 2.5mm | Maximum
Loan delinquency information
Interest rate	6.75%	6.35%
Adjustable rate loans held for investment | 2mm - 2.5mm | 30-89 Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 2,253
Adjustable rate loans held for investment | >2.5mm
Loan delinquency information
Number of loans held for investment | loan	51	42
Loans, Held-for-investment net of specific allowance for loan losses	$ 328,689	$ 205,312
Adjustable rate loans held for investment | >2.5mm | Minimum
Loan delinquency information
Interest rate	2.43%	2.14%
Adjustable rate loans held for investment | >2.5mm | Maximum
Loan delinquency information
Interest rate	9.13%	8.32%
Adjustable rate loans held for investment | >2.5mm | 90+ Days
Loan delinquency information
Loans, Held-for-investment net of specific allowance for loan losses	$ 7,666	$ 8,096