Fair Value of Assets and Liabilities - Not Measured at Fair Value (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total assets	$ 1,725,323	$ 2,205,970
Total liabilities	992,713	1,133,175
Level 1 Inputs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total assets	0	0
Total liabilities	0	0
Level 2 Inputs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total assets	126,340	160,131
Total liabilities	0	0
Level 3 Inputs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total assets	1,598,983	2,045,839
Total liabilities	992,713	1,133,175
Carrying Amount
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Cash and cash equivalents	338,394	243,779
Restricted cash (1)	134,345	243,343
Servicer reserve receivable	41	73
Deposits	892	953
Total assets	473,672	488,148
Accrued expenses and other liabilities	18,414	24,899
Accounts payable	2,951	10,855
Payables to investors	49,405	97,530
Credit facilities and securities sold under repurchase agreements	480,079	587,453
Total liabilities	550,849	720,737
Portion Not Recorded at Fair Value | Level 1 Inputs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Cash and cash equivalents	0	0
Restricted cash (1)	0	0
Servicer reserve receivable	0	0
Deposits	0	0
Total assets	0	0
Accrued expenses and other liabilities	0	0
Accounts payable	0	0
Payables to investors	0	0
Credit facilities and securities sold under repurchase agreements	0	0
Total liabilities	0	0
Portion Not Recorded at Fair Value | Level 2 Inputs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Cash and cash equivalents	338,394	243,779
Restricted cash (1)	134,345	243,343
Servicer reserve receivable	41	73
Deposits	892	953
Total assets	473,672	488,148
Accrued expenses and other liabilities	0	0
Accounts payable	2,951	10,855
Payables to investors	49,405	97,530
Credit facilities and securities sold under repurchase agreements	66,378	77,143
Total liabilities	118,734	185,528
Portion Not Recorded at Fair Value | Level 3 Inputs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Cash and cash equivalents	0	0
Restricted cash (1)	0	0
Servicer reserve receivable	0	0
Deposits	0	0
Total assets	0	0
Accrued expenses and other liabilities	18,414	24,899
Accounts payable	0	0
Payables to investors	0	0
Credit facilities and securities sold under repurchase agreements	413,701	510,310
Total liabilities	432,115	535,209
Balance at Fair Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Cash and cash equivalents	338,394	243,779
Restricted cash (1)	134,345	243,343
Servicer reserve receivable	41	73
Deposits	892	953
Total assets	473,672	488,148
Accrued expenses and other liabilities	18,414	24,899
Accounts payable	2,951	10,855
Payables to investors	49,405	97,530
Credit facilities and securities sold under repurchase agreements	480,079	587,453
Total liabilities	$ 550,849	$ 720,737