Residential Loans - Loan Product Type and Accompanying Loan Characteristics of Residential Loans (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2016 USD ($) loan	Dec. 31, 2015 USD ($) loan
Mortgage Loans on Real Estate [Line Items]
Interest Rate		0.00%
Residential Loans Held For Investment at Fair Value
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan	3,068	2,398
Principal value	$ 2,233,797	$ 1,758,990
30-89 Days DQ	4,299	5,357
90+ Days DQ	$ 237	$ 0
Residential Loans Held For Investment at Fair Value | ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan		5
Principal value		$ 3,501
30-89 Days DQ		0
90+ Days DQ		0
Residential Loans Held For Investment at Fair Value | ARM Loans | $251 to $500
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum		251
Loan Balance, maximum		$ 500
Number of loans | loan		2
Principal value		$ 563
30-89 Days DQ		0
90+ Days DQ		$ 0
Residential Loans Held For Investment at Fair Value | ARM Loans | $251 to $500 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.63%
Residential Loans Held For Investment at Fair Value | ARM Loans | $251 to $500 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.75%
Residential Loans Held For Investment at Fair Value | ARM Loans | $501 to $750
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum		$ 501
Loan Balance, maximum		$ 750
Number of loans | loan		2
Principal value		$ 1,671
30-89 Days DQ		0
90+ Days DQ		$ 0
Residential Loans Held For Investment at Fair Value | ARM Loans | $501 to $750 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.50%
Residential Loans Held For Investment at Fair Value | ARM Loans | $501 to $750 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.50%
Residential Loans Held For Investment at Fair Value | ARM Loans | $751 to $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum		$ 751
Loan Balance, maximum		$ 1,000
Number of loans | loan		1
Principal value		$ 1,267
30-89 Days DQ		0
90+ Days DQ		$ 0
Residential Loans Held For Investment at Fair Value | ARM Loans | $751 to $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.63%
Residential Loans Held For Investment at Fair Value | ARM Loans | $751 to $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.63%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan	11	56
Principal value	$ 10,257	$ 42,268
30-89 Days DQ	0	0
90+ Days DQ	0	0
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $0 to $250
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	251
Loan Balance, maximum	$ 500
Number of loans | loan	1
Principal value	$ 264
30-89 Days DQ	0
90+ Days DQ	$ 0
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $0 to $250 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.63%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $0 to $250 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.63%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $251 to $500
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum		251
Loan Balance, maximum		$ 500
Number of loans | loan		7
Principal value		$ 2,963
30-89 Days DQ		0
90+ Days DQ		$ 0
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $251 to $500 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		2.88%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $251 to $500 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate		3.88%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $501 to $750
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 501	$ 501
Loan Balance, maximum	$ 750	$ 750
Number of loans | loan	4	28
Principal value	$ 2,722	$ 17,514
30-89 Days DQ	0	0
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $501 to $750 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.88%	2.63%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $501 to $750 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	4.65%	4.90%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $751 to $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 751	$ 751
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	2	15
Principal value	$ 1,726	$ 12,994
30-89 Days DQ	0	0
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $751 to $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.50%	2.75%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | $751 to $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	4.00%	5.05%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | Over $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	4	6
Principal value	$ 5,545	$ 8,797
30-89 Days DQ	0	0
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | Over $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%	2.88%
Residential Loans Held For Investment at Fair Value | Hybrid ARM Loans | Over $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	4.20%	5.20%
Residential Loans Held For Investment at Fair Value | Fixed Loans
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan	3,057	2,337
Principal value	$ 2,223,540	$ 1,713,221
30-89 Days DQ	4,299	5,357
90+ Days DQ	237	0
Residential Loans Held For Investment at Fair Value | Fixed Loans | $0 to $250
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	0	0
Loan Balance, maximum	$ 250	$ 250
Number of loans | loan	26	29
Principal value	$ 4,643	$ 5,295
30-89 Days DQ	0	242
90+ Days DQ	$ 237	$ 0
Residential Loans Held For Investment at Fair Value | Fixed Loans | $0 to $250 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.67%	3.64%
Residential Loans Held For Investment at Fair Value | Fixed Loans | $0 to $250 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.08%	5.38%
Residential Loans Held For Investment at Fair Value | Fixed Loans | $251 to $500
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 251	$ 251
Loan Balance, maximum	$ 500	$ 500
Number of loans | loan	633	484
Principal value	$ 278,560	$ 212,732
30-89 Days DQ	264	913
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Investment at Fair Value | Fixed Loans | $251 to $500 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.80%	3.13%
Residential Loans Held For Investment at Fair Value | Fixed Loans | $251 to $500 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.13%	5.13%
Residential Loans Held For Investment at Fair Value | Fixed Loans | $501 to $750
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 501	$ 501
Loan Balance, maximum	$ 750	$ 750
Number of loans | loan	1,306	959
Principal value	$ 807,714	$ 595,863
30-89 Days DQ	2,803	3,213
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Investment at Fair Value | Fixed Loans | $501 to $750 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.75%	2.94%
Residential Loans Held For Investment at Fair Value | Fixed Loans | $501 to $750 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%	5.25%
Residential Loans Held For Investment at Fair Value | Fixed Loans | $751 to $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 751	$ 751
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	690	552
Principal value	$ 597,002	$ 480,557
30-89 Days DQ	0	989
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Investment at Fair Value | Fixed Loans | $751 to $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.75%	2.90%
Residential Loans Held For Investment at Fair Value | Fixed Loans | $751 to $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.63%	5.00%
Residential Loans Held For Investment at Fair Value | Fixed Loans | Over $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	402	313
Principal value	$ 535,621	$ 418,774
30-89 Days DQ	1,232	0
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Investment at Fair Value | Fixed Loans | Over $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.80%	3.14%
Residential Loans Held For Investment at Fair Value | Fixed Loans | Over $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.00%	5.00%
Residential loans, held-for-investment
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan	3,735	4,545
Principal value	$ 886,981	$ 1,122,415
30-89 Days DQ	32,356	30,328
90+ Days DQ	$ 30,166	$ 58,823
Residential loans, held-for-investment | ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan	3,698	4,504
Principal value	$ 869,290	$ 1,101,588
30-89 Days DQ	31,687	29,786
90+ Days DQ	30,166	58,823
Residential loans, held-for-investment | ARM Loans | $0 to $250
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	0	0
Loan Balance, maximum	$ 250	$ 250
Number of loans | loan	2,623	3,133
Principal value	$ 297,646	$ 355,415
30-89 Days DQ	9,158	10,661
90+ Days DQ	$ 7,410	$ 13,078
Residential loans, held-for-investment | ARM Loans | $0 to $250 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.63%	0.38%
Residential loans, held-for-investment | ARM Loans | $0 to $250 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.60%	5.16%
Residential loans, held-for-investment | ARM Loans | $251 to $500
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 251	$ 251
Loan Balance, maximum	$ 500	$ 500
Number of loans | loan	694	858
Principal value	$ 241,253	$ 296,425
30-89 Days DQ	9,177	9,620
90+ Days DQ	$ 10,059	$ 15,345
Residential loans, held-for-investment | ARM Loans | $251 to $500 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.25%	0.00%
Residential loans, held-for-investment | ARM Loans | $251 to $500 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.75%	5.63%
Residential loans, held-for-investment | ARM Loans | $501 to $750
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 501	$ 501
Loan Balance, maximum	$ 750	$ 750
Number of loans | loan	203	269
Principal value	$ 121,919	$ 161,273
30-89 Days DQ	5,812	4,578
90+ Days DQ	$ 5,069	$ 7,209
Residential loans, held-for-investment | ARM Loans | $501 to $750 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.88%	0.63%
Residential loans, held-for-investment | ARM Loans | $501 to $750 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.89%	4.66%
Residential loans, held-for-investment | ARM Loans | $751 to $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 751	$ 751
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	100	135
Principal value	$ 86,988	$ 118,983
30-89 Days DQ	2,750	3,586
90+ Days DQ	$ 3,322	$ 8,473
Residential loans, held-for-investment | ARM Loans | $751 to $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.63%	0.38%
Residential loans, held-for-investment | ARM Loans | $751 to $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%	2.38%
Residential loans, held-for-investment | ARM Loans | Over $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	78	109
Principal value	$ 121,484	$ 169,492
30-89 Days DQ	4,790	1,341
90+ Days DQ	$ 4,306	$ 14,718
Residential loans, held-for-investment | ARM Loans | Over $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.25%	0.00%
Residential loans, held-for-investment | ARM Loans | Over $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.75%	2.63%
Residential loans, held-for-investment | Hybrid ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan	37	41
Principal value	$ 17,691	$ 20,827
30-89 Days DQ	669	542
90+ Days DQ	0	0
Residential loans, held-for-investment | Hybrid ARM Loans | $0 to $250
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	0	0
Loan Balance, maximum	$ 250	$ 250
Number of loans | loan	4	3
Principal value	$ 453	$ 317
30-89 Days DQ	0	0
90+ Days DQ	$ 0	$ 0
Residential loans, held-for-investment | Hybrid ARM Loans | $0 to $250 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%	2.75%
Residential loans, held-for-investment | Hybrid ARM Loans | $0 to $250 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%	2.88%
Residential loans, held-for-investment | Hybrid ARM Loans | $251 to $500
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 251	$ 251
Loan Balance, maximum	$ 500	$ 500
Number of loans | loan	18	20
Principal value	$ 6,516	$ 7,523
30-89 Days DQ	0	0
90+ Days DQ	$ 0	$ 0
Residential loans, held-for-investment | Hybrid ARM Loans | $251 to $500 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.63%	2.63%
Residential loans, held-for-investment | Hybrid ARM Loans | $251 to $500 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.13%	2.88%
Residential loans, held-for-investment | Hybrid ARM Loans | $501 to $750
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 501	$ 501
Loan Balance, maximum	$ 750	$ 750
Number of loans | loan	13	15
Principal value	$ 8,483	$ 9,874
30-89 Days DQ	669	542
90+ Days DQ	$ 0	$ 0
Residential loans, held-for-investment | Hybrid ARM Loans | $501 to $750 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.75%	2.63%
Residential loans, held-for-investment | Hybrid ARM Loans | $501 to $750 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.13%	2.88%
Residential loans, held-for-investment | Hybrid ARM Loans | $751 to $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 751	$ 751
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	1	2
Principal value	$ 751	$ 1,547
30-89 Days DQ	0	0
90+ Days DQ	$ 0	$ 0
Residential loans, held-for-investment | Hybrid ARM Loans | $751 to $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.13%	2.75%
Residential loans, held-for-investment | Hybrid ARM Loans | $751 to $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.13%	2.75%
Residential loans, held-for-investment | Hybrid ARM Loans | Over $1,000
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum
Loan Balance, maximum	$ 1,000	$ 1,000
Number of loans | loan	1	1
Principal value	$ 1,488	$ 1,566
30-89 Days DQ	0	0
90+ Days DQ	$ 0	$ 0
Residential loans, held-for-investment | Hybrid ARM Loans | Over $1,000 | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%	2.75%
Residential loans, held-for-investment | Hybrid ARM Loans | Over $1,000 | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%	2.75%
Residential Loans Held For Sale
Mortgage Loans on Real Estate [Line Items]
Number of loans | loan	1,121	1,772
Principal value	$ 833,385	$ 1,091,518
30-89 Days DQ	0	4,346
90+ Days DQ	300	1,852
Residential Loans Held For Sale | ARM Loans
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	61	64
Loan Balance, maximum	$ 396	$ 1,298
Number of loans | loan	6	14
Principal value	$ 882	$ 5,258
30-89 Days DQ	0	0
90+ Days DQ	$ 300	$ 415
Residential Loans Held For Sale | ARM Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.88%	1.50%
Residential Loans Held For Sale | ARM Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.75%	4.00%
Residential Loans Held For Sale | Hybrid ARM Loans
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 2	$ 164
Loan Balance, maximum	$ 1,947	$ 1,989
Number of loans | loan	173	356
Principal value	$ 144,174	$ 276,457
30-89 Days DQ	0	2,249
90+ Days DQ	$ 0	$ 0
Residential Loans Held For Sale | Hybrid ARM Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.50%	2.50%
Residential Loans Held For Sale | Hybrid ARM Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.00%	4.25%
Residential Loans Held For Sale | Fixed Loans
Mortgage Loans on Real Estate [Line Items]
Loan Balance, minimum	$ 404	$ 30
Loan Balance, maximum	$ 1,997	$ 2,332
Number of loans | loan	942	1,402
Principal value	$ 688,329	$ 809,803
30-89 Days DQ	0	2,097
90+ Days DQ	$ 0	$ 1,437
Residential Loans Held For Sale | Fixed Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.99%	2.75%
Residential Loans Held For Sale | Fixed Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%	5.25%