Loans Held for Investment - Loan Products Type and Characteristics (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Unpaid Principal Balance	$ 13,129,822	$ 13,169,843
Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 2,690,313	$ 2,379,192
% of Unpaid Principal Balance	20.50%	18.10%
30 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 1,058,059	$ 1,070,173
% of Unpaid Principal Balance	8.10%	8.10%
60 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 485,036	$ 336,950
% of Unpaid Principal Balance	3.70%	2.60%
90 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 540,755	$ 291,866
% of Unpaid Principal Balance	4.10%	2.20%
Bankruptcy | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 297,249	$ 322,288
% of Unpaid Principal Balance	2.30%	2.40%
Foreclosure | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 266,443	$ 298,526
% of Unpaid Principal Balance	2.00%	2.30%
REO | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 42,771	$ 59,389
% of Unpaid Principal Balance	0.30%	0.50%