Residential Whole Loans (Interest Income Components) (Details) - Residential whole loans - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 57,879	$ 17,935	$ 107,499	$ 32,264
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	26,578	4,185	48,992	5,893
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	13,256	2,270	23,189	3,615
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,926	570	6,627	815
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,122	0	6,295	0
Purchased Performing Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	46,882	7,025	85,103	10,323
Purchased credit impaired loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 10,997	$ 10,910	$ 22,396	$ 21,941