Mortgage Banking Activities, Net - Components of Mortgage Banking Activities (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Other income, net	$ 955	$ 4,859	$ 3,392	$ 9,484
Mortgage banking activities, net	(5,772)	19,160	(34,183)	31,469
Residential Mortgage Banking Activities, Net
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Other income, net	(6,612)	852	(6,354)	973
Mortgage banking activities, net	(8,005)	15,359	(31,086)	26,186
Residential Mortgage Banking Activities, Net | Residential loans, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Changes in fair value of assets	(1,393)	20,267	6,562	35,111
Residential Mortgage Banking Activities, Net | Risk management derivatives
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Risk management derivatives	0	(5,760)	(31,294)	(9,898)
Business Purpose Mortgage Banking Activities, Net:
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Changes in fair value of assets	(1,260)	1,012	(5,194)	1,098
Other income, net	2,283	2,578	10,617	3,076
Mortgage banking activities, net	2,233	3,801	(3,097)	5,283
Business Purpose Mortgage Banking Activities, Net: | Risk management derivatives
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Risk management derivatives	0	(1,671)	(21,538)	(2,517)
Business Purpose Mortgage Banking Activities, Net: | Single-family rental loans, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Changes in fair value of assets	$ 1,210	$ 1,882	$ 13,018	$ 3,626