Concentration of Credit and Other Risks - Seller Concentration (Details)	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Top ten Single-family sellers | Single-family loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	55.00%	57.00%
JPMorgan Chase Bank, National Association | Single-family loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	12.00%	16.00%
United Shore Financial Services, LLC | Single-family loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	10.00%
Other top 10 sellers | Single-family loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	33.00%	31.00%
Top ten multifamily sellers | Multifamily loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	78.00%	77.00%
CBRE Capital Markets, Inc. | Multifamily loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	17.00%	15.00%
Berkadia Commercial Mortgage LLC | Multifamily loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	14.00%	13.00%
Other top 10 sellers | Multifamily loan purchase volume
Concentration Risk [Line Items]
Concentration risk percentage	47.00%	49.00%