Fair Value - Financial Instruments Not Carried at Fair Value (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financial assets not carried at fair value:
HTM securities	$ 570,888	$ 570,638
Carrying Value
Financial assets not carried at fair value:
Loans held-for-investment, loans held-for-sale and loans transferred as secured borrowings	10,316,073	10,034,030
HTM securities	565,851	570,638
Financial liabilities not carried at fair value:
Secured financing agreements and secured borrowings on transferred loans	10,618,366	9,834,108
Unsecured senior notes	1,930,584	1,928,622
Fair Value
Financial assets not carried at fair value:
Loans held-for-investment, loans held-for-sale and loans transferred as secured borrowings	10,165,113	10,086,372
HTM securities	540,268	568,727
Financial liabilities not carried at fair value:
Secured financing agreements and secured borrowings on transferred loans	10,361,814	9,826,511
Unsecured senior notes	$ 1,773,382	$ 2,022,283