Fair Value - Financial Instruments Not Carried at Fair Value (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financial assets not carried at fair value:
HTM securities	$ 551,314	$ 570,638
Carrying Value
Financial assets not carried at fair value:
Loans held-for-investment, loans held-for-sale and loans transferred as secured borrowings	10,197,948	10,034,030
HTM securities	544,423	570,638
Financial liabilities not carried at fair value:
Secured financing agreements and secured borrowings on transferred loans	9,765,627	9,834,108
Unsecured senior notes	1,932,560	1,928,622
Fair Value
Financial assets not carried at fair value:
Loans held-for-investment, loans held-for-sale and loans transferred as secured borrowings	10,096,859	10,086,372
HTM securities	512,493	568,727
Financial liabilities not carried at fair value:
Secured financing agreements and secured borrowings on transferred loans	9,663,737	9,826,511
Unsecured senior notes	$ 1,873,867	$ 2,022,283