Mortgage Loans - Schedule of Change in Accretable Yield (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Accretable Yield Movement Schedule [Roll Forward]
Reclassification from (to) non-accretable amount, net	$ 33,200	$ (33,100)
Re-performing loans
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Accretable Yield Movement Schedule [Roll Forward]
Balance at beginning of period	311,806	344,141
Accretable yield additions	31,917	35,471
Accretion	(94,334)	(101,195)
Reclassification from (to) non-accretable amount, net	(30,273)	33,389
Balance at end of period	219,116	311,806
Non-performing loans
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Accretable Yield Movement Schedule [Roll Forward]
Balance at beginning of period	6,459	7,370
Accretable yield additions	957	675
Accretion	(787)	(1,316)
Reclassification from (to) non-accretable amount, net	(2,922)	(270)
Balance at end of period	$ 3,707	$ 6,459