Other Assets - Schedule of Other Assets (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Deferred Costs, Capitalized, Prepaid, and Other Assets Disclosure [Abstract]
FHLBI stock	$ 74,250	$ 74,250
Loan participation interest	44,654	54,981
Commercial loans, held-for-investment	24,055	31,582
Investments in unconsolidated ventures	21,998	24,012
Prepaid expenses and other assets	1,223	1,234
Total	$ 166,180	$ 186,059