Mortgage Loans at Fair Value - Schedule of Mortgage Loans (Details) - Residential mortgage - Residential portfolio segment - Nonperforming financing receivable $ in Thousands	Dec. 31, 2017 USD ($) loan	Dec. 31, 2016 USD ($) loan
Number of Loans
Current | loan	17	211
Foreclosure | loan	67	2,180
Mortgage loans | loan	111	2,891
Fair Value and Carrying Value
Current	$ 1,528	$ 33,992
Foreclosure	8,874	365,772
Mortgage loans at fair value	11,477	460,444
Unpaid Principal Balance
Current	2,380	45,568
Foreclosure	18,813	551,243
Mortgage loans	29,350	697,716
Market Value of Underlying Properties
Current	3,156	58,842
Foreclosure	20,820	574,546
Mortgage loans	$ 31,174	$ 746,272
Real estate loan, held-for-sale
Number of Loans
Current | loan		519
Foreclosure | loan		33
Mortgage loans | loan		583
Fair Value and Carrying Value
Current		$ 100,558
Foreclosure		4,488
Mortgage loans at fair value		108,036
Unpaid Principal Balance
Current		114,757
Foreclosure		6,023
Mortgage loans		125,605
Market Value of Underlying Properties
Current		140,471
Foreclosure		6,675
Mortgage loans		$ 153,568
30
Number of Loans
Past Due | loan	1	66
Fair Value and Carrying Value
Past Due	$ 51	$ 7,898
Unpaid Principal Balance
Past Due	139	11,836
Market Value of Underlying Properties
Past Due	$ 70	$ 13,576
30 | Real estate loan, held-for-sale
Number of Loans
Past Due | loan		10
Fair Value and Carrying Value
Past Due		$ 1,082
Unpaid Principal Balance
Past Due		1,911
Market Value of Underlying Properties
Past Due		$ 2,329
60
Number of Loans
Past Due | loan	3	34
Fair Value and Carrying Value
Past Due	$ 304	$ 4,444
Unpaid Principal Balance
Past Due	344	6,364
Market Value of Underlying Properties
Past Due	$ 630	$ 7,536
60 | Real estate loan, held-for-sale
Number of Loans
Past Due | loan		4
Fair Value and Carrying Value
Past Due		$ 286
Unpaid Principal Balance
Past Due		623
Market Value of Underlying Properties
Past Due		$ 663
90
Number of Loans
Past Due | loan	23	400
Fair Value and Carrying Value
Past Due	$ 720	$ 48,338
Unpaid Principal Balance
Past Due	7,674	82,705
Market Value of Underlying Properties
Past Due	$ 6,498	$ 91,772
90 | Real estate loan, held-for-sale
Number of Loans
Past Due | loan		17
Fair Value and Carrying Value
Past Due		$ 1,622
Unpaid Principal Balance
Past Due		2,291
Market Value of Underlying Properties
Past Due		$ 3,430