Debt Obligations, Credit Facilities and Repurchase Agreements (Details) - USD ($) $ in Thousands	1 Months Ended	3 Months Ended	12 Months Ended
	Jan. 31, 2018	Mar. 31, 2018	Dec. 31, 2017
Debt Obligations
Debt Carrying Value		$ 626,063	$ 528,573
Collateral Carrying Value		$ 684,016	$ 533,584
Weighted Average Note Rate (as a percent)		3.74%	3.34%
Structured Business
Debt Obligations
Debt Carrying Value		$ 344,725	$ 237,037
Collateral Carrying Value		$ 402,395	$ 241,868
Weighted Average Note Rate (as a percent)		4.21%	4.02%
Weighted average note rate including certain fees and costs (as a percent)		4.55%	4.51%
Unamortized deferred finance costs		$ 3,400	$ 2,200
Leverage on loans and investment portfolio, excluding the $3.0 million and $2.2 million master security agreement used to finance leasehold and capital expenditure improvements to corporate office (as a percent)		71.00%	72.00%
Maximum borrowing capacity		$ 2,200
Fixed interest rate (as a percent)		4.60%
Structured Business | $300 million repurchase facility
Debt Obligations
Debt Carrying Value		$ 115,226	$ 102,350
Collateral Carrying Value		$ 163,000	$ 145,850
Weighted Average Note Rate (as a percent)		3.96%	3.90%
Maximum borrowing capacity		$ 300,000	$ 300,000
Structured Business | $300 million repurchase facility | Minimum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		1.75%
Structured Business | $300 million repurchase facility | Maximum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		3.50%
Structured Business | $100 million repurchase facility
Debt Obligations
Debt Carrying Value		$ 37,871	2,445
Collateral Carrying Value		$ 50,827	$ 6,600
Weighted Average Note Rate (as a percent)		3.94%	3.61%
Maximum borrowing capacity		$ 100,000	$ 100,000
Structured Business | $100 million repurchase facility | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.00%
Structured Business | $75 million credit facility - One
Debt Obligations
Debt Carrying Value		$ 13,212
Collateral Carrying Value		$ 18,500
Weighted Average Note Rate (as a percent)		4.06%
Maximum borrowing capacity		$ 75,000	75,000
Structured Business | $75 million credit facility - One | Minimum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.00%
Structured Business | $75 million credit facility - One | Maximum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.50%
Structured Business | $75 million credit facility - two
Debt Obligations
Debt Carrying Value		$ 9,000	8,999
Collateral Carrying Value		$ 16,000	$ 16,000
Weighted Average Note Rate (as a percent)		3.94%	3.61%
Maximum borrowing capacity		$ 75,000	$ 75,000
Structured Business | $75 million credit facility - two | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.00%
Structured Business | $50 million credit facility - one
Debt Obligations
Debt Carrying Value		$ 38,027	32,538
Collateral Carrying Value		$ 48,800	$ 40,700
Weighted Average Note Rate (as a percent)		3.94%	3.61%
Maximum borrowing capacity		$ 50,000	$ 50,000
Structured Business | $50 million credit facility - one | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.00%
Structured Business | $50 million credit facility - two
Debt Obligations
Debt Carrying Value		$ 3,599	3,581
Collateral Carrying Value		$ 4,625	$ 4,625
Weighted Average Note Rate (as a percent)		5.20%	4.88%
Maximum borrowing capacity		$ 50,000	$ 50,000
Structured Business | $50 million credit facility - two | Minimum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.50%
Structured Business | $50 million credit facility - two | Maximum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		3.25%
Structured Business | $25.5 million credit facility
Debt Obligations
Debt Carrying Value		$ 13,940	13,920
Collateral Carrying Value		$ 18,753	$ 18,753
Weighted Average Note Rate (as a percent)		4.44%	4.12%
Maximum borrowing capacity		$ 25,500
Structured Business | $25.5 million credit facility | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.50%
Structured Business | $23.2 million credit facility
Debt Obligations
Debt Carrying Value		$ 23,072
Collateral Carrying Value		$ 30,900
Weighted Average Note Rate (as a percent)		4.24%
Maximum borrowing capacity		$ 23,200	$ 23,200
Structured Business | $23.2 million credit facility | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.30%
Structured Business | $20.0 million credit facility
Debt Obligations
Debt Carrying Value		$ 19,884
Collateral Carrying Value		$ 41,650
Weighted Average Note Rate (as a percent)		4.44%
Variable rate, spread (as a percent)		2.50%
Maximum borrowing capacity		$ 20,000	20,000
Structured Business | $20.0 million credit facility | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.50%
Structured Business | $10 million working capital facility
Debt Obligations
Debt Carrying Value			$ 10,000
Weighted Average Note Rate (as a percent)			4.12%
Maximum borrowing capacity		$ 10,000	$ 10,000
Structured Business | $10 million working capital facility | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.50%
Structured Business | $10 million credit facility
Debt Obligations
Maximum borrowing capacity			10,000
Structured Business | $7.5 million credit facility
Debt Obligations
Debt Carrying Value		$ 7,444	7,432
Collateral Carrying Value		$ 9,340	$ 9,340
Weighted Average Note Rate (as a percent)		4.70%	4.37%
Maximum borrowing capacity		$ 7,500	$ 7,500
Structured Business | $7.5 million credit facility | Minimum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.75%
Structured Business | Repurchase facility
Debt Obligations
Debt Carrying Value		$ 60,390	$ 53,938
Weighted Average Note Rate (as a percent)		4.84%	4.45%
Structured Business | Repurchase facility | Minimum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		2.50%
Structured Business | Repurchase facility | Maximum | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		3.50%
Structured Business | $3 million master security agreement
Debt Obligations
Debt Carrying Value		$ 1,670	$ 1,834
Weighted Average Note Rate (as a percent)		3.20%	3.21%
Maximum borrowing capacity		$ 3,000	$ 3,000
Structured Business | $3 million master security agreement | Minimum
Debt Obligations
Variable rate, spread (as a percent)		2.96%
Structured Business | $3 million master security agreement | Maximum
Debt Obligations
Variable rate, spread (as a percent)		3.42%
Structured Business | $2.2 million master security agreement
Debt Obligations
Debt Carrying Value		$ 1,390
Weighted Average Note Rate (as a percent)		4.66%
Variable rate, spread (as a percent)		4.60%
Maximum borrowing capacity		$ 2,200	2,200
Structured Business | $225 million repurchase facility
Debt Obligations
Variable rate, spread (as a percent)		0.50%
Maximum borrowing capacity		$ 300,000	225,000
Additional borrowing capacity		75,000
Agency Business
Debt Obligations
Debt Carrying Value		281,338	291,536
Collateral Carrying Value		$ 281,621	$ 291,716
Weighted Average Note Rate (as a percent)		3.16%	2.78%
Variable rate, spread (as a percent)		0.35%	0.35%
Unamortized deferred finance costs		$ 300	$ 200
Agency Business | $100 million repurchase facility
Debt Obligations
Debt Carrying Value		19,961	24,827
Collateral Carrying Value		$ 19,990	$ 24,873
Weighted Average Note Rate (as a percent)		3.23%	2.91%
Agency Business | $100 million repurchase facility | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		1.35%
Agency Business | $500 million multifamily ASAP agreement
Debt Obligations
Debt Carrying Value		$ 40,470	$ 121,880
Collateral Carrying Value		$ 40,470	$ 121,880
Weighted Average Note Rate (as a percent)		2.93%	2.61%
Maximum borrowing capacity		$ 500,000	$ 500,000
Agency Business | $500 million multifamily ASAP agreement | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		1.05%
Agency Business | $150 million credit facility - one
Debt Obligations
Debt Carrying Value		$ 139,763	21,802
Collateral Carrying Value		$ 139,955	$ 21,821
Weighted Average Note Rate (as a percent)		3.18%	2.96%
Maximum borrowing capacity	$ 150,000	$ 150,000	$ 150,000
Decrease in interest rate (as a percent)	0.10%
Agency Business | $150 million credit facility - one | LIBOR
Debt Obligations
Variable rate, spread (as a percent)	1.35%	1.30%
Agency Business | $150 million credit facility - two
Debt Obligations
Debt Carrying Value		$ 76,144	99,242
Collateral Carrying Value		$ 76,206	$ 99,357
Weighted Average Note Rate (as a percent)		3.23%	2.91%
Maximum borrowing capacity		$ 150,000	$ 150,000
Agency Business | $150 million credit facility - two | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		1.35%
Agency Business | $100 million credit facility - two
Debt Obligations
Debt Carrying Value		$ 5,000	23,785
Collateral Carrying Value		$ 5,000	$ 23,785
Weighted Average Note Rate (as a percent)		3.18%	2.86%
Maximum borrowing capacity	$ 250,000	$ 100,000	$ 100,000
Agency Business | $100 million credit facility - two | LIBOR
Debt Obligations
Variable rate, spread (as a percent)		1.30%