Segment Reporting (Details) - USD ($) $ in Thousands		3 Months Ended
		Mar. 31, 2022	Mar. 31, 2021	Dec. 31, 2021
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		$ 5,519	$ 3,301
Interest expense		1,640	1,454
Net interest income (expense)		3,879	1,847
Servicing fee income		13,116	13,540
Servicing costs		3,193	3,082
Net servicing income		9,923	10,458
Other income (expense)		22,339	15,964
Other operating expenses		3,537	3,578
Provision for corporate business taxes	[1]	3,875	3,463
Net Income		28,729	21,228
Investments		1,020,216		$ 1,172,223
Other assets		191,804		130,639
Total Assets		1,212,020		1,302,862
Debt		923,953		1,010,762
Other liabilities		28,416		13,284
Total Liabilities		952,369		1,024,046
Net assets		259,651		278,816
Servicing Related Assets [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		0	120
Interest expense		1,253	932
Net interest income (expense)		(1,253)	(812)
Servicing fee income		13,116	13,540
Servicing costs		3,193	3,082
Net servicing income		9,923	10,458
Other income (expense)		(3,366)	(4,762)
Other operating expenses		522	566
Provision for corporate business taxes		3,875	3,463
Net Income		907	855
Investments		246,103		218,727
Other assets		36,101		44,506
Total Assets		282,204		263,233
Debt		159,068		145,268
Other liabilities		7,308		1,847
Total Liabilities		166,376		147,115
Net assets		115,828		116,118
RMBS [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		5,519	3,181
Interest expense		387	522
Net interest income (expense)		5,132	2,659
Servicing fee income		0	0
Servicing costs		0	0
Net servicing income		0	0
Other income (expense)		25,705	20,726
Other operating expenses		228	171
Provision for corporate business taxes		0	0
Net Income		30,609	23,214
Investments		774,113		953,496
Other assets		102,837		21,611
Total Assets		876,950		975,107
Debt		764,885		865,494
Other liabilities		9,371		1,411
Total Liabilities		774,256		866,905
Net assets		102,694		108,202
All Other [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		0	0
Interest expense		0	0
Net interest income (expense)		0	0
Servicing fee income		0	0
Servicing costs		0	0
Net servicing income		0	0
Other income (expense)		0	0
Other operating expenses		2,787	2,841
Provision for corporate business taxes		0	0
Net Income		(2,787)	$ (2,841)
Investments		0		0
Other assets		52,866		64,522
Total Assets		52,866		64,522
Debt		0		0
Other liabilities		11,737		10,026
Total Liabilities		11,737		10,026
Net assets		$ 41,129		$ 54,496

[1] The provision for income taxes is recorded at the TRS level.