Segment Reporting (Details) - USD ($) $ in Thousands		3 Months Ended				6 Months Ended
		Jun. 30, 2022	Mar. 31, 2022	Jun. 30, 2021	Mar. 31, 2021	Jun. 30, 2022	Jun. 30, 2021	Dec. 31, 2021
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		$ 6,004		$ 3,526		$ 11,523	$ 6,827
Interest expense		2,502		1,341		4,142	2,795
Net interest income (expense)		3,502		2,185		7,381	4,032
Servicing fee income		13,188		13,748		26,304	27,288
Servicing costs		2,615		4,072		5,808	7,154
Net servicing income		10,573		9,676		20,496	20,134
Other income (expense)		(25,003)		(21,472)		(2,664)	(5,508)
Other operating expenses		3,113		3,832		6,650	7,410
Provision for (Benefit from) corporate business taxes	[1]	1,423		(1,830)		5,298	1,633
Net Income (Loss)		(15,464)	$ 28,729	(11,613)	$ 21,228	13,265	9,615
Investments		992,403				992,403		$ 1,172,223
Other assets		207,014				207,014		130,639
Total Assets		1,199,417				1,199,417		1,302,862
Debt		861,481				861,481		1,010,762
Other liabilities		82,852				82,852		13,284
Total Liabilities		944,333				944,333		1,024,046
Net assets		255,084				255,084		278,816
Servicing Related Assets [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		0		106		0	226
Interest expense		1,131		1,074		2,384	2,006
Net interest income (expense)		(1,131)		(968)		(2,384)	(1,780)
Servicing fee income		13,188		13,748		26,304	27,288
Servicing costs		2,615		4,072		5,808	7,154
Net servicing income		10,573		9,676		20,496	20,134
Other income (expense)		(5,530)		(11,147)		(8,896)	(15,909)
Other operating expenses		510		846		1,032	1,412
Provision for (Benefit from) corporate business taxes		1,423		(1,830)		5,298	1,633
Net Income (Loss)		1,979		(1,455)		2,886	(600)
Investments		263,578				263,578		218,727
Other assets		30,284				30,284		44,506
Total Assets		293,862				293,862		263,233
Debt		178,308				178,308		145,268
Other liabilities		14,268				14,268		1,847
Total Liabilities		192,576				192,576		147,115
Net assets		101,286				101,286		116,118
RMBS [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		6,004		3,420		11,523	6,601
Interest expense		1,371		267		1,758	789
Net interest income (expense)		4,633		3,153		9,765	5,812
Servicing fee income		0		0		0	0
Servicing costs		0		0		0	0
Net servicing income		0		0		0	0
Other income (expense)		(19,473)		(10,325)		6,232	10,401
Other operating expenses		151		158		379	329
Provision for (Benefit from) corporate business taxes		0		0		0	0
Net Income (Loss)		(14,991)		(7,330)		15,618	15,884
Investments		728,825				728,825		953,496
Other assets		114,913				114,913		21,611
Total Assets		843,738				843,738		975,107
Debt		683,173				683,173		865,494
Other liabilities		57,817				57,817		1,411
Total Liabilities		740,990				740,990		866,905
Net assets		102,748				102,748		108,202
All Other [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income		0		0		0	0
Interest expense		0		0		0	0
Net interest income (expense)		0		0		0	0
Servicing fee income		0		0		0	0
Servicing costs		0		0		0	0
Net servicing income		0		0		0	0
Other income (expense)		0		0		0	0
Other operating expenses		2,452		2,828		5,239	5,669
Provision for (Benefit from) corporate business taxes		0		0		0	0
Net Income (Loss)		(2,452)		$ (2,828)		(5,239)	$ (5,669)
Investments		0				0		0
Other assets		61,817				61,817		64,522
Total Assets		61,817				61,817		64,522
Debt		0				0		0
Other liabilities		10,767				10,767		10,026
Total Liabilities		10,767				10,767		10,026
Net assets		$ 51,050				$ 51,050		$ 54,496

[1] The provision for income taxes is recorded at the TRS level.