Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2021	Mar. 31, 2021	Jun. 30, 2020	Mar. 31, 2020	Jun. 30, 2021		Jun. 30, 2020		Dec. 31, 2020
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	$ 7,085		$ 10,132		$ 13,070		$ 30,381
Interest expense	449		3,425		561		15,716
Net interest income (expense)	6,636		6,707		12,509		14,665
Servicing fee income	13,748		18,032		27,288		37,551
Servicing costs	4,072		6,594		7,154		12,434
Net servicing income	9,676		11,438		20,134		25,117
Other income (expense)	(26,080)		(30,712)		(14,317)		(108,618)
Other operating expenses	3,675		3,394		7,078		8,115
Provision for (Benefit from) corporate business taxes	(1,830)		(3,278)		1,633	[1]	(15,432)	[1]
Net Income (Loss)	(11,613)	$ 21,228	(12,683)	$ (48,836)	9,615		(61,519)
Investments	1,210,794				1,210,794				$ 1,402,665
Other assets	123,962				123,962				190,823
Total Assets	1,334,756				1,334,756				1,593,488
Debt	1,032,025				1,032,025				1,261,357
Other liabilities	15,683				15,683				19,565
Total Liabilities	1,047,708				1,047,708				1,280,922
Book value	287,048				287,048				312,566
Servicing Related Assets [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	105		479		225		2,120
Interest expense	(771)		438		(1,959)		2,147
Net interest income (expense)	876		41		2,184		(27)
Servicing fee income	13,748		18,032		27,288		37,551
Servicing costs	4,072		6,594		7,154		12,434
Net servicing income	9,676		11,438		20,134		25,117
Other income (expense)	(12,995)		(24,746)		(19,880)		(56,602)
Other operating expenses	846		1,135		1,408		1,735
Provision for (Benefit from) corporate business taxes	(1,830)		(3,278)		1,633		(15,432)
Net Income (Loss)	(1,459)		(11,124)		(603)		(17,815)
Investments	211,995				211,995				174,414
Other assets	48,668				48,668				51,063
Total Assets	260,663				260,663				225,477
Debt	134,978				134,978				111,379
Other liabilities	2,575				2,575				2,392
Total Liabilities	137,553				137,553				113,771
Book value	123,110				123,110				111,706
RMBS [Member] | Operating Segments [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	6,980		9,653		12,845		28,261
Interest expense	1,220		2,987		2,520		13,569
Net interest income (expense)	5,760		6,666		10,325		14,692
Servicing fee income	0		0		0		0
Servicing costs	0		0		0		0
Net servicing income	0		0		0		0
Other income (expense)	(13,085)		(5,966)		5,563		(52,016)
Other operating expenses	0		0		0		0
Provision for (Benefit from) corporate business taxes	0		0		0		0
Net Income (Loss)	(7,325)		700		15,888		(37,324)
Investments	998,799				998,799				1,228,251
Other assets	20,638				20,638				55,260
Total Assets	1,019,437				1,019,437				1,283,511
Debt	897,047				897,047				1,149,978
Other liabilities	837				837				6,370
Total Liabilities	897,884				897,884				1,156,348
Book value	121,553				121,553				127,163
All Other [Member]
Segment Reporting Profit (Loss) and Other Information [Abstract]
Interest income	0		0		0		0
Interest expense	0		0		0		0
Net interest income (expense)	0		0		0		0
Servicing fee income	0		0		0		0
Servicing costs	0		0		0		0
Net servicing income	0		0		0		0
Other income (expense)	0		0		0		0
Other operating expenses	2,829		2,259		5,670		6,380
Provision for (Benefit from) corporate business taxes	0		0		0		0
Net Income (Loss)	(2,829)		$ (2,259)		(5,670)		$ (6,380)
Investments	0				0				0
Other assets	54,656				54,656				84,500
Total Assets	54,656				54,656				84,500
Debt	0				0				0
Other liabilities	12,271				12,271				10,803
Total Liabilities	12,271				12,271				10,803
Book value	$ 42,385				$ 42,385				$ 73,697

[1] The provision for income taxes is recorded at the TRS level.