Condensed Quarterly Financial Data (Unaudited) (Details) - USD ($)	3 Months Ended								12 Months Ended
	Dec. 31, 2021	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2021		Dec. 31, 2020		Dec. 31, 2019
Text Block [Abstract]
Interest income	$ 49,390,000	$ 40,146,000	$ 45,890,000	$ 40,079,000	$ 39,067,000	$ 43,075,000	$ 39,281,000	$ 52,108,000	$ 175,505,000		$ 173,531,000		$ 159,901,000
Interest expense	(10,918,000)	(10,604,000)	(11,166,000)	(11,342,000)	(11,952,000)	(12,937,000)	(14,686,000)	(22,090,000)	(44,030,000)		(61,665,000)		(78,479,000)
Interest Income (Expense), Net	38,472,000	29,542,000	34,724,000	28,737,000	27,115,000	30,138,000	24,595,000	30,018,000	131,475,000		111,866,000		81,422,000
Realized gains (losses) on securities and loans, financial derivatives, and real estate owned, net	5,229,000	4,527,000	(1,658,000)	10,132,000	(9,551,000)	(192,000)	(27,927,000)	204,000
Unrealized gains (losses) on securities and loans, financial derivatives, and real estate owned, net	(31,462,000)	(1,385,000)	3,003,000	8,138,000	42,547,000	24,032,000	52,057,000	(144,079,000)
Base management fee to affiliate (Net of fee rebates)	3,115,000	3,675,000	3,355,000	3,277,000	3,178,000	2,981,000	2,906,000	2,443,000	13,422,000	[1]	11,508,000	[1]	7,988,000
Incentive Fee Expense	3,246,000	5,255,000	7,157,000	0	0	0	0	0	15,658,000		0		116,000
Servicing expense	1,280,000	1,182,000	974,000	986,000	1,736,000	2,379,000	2,493,000	2,531,000	4,422,000		9,139,000		8,632,000
Debt issuance costs related to Other secured borrowings, at fair value	1,586,000	0	2,039,000	1,665,000	1,819,000	0	2,075,000	0	5,290,000		3,894,000		3,536,000
Other	3,113,000	1,697,000	1,818,000	2,204,000	1,782,000	1,199,000	707,000	1,423,000	8,832,000		5,111,000		5,609,000
Professional fees	1,979,000	1,202,000	1,037,000	1,198,000	1,186,000	1,209,000	1,333,000	1,277,000	5,416,000		5,005,000		4,853,000
Compensation expense	1,357,000	1,554,000	1,412,000	1,420,000	962,000	1,085,000	941,000	788,000	5,743,000		3,776,000		3,649,000
Other expenses	1,764,000	1,682,000	1,633,000	1,579,000	1,531,000	1,625,000	1,497,000	1,752,000	6,658,000		6,405,000		4,354,000
Operating Expenses	17,440,000	16,247,000	19,425,000	12,329,000	12,194,000	10,478,000	11,952,000	10,214,000	65,441,000		44,838,000		38,737,000
Income (Loss) from Continuing Operations before Equity Method Investments, Income Taxes, Noncontrolling Interest	8,528,000	19,423,000	21,007,000	36,638,000	47,122,000	40,753,000	36,338,000	(122,392,000)	85,596,000		1,821,000		54,526,000
Income Tax Expense (Benefit)	(4,000)	(2,009,000)	3,140,000	2,017,000	7,888,000	2,494,000	1,542,000	(547,000)	3,144,000		11,377,000		1,558,000
Earnings (losses) from investments in unconsolidated entities	30,318,000	2,549,000	18,602,000	6,635,000	27,344,000	11,443,000	5,643,000	(6,497,000)	58,104,000		37,933,000		10,209,000
Net Income (Loss), Including Portion Attributable to Noncontrolling Interest	38,850,000	23,981,000	36,469,000	41,256,000	66,578,000	49,702,000	40,439,000	(128,342,000)	140,556,000		28,377,000		63,177,000
Net income (loss) attributable to non-controlling interests	2,284,000	1,476,000	1,874,000	1,459,000	1,475,000	1,559,000	1,220,000	(885,000)	7,093,000		3,369,000		5,244,000
Dividends on preferred stock	2,295,000	1,941,000	1,940,000	1,941,000	1,941,000	1,940,000	1,941,000	1,941,000	8,117,000		7,763,000		1,466,000
Net income (loss) attributable to common stockholders	$ 34,271,000	$ 20,564,000	$ 32,655,000	$ 37,856,000	$ 63,162,000	$ 46,203,000	$ 37,278,000	$ (129,398,000)	$ 125,346,000		$ 17,245,000		$ 56,467,000
Basic and Diluted (USD per share)	$ 0.61	$ 0.41	$ 0.75	$ 0.86	$ 1.44	$ 1.06	$ 0.85	$ (3.04)	$ 2.58		$ 0.39		$ 1.76
Other, net	$ 13,729,000	$ 2,986,000	$ 4,363,000	$ 1,960,000	$ (795,000)	$ (2,747,000)	$ (435,000)	$ 1,679,000	$ 23,038,000		$ (2,298,000)		$ 5,350,000
Noninterest Income	$ (12,504,000)	$ 6,128,000	$ 5,708,000	$ 20,230,000	$ 32,201,000	$ 21,093,000	$ 23,695,000	$ (142,196,000)	$ 19,562,000		$ (65,207,000)		$ 11,841,000

[1] See Note 13 for further details on management fee rebates.