Collateralized Borrowings (Details) - USD ($)	Sep. 30, 2020	Dec. 31, 2019
Line of Credit Facility [Line Items]
Repurchase agreements	$ 1,850,845,000	$ 1,924,021,000
Asset-specific financings	123,091,000	116,465,000
Revolving credit facilities	0	42,008,000
Unused capacity under collateralized borrowings	428,261,000	320,236,000
Total capacity under collateralized borrowings	2,279,106,000	2,225,000,000
Carrying value of assets pledged as collateral for repurchase agreements	2,549,281,000	2,585,896,000
Loans held-for-investment pledged as collateral for borrowings	3,900,000,000	4,100,000,000
Citibank
Line of Credit Facility [Line Items]
Revolving credit facilities		42,008,000
Unused capacity under collateralized borrowings		32,992,000
Total capacity under collateralized borrowings		75,000,000
Citibank | Revolving Credit Facility
Line of Credit Facility [Line Items]
Loans held-for-investment pledged as collateral for borrowings		$ 80,473,000
Weighted average borrowing rate of facility		4.00%
Canadian Imperial Bank of Commerce
Line of Credit Facility [Line Items]
Asset-specific financings	123,091,000	$ 116,465,000
Unused capacity under collateralized borrowings	26,909,000	33,535,000
Total capacity under collateralized borrowings	150,000,000	150,000,000
Canadian Imperial Bank of Commerce | Secured Debt
Line of Credit Facility [Line Items]
Loans held-for-investment pledged as collateral for borrowings	$ 152,262,000	$ 144,322,000
Weighted average borrowing rate of facility	2.50%	3.50%
Loans Held-for-Investment | Morgan Stanley Bank
Line of Credit Facility [Line Items]
Repurchase agreements	$ 498,181,000	$ 556,887,000
Unused capacity under collateralized borrowings	101,819,000	43,113,000
Total capacity under collateralized borrowings	600,000,000	600,000,000
Carrying value of assets pledged as collateral for repurchase agreements	$ 703,898,000	$ 740,791,000
Weighted average borrowing rate of repurchase agreements	2.30%	3.90%
Loans Held-for-Investment | Goldman Sachs Bank
Line of Credit Facility [Line Items]
Repurchase agreements	$ 405,971,000	$ 405,057,000
Unused capacity under collateralized borrowings	94,029,000	94,943,000
Total capacity under collateralized borrowings	500,000,000	500,000,000
Carrying value of assets pledged as collateral for repurchase agreements	$ 582,376,000	$ 541,640,000
Weighted average borrowing rate of repurchase agreements	2.50%	3.80%
Loans Held-for-Investment | JPMorgan Chase Bank
Line of Credit Facility [Line Items]
Repurchase agreements	$ 400,880,000	$ 408,819,000
Unused capacity under collateralized borrowings	103,226,000	41,181,000
Total capacity under collateralized borrowings	504,106,000	450,000,000
Carrying value of assets pledged as collateral for repurchase agreements	$ 550,025,000	$ 553,020,000
Weighted average borrowing rate of repurchase agreements	2.80%	3.70%
Loans Held-for-Investment | Citibank
Line of Credit Facility [Line Items]
Repurchase agreements	$ 390,767,000	$ 339,888,000
Unused capacity under collateralized borrowings	9,233,000	60,112,000
Total capacity under collateralized borrowings	400,000,000	400,000,000
Carrying value of assets pledged as collateral for repurchase agreements	$ 492,830,000	$ 432,867,000
Weighted average borrowing rate of repurchase agreements	1.80%	3.40%
Loans Held-for-Investment | Wells Fargo Bank
Line of Credit Facility [Line Items]
Repurchase agreements	$ 155,046,000	$ 194,113,000
Unused capacity under collateralized borrowings	119,954,000	80,887,000
Total capacity under collateralized borrowings	275,000,000	275,000,000
Carrying value of assets pledged as collateral for repurchase agreements	$ 220,152,000	$ 286,672,000
Weighted average borrowing rate of repurchase agreements	1.90%	3.50%
Maximum capacity under collateralized borrowings upon exercise of option to increase	$ 350,000,000
Commercial Mortgage Backed Securities
Line of Credit Facility [Line Items]
Asset-specific financings	$ 123,091,000
Commercial Mortgage Backed Securities | JPMorgan Chase Bank
Line of Credit Facility [Line Items]
Repurchase agreements		$ 19,257,000
Carrying value of assets pledged as collateral for repurchase agreements		$ 30,906,000
Weighted average borrowing rate of repurchase agreements		4.10%