Fair Value of Financial Instruments - Fair Value (Details) - USD ($) $ in Thousands	Jun. 30, 2022	Dec. 31, 2021
Assets
Derivative assets	$ 2,476	$ 1,163
Liabilities
Derivative liabilities	1,872	602
Fair Value, Measurements, Recurring
Assets
Derivative assets	1,748	105
Total Assets	2,741,101	2,701,362
Liabilities
Derivative liabilities	1,872	602
Securitized debt	1,574,468	1,344,370
Total Liabilities	1,576,340	1,344,972
Residential Whole Loans
Assets
Fair value	1,195,853	1,023,502
Residential Whole Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	1,195,853	1,023,502
Residential Bridge Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	5,095	5,428
Securitized commercial loan \| Fair Value, Measurements, Recurring
Assets
Fair value	1,243,371	1,355,808
Commercial Loans
Assets
Fair value	128,421
Commercial Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	128,421	130,572
Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	57	114
Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	57	114
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS
Assets
Estimated Fair Value	728	1,058
Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	728	1,058
Subtotal Agency MBS
Assets
Estimated Fair Value	785	1,172
Subtotal Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	785	1,172
Non-Agency CMBS
Assets
Estimated Fair Value	93,096	105,358
Non-Agency CMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	93,096	105,358
Non-Agency RMBS
Assets
Estimated Fair Value	31,017	25,652
Non-Agency RMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	31,017	25,652
Non-Agency RMBS Interest-Only Strips
Assets
Estimated Fair Value	1,181	2,117
Non-Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	1,181	2,117
Subtotal Non-Agency MBS
Assets
Estimated Fair Value	125,294	133,127
Subtotal Non-Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	125,294	133,127
Other securities
Assets
Estimated Fair Value	40,534	51,648
Other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	40,534	51,648
Total mortgage-backed securities and other securities
Assets
Estimated Fair Value	166,613	185,947
Total mortgage-backed securities and other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	166,613	185,947
Level I \| Fair Value, Measurements, Recurring
Assets
Derivative assets	0	0
Total Assets	0	0
Liabilities
Derivative liabilities	0	0
Securitized debt	0	0
Total Liabilities	0	0
Level I \| Residential Whole Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level I \| Residential Bridge Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level I \| Securitized commercial loan \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level I \| Commercial Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level I \| Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Subtotal Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Non-Agency CMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Non-Agency RMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Non-Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Subtotal Non-Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level I \| Total mortgage-backed securities and other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level II \| Fair Value, Measurements, Recurring
Assets
Derivative assets	1,748	105
Total Assets	166,395	177,035
Liabilities
Derivative liabilities	1,872	602
Securitized debt	1,559,549	1,329,451
Total Liabilities	1,561,421	1,330,053
Level II \| Residential Whole Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level II \| Residential Bridge Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level II \| Securitized commercial loan \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level II \| Commercial Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	0	0
Level II \| Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level II \| Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level II \| Subtotal Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level II \| Non-Agency CMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	93,096	99,630
Level II \| Non-Agency RMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	31,017	25,652
Level II \| Non-Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level II \| Subtotal Non-Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	124,113	125,282
Level II \| Other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	40,534	51,648
Level II \| Total mortgage-backed securities and other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	164,647	176,930
Level III \| Fair Value, Measurements, Recurring
Assets
Derivative assets	0	0
Total Assets	2,574,706	2,524,327
Liabilities
Derivative liabilities	0	0
Securitized debt	14,919	14,919
Total Liabilities	14,919	14,919
Level III \| Residential Whole Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	1,195,853	1,023,502
Level III \| Residential Bridge Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	5,095	5,428
Level III \| Securitized commercial loan \| Fair Value, Measurements, Recurring
Assets
Fair value	1,243,371	1,355,808
Level III \| Commercial Loans \| Fair Value, Measurements, Recurring
Assets
Fair value	128,421	130,572
Level III \| Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	57	114
Level III \| Agency RMBS Interest-Only Strips accounted for as derivatives, included in MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	728	1,058
Level III \| Subtotal Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	785	1,172
Level III \| Non-Agency CMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	5,728
Level III \| Non-Agency RMBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level III \| Non-Agency RMBS Interest-Only Strips \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	1,181	2,117
Level III \| Subtotal Non-Agency MBS \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	1,181	7,845
Level III \| Other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	0	0
Level III \| Total mortgage-backed securities and other securities \| Fair Value, Measurements, Recurring
Assets
Estimated Fair Value	$ 1,966	$ 9,017

Fair value portion of probable future economic benefits obtained or controlled by an entity as a result of past transactions or events.

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 820
-SubTopic 10
-Section 50
-Paragraph 2
-Subparagraph (a)
-URI https://asc.fasb.org/extlink&oid=126976982&loc=d3e19207-110258

+ Details

Name:	us-gaap_AssetsFairValueDisclosure
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- References

No definition available.

+ Details

Name:	us-gaap_AssetsFairValueDisclosureAbstract
Namespace Prefix:	us-gaap_
Data Type:	xbrli:stringItemType
Balance Type:	na
Period Type:	duration

- Definition

Fair value portion of debt instrument payable, including, but not limited to, notes payable and loans payable.

+ References

Reference 1: http://www.xbrl.org/2003/role/exampleRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 470
-SubTopic 20
-Section 55
-Paragraph 69C
-URI https://asc.fasb.org/extlink&oid=123466577&loc=SL123495737-112612

Reference 2: http://www.xbrl.org/2003/role/exampleRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 470
-SubTopic 20
-Section 55
-Paragraph 69B
-URI https://asc.fasb.org/extlink&oid=123466577&loc=SL123495735-112612

Reference 3: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 825
-SubTopic 10
-Section 50
-Paragraph 10
-URI https://asc.fasb.org/extlink&oid=123594938&loc=d3e13433-108611

Reference 4: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 470
-SubTopic 20
-Section 50
-Paragraph 1D
-Subparagraph (c)
-URI https://asc.fasb.org/extlink&oid=123466505&loc=SL123495340-112611

+ Details

Name:	us-gaap_DebtInstrumentFairValue
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	credit
Period Type:	instant

- Definition

Fair value, after the effects of master netting arrangements, of a financial asset or other contract with one or more underlyings, notional amount or payment provision or both, and the contract can be net settled by means outside the contract or delivery of an asset. Includes assets not subject to a master netting arrangement and not elected to be offset.

+ References

Reference 1: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 10
-URI https://asc.fasb.org/extlink&oid=99393222&loc=SL20226008-175313

Reference 2: http://www.xbrl.org/2003/role/exampleRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 22
-URI https://asc.fasb.org/extlink&oid=99393222&loc=SL20226052-175313

Reference 3: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 825
-SubTopic 10
-Section 50
-Paragraph 10
-URI https://asc.fasb.org/extlink&oid=123594938&loc=d3e13433-108611

Reference 4: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 50
-Paragraph 3
-Subparagraph (c)
-URI https://asc.fasb.org/extlink&oid=51824906&loc=SL20225862-175312

+ Details

Name:	us-gaap_DerivativeAssets
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Fair value, after the effects of master netting arrangements, of a financial liability or contract with one or more underlyings, notional amount or payment provision or both, and the contract can be net settled by means outside the contract or delivery of an asset. Includes liabilities not subject to a master netting arrangement and not elected to be offset.

+ References

Reference 1: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 50
-Paragraph 3
-Subparagraph (c)
-URI https://asc.fasb.org/extlink&oid=51824906&loc=SL20225862-175312

Reference 2: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 10
-URI https://asc.fasb.org/extlink&oid=99393222&loc=SL20226008-175313

Reference 3: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 825
-SubTopic 10
-Section 50
-Paragraph 10
-URI https://asc.fasb.org/extlink&oid=123594938&loc=d3e13433-108611

Reference 4: http://www.xbrl.org/2003/role/exampleRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 20
-Section 55
-Paragraph 22
-URI https://asc.fasb.org/extlink&oid=99393222&loc=SL20226052-175313

+ Details

Name:	us-gaap_DerivativeLiabilities
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	credit
Period Type:	instant

- Definition

Fair value of financial and nonfinancial obligations.

+ References

+ Details

Name:	us-gaap_LiabilitiesFairValueDisclosure
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	credit
Period Type:	instant

- References

No definition available.

+ Details

Name:	us-gaap_LiabilitiesFairValueDisclosureAbstract
Namespace Prefix:	us-gaap_
Data Type:	xbrli:stringItemType
Balance Type:	na
Period Type:	duration

- Definition

Fair value portion of loan receivable, including, but not limited to, mortgage loans held for investment, finance receivables held for investment, policy loans on insurance contracts.

+ References

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-URI https://asc.fasb.org/subtopic&trid=2196772

+ Details

Name:	us-gaap_LoansReceivableFairValueDisclosure
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Fair value of securitized, pay-through debt securities collateralized by real estate mortgage loans classified as available-for-sale.

+ References

No definition available.

+ Details

Name:	us-gaap_MortgageBackedSecuritiesAvailableForSaleFairValueDisclosure
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant