Transactions with Related Parties (Details)	3 Months Ended		6 Months Ended
	Jun. 30, 2021 USD ($) Loan	Jun. 30, 2020 USD ($) Loan	Jun. 30, 2021 USD ($) Employee Loan	Jun. 30, 2020 USD ($) Loan
Related Party Transactions [Abstract]
Percentage of annual management fee paid equal to gross equity			1.50%
Renew of management agreement subject to termination			1 year
Management agreement subject to termination, notice period for termination to manager			30 days
Management agreement subject to termination, period of notice by manager in the event of default			60 days
Management agreement subject to termination, period of termination fee payment in the event of default			30 days
Carrying value of MSR Portfolio sold		$ 15,700,000		$ 15,700,000
Proceeds from sale of MSRs			$ 0	15,831,000
Realized loss on investments in MSRs, net	$ 0	(11,347,000)	0	(11,347,000)
Unrealized losses on market value in MSRs				11,500,000
Management Fees and Compensation Reimbursement to Manager [Abstract]
Management fees	1,699,000	1,736,000	3,410,000	3,463,000
Compensation reimbursement	250,000	238,000	500,000	476,000
Total	$ 1,949,000	$ 1,974,000	$ 3,910,000	$ 3,939,000
Information about Loans Refinancing [Abstract]
Number of loans refinanced | Loan	2,173	691	4,926	1,443
Aggregate unpaid principal balance of refinanced loans	$ 547,290,000	$ 178,409,000	$ 1,240,012,000	$ 377,764,000
Minimum [Member]
Related Party Transactions [Abstract]
Management agreement subject to non-renewal, notice period			180 days
Maximum [Member]
Related Party Transactions [Abstract]
Management agreement subject to non-renewal, notice period			270 days
RoundPoint Mortgage Servicing Corporation [Member]
Related Party Transactions [Abstract]
Subservicing agreement term			2 years
Term Loan Facility [Member]
Related Party Transactions [Abstract]
Term loan facility	11,200,000		$ 11,200,000
Freedom Mortgage Excess Service Right [Member]
Related Party Transactions [Abstract]
Number of employees leases from mortgage | Employee			3
Remaining holdback amount	$ 757,000		$ 757,000