Transactions with Affiliates and Affiliated Entities (Details)		3 Months Ended								12 Months Ended
	Jan. 31, 2017 USD ($)	Dec. 31, 2019 USD ($)	Sep. 30, 2019 USD ($)	Jun. 30, 2019 USD ($)	Mar. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)	Sep. 30, 2018 USD ($)	Jun. 30, 2018 USD ($)	Mar. 31, 2018 USD ($)	Dec. 31, 2019 USD ($) Loan Employee	Dec. 31, 2018 USD ($) Loan	Dec. 31, 2017 USD ($)
Related Party Transactions [Abstract]
Percentage of annual management fee paid equal to gross equity										1.50%
Renew of management agreement subject to termination										1 year
Management agreement subject to termination, notice period for termination to manager										30 days
Management agreement subject to termination, period of notice by manager in the event of default										60 days
Management agreement subject to termination, period of termination fee payment in the event of default										30 days
Subservicing agreement term										3 years
Losses on loans repurchased										$ 382,000	$ 0
Foreclosure related losses not reimbursable										56,000
Management fees										6,832,000	5,088,000	$ 3,583,000
Compensation reimbursement										952,000	858,000	764,000
Total		$ 1,999,000	$ 2,042,000	$ 1,934,000	$ 1,809,000	$ 1,649,000	$ 1,599,000	$ 1,383,000	$ 1,315,000	$ 7,784,000	$ 5,946,000	$ 4,347,000
Minimum [Member]
Related Party Transactions [Abstract]
Management agreement subject to non-renewal, notice period										180 days
Maximum [Member]
Related Party Transactions [Abstract]
Management agreement subject to non-renewal, notice period										270 days
Freedom Mortgage Excess Service Right [Member]
Related Party Transactions [Abstract]
Joint marketing recapture agreement initial term										1 year
Joint marketing recapture agreement automatic renewals term										1 year
Number of MSRs loans | Loan										19	98
Aggregate unpaid principal balance										$ 4,400,000	$ 21,500,000
Amount due to affiliated entity		5,200				32,000				$ 5,200	32,000
Number of employees leases from mortgage | Employee										5
Remaining holdback amount		757,000				$ 757,000				$ 757,000	$ 757,000
Freedom Mortgage Excess Service Right [Member] | Receivable and Other Assets [Member]
Related Party Transactions [Abstract]
Foreclosure related losses recoverable		$ 326,000								$ 326,000
Ginnie Mae MSRs [Member] | Aurora Financial Group, Inc. [Member]
Related Party Transactions [Abstract]
Aggregate unpaid principal balance	$ 4,500,000,000
Weighted average servicing fee	0.30%