Transactions with Affiliates and Affiliated Entities (Details)			3 Months Ended								12 Months Ended
	Jan. 31, 2017 USD ($)	Nov. 15, 2016 USD ($)	Dec. 31, 2018 USD ($)	Sep. 30, 2018 USD ($)	Jun. 30, 2018 USD ($)	Mar. 31, 2018 USD ($)	Dec. 31, 2017 USD ($)	Sep. 30, 2017 USD ($)	Jun. 30, 2017 USD ($)	Mar. 31, 2017 USD ($)	Dec. 31, 2018 USD ($) Loan Employee	Dec. 31, 2017 USD ($) Loan	Dec. 31, 2016 USD ($)
Related Party Transactions [Abstract]
Renew of management agreement subject to termination											1 year
Management agreement subject to termination, notice period for termination to manager											30 days
Management agreement subject to termination, period of notice by manager in the event of default											60 days
Management agreement subject to termination, period of termination fee payment in the event of default											30 days
Percentage of annual management fee paid equal to gross equity											1.50%
Management fees											$ 5,088,000	$ 3,583,000	$ 2,303,000
Compensation reimbursement											858,000	764,000	643,000
Total			$ 1,649,000	$ 1,599,000	$ 1,383,000	$ 1,315,000	$ 1,345,000	$ 948,000	$ 1,162,000	$ 892,000	$ 5,946,000	$ 4,347,000	$ 2,946,000
Minimum [Member]
Related Party Transactions [Abstract]
Management agreement subject to non-renewal, notice period											180 days
Maximum [Member]
Related Party Transactions [Abstract]
Management agreement subject to non-renewal, notice period											270 days
Freedom Mortgage Excess Service Right [Member]
Related Party Transactions [Abstract]
Joint marketing recapture agreement initial term											1 year
Joint marketing recapture agreement automatic renewals term											1 year
Number of MSRs loans | Loan											98	116
Aggregate unpaid principal balance											$ 21,500,000	$ 27,600,000
Amount due to affiliated entity			32,000				43,000				$ 32,000	43,000
Cash proceeds from sale of mortgage loans		$ 38,000,000
Period of monthly yield maintenance payment											12 months
Yield maintenance payment			3,000,000								$ 3,000,000
Number of employees leases from mortgage | Employee											4
Remaining holdback amount			$ 757,000				$ 1,000,000				$ 757,000	$ 1,000,000
Freedom Mortgage Excess Service Right [Member] | NexBank Term Loan [Member]
Related Party Transactions [Abstract]
Repayment of outstanding borrowings		12,000,000
Outstanding borrowings		$ 25,000,000
Ginnie Mae MSRs [Member] | Aurora Financial Group Inc [Member]
Related Party Transactions [Abstract]
Weighted average servicing fee	0.30%
Aggregate unpaid principal balance	$ 4,500,000,000