Loans and Allowance for Losses and Concentrations of Credit Risk - Certain Loans Acquired in Transfer (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2017 USD ($) defaulted_loan	Dec. 31, 2016 USD ($) defaulted_loan	Dec. 31, 2015 USD ($) defaulted_loan
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities Acquired During Period [Line Items]
Certain Loans Acquired In Transfer Not Accounted For As Debt Securities, Number Defaulted | defaulted_loan	13	8	6
Payments to Acquire Other Loans and Leases Held-for-investment	$ 5,981	$ 2,516	$ 16,907
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Acquired During Period, Contractually Required Payments Receivable at Acquisition, During the Period	6,018	2,544	17,036
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Provision for Loan Losses	60	208	3,772
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Allowance for Loan Losses, Decreases	171	67	1,019
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Outstanding Balance	18,866	14,669
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Carrying Amount, Net	17,691	13,069
Long-Term Standby Purchase Commitments [Member]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities Acquired During Period [Line Items]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, UPB at acquisition	311	398	13,500
Farmer Mac Guaranteed Securities [Member]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities Acquired During Period [Line Items]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, UPB at acquisition	$ 5,670	$ 2,118	$ 3,407