Loans and Allowance for Losses and Concentrations of Credit Risk - Certain Loans Acquired in Transfer (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2016 USD ($) defaulted_loan	Dec. 31, 2015 USD ($) defaulted_loan	Dec. 31, 2014 USD ($) defaulted_loan
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities Acquired During Period [Line Items]
Certain Loans Acquired In Transfer Not Accounted For As Debt Securities, Number Defaulted | defaulted_loan	8	6	2
Payments to Acquire Other Loans and Leases Held-for-investment	$ 2,516	$ 16,907	$ 705
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Acquired During Period, Contractually Required Payments Receivable at Acquisition, During the Period	2,544	17,036	705
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Provision for Loan Losses	208	3,772	69
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Allowance for Loan Losses, Decreases	67	1,019	233
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Outstanding Balance	14,669	36,195
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, Carrying Amount, Net	13,069	34,015
Long-Term Standby Purchase Commitments [Member]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities Acquired During Period [Line Items]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, UPB at acquisition	2,118	13,500	705
Farmer Mac Guaranteed Securities [Member]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities Acquired During Period [Line Items]
Certain Loans Acquired in Transfer Not Accounted for as Debt Securities, UPB at acquisition	$ 398	$ 3,407	$ 0