Multifamily Loans - Product Types and Characteristics of Multifamily Loans (Details) $ in Thousands			12 Months Ended
	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of Loans | loan			669	1,484
Total Principal	$ 524,928	$ 1,034,284	$ 524,928	$ 1,034,284
30-89 Days DQ	1,650	559	1,650	559
90 Days DQ	$ 747	$ 747	$ 747	$ 747
Multifamily loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of Loans | loan	279	162	279	162
Total Principal	$ 4,195,000	$ 2,126,698	$ 4,195,000	$ 2,126,698
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	0	0
Multifamily loans | $1,000 to $10,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	1,000	1,000	1,000	1,000
Loan Balance, maximum	10,000	10,000	$ 10,000	$ 10,000
Number of Loans | loan			114	70
Total Principal	674,666	394,373	$ 674,666	$ 394,373
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	$ 0	$ 0
Multifamily loans | $1,000 to $10,000 | Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			3.29%	3.29%
Multifamily loans | $1,000 to $10,000 | Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.73%	4.73%
Multifamily loans | $10,001 to $20,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	10,001	10,001	$ 10,001	$ 10,001
Loan Balance, maximum	20,000	20,000	$ 20,000	$ 20,000
Number of Loans | loan			102	66
Total Principal	1,489,118	960,992	$ 1,489,118	$ 960,992
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	$ 0	$ 0
Multifamily loans | $10,001 to $20,000 | Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			3.54%	3.54%
Multifamily loans | $10,001 to $20,000 | Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.94%	4.61%
Multifamily loans | $20,001 to $30,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	20,001	20,001	$ 20,001	$ 20,001
Loan Balance, maximum	30,000	30,000	$ 30,000	$ 30,000
Number of Loans | loan			32	16
Total Principal	750,712	373,036	$ 750,712	$ 373,036
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	$ 0	$ 0
Multifamily loans | $20,001 to $30,000 | Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			3.54%	3.65%
Multifamily loans | $20,001 to $30,000 | Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.69%	4.72%
Multifamily loans | $30,001 to $40,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	30,001	30,001	$ 30,001	$ 30,001
Loan Balance, maximum	40,000	40,000	$ 40,000	$ 40,000
Number of Loans | loan			19	7
Total Principal	654,729	244,074	$ 654,729	$ 244,074
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	$ 0	$ 0
Multifamily loans | $30,001 to $40,000 | Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			3.52%	3.62%
Multifamily loans | $30,001 to $40,000 | Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.79%	4.71%
Multifamily loans | Over $40,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, maximum	40,000	40,000	$ 40,000	$ 40,000
Number of Loans | loan			12	3
Total Principal	625,775	154,223	$ 625,775	$ 154,223
30-89 Days DQ	0	0	0	0
90 Days DQ	$ 0	$ 0	$ 0	$ 0
Multifamily loans | Over $40,000 | Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			3.55%	3.74%
Multifamily loans | Over $40,000 | Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.65%	4.18%