Secured Borrowings (Details) - USD ($) $ in Thousands	9 Months Ended
	Sep. 30, 2019	Dec. 31, 2018
Secured borrowings and promissory note
Pledged Assets Carrying Value	$ 1,707,541	$ 1,112,522
Carrying Value, Secured borrowings	1,315,534	834,547
Secured borrowings
Secured borrowings and promissory note
Maximum Facility Size	1,999,440
Pledged Assets Carrying Value	1,707,541	1,112,522
Carrying Value, Secured borrowings	1,315,534	834,547
Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	781,274
Pledged Assets Carrying Value	703,953	279,012
Carrying Value, Secured borrowings	$ 439,371	$ 199,315
Weighted average interest rate of borrowings (as a percent)	4.20%	4.60%
Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	$ 1,218,166
Pledged Assets Carrying Value	1,003,588	$ 833,510
Carrying Value, Secured borrowings	$ 876,163	$ 635,232
Weighted average interest rate of borrowings (as a percent)	4.30%	4.00%
Commercial | JPMorgan | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	$ 250,000
Pledged Assets Carrying Value	382,718	$ 85,393
Carrying Value, Secured borrowings	$ 170,461	68,417
Commercial | JPMorgan | Borrowings under credit facilities | Minimum | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.00%
Commercial | JPMorgan | Borrowings under credit facilities | Maximum | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
Commercial | JPMorgan | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	$ 200,000
Pledged Assets Carrying Value	43,850	144,337
Carrying Value, Secured borrowings	$ 134,036	96,343
Commercial | JPMorgan | Borrowings under repurchase agreements | Minimum | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.25%
Commercial | JPMorgan | Borrowings under repurchase agreements | Maximum | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	4.00%
Commercial | Keybank | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	$ 125,000
Pledged Assets Carrying Value	30,412	23,322
Carrying Value, Secured borrowings	$ 29,828	22,973
Commercial | Keybank | Borrowings under credit facilities | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.50%
Commercial | East West Bank | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	$ 50,000
Pledged Assets Carrying Value	65,057	37,255
Carrying Value, Secured borrowings	$ 47,885	27,443
Commercial | East West Bank | Borrowings under credit facilities | Minimum | Administrative agent's prime rate
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	(0.821%)
Commercial | East West Bank | Borrowings under credit facilities | Maximum | Administrative agent's prime rate
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	0.029%
Commercial | Rabobank | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.22%
Maximum Facility Size	$ 14,500
Pledged Assets Carrying Value	19,950
Carrying Value, Secured borrowings	$ 12,584
Commercial | FCB | Borrowings under credit facilities
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	2.75%
Maximum Facility Size	$ 1,774
Pledged Assets Carrying Value	1,822	3,219
Carrying Value, Secured borrowings	1,774	2,974
Commercial | Deutsche Bank | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	300,000
Pledged Assets Carrying Value	397,874	310,797
Carrying Value, Secured borrowings	$ 298,584	239,972
Commercial | Deutsche Bank | Borrowings under repurchase agreements | Minimum | Three Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.30%
Commercial | Deutsche Bank | Borrowings under repurchase agreements | Maximum | Three Month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.80%
Commercial | Citibank | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	$ 500,000
Pledged Assets Carrying Value	269,077	230,140
Carrying Value, Secured borrowings	$ 225,378	194,117
Commercial | Citibank | Borrowings under repurchase agreements | Minimum | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.875%
Commercial | Citibank | Borrowings under repurchase agreements | Maximum | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.125%
Residential | East West Bank | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	$ 50,000
Pledged Assets Carrying Value	56,513	63,479
Carrying Value, Secured borrowings	$ 35,800	8,500
Residential | East West Bank | Borrowings under credit facilities | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	2.50%
Residential | Comerica | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	$ 100,000
Pledged Assets Carrying Value	75,424	35,860
Carrying Value, Secured borrowings	$ 71,090	40,231
Residential | Comerica | Borrowings under credit facilities | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.75%
Residential | Associated Bank | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	$ 40,000
Pledged Assets Carrying Value	13,953	13,653
Carrying Value, Secured borrowings	$ 13,283	15,907
Residential | Associated Bank | Borrowings under credit facilities | One month LIBOR
Secured borrowings and promissory note
Pricing, spread on variable (as a percent)	1.50%
Residential | Origin Bank | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	$ 50,000
Pledged Assets Carrying Value	17,502	16,831
Carrying Value, Secured borrowings	16,733	12,870
Residential | TBK Bank | Borrowings under credit facilities
Secured borrowings and promissory note
Maximum Facility Size	100,000
Pledged Assets Carrying Value	40,602
Carrying Value, Secured borrowings	39,933
MBS | Borrowings under repurchase agreements
Secured borrowings and promissory note
Pledged Assets Carrying Value	92,189	112,552
MBS | JPMorgan | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	41,375
Pledged Assets Carrying Value	65,803	39,882
Carrying Value, Secured borrowings	$ 41,375	24,881
MBS | JPMorgan | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.54%
MBS | JPMorgan | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.84%
MBS | Deutsche Bank | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	$ 48,260
Pledged Assets Carrying Value	69,644	24,536
Carrying Value, Secured borrowings	$ 48,260	17,425
MBS | Deutsche Bank | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.14%
MBS | Deutsche Bank | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.27%
MBS | Citibank | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	$ 25,289
Pledged Assets Carrying Value	29,845
Carrying Value, Secured borrowings	$ 25,289
MBS | Citibank | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.29%
MBS | Citibank | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.44%
MBS | Bank of America | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	$ 43,277
Pledged Assets Carrying Value	49,999
Carrying Value, Secured borrowings	$ 43,277
MBS | Bank of America | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	2.76%
MBS | Bank of America | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.06%
MBS | RBC Bank | Borrowings under repurchase agreements
Secured borrowings and promissory note
Maximum Facility Size	$ 59,965
Pledged Assets Carrying Value	77,496	83,818
Carrying Value, Secured borrowings	$ 59,965	$ 62,494
MBS | RBC Bank | Borrowings under repurchase agreements | Minimum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	3.44%
MBS | RBC Bank | Borrowings under repurchase agreements | Maximum
Secured borrowings and promissory note
Pricing, stated rate (as a percent)	4.08%