Secured Borrowings - Collateral Pledged (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Collateral pledged
Pledged Assets Carrying Value	$ 1,668,889	$ 1,668,460
Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	628,903	561,257
Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	1,039,986	1,107,203
PPP Loans | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	105,248
Investment in unconsolidated joint ventures | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	28,199
Loans, held for sale, at fair value | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	289,377	159,928
Loans, net | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	138,436	276,810
Loans, net | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	697,913	721,887
Mortgage servicing rights | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	49,699	61,304
Purchased future receivables | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	27,190	43,265
Real estate, held for sale | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	18,953	19,950
Mortgage backed securities | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	71,546	113,436
Retained interest in assets of consolidated VIEs | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	$ 242,328	$ 271,880