Secured Borrowings - Collateral Pledged (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Secured borrowings
Collateral pledged
Pledged Assets Carrying Value	$ 1,767,319	$ 1,668,460
Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	631,014	561,257
Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	1,136,305	1,107,203
Real estate, held for sale | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	32,948	19,950
Real estate acquired in settlement of loans | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	829
Loans, held for sale, at fair value | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	313,844	159,928
Loans, held for sale, at fair value | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	17,850
Loans, net | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	159,482	276,810
Loans, net | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	815,603	721,887
Loans, held at fair value | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	73,799
Loans, held at fair value | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	3,071
Mortgage servicing rights | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value	50,941	61,304
Purchased future receivables | Borrowings under credit facilities
Collateral pledged
Pledged Assets Carrying Value		43,265
Mortgage backed securities | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	72,179	113,436
Retained interest in assets of consolidated VIEs | Borrowings under repurchase agreements
Collateral pledged
Pledged Assets Carrying Value	$ 226,773	$ 271,880