Fair Value Measurements - Valuation and Inputs, at FV (Details) - Recurring basis - USD ($) $ / shares in Units, $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2017	Dec. 31, 2016
Fair value inputs, quantitative information
Asset, fair value	$ 492,327	$ 683,557
Liabilities, fair value	9,870	15,130
Level 3
Fair value inputs, quantitative information
Asset, fair value	340,987	195,361
Liabilities, fair value	8,939	14,487
Loans, held at fair value | Level 3
Fair value inputs, quantitative information
Asset, fair value	$ 170,128	$ 81,592
Loans, held at fair value | Level 3 | Minimum
Fair value inputs, quantitative information
Price	$ 100.00	$ 98.47
Loans, held at fair value | Level 3 | Maximum
Fair value inputs, quantitative information
Price	104.00	105.00
Loans, held at fair value | Level 3 | Weighted Average
Fair value inputs, quantitative information
Price	$ 102.80	$ 103.16
Loans, held for sale, at fair value | Level 3
Fair value inputs, quantitative information
Asset, fair value	$ 57,466	$ 17,312
Loans, held for sale, at fair value | Level 3 | Minimum
Fair value inputs, quantitative information
Price	$ 99.89	$ 100.04
Loans, held for sale, at fair value | Level 3 | Maximum
Fair value inputs, quantitative information
Price	107.06	102.97
Loans, held for sale, at fair value | Level 3 | Weighted Average
Fair value inputs, quantitative information
Price	$ 101.72	$ 100.87
MBS | Level 3 | Broker Quotes
Fair value inputs, quantitative information
Asset, fair value	$ 43,741	$ 29,883
MBS | Level 3 | Broker Quotes | Minimum
Fair value inputs, quantitative information
Price	$ 29.96	$ 17.91
MBS | Level 3 | Broker Quotes | Maximum
Fair value inputs, quantitative information
Price	104.01	101.00
MBS | Level 3 | Broker Quotes | Weighted Average
Fair value inputs, quantitative information
Price	$ 70.27	$ 68.90
MBS | Level 3 | Transaction Price
Fair value inputs, quantitative information
Asset, fair value	$ 136	$ 2,508
MBS | Level 3 | Transaction Price | Minimum
Fair value inputs, quantitative information
Price	$ 99.00	$ 99.00
MBS | Level 3 | Transaction Price | Maximum
Fair value inputs, quantitative information
Price	99.00	99.00
MBS | Level 3 | Transaction Price | Weighted Average
Fair value inputs, quantitative information
Price	$ 99.00	$ 99.00
Interest-Only-Strips | Level 3 | Transaction Price
Fair value inputs, quantitative information
Asset, fair value	$ 1,400	$ 1,500
Residential MRSs | Level 3
Fair value inputs, quantitative information
Asset, fair value	66,797	61,376
Contingent Consideration | Level 3
Fair value inputs, quantitative information
Liabilities, fair value	$ 8,939	$ 14,487