FAIR VALUE OF FINANCIAL INSTRUMENTS - Summary of Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Assets:
Fair Value	$ 2,452,785	$ 2,416,966
Liabilities:
Fair Value	3,940,704	4,271,537
Repurchase agreements - short-term
Liabilities:
Outstanding Face Amount	629,430	1,224,942
Fair Value	629,430	1,224,942
Repurchase agreements - long-term
Liabilities:
Outstanding Face Amount	477,756	35,813
Fair Value	477,756	35,813
Revolving credit facility
Liabilities:
Outstanding Face Amount	25,000
Fair Value	25,000
Mortgage loan financing
Liabilities:
Outstanding Face Amount	589,152	540,764
Fair Value	595,778	557,841
Borrowings from the FHLB
Liabilities:
Outstanding Face Amount	1,660,000	1,856,700
Fair Value	1,662,178	1,861,584
Senior unsecured notes
Liabilities:
Outstanding Face Amount	563,872	619,555
Fair Value	550,562	591,357
Mortgage loan receivables held for investment, at amortized cost
Assets:
Outstanding Face Amount	2,011,309	1,749,556
Fair Value	2,014,973	1,756,774
Mortgage loan receivables held for sale
Assets:
Outstanding Face Amount	360,518	571,638
Fair Value	359,897	582,277
FHLB stock
Assets:
Outstanding Face Amount	77,915	77,915
Fair Value	77,915	77,915
Level 3
Assets:
Fair Value	2,452,785	2,416,966
Liabilities:
Fair Value	3,940,704	4,271,537
Level 3 | Repurchase agreements - short-term
Liabilities:
Fair Value	629,430	1,224,942
Level 3 | Repurchase agreements - long-term
Liabilities:
Fair Value	477,756	35,813
Level 3 | Revolving credit facility
Liabilities:
Fair Value	25,000
Level 3 | Mortgage loan financing
Liabilities:
Fair Value	595,778	557,841
Level 3 | Borrowings from the FHLB
Liabilities:
Fair Value	1,662,178	1,861,584
Level 3 | Senior unsecured notes
Liabilities:
Fair Value	550,562	591,357
Level 3 | Mortgage loan receivables held for investment, at amortized cost
Assets:
Fair Value	2,014,973	1,756,774
Level 3 | Mortgage loan receivables held for sale
Assets:
Fair Value	359,897	582,277
Level 3 | FHLB stock
Assets:
Fair Value	77,915	77,915
Recurring
Assets:
Fair Value	2,105,965	2,410,038
Recurring | Nonhedge derivatives
Liabilities:
Derivative liability face amount	100,400	374,200
Fair Value	3,446	5,504
Recurring | CMBS
Assets:
Outstanding Face Amount	1,676,680	1,972,492
Fair Value	1,701,395	1,991,506
Recurring | CMBS interest-only
Assets:
Outstanding Face Amount	8,160,458	7,436,379
Fair Value	342,171	344,423
Recurring | GNMA interest-only
Assets:
Outstanding Face Amount	478,577	632,175
Fair Value	16,821	26,194
Recurring | GNMA construction securities
Assets:
Outstanding Face Amount		27,091
Fair Value		28,639
Recurring | Agency securities
Assets:
Outstanding Face Amount	774
Fair Value	780
Recurring | GNMA permanent securities
Assets:
Outstanding Face Amount	38,327	16,249
Fair Value	39,780	16,455
Recurring | Nonhedge derivatives
Assets:
Derivative asset face amount	847,000	868,700
Fair Value	5,018	2,821
Recurring | Level 2
Assets:
Fair Value	5,018	2,821
Recurring | Level 2 | Nonhedge derivatives
Liabilities:
Fair Value	3,446	5,504
Recurring | Level 2 | Nonhedge derivatives
Assets:
Fair Value	5,018	2,821
Recurring | Level 3
Assets:
Fair Value	2,100,947	2,407,217
Recurring | Level 3 | CMBS
Assets:
Fair Value	1,701,395	1,991,506
Recurring | Level 3 | CMBS interest-only
Assets:
Fair Value	342,171	344,423
Recurring | Level 3 | GNMA interest-only
Assets:
Fair Value	16,821	26,194
Recurring | Level 3 | GNMA construction securities
Assets:
Fair Value		28,639
Recurring | Level 3 | Agency securities
Assets:
Fair Value	780
Recurring | Level 3 | GNMA permanent securities
Assets:
Fair Value	$ 39,780	$ 16,455