FAIR VALUE OF FINANCIAL INSTRUMENTS - Summary of Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2015	Dec. 31, 2014
Assets:
Fair Value	$ 2,416,966	$ 2,034,719
Liabilities:
Fair Value	4,271,538	4,198,379
Repurchase agreements - short-term
Liabilities:
Outstanding Face Amount	1,224,942	1,331,603
Fair Value	1,224,942	1,331,603
Repurchase agreements - long-term
Liabilities:
Outstanding Face Amount	35,814	100,062
Fair Value	35,814	100,062
Borrowings under credit agreement
Liabilities:
Outstanding Face Amount		11,000
Fair Value		11,000
Borrowings under credit and security agreement
Liabilities:
Outstanding Face Amount		46,750
Fair Value		46,750
Revolving credit facility
Liabilities:
Outstanding Face Amount		25,000
Fair Value		25,000
Mortgage loan financing
Liabilities:
Outstanding Face Amount	540,764	442,753
Fair Value	557,841	455,846
Borrowings from the FHLB
Liabilities:
Outstanding Face Amount	1,856,700	1,611,000
Fair Value	1,861,584	1,616,373
Senior unsecured notes
Liabilities:
Outstanding Face Amount	619,555	619,555
Fair Value	591,357	611,745
Mortgage loan receivables held for investment
Assets:
Outstanding Face Amount	1,749,556	1,536,923
Fair Value	1,756,774	1,540,388
Mortgage loan receivables held for sale
Assets:
Outstanding Face Amount	571,638	417,955
Fair Value	582,277	421,991
FHLB stock
Assets:
Outstanding Face Amount	77,915	72,340
Fair Value	77,915	72,340
Level 2
Liabilities:
Fair Value		68,357
Level 2 | Repurchase agreements - short-term
Liabilities:
Fair Value		68,357
Level 3
Assets:
Fair Value	2,416,966	2,034,719
Liabilities:
Fair Value	4,271,538	4,130,022
Level 3 | Repurchase agreements - short-term
Liabilities:
Fair Value	1,224,942	1,263,246
Level 3 | Repurchase agreements - long-term
Liabilities:
Fair Value	35,814	100,062
Level 3 | Borrowings under credit agreement
Liabilities:
Fair Value		11,000
Level 3 | Borrowings under credit and security agreement
Liabilities:
Fair Value		46,750
Level 3 | Revolving credit facility
Liabilities:
Fair Value		25,000
Level 3 | Mortgage loan financing
Liabilities:
Fair Value	557,841	455,846
Level 3 | Borrowings from the FHLB
Liabilities:
Fair Value	1,861,584	1,616,373
Level 3 | Senior unsecured notes
Liabilities:
Fair Value	591,357	611,745
Level 3 | Mortgage loan receivables held for investment
Assets:
Fair Value	1,756,774	1,540,388
Level 3 | Mortgage loan receivables held for sale
Assets:
Fair Value	582,277	421,991
Level 3 | FHLB stock
Assets:
Fair Value	77,915	72,340
Recurring
Assets:
Fair Value	2,410,038	2,815,989
Recurring | Nonhedge derivatives
Liabilities:
Derivative liability face amount	374,200	1,428,700
Fair Value	5,504	13,446
Recurring | CMBS
Assets:
Outstanding Face Amount	1,972,492	2,247,565
Fair Value	1,991,506	2,305,410
Recurring | CMBS interest-only
Assets:
Outstanding Face Amount	7,436,379	7,239,503
Fair Value	344,423	378,335
Recurring | GNMA interest-only
Assets:
Outstanding Face Amount	632,175	1,400,141
Fair Value	26,194	66,642
Recurring | GN construction securities
Assets:
Outstanding Face Amount	27,091	27,538
Fair Value	28,639	28,406
Recurring | GN permanent securities
Assets:
Outstanding Face Amount	16,249	36,232
Fair Value	16,455	36,773
Recurring | Nonhedge derivatives
Assets:
Derivative asset face amount	868,700	125,050
Fair Value	2,821	423
Recurring | Level 2
Assets:
Fair Value	2,821	132,244
Recurring | Level 2 | Nonhedge derivatives
Liabilities:
Fair Value	5,504	13,446
Recurring | Level 2 | GNMA interest-only
Assets:
Fair Value		66,642
Recurring | Level 2 | GN construction securities
Assets:
Fair Value		28,406
Recurring | Level 2 | GN permanent securities
Assets:
Fair Value		36,773
Recurring | Level 2 | Nonhedge derivatives
Assets:
Fair Value	2,821	423
Recurring | Level 3
Assets:
Fair Value	2,407,217	2,683,745
Recurring | Level 3 | CMBS
Assets:
Fair Value	1,991,506	2,305,410
Recurring | Level 3 | CMBS interest-only
Assets:
Fair Value	344,423	$ 378,335
Recurring | Level 3 | GNMA interest-only
Assets:
Fair Value	26,194
Recurring | Level 3 | GN construction securities
Assets:
Fair Value	28,639
Recurring | Level 3 | GN permanent securities
Assets:
Fair Value	$ 16,455