Throttle stall / deadlock

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Throttle stall / deadlock

Ron Hitchens
I know I'm probably doing this wrong, but I'm trying to understand how throttles work and how best to use them.  I have a need for queuing up lots of work that should be drained at a steady rate.  I have a scheme in place that is imperfect but generally works well enough for now.  However, it occasionally stalls where everything comes to a stop and all the jobs are in QUEUED state (which in this context means the first promise in the job sequence has not been started).  The Throttle I'm using allows three concurrent jobs.

As an exploratory exercise, I wrote this bit of test code to try stressing the Throttle class because I was wondering if there might be a race condition of some kind.  Now, I do know that these 10 Promises will run sequentially because of the RatPack execution model, but I was surprised to find that this code actually appears to deadlock.

Tracing this in the debugger, it appears that once the Throttle is full, none of the following Promises can get themselves queued, they just never do anything.  And the "active" Promises won't start until the current execution segment finishes.  Is this expected behavior?  Why do the first, active ones never seem to start?  I would expect all these Promises to neatly queue up and run in order, with the Throttle not having any net effect.

Am I just missing the point here?  Any help appreciated.

@Test
public void throttleTest()
{
        Throttle t = Throttle.ofSize (3)

        ExecResult<String> result = ExecHarness.yieldSingle { Execution exec ->
                final AtomicInteger counter = new AtomicInteger();

                println "Creating promises"

                (1..10).each { i ->
                        counter.incrementAndGet()
                        exec.promise { f ->
                                println "Enter ${i}: t.active=${t.active}, t.waiting=${t.waiting}, net=${counter.get()}"
                                f.success ("Promise ${i}".toString())
                                println "Exit ${i}: t.active=${t.active}, t.waiting=${t.waiting}, net=${counter.get()}"
// This deadlocks
                        }.throttled (t).then {
// This does not
// }.then {

                                println "Done ${i}: t.active=${t.active}, t.waiting=${t.waiting}, net=${counter.get()}"
                                counter.decrementAndGet()
                        }
                        println "Prestart ${i}: t.active=${t.active}, t.waiting=${t.waiting}"
                }

                println "Promises created"

                exec.promise { f ->
                        f.success ("${counter.get()}".toString())
                }
        }

        assert (result.value) == '0'
}
Reply | Threaded
Open this post in threaded view
|

Re: Throttle stall / deadlock

Ron Hitchens
Update: If I set the throttle size to 30, then all the Promises run, but apparently differently.  I get the following output and my assertion fails.  Setting to 10 gives the same result, 9 deadlocks.

Creating promises
Prestart 1: t.active=0, t.waiting=0
Prestart 2: t.active=0, t.waiting=0
Prestart 3: t.active=0, t.waiting=0
Prestart 4: t.active=0, t.waiting=0
Prestart 5: t.active=0, t.waiting=0
Prestart 6: t.active=0, t.waiting=0
Prestart 7: t.active=0, t.waiting=0
Prestart 8: t.active=0, t.waiting=0
Prestart 9: t.active=0, t.waiting=0
Prestart 10: t.active=0, t.waiting=0
Promises created
Enter 1: t.active=10, t.waiting=0, counter=10
Exit 1: t.active=10, t.waiting=0, counter=10
Done 1: t.active=9, t.waiting=0, counter=10
Enter 2: t.active=9, t.waiting=0, counter=9
Exit 2: t.active=9, t.waiting=0, counter=9
Done 2: t.active=8, t.waiting=0, counter=9
Enter 3: t.active=8, t.waiting=0, counter=8
Exit 3: t.active=8, t.waiting=0, counter=8
Done 3: t.active=7, t.waiting=0, counter=8
Enter 4: t.active=7, t.waiting=0, counter=7
Exit 4: t.active=7, t.waiting=0, counter=7
Done 4: t.active=6, t.waiting=0, counter=7
Enter 5: t.active=6, t.waiting=0, counter=6
Exit 5: t.active=6, t.waiting=0, counter=6
Done 5: t.active=5, t.waiting=0, counter=6
Enter 6: t.active=5, t.waiting=0, counter=5
Exit 6: t.active=5, t.waiting=0, counter=5
Done 6: t.active=4, t.waiting=0, counter=5
Enter 7: t.active=4, t.waiting=0, counter=4
Exit 7: t.active=4, t.waiting=0, counter=4
Done 7: t.active=3, t.waiting=0, counter=4
Enter 8: t.active=3, t.waiting=0, counter=3
Exit 8: t.active=3, t.waiting=0, counter=3
Done 8: t.active=2, t.waiting=0, counter=3
Enter 9: t.active=2, t.waiting=0, counter=2
Exit 9: t.active=2, t.waiting=0, counter=2
Done 9: t.active=1, t.waiting=0, counter=2
Enter 10: t.active=1, t.waiting=0, counter=1
Exit 10: t.active=1, t.waiting=0, counter=1
Done 10: t.active=0, t.waiting=0, counter=1

Assertion failed:

assert (result.value) == '0'
        |      |      |
        |      10     false
        ExecResult{complete=false, error=null, value=10}
Reply | Threaded
Open this post in threaded view
|

Re: Throttle stall / deadlock

Luke Daley
Administrator
For anyone finding this thread, the problem was fixed in Ratpack 0.9.19.