HttpClient map/reduce alike behavior

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

HttpClient map/reduce alike behavior

Massimo Lusetti
I need to perform two or more http requests and apply a function to aggregate the results.

Which is the best (idiomatic) way of doing this with ratpack HttpClient ?

Using RxRatpack, or to work with Promise/PromiseOperations APIs ?

Thanks for any pointer.
rus
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

rus
I would say rx and zip is your friend here
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

Massimo Lusetti
I'm going deeper following the white rx rabbit in the hole, thanks rus.
rus
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

rus
The promise and streams api's are getting more and more feature rich but the more complex you need to go the more rx comes into play. The 3 work with each other too
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

Luke Daley
Administrator
In reply to this post by Massimo Lusetti
It depends a lot on the details. Do you want the requests to be concurrent? What if one request fails?

If it's just two request, then that's trivial with promises…

String merged = httpClient.get(«url»).flatMap(result1 ->
  httpClient.get(«url»).map(result2 ->
    result1.getBody().getText() + result2.getBody().getText()
  )
)

That gets a little less trivial as the number of jobs grows, but you can build up the promise in a loop easily enough.

Another approach is to use a stream (in Groovy this time)…

context.stream(Streams.publish(«url1», «url2», «url3»))
  .flatMap { httpClient.get(it).map { it.body.text } }
  .toList()
  .map { responses ->
    responses.join(",")
  }
  .then {
    context.render(it)
  }

If you do want parallelism, your best be is to use the RxJava and RxRatpack.forkOnNext().
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

Massimo Lusetti
Thanks Rus and Luke for the answers.

So if I understand correctly the actual HTTP request is not fired until I call then() on the Promise from httpClient.

Is that right?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

Luke Daley
Administrator
That's right, and it's the same with Rx.

You can think about the promise API (or Rx) as being used to build up a processing pipeline, as a series of functions. When the promise is “subscribed to”, the request for the data travels back up to the start of the pipeline and then the data travels down the pipeline.
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

Massimo Lusetti
I got stuck in a cyclic issue.

I need to call n number of times an API depending on the number, the same API, reply to me the first time.

I mean: httpClient.get(url).map( r -> parse_json), now on parse_json I can get the actual number of results I can get, the total count, cause the API is paged.

So only after this first call I can determine how many calls I've to do to retrieve all my data.

I ended up doing a first httpClient.get().then() and inside that then calculate the total amount of requests building each URL I need and then using Stream as you showed.

httpClient.get(new URI(url), action ->
                            action.headers(headers -> headers.set("X-API-Key", API_KEY))
            ).then(response2 -> {

                JsonNode timeEntries = mapper.readTree(response2.getBody().getText());
                int timeEntriesCount = timeEntries.findValue("total_count").asInt();
                int limit = timeEntries.findValue("limit").asInt();
                int offset = 0;
                int count = timeEntriesCount / limit;

                List<String> urls = new ArrayList<String>();
                while (count > 0)
                {
                    offset += limit;
                    urls.add(url + "?offset=" + offset);
                    count--;
                }

                Streams.publish(urls)
                        .flatMap( url ->
                            httpClient.get(new URI(url), action ->
                                    action.headers(headers -> headers.set("X-API-Key", API_KEY))
                            ).map(respo -> parse_json(respo.getBody().getText())
                        )
                        .toList()
                        .map( jsonList -> {
                            // read each json and build a specific pdf file which can be returned
                        })
                        .then( res -> {
                            context.render(res.getPath);
                        });
            });


This feels cumbersome to me.

Is there a better way?
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

Luke Daley
Administrator
Sorry for the late reply, I lost track of  this one.

This is roughly the approach to take. I'd probably do this as a series of promise transforms though. You can Convert a Publisher<T> to a Promise<List<T>> pretty easy, which you could then flat map. So you really only need to go to a stream to get the list.

Alternatively, you can just call execControl.promise() in a loop for the nested item, where the .then() of each just adds the item to to a shared list. Something like…

httpClient.get("something")
  .map { /* extract list of urls to fetch /* }
  .flatMap { List urls ->
    execControl.promise { outer ->
      def things = []
      urls.each { url ->
        httpClient.get(url).then {
          things << /* extract from response */
          if (url == things.last()) {
            outer.success(things)
          }
        }
      }
    }
  }.then { things ->
 
  }
Reply | Threaded
Open this post in threaded view
|

Re: HttpClient map/reduce alike behavior

Massimo Lusetti
Thank you for taking time to answer me. Really appreciated.

I think I'm going to explore more on this soon as now I've got my own prototype fully working.