Dask Persist Behavior Inconsistent

I found weird behavior with dask persist, if I comment out this line # client = Client(memory_limit='20GB',n_workers=1) # Connect to distributed cluster and override default

Solution 1:

When we persist an object with client.persist, we get back a future, that refers to the results of the computation. Once computed, the result will be stored on a worker or multiple workers, as appropriate. Running client.persist on an existing future will give back another future... so a reference to a reference to another computation, which is likely unnecessary.

To get the result of a future, one can run .result() on the future itself. This will block further commands until the future is computed and the results are returned.