{"base_url":"https://datafy.id/","mode":"build","title":null,"description":null,"languages":{},"default_language":"en","generate_feed":false,"generate_feeds":false,"feed_filenames":["atom.xml"],"taxonomies":[{"name":"lang","slug":"lang","paginate_by":null,"paginate_path":null,"render":true,"feed":true},{"name":"lib","slug":"lib","paginate_by":null,"paginate_path":null,"render":true,"feed":true}],"author":null,"build_search_index":true,"extra":{"links":[{"href":"https://datafy.id","title":"Datafy"}]},"markdown":{"highlight_code":true,"highlight_theme":"base16-ocean-dark","highlight_themes_css":[],"render_emoji":false,"external_links_target_blank":false,"external_links_no_follow":false,"external_links_no_referrer":false,"smart_punctuation":false,"bottom_footnotes":false,"extra_syntaxes_and_themes":[],"lazy_async_image":false},"search":{"index_format":"elasticlunr_javascript"}}

{"name":"lib","slug":"lib","paginate_by":null,"paginate_path":null,"render":true,"feed":true}

"https://datafy.id/lib/missionary/"

"/lib/missionary/"

{"name":"missionary","slug":"missionary","path":"/lib/missionary/","permalink":"https://datafy.id/lib/missionary/","pages":[{"relative_path":"blog/20240801_paged_listing.md","colocated_path":null,"content":"

This function demonstrated how we can create paging data out of a collection of\nentity id (eid), utilizing the\nmissionary flow to achieve safe and\nperformant concurrent processing.

\n

One important thing is that we want to reduce the overheads of pulling the\nentity attributes first, and then filtering them down to just a small subset\nbased on the requested page.

\n

So, basically, we query first to return only the eids, do page filtering and\nthen pull the necessary attributes just for the requested items. Bonus points,\nwe can make use the excellent missionary forking mechanism to parallelize the\nattribute pull and data transform operation on it.

\n
(defn paged-listing\n  "Return a task completing with paged listing of running `eid->task` for each eid\n  in eids concurrently."\n  [conn\n   eid->task ;; this will be called as `(eid->task conn x)`\n   eids {:keys [currentPage\n                itemsPerPage]\n         :or {currentPage 1\n              itemsPerPage PNR_ITEMS_PER_PAGE}}\n   ]\n  (let [paged-eids (->> eids\n                        (drop (* itemsPerPage (dec currentPage)))\n                        (take itemsPerPage))\n        total-items (count eids)\n        [q r] ((juxt quot rem) total-items itemsPerPage)\n        total-pages (+ q (if (zero? r) 0 1))\n        ]\n    (m/sp\n      {:currentPage currentPage\n       :totalItems total-items\n       :totalPage total-pages\n       :itemsPerPage itemsPerPage\n       :pnrReportSummaries\n       (m/? (->> (m/ap (let [x (m/?> 4 (m/seed paged-eids))]\n                         (m/? (eid->task conn x))))\n                 (m/reduce conj)))})))\n
\n
\n

The eids argument is, in this case, could be a values returned from\ndatalevin query shown below:

\n
(->> (d/q '[:find [?e ...]\n            :where\n            [?e :locatorCode]]\n          (d/db conn))\n     (sort #(compare %2 %1))\n     (take 1000000))\n
\n
\n

Argument eid->task is a function returning missionary task.

\n
(defn pull-detail-task\n  [conn eid]\n  (m/via m/blk\n         (-> (d/pull (d/db conn) '[*] eid)\n             ,,, ;; some data transformation function here\n             )))\n
\n
\n

And, finally, the typical usage could be some thing like this:

\n
(-> eids\n    (as-> $$ (paged-listing conn pull-detail-task $$)\n    )\n;; =>\n{:totalItems 6,\n :pnrReportSummaries\n [{:locatorCode "9EA5F385" ,,, } {:locatorCode "YCTKQZJ2" ,,, }],\n :currentPage 2,\n :totalPage 3,\n :itemsPerPage 2}\n
\n","permalink":"https://datafy.id/blog/20240801-paged-listing/","slug":"20240801-paged-listing","ancestors":["_index.md","blog/_index.md"],"title":"Create paged listing from Datalevin query results with Missionary concurrent flow processing","description":null,"updated":null,"date":"2024-08-01","year":2024,"month":8,"day":1,"taxonomies":{"lib":["missionary","datalevin"],"lang":["clojure"]},"authors":[],"extra":{},"path":"/blog/20240801-paged-listing/","components":["blog","20240801-paged-listing"],"summary":null,"toc":[],"word_count":320,"reading_time":2,"assets":[],"draft":false,"lang":"en","lower":null,"higher":null,"translations":[],"backlinks":[]}],"page_count":1}

"en"