{"base_url":"https://datafy.id/","mode":"build","title":null,"description":null,"languages":{},"default_language":"en","generate_feed":false,"generate_feeds":false,"feed_filenames":["atom.xml"],"taxonomies":[{"name":"lang","slug":"lang","paginate_by":null,"paginate_path":null,"render":true,"feed":true},{"name":"lib","slug":"lib","paginate_by":null,"paginate_path":null,"render":true,"feed":true}],"author":null,"build_search_index":true,"extra":{"links":[{"href":"https://datafy.id","title":"Datafy"}]},"markdown":{"highlight_code":true,"highlight_theme":"base16-ocean-dark","highlight_themes_css":[],"render_emoji":false,"external_links_target_blank":false,"external_links_no_follow":false,"external_links_no_referrer":false,"smart_punctuation":false,"bottom_footnotes":false,"extra_syntaxes_and_themes":[],"lazy_async_image":false},"search":{"index_format":"elasticlunr_javascript"}}

{"name":"lang","slug":"lang","paginate_by":null,"paginate_path":null,"render":true,"feed":true}

"https://datafy.id/lang/"

"/lang/"

[{"name":"clojure","slug":"clojure","path":"/lang/clojure/","permalink":"https://datafy.id/lang/clojure/","pages":[{"relative_path":"blog/20240802_range_query.md","colocated_path":null,"content":"

So, at one moment i think what of kind of query the range query is? It turn\nout just yet another fancy name and don't over thinking ot it ever again. This\nis the principle.

Range query is for when the clause is not an equal (=).

Or, in more generic sense: Range query is for whatever that could return more\nthan one exact result.

SQL:

select * from users where id>1;\nselect * from users where created_at between '2024-08-01' and '2024-08-02';\n

Datalog:

(d/q '[:find ?e\n       :in $ n\n       :where\n       [?e :user/id ?x]\n       [(> ?x n)]\n       ]\n     (d/db (:qln.pnr/datalevin-conn qln.sys/*sys*))\n     1)\n\n(d/q '[:find ?e\n       :in $ dt1 dt2\n       :where\n       [?e :user/id]\n       [?e :user/createdAt ?x]\n       [(> ?x dt1)]\n       [(< ?x dt2)]\n       ]\n     (d/db (:qln.pnr/datalevin-conn qln.sys/*sys*))\n     (java.util.Date/from (java.time.Instant/parse "2024-08-01T00:00:00Z"))\n     (java.util.Date/from (java.time.Instant/parse "2024-08-02T00:00:00Z")))\n

This function demonstrated how we can create paging data out of a collection of\nentity id (eid), utilizing the\nmissionary flow to achieve safe and\nperformant concurrent processing.

One important thing is that we want to reduce the overheads of pulling the\nentity attributes first, and then filtering them down to just a small subset\nbased on the requested page.

So, basically, we query first to return only the eids, do page filtering and\nthen pull the necessary attributes just for the requested items. Bonus points,\nwe can make use the excellent missionary forking mechanism to parallelize the\nattribute pull and data transform operation on it.

(defn paged-listing\n  "Return a task completing with paged listing of running `eid->task` for each eid\n  in eids concurrently."\n  [conn\n   eid->task ;; this will be called as `(eid->task conn x)`\n   eids {:keys [currentPage\n                itemsPerPage]\n         :or {currentPage 1\n              itemsPerPage PNR_ITEMS_PER_PAGE}}\n   ]\n  (let [paged-eids (->> eids\n                        (drop (* itemsPerPage (dec currentPage)))\n                        (take itemsPerPage))\n        total-items (count eids)\n        [q r] ((juxt quot rem) total-items itemsPerPage)\n        total-pages (+ q (if (zero? r) 0 1))\n        ]\n    (m/sp\n      {:currentPage currentPage\n       :totalItems total-items\n       :totalPage total-pages\n       :itemsPerPage itemsPerPage\n       :pnrReportSummaries\n       (m/? (->> (m/ap (let [x (m/?> 4 (m/seed paged-eids))]\n                         (m/? (eid->task conn x))))\n                 (m/reduce conj)))})))\n

The eids argument is, in this case, could be a values returned from\ndatalevin query shown below:

(->> (d/q '[:find [?e ...]\n            :where\n            [?e :locatorCode]]\n          (d/db conn))\n     (sort #(compare %2 %1))\n     (take 1000000))\n

Argument eid->task is a function returning missionary task.

(defn pull-detail-task\n  [conn eid]\n  (m/via m/blk\n         (-> (d/pull (d/db conn) '[*] eid)\n             ,,, ;; some data transformation function here\n             )))\n

And, finally, the typical usage could be some thing like this:

(-> eids\n    (as-> $$ (paged-listing conn pull-detail-task $$)\n    )\n;; =>\n{:totalItems 6,\n :pnrReportSummaries\n [{:locatorCode "9EA5F385" ,,, } {:locatorCode "YCTKQZJ2" ,,, }],\n :currentPage 2,\n :totalPage 3,\n :itemsPerPage 2}\n

"en"