Making kdb+ work with Apache Spark

kdb+, spark

Something exposed by Hugh Hyndman in his blog is the perfect fit between kdb+ tabular format and Apache Spark. He has created a Spark data source for kdb+ to this end. In this post, we will test his work in a simple way – before opening the door to a distributed system.

This data source makes Spark a powerful addition to kdb+ capabilities. In fact, kdb+ can be really limited sometimes as it is not scalable horizontally. Your limitation is often the hardware. Using Apache Spark in addition to kdb+ can help you alleviate the workload on your host machine and makes it possible to do some distributed computing.
More

How to uppercase the first letter of a string in kdb+

kdb+

One of the most common operations with strings is to make the string capitalized: uppercase its first letter, and leave the rest of the string as-is.

The best way to do this is through a combination of two functions. One uppercases the first letter (upper function), and the second slices the string and returns it starting from the second character (function cut, drop).

yourTable:update `${(upper x[0]),x[1]} each {0 1_string x} each yourColumn from yourTable;

More

Query Rest API with kdb+

kdb+

My first post is going to be a light one. Today, I would like to share with you a useful tip. In my case, I have always struggled with kdb+ to use any Rest API, until I got this code working.

httpGet:{[cmd] (`$":http://hostname:port")"GET ",cmd," HTTP/1.0\r\nHost:hostname\r\n\r\n"};
res:httpGet[ "/entry/point" ];

In this code, we are setting a handle to the API host with a GET method. Note that POST would work too. The text following the handle is the header of the query. This function is an alternative version of curl where one can query any URI he would like.
More