![\"Image\"](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/mae\u0308l-deschamps.jpeg\")
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/04\/Medium-Blog.png\" "\\"Medium")
<\/span><\/div>Introduction | String filters in pandas<\/h2><\/div>The simplification of hardware management, brought by cloud solutions, pushes us more and more to turn away from the problems of code optimisation.<\/p>\n<\/div>
![\"The](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l1.png\" "\\"article-mae\u0308l1\\"")
<\/span><\/div>But scaling up and\u00a0increasing computing power is not always the solution <\/strong>as it leads to an escalation of costs and computing power is not infinite.<\/p>\nBy giving myself the challenge to fit all my data preparation on a simple container, I quickly realised that with the knowledge of a few tricks,\u00a0optimising your code can sometimes be just as simple as instancing a bigger instance.<\/strong><\/p>\nIn this article I\u2019d like to come back to a single change of code that allowed me to reduce drastically the time spent on the features calculation during the development phase of a propensity model. This change is common enough to be applied to many other situations.<\/p>\n<\/div>
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l2.png\" "\\"article-mae\u0308l2\\"")
<\/span><\/div>It doesn\u2019t aim to be the most optimal solution, but seek to be a quick option to decrease the computation time in an efficient way, in the spirit of the\u00a0Pareto principle.<\/strong><\/p>\n<\/div>Context<\/h2><\/div>During this mission, I was in charge of automating the data preparation process and the prediction of models developed by our Data Scientist team. To simplify the data-flow, transactional data was uploaded everyday and had to go through a first\u00a0Preprocessing<\/strong>\u00a0step, followed by a second step of\u00a0Feature<\/strong> Computation<\/strong>\u00a0before reaching the last step of\u00a0Model Prediction<\/strong>\u00a0using the trained model.<\/p>\n<\/div>![\"The](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l3.png\" "\\"article-mae\u0308l3\\"")
<\/span><\/div>The Feature computation<\/strong>\u00a0phase is the one that took the longest time to execute: indeed many features were computed at the customer level, which resulted in the recurrent execution of a surprisingly time-consuming line in the code :<\/p>\n<\/div>![\"which](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l4.png\" "\\"article-mae\u0308l4\\"")
<\/span><\/div>This single line was timed 18ms, which meant that with my 33.717 clients to evaluate daily, I was spending around 10 min of raw computation time per customer level features, reduced to 1 min and 16 second per feature thanks to the parallelisation of the operation on my 8 available CPUs.<\/p>\n
As we were working with B2B data, it was necessary to compute features at the customer level, as one customer was representing in fact one business with sometimes multiple orders per day.<\/p>\n<\/div>
Experimentation | String filters in pandas<\/h2><\/div>After investigating a little using the\u00a0%%prun<\/strong>\u00a0magic command, I was able to identify the source of this processing bottleneck : the\u00a0pandas._libs.ops.scalar_compare<\/strong>\u00a0operator which was under-optimised in my version of Pandas (1.3.1).<\/p>\n<\/div>![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l5.png\" "\\"article-mae\u0308l5\\"")
<\/span><\/div>By simply replacing this \u201c==<\/strong>\u201d operator by \u201cisin<\/strong>\u201d, which is not very intuitive as I was comparing a single string, I already divided the computing time by 2.5 times, going from 18ms by operation to 7.95ms.<\/p>\n<\/div>![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l6.png\" "\\"article-mae\u0308l6\\"")
<\/span><\/div>Still looking for optimisation, I came across a\u00a0Stackoverflow<\/a>\u00a0post pushing the use of\u00a0Categorical type<\/a>\u00a0to improve further the operation.<\/p>\n<\/div>![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l7.png\" "\\"article-mae\u0308l7\\"")
<\/span><\/div>This last implementation allowed me to divide the computing time more than\u00a036 times<\/strong>. I could however observe a nuance to this trick, as\u00a0the category type does not behave like a classic str during all operations<\/strong>\u00a0(see example bellow when using .groupby() in pandas) so I had to convert it back to str at one point.<\/p>\n<\/div>![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l8.png\" "\\"article-mae\u0308l8\\"")
<\/span><\/div>Hypothesis<\/h2><\/div>But why is that ? How can a simple\u00a0==<\/strong>\u00a0operation between two string takes more time than an\u00a0isin()<\/strong>\u00a0operation comparing lists, or a\u00a0categorical<\/strong>\u00a0one ?
\nWell, to answer the first question we would definitely need to dig out the code behind the execution of\u00a0scalar_compare<\/a>\u00a0which uses mostly Cython and compare it to the code behind the\u00a0isin()<\/a>\u00a0method.<\/p>\n<\/div>Fortunately, the answer to the second part seems to be more intuitive: when comparing two string values, we are comparing an\u00a0infinite number<\/strong>\u00a0of possibilities together, whereas when comparing two categories, the\u00a0number of options is set<\/strong>\u00a0by the different unique categories that exist. It seems much\u00a0easier to compare two entities when our number of options is fixed<\/strong>.<\/p>\n<\/div>My thirst for optimisation was still not quenched, I decided to take a step back from the current method. I came up with a new approach:\u00a0partitioning my Dataframe into a dictionary<\/strong>\u00a0that I will then use to filter my customers when computing my features.
\nIn terms of code, this simply translated into the following lines:<\/p>\n<\/div>customers_list = list(df.ID_Customer.unique())\ndf_dict = {elem: df[df.ID_Customer == elem] for elem in customers_list}Building that dictionary cost me 32 seconds of computing time, but using this partitioned Dataframe I was now able to filter my data in a few nanoseconds.<\/p>\n<\/div>
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l9.png\" "\\"article-mae\u0308l9\\"")
<\/span><\/div>Conclusion | String filters in pandas<\/h2><\/div>After spending a couple of hours in the experimentation phase, I was happy with the result :<\/p>\n<\/div>
![\"After](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l10.png\" "\\"article-mae\u0308l10\\"")
<\/span><\/div>The initial computing time per customer filtering was now divided 348 000\u00a0times<\/strong>, going from\u00a018ms to 51.7ns<\/strong>, or from\u00a010min to 2.65ms<\/strong>\u00a0per feature computed in my case, taking into account the time spend on the partitioning.<\/p>\n<\/div>![\"Immediately,](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2022\/03\/article-mae\u0308l11.png\" "\\"article-mae\u0308l11\\"")
<\/span><\/div>Immediately, the impact of this small change allowed me to\u00a0reduce the calculation time of my complete Feature computation phase by 90%<\/strong>, from 40’49” to 7’27”. Using a CO2eq estimation method that I will detail in my next article, this modification\u00a0saved at least 170$\/year + 22kgCO2\/year<\/strong>\u00a0and potentially way more with the growing customer list and the roll-out of the project in other countries.<\/p>\n<\/div><\/div><\/div><\/div><\/article>
The simplification of hardware management, brought by cloud solutions, pushes us more and more to turn away from the problems of code optimisation.<\/p>\n<\/div>
<\/span><\/div>But scaling up and\u00a0increasing computing power is not always the solution <\/strong>as it leads to an escalation of costs and computing power is not infinite.<\/p>\n By giving myself the challenge to fit all my data preparation on a simple container, I quickly realised that with the knowledge of a few tricks,\u00a0optimising your code can sometimes be just as simple as instancing a bigger instance.<\/strong><\/p>\n In this article I\u2019d like to come back to a single change of code that allowed me to reduce drastically the time spent on the features calculation during the development phase of a propensity model. This change is common enough to be applied to many other situations.<\/p>\n<\/div> It doesn\u2019t aim to be the most optimal solution, but seek to be a quick option to decrease the computation time in an efficient way, in the spirit of the\u00a0Pareto principle.<\/strong><\/p>\n<\/div> During this mission, I was in charge of automating the data preparation process and the prediction of models developed by our Data Scientist team. To simplify the data-flow, transactional data was uploaded everyday and had to go through a first\u00a0Preprocessing<\/strong>\u00a0step, followed by a second step of\u00a0Feature<\/strong> Computation<\/strong>\u00a0before reaching the last step of\u00a0Model Prediction<\/strong>\u00a0using the trained model.<\/p>\n<\/div> The Feature computation<\/strong>\u00a0phase is the one that took the longest time to execute: indeed many features were computed at the customer level, which resulted in the recurrent execution of a surprisingly time-consuming line in the code :<\/p>\n<\/div> This single line was timed 18ms, which meant that with my 33.717 clients to evaluate daily, I was spending around 10 min of raw computation time per customer level features, reduced to 1 min and 16 second per feature thanks to the parallelisation of the operation on my 8 available CPUs.<\/p>\n As we were working with B2B data, it was necessary to compute features at the customer level, as one customer was representing in fact one business with sometimes multiple orders per day.<\/p>\n<\/div> After investigating a little using the\u00a0%%prun<\/strong>\u00a0magic command, I was able to identify the source of this processing bottleneck : the\u00a0pandas._libs.ops.scalar_compare<\/strong>\u00a0operator which was under-optimised in my version of Pandas (1.3.1).<\/p>\n<\/div> By simply replacing this \u201c==<\/strong>\u201d operator by \u201cisin<\/strong>\u201d, which is not very intuitive as I was comparing a single string, I already divided the computing time by 2.5 times, going from 18ms by operation to 7.95ms.<\/p>\n<\/div> Still looking for optimisation, I came across a\u00a0Stackoverflow<\/a>\u00a0post pushing the use of\u00a0Categorical type<\/a>\u00a0to improve further the operation.<\/p>\n<\/div> This last implementation allowed me to divide the computing time more than\u00a036 times<\/strong>. I could however observe a nuance to this trick, as\u00a0the category type does not behave like a classic str during all operations<\/strong>\u00a0(see example bellow when using .groupby() in pandas) so I had to convert it back to str at one point.<\/p>\n<\/div> But why is that ? How can a simple\u00a0==<\/strong>\u00a0operation between two string takes more time than an\u00a0isin()<\/strong>\u00a0operation comparing lists, or a\u00a0categorical<\/strong>\u00a0one ? Fortunately, the answer to the second part seems to be more intuitive: when comparing two string values, we are comparing an\u00a0infinite number<\/strong>\u00a0of possibilities together, whereas when comparing two categories, the\u00a0number of options is set<\/strong>\u00a0by the different unique categories that exist. It seems much\u00a0easier to compare two entities when our number of options is fixed<\/strong>.<\/p>\n<\/div> My thirst for optimisation was still not quenched, I decided to take a step back from the current method. I came up with a new approach:\u00a0partitioning my Dataframe into a dictionary<\/strong>\u00a0that I will then use to filter my customers when computing my features. Building that dictionary cost me 32 seconds of computing time, but using this partitioned Dataframe I was now able to filter my data in a few nanoseconds.<\/p>\n<\/div> After spending a couple of hours in the experimentation phase, I was happy with the result :<\/p>\n<\/div> The initial computing time per customer filtering was now divided 348 000\u00a0times<\/strong>, going from\u00a018ms to 51.7ns<\/strong>, or from\u00a010min to 2.65ms<\/strong>\u00a0per feature computed in my case, taking into account the time spend on the partitioning.<\/p>\n<\/div> Immediately, the impact of this small change allowed me to\u00a0reduce the calculation time of my complete Feature computation phase by 90%<\/strong>, from 40’49” to 7’27”. Using a CO2eq estimation method that I will detail in my next article, this modification\u00a0saved at least 170$\/year + 22kgCO2\/year<\/strong>\u00a0and potentially way more with the growing customer list and the roll-out of the project in other countries.<\/p>\n<\/div><\/div><\/div><\/div><\/article><\/span><\/div>Context<\/h2><\/div>
<\/span><\/div><\/span><\/div>Experimentation | String filters in pandas<\/h2><\/div>
<\/span><\/div><\/span><\/div><\/span><\/div><\/span><\/div>Hypothesis<\/h2><\/div>
\nWell, to answer the first question we would definitely need to dig out the code behind the execution of\u00a0scalar_compare<\/a>\u00a0which uses mostly Cython and compare it to the code behind the\u00a0isin()<\/a>\u00a0method.<\/p>\n<\/div>
\nIn terms of code, this simply translated into the following lines:<\/p>\n<\/div>customers_list = list(df.ID_Customer.unique())\ndf_dict = {elem: df[df.ID_Customer == elem] for elem in customers_list}<\/span><\/div>Conclusion | String filters in pandas<\/h2><\/div>
<\/span><\/div><\/span><\/div>
![](\"https:\/\/www.artefact.com\/\/wp-content\/uploads\/2021\/03\/medium.png\" "\\"medium\\"")
<\/span><\/div>