Reading time ~2 minutes This post was inspired by a discussion on Reddit that followed my previous post In this post, I will cover a suggestion by BelowAverageITGuy that cut down the total execution time by almost one hour. Saga Before I go further here are some link to the previous posts on the problem of calculating similarities and then optimizing it grew to few post. Here are all of them:How I calculate similarities in cookit?How to calculate 17 billion similaritiesIndependent code in ...

Reading time ~6 minutes Last time I’ve shown how I’ve gone from 34 hours to 11. This time we go faster. To go faster I have to do less. The current implementation of Similarity iterates over one vector and checks if that ingredient exists in the second one. Since those vectors are sparse the chance of a miss is big. This means that I am losing computational power on iterating and calling TryGetValue. How to iterate only over the mutually owned ones and do it fast? Saga Before I go furth...

Reading time ~5 minutes Warning this post contains some math. Even more, it shows how to use it for solving real life problems. This post describes how I calculate similarity between recipes in my pet project cookit.pl. For those of you that don’t know, cookit is a search engine for recipes. It crawls websites extracting recipe texts, then it parses it and tries to create a precise ingredient list with amounts and units. 182 184 recipes2936 ingredients This scale may not seem huge, but tr...

Reading time ~4 minutes Diagnosing high memory usage can be tricky, here is the second part of how I found what was hogging to much memory in our system. In the previous post I’ve wrote how to create a memory dump and how many possibilities of catching just the right moment for it ProcDump has. When trying to analyze memory leaks, or high memory usage (not necessary meaning a leak) we have a few ways to approach it: Attach a debugger There are many problems with this approach, to name a fe...

Reading time ~6 minutes This is a sixth part of a series:part 1 - Why schedule and procrastinate jobs?part 2 - Overview of Hangfiepart 3 - Scheduling and Queuing jobs in Hangfirepart 4 - Dashboard, retries and job cancellationpart 5 - Job continuation with ContinueWithpart 6 - Recurring jobs and cron expressions Parts 3, 4, and 5 covered the BackgroundJob class responsible for enqueuing single jobs (fire and forget). This post will cover RecurringJob class exposing API for recurring jobs (as the name ...

Reading time ~3 minutes This is the fourth part of a series discussing job scheduling and Hangfire details:part 1 - Why schedule and procrastinate jobs?part 2 - Overview of Hangfiepart 3 - Scheduling and Queuing jobs in Hangfirepart 4 - Dashboard, retries and job cancellation This part will cover few small topics:dashboardretriesmore technical part of the Hangfire.BackgroundJob class APIjob cancellationDashboard Let’s start with the administrative dashboard because it gives a good background for the ...

Reading time ~2 minutes In the previous post I’ve wrote about why I think the ability to schedule tasks for later execution is a fundamental technical feature, but also a must have from a business point of view. We are passed the whys, so lets get to the hows. The answer is simple - Hangfire. I’ve wrote about it here, here and here, so yeah, I like it. Hangfire is an amazing library. It has proved itself in my pet project (cookit.pl) and in a huge ERP system that we are building at work, where we repla...

One of the steps in cookit is calculating similar recipes. This is what you can see on the left on the recipe page like this For the sake of clarity and manageability it’s scheduled as separate Hangfire jobs. Because cookit is running 5 workers, so similarities are calculated for 5 websites concurrently. The process uses cosine similarity, so it allocates a huge list at start and calculates similarities. A very CPU heavy operation. So some time after triggering all recipes recalculation I saw this in...

One of the main processes in cookit is dealing with extracting recipe information from raw html. I know it isn’t the most elegant solution but it is the only universal one. But to the point. Every web page goes through a process involving html parsing, stemming, parsing, and n-gram token matching. Then it’s saved to Sql Server and after transformation to Solr. So a lot of string manipulation, math calculations and from time to time mostly 0-gen GC. In the most pessimistic case this process has to be r...

Last week I had the opportunity to attend Graph Connect Europe. Many great sessions, but one thing topped them all - Neo4j 3.0 is out! And as with previous major release (it introduced Cypher) there are many bug fixes, tweaks, speed improvements, but here are my personal favorites.