Cute Programming - Simple Parallellism on Qt 5

How to create a progress bar for an heavy operation using QtConcurrenRun
Background
Playing Guild Wars 2 and specifically running to craft the Ascended weapons I had the need to find a proper tool that produce a list of required resources of various items. These items are crafted by other items which need more other resources and so, other items. In the end (after 5 or more iterations) I need the collection of all the resources needed to create those item. There are several tools on the Internet that gives almost such result but most of them don't work as I intend and most of all they are online website. I hate website, expecially mine. I want a "classic" damn application.

When you work on "classic" applications it's common that an heavy function hangs the user interface and you have to wait until the end of this function to use the UI again. This is bad programming (though sometimes just unluck of unpredictable behaviours).
When you program a heavy function/system you must run that function on another thread and let the system balance the loads, using parallel programming.

I'm working on a program (using the Qt framework) that downloads the item database using the Guild Wars 2 API and can work with those datas.
Since I have not my own database ready yet to store the data from those APIs and I need some information on those data structures I first coded a program that retreive all the JSon objects returned from those API and then store them in simple text files. Then I wrote a tool to retreive those text files in memory and parse them to obtain the informations I need.

The following tutorial is a little extrapolation of my program that reads a lot of (small) files from the disk and look for a specific pattern once those files are in memory. It's simple.

The cute solution: QtConcurrentRun
When I program with Unity3D I often use the StartCoroutine() function that get any other function and run it in another loop out of the main update thread. Qt 5 offer an even better solution through the QFuture and QFutureWatcher classes.

The QFuture class gives you the power to run any function outside the main thread and the QFutureWatcher gives you some control over it's events (run, stop, termination and so on).
Specifically the QFutureWatcher have some predefined behaviour to indicate the progress of the running operation if you use a data structure like a map and an operation on it's datas. The problem is, it lacks any method to indicate the progress of a custom function. You have to write something by yourself.

In this tutorial I'll show how to do it using Qt 5, it's Signal and Slot system and the QtConcurrentRun APIs.

UI and Usage
The user interface is simple and pretty straightforward: you can input the directory where read the files, filter by extension, if you wish, and prompt a search term.
The location must exist otherwise the operation will terminate immediately (prompting an error). This program works and compile for any OS supported by Qt5 but since Qt is a great framework it accept both Unix and Windows location.
In this screenshot, for example, the program has been compiled for Windows but I'm using Unix path style.

The user interface of this example.

The "Previoux File" and "Next File" buttons will scroll the list of files and show the content, filtering the search term. Note that changing the term after a research will not change the highlight, you need to run again the search to change the highlight. This behaviour is wrong but intented: it's left as exercise of improve this behaviour.

Workflow
The two most important classes we need are QFuture and QFutureWatcher. The first is the holder of the running job in the external thread while the second is optional and it's used to fire events when the function ends.

  1. void MainWindow::on_begin_button_clicked()
  2. {
  3.     ui->begin_button->setEnabled(false);
  4.     ui->prev_button->setEnabled(false);
  5.     ui->next_button->setEnabled(false);
  6.  
  7.     ui->status_label->setText(g_status_running);
  8.     _load_status = idle;
  9.     _files.clear();
  10.  
  11.     QString temp = ui->input_terms->toPlainText();
  12.     _highlighter->setPattern(temp);
  13.  
  14.     connect(&_futureWatcher, SIGNAL(finished()), this, SLOT(load_files_finished()));
  15.     _future = QtConcurrent::run(this, &MainWindow::load_files);
  16.     _futureWatcher.setFuture(_future);
  17. }

When an user click the "Begin a new search!" button the proper event is fired. This event will disable all the buttons in the UI until the operation is finished to prevent creating another job. This program is just and example and can handle only one concurrent function but you are free to create as many as you wish (but then you have to handle them).

The _future and _futureWatcher are set to fire the load_files function to run in another thread and to call load_files_finished when it's over.

During the run of load_files we use the Singal progress_load to increase the value of our progress bar.

  1. void MainWindow::load_files()
  2. {
  3.     // Omitted code that set the directory and get the file paths.
  4.     // Download the source code in the end of this article!
  5.    
  6.     // Let's filter by extension and read those file
  7.     for (int i = 0; i < maximum; i++)
  8.     {
  9.         emit progress_load(i, maximum);
  10.  
  11.         // Omitted code that read the file content.
  12.     }
  13. }

When everything is finished and the load_file is done the QFutureWatcher will fire the finished() signal which we previously attached to our load_files_finished slot.
This function will take care of reenabling the buttons and display an error or sucess message.

Handling Error and States
QFutureWatcher don't give us much control of what happen inside the function. The only event we can register for a custom function is finished() and while we can control the ongoing status with methods like isRunning() and similar (refer to the docs page for more information) it doesn't give us any information.

I've choose the option to set a member variable that will give some information of what happen through a simple enumeration:

  1. enum load_file_status_t
  2. {
  3.     idle,
  4.     no_dir,
  5.     no_file,
  6.     done,
  7.     error
  8. };
  9.  
  10.  
  11. void MainWindow::load_files()
  12. {
  13.     // Let's check eveything is ok.
  14.     QString dir_name(ui->input_location->toPlainText());
  15.     QString extension(ui->input_file_ext->toPlainText());
  16.  
  17.     QDir dir(dir_name);
  18.  
  19.     // Directory exist? Check!
  20.     if (!dir.exists())
  21.     {
  22.         _load_status = no_dir;
  23.         return;
  24.     }
  25.  
  26.     // Omitted code.
  27. }

Then the load_files_finished will then handle the results. Usually member variable are evil since everything can modify it and it's easy to lose control of what is going on but this is just an example.

Highlight
The highlight is done on request by the SimpleSearchHighlighter class I've wrote. The class is really simple and it's based on the example found on the docs.

This class accept an string in input through the SetPattern setter and will use it when the widget automatically call the highlightBlock function. Changing this string will change the highlight: infact this is the string we set every time we begin a new search, when it's done and we scroll the texts setting one of them to the TextBox widget, this component will highlight the search term.

To change the color, font or any other text elements just change the setup on the class constructor.

  1. SimpleSearchHighlighter::SimpleSearchHighlighter(QObject *parent) :
  2.     QSyntaxHighlighter(parent)
  3. {
  4.     _format.setBackground(Qt::gray);
  5.     _format.setForeground(Qt::magenta);
  6.     _format.setFontItalic(true);
  7. }

Queued Signals
If you pay attention on MainWindow's constructor I use QueuedConnection instead of normal signal connection.

  1. MainWindow::MainWindow(QWidget *parent) :
  2.     QMainWindow(parent),
  3.     ui(new Ui::MainWindow),
  4.     _counter(0),
  5.     _load_status(idle)
  6. {
  7.     ui->setupUi(this);
  8.     _buffer = new char[g_file_size_limit];
  9.  
  10.     _highlighter = new SimpleSearchHighlighter(ui->result_box);
  11.     connect(this, SIGNAL(progress_load(int,int)), this, SLOT(on_progress_load(int,int)), Qt::QueuedConnection);
  12. }

The reason is simple: a queue signal is... queued. If you send a signal it will wait until the end of it's operation (if a slot is connected) while the queued will instead return immediately and the slot is executed in a separate thread (or program loop, I'm not sure how it's implemented).
Try both: you'll notice a slightly improvements using the queued connection instead of standard ones. The increase in performance is up to the slot's (computational) weight. Since this example program handles only one operation at time there are no problems on not using a queue connetion but try to remove the guard that disable the "Begin new search!" button and fire twice the event to see what happen.

Queue connection can also lead in a strange improvements: the concurrent run can finish before all the signals are correctly emitted.
For example: updating the progress bar can take more time then reading the files. If this happen the files are correctly read and the function will terminate but the progress bar will still update. This is not the case though: reading a file is always slower then updating a graphic bar. We can find this behaviour in a producer-consumer example instead.

Source and license.
A 7z file with all the source code and QtCreator project files is available here. The code is open-source, licensed by GPLv3. Exercises for the reader
If you are a student and want to improve your skills in C++ and Qt (always try to improve your skills!) here some exercies for you:
  • Easy - The search actually don't look in sub-folder. Make the search recursive
  • Medium - Remove the highlighter and instead add a function that store the text in Rich Text or HTML to highlight the search term (the TextBox widget accept HTML code or Rich Text).
  • Hard - Create two different concurrent functions: one will read files and meanwhile the other will parse them (and consequentially one more progress bar). The parse should use the function of the previous exercise.
    Hint: you don't need any particular mechanism of data preservation. Use one array for both functions and an index: while the first function read the files and store them in the array make the index advance. The second function will surely be faster then the first so create a loop that parse the array items until the index position, when it stop to the index make this function wait (a sleep function is fine) and then continue until the index reach the end of the array.
  • Harder - Drop everything and use proper threads to create a typical producer - consumer problem. :P
English

Add new comment

Comment Text

  • Allowed HTML tags: <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
To prevent automated spam submissions leave this field empty.
CAPTCHA
Answer the question and don't bother me with spam.