Posts Tagged ‘KGet’


November 17, 2010

Often when I follow discussions on KDE on the internet I see people mentioning that they think that KDE-devs do not care about bugs and only about features. Also often mentioned is that applications aren’t polished. Well in fact then there are also people who disagree on this asseration.

So this time instead of "new features" I show you some polishing work that I did lately. In fact I do that quite often (mostly for KGet) as do most other devs I know. Just because polishing achievments aren’t blogged about that often, doesn’t mean that they don’t happen.

Same is true with bugs. Just take a look at bko and the statistics there, especially compare that with the last few years. You’ll notice that the number of bugs stays mostly constant in total despite all the new features and new programs and that there are people fixing them like crazy.


First of all after the last blog entry I still improved the speed in some areas, now (4.6) adding many urls to KGet is blazingly fast. The interesting thing is that all the profiling work started because of a bug entry. A user reported that KGet crashed for him when trying to download 602 files, he also mentioned that it was quite slow with that many files.

This rather parenthetic statement — and in fact the hope to reproduce the bug — triggered me to try it myself as I never downloaded that many files at once before. And well I was shocked. So I started fixing that step by step what took many seconds before takes less than one now and is hardly noticeable. Baseline is we do care about our users and that reporting bugs is really important.

Now there was one thing that annoyed me and Lukas to some degree. Adding Transfers to KGet is not always consistent — and still isn’t now, because we have to keep our DBus-Interface — different dialogs would be displayed. And what I did not like is that if you had existing transfers or files you’d get a dialog for every such occurence. When downloading the 602 files this could get annoying if I forgot to delete them before.

So here you go, now you’ll get one dialog for Transfers with the option to "Yes All" or "No All". Now you have to know that KGet does not ask you for a location for transfers if one is specified already (e.g. via settings in a group), so if a location exists already you’ll get the dialog you know already from moving files around in Dolphin.

In the case that no location has been specified you will see a dialog with all the files you want to download and a location pre-selected. Once you come that far you won’t be bothered with any dialogs. Instead all the information is displayed in the existing dialog.

Yes you can’t rename those files now, but you could choose a different location. Still having a little option less in this case is imo better than having to constantly deal with dialogs. Maybe I’ll simply add another column with the filename, there you could specify it and also an auto-rename-button. Though that is something to think about.


In the above case I mentioned that I do not really like dialogs — despite using them quite often, yeah not easy to do without them πŸ˜€ — but what I like even less are dialogs that make me fear of doing something wrong.

The Dolphin rename-dialog was one of those and many dialogs I created in KGet were such too and many still are. Often you don’t think about the problems and often you don’t have the time to fix them.

So now, what was bad in the rename-dialog? It allowed you to press "Ok" if no name was entered, and if the name was not changed.

Especially the first case is confusing. What would happen if you press "Ok"? Would the file be renamed to nothing and cease to exist? If you tried you’d realize that nothing would happen but you would be informed that renaming to nothing is not possible. Yet why should you try if you are unsure? The whole situation could lead to you being extra careful at this operation and having an uneasy feeling, even though you could basically do nothing wrong. But you’d only know that if you risked doing something wrong.

What I changed is that "Ok" will be disabled if the name was empty or not changed at all. In fact you could argue that the same uneasy feeling applies to if you rename the file to another existing one by accident and I agree though have no solution for that yet.

Spell Check Runner

I was fixing a bug for this runner and then I thought about a feature that I could implement. Now I mentioned above that this is not about features but rather polishing. Though often polishing means adding small features that make the user’s life easier.

Imagine you are me, so you speak German and have your locale set to German. Yet you want to know if you spelled "coding" correctly, So you fire up krunner and press "spell coding" and this is what you get:

Now you’ll realise that it will use a german dictionary by default and not the one you want it to use at the moment. You also wonder what all those suggestions have to do with the term, but that is a different matter. Now you could fire up KWrite, change the spelling locale and then see if it is correct, or you’d search for that term on google and hope that it would recoginse any mistakes. Both cases are not comfortable. What would be great though was if "spell en coding", "spell englisch coding", "spell englisch coding" or "spell en_US coding" would work.

Guess what with 4.6 it will:

You probably have noticed that I wrote "englisch" instead of "English". Simply because "englisch" is the German word for "English" and since I have the German locale set it expects me to enter German words.

Now when you stumble over a different term like "Sackerl" you try it again, though you fail, since that is an Austrian specific term, you change the input to:

In fact if you don’t have the correct dictionary installed it won’t work:

The solution to implement support for language names in the users’ locale is a little hacky, so it won’t recognise any terms you throw it at, e.g. "austrian" or "ΓΆsterreichisch" would not work, in those cases you have to use the iso form "de_AT" for example.

Shell Runner

Here again I was fixing the shell runner and did a small polishing improvement. Now when clicking "run as different user" the focus will directly go to the input field for the user-name. One click less and exactly what a user wants to do when pressing this option in the first place.


So a small feature as in the case of the spell checking runner can make an existing (great) one a lot more useable and makes it more complete. And that is polishing for me. Makeing something existing complete. Be it by little refactorisations of the UI or by adding a small feature that held the original back. All this are things the KDE devs do on a regular basis, they often just don’t report on it.

A way to realise that would be to use the newest KDE version for a long time and then move to an older version, but I am sure you don’t really want to do that. πŸ™‚

Speeding up KGet β€” using Callgrind

September 11, 2010

Old slowness

If you have used KGet you might have noticed that KGet tends to be very slow, or rather sluggish if you have many items in your transferlist. For one it can take very long to start, then long for an operation you want to do and again long to remove many transfers.

I have used callgrind before to track the issues down. Though it turned out that the callgrind output was so confusing — even combined with KCachegrind — that it was hard to find the parts that were of interest for a certain task I wanted to look at.

Efficiently using Callgrind

Now triggered by Jonathan’s blog post (and the followup) on speeding up KRatingPointer via finding bottlenecks through callgrind I did the same again for KGet. But this time I used more sophisticated methods that really make it shine, an easy introduction can be found here.

The idea is that I define the places that should be profiled in KGet itself via some macros (CALLGRIND_START_INSTRUMENTATION; and CALLGRIND_STOP_INSTRUMENTATION; from #include <valgrind/callgrind>). That way only what I want is being profiled and I don’t get profiling information for areas I don’t want. The later part is really important since otherwise the result would be completely distorted (e.g. because of the starting KGet phase etc.). Now I only need to start callgrind with valgrind --tool=callgrind --instr-atstart=no kget --nofork and I get some useful dumps.


So what does using callgrind look like?

I wanted to know what takes long when starting KGet, in specific loading all Transfers. So I added the above mentioned macros to KGet::load(…) at the beginning of the method and at the end. Then running callgrind and then kcachegrind on the data generated something like you can see in the screenshot (sorry for the German).

As you see loading 604 Transfers was quite expensive and most of the cycles happened in TransferTreeModel::addTransfer and there what took so much time was QStandardItem::appendRow — or rather the actions caused by the signals emitted by appendRow. What I did to improve this was adding a TransferTreeModel::addTransfers to add all the transfers at once. I did it a little hacky, i.e. I start a beginInsertRows then deactivate emitting signals for the model as those resulted in constant redrawing of the view connected to the model, then call appendRow multiple times, reactivate the signals again and do a endInsertRows. I only did that hacky way because I did not manage to use QStandardItem::insertRows, it appears not to be what I need.

Those measurs alone resulted in a huge speedup. Once done even more I got something like this:

As you can see the total cost decreased drastically and a lot of time is spend in parsing xml-data. Still there are areas left that can be improved easily. DataSourceFactory::setStatus takes 16% of all cycles — 99% because of signals — taking a closer look one can see that both DataSourceFactory::processedSize and DataSourceFactory::percent are emited 1208 times while the rest is emited 604 times. That means that for each Transfer those two are emitted twice. Emitting them just once gave a 5% speedup, more would be possible by emitting just one signal something like changeEvent(Tc_Status | Tc_Speed | Tc_TotalSize | Tc_Downloadedsize), so not having a signal for each event but rather defining in the send data what changed.


All this lead to many speed improvments in KGet. KGet should now (4.5.2) be a lot faster in the startup process, when scrolling and significantly faster when removing many transfers. I will look into speeding up starting/stopping many transfers too.

Before I go into any details remember that all that was tested on my machine so the results may vary on yours. Also keep in mind that faster here means less cycles, so that does not translate 1:1 in real time, as io operations — Callgrind does not help here — could use few cycles yet still take a lot of time as a HD is very slow compared with other memory. An example for this is the SQlite history backend that, looking at the cycles, only improved marginally (~20%) while the time when deleting ~600 downloads — all those get added to the store — shrank from 51942 msec to 138 msec.

Especially removing transfers got a lot faster. When removing many unfinished transfers it can now be more than 170 times faster than before, also removing finished transfers can be around 85 times faster. In the last case the speedup depends on if the removed transfers are neighbours, i.e. there are no transfers inbetween them, in the worst case it should still be a little faster. At the same time starting KGet with ~600 downloads was improved 35 times.

In fact there are still a lot of areas for further improvement, though I am quite happy with the result already.

So thank you Callgrind devs, Jonathan for your blog entry and especially Kyle Fransham for his great tutorial which should be put on techbase imo! And in fact all others who helped me deserve a thank you was well. πŸ™‚

PS.: No this is not the large feature I was talking about last time, haven’t worked on that for a while. This is rather like bug fixing. πŸ˜‰

KGet is alive too

August 26, 2010

Alive and kicking

There haven’t been blogs on KGet for a long time, but KGet is alive too.

The last few months and weeks were quite busy for us, though we still managed to fix a lot of bugs and 4.5.1 will also see some further fixes. Still there are some serious bugs left that I hope to be able to track down with our users, since sometimes bugs aren’t reproduceable for me.

Besides the bug fixes some ui improvements have entered KGet that should make it more comfortable to use. Now if you resize the header of a list its size and position will be restored the next time you look at the list. This also holds true for some dialogs. Sometimes small details really can change a lot.

KDE Brainstorm

In any case this post is also about a new feature I am working on. I stumbled upon an idea on KDE Brainstorm that made quite some sense. You can set up KGet to monitor your clipboard and add urls automatically as downloads. The main problem is that any url — that is not a local file — would be taken.

What I added is a combined whitelist and blacklist. The user only has to add some rules, and the order of those rules defines their priority. Rules higher in the list have a higher priority than those being lower. Also supporting both Wildcards and RegExp makes it possible to realise some interesting usecases. In fact the added rules can also be edited later on, if the user wants to change them.

At first KGet will look if the url is not a local file, has a protocoll etc. and will then continue with your set rules, in our case it will automatically start downloads for any rar or zip file unless it is from http://kde*

A backdraw you can see in the screenshot is that “Add” has no shortcut — it is a default button provided by KDE — though this is not that bad, you simply enter the pattern and press return and then you can enter the next item.

As you can see I also changed the advanced configuration page a bit, I guess it still needs some work though.

The most interesting part imo is the code though. The feature was quite easy to implement but it is the GUI that took most of the time. Yeah 95% of the new code are just for GUI, not that this code was hard, since it is always the same: 1. create a model 2. create a delegate 3. add means to add/remove data

That really shows some of the strengths of ignoring GUI altogether and resorting on config files. Yet that wouldn’t be “user-friendly” nor discoverable, so we keep it at that. πŸ˜‰ And I really try to make things useable and especially safe. “Safe” that it is hard to make wrong stuff, no adding empty items, editing can’t result in empty items etc. etc. Imo this is one of the keys to make features discoverable. If a user fears that any “wrong” click could destroy something they won’t even look for features that could be of use for them. Often my target is to have a KMines with as few mines as possible. πŸ˜€

I also have another new — very large — feature in my pipeline though I’ll work on it more before disclosing it here.

Git bisect

January 25, 2010

KDE is supposed to move over to git and there has been lots of discussion and blog posts about that lately. Now it is on me to provide a small post myself. Those that know git bisect can skip this one.

Git bisect is a great tool that saved my day several times already and again just today, shortly before the release of KDE 4.4 SC. In short git bisect is there to find faulty commits.

Often when developing you encounter bugs where you know that it worked many revisions ago, but how do you find the revision that introduced the bug? When using git a few commands are enough:

  1. start it: git bisect start
  2. tell git which version is bad: git bisect bad {sha1/tag}?
  3. tell git which version is good: git bisect good {sha1/tag}?
  4. end it: git bisect reset

Basically that is all you need to know and here is an example session (I was at master):

  1. git bisect start
  2. git bisect bad //latest commit is bad
  3. git bisect good 73ea4b2fd5ae39993009dd765c6ff562ceec09da//this commit is good
  4. XY revisions left to test after this (roughly Z steps)
  5. recompiling
  6. testing
  7. either git bisect bad or git bisect good depending if it did not work or if it did work
  8. go to 4 or 9
  9. d4f650537917441fcfd3aa71e0c646b8fc7464ec is the first bad commit//yeah it was me who did crap πŸ˜‰
  10. git reset
  11. Fix and commit

That process is very fast as git uses bisecting as the name suggests e.g. there are 100 commits, 1 is working 100 not, now it checks commit 50, if it works it is checking commit 75 otherwise commit 25 …

Whee and another really stupid and bad bug that was only triggered ocassionally bites the dust.

Baseline? Using git bisect when a commit is wrong or even better use test-cases to avoid wrong commits in the first place.

And another KGet entry

November 9, 2009

There have been some blog posts on KGet and now it is my turn to add another one. πŸ™‚

The last few weeks/months I kept polishing all the changes I did for KGet during GSOC and also introduced new stuff. It is fantastic to see how KGet improved the last few months with all the work we (the KGet team) put into it.

Speeding up downloads

Multisource-Downloading worked and still can work a way that a file is split into segments, then a TransferDataSource is assigned a segment and then downloads it. What that means is that whenever a TransferDataSource finishes the download of a segment it is assigned a different segment and connects again to the server … In fact that is not ideal as connecting to the server takes time, so I changed that.

Now multiple segments e.g. segment 1 to 10 can be assigned to a TransferDataSource, thus the connection is not always closed and recreated once a segment is finished, but only closes once the segment range finished resulting in less resource usage and faster downloads. [1]

Whenever the user decides to use more connections per mirror or when they add another mirror the TransferDataSource that has most undowloaded segments splits the range.


One of those polishing changes I did was to use threads when creating checksums, like when a download is being verified, or when you create a metalink and chose to automatically create checksums. Now these calls are not blocking anymore and you can control KGet during this kind of operations normally.

I also experimented with OpenMP and optimizing the loading of files when calculating the checksums, though the very small speedup does not justify the changes in the code, so it won’t be there.


Other than that I implemented PGP signature checking.

The user has the posibility to enter a signature or in case of a metalink this happens automatically — in fact only if the signature is embed in the metalink. If there is no key for that signature then the user is asked if the key should be searched for. It can even be specified (in the Preferences) in which order the servers should be tried.

Verification preferences

Yes, now all that can be configured. Btw. “Strong” is default.

Ok, so what does it all look like, you may ask? First I changed the transfer settings dialog, to give the user a better overview of what the situation is like for all the files of a transfer, which ones have been verified etc.


The verification dialog now shows which checksums have been verified (in the screenshot I used the “Strongest” option):

Enough talking about side aspects, here is the signature-dialog. I tried to use icons where feasable — to help the user find what could be a problem — it will see some improvments in the following weeks, though I think it is in a releasable state already.

In fact the newly added keys will also appear in KGpg or Kleopatra or any other program using gpg one way or the other, as I’m using gpgme++, btw. thanks to the everyone who worked on gpgme++ makes life way easier than gpgme [3] and thanks to the people who helped me on the ml.

Before you ask so far I do not plan to add support for decrypting, imo you should use other tools for that than a download program.


I also fixed some bugs, now that annoying one where the details of transfers would still be shown even if they were (re)moved is finally gone.

Lukas also worked a lot on KGet like changing the basic model that is used in the view etc. (up to him to blog about that πŸ˜‰ ) and Dario tracked down some rare crashes via using QTests.

On Bugs, if you find some in trunk please report them so that we can fix them before 4.4.


All in all I have the feeling that the next KGet release (with KDE 4.4) will be a great one. That also means that you are encouraged to test KGet from trunk and report bugs and that I’m encouraged to try fixing them in the weeks to follow. πŸ™‚

PS.: This blog post has been in the work for quite a while, but rl caught up,

[1] Thanks to the people on #kde-devel who discussed with me on this issue! And thanks in general for all the help you gave me on the several #kde channels. πŸ™‚

[2] Btw. do you know an easy way to find a good size for a dialog? E.g. all columns in a view should not be smaller than their prefered size, when there is enough space the dialog should expand to show everything –> the dialog you see in the screenshot has been manually resized by me, otherwise it would not look that way.

[3] In search of what I thought was a bug I reprogrammed large parts with gpgme (yummy, error handling with c-libs *turns crazy*), later on it turned out that I just misunderstood something. Meh!

GSOC KGet — wrapup

August 18, 2009

Now that GSOC is over everyone expects to see the results, so here you go. πŸ™‚

The last few weeks were mostly filled with improving the newly added features and adding some small features here and there. I was able to fix lots of bugs in my code I sometimes experienced and hope that the user experience won’t be that bad. πŸ˜€

KGet now supports:

  • multisource downloading
  • changing the destination of a download while downloading
  • adding/removing mirrors to downloads while you download and changing the number of connections to the mirrors
  • adding checksums to downloads and manually verifiying them
  • automatically searching for checksums on the server (by e.g. appending .md5 to the url)
  • automatically using present checksums to verify a download that finished
  • repairing a broken download (redownloading foul parts or whole download)
  • Downloading a metalink you can define which files should be downloaded and which not, can be changed later in the transfer settings
  • some data is passed from Metalink to Nepomuk, like Publisher etc.
  • MetalinkCreator –> an assistant to create metalinks, currently it is based on the most recent Metalink Draft (v. 12 — not all parts are supported in the GUI yet) so it is bound to change — I’ll probably show a video of the metalinkcreator in action once I have time
  • the parser used to parse the metalink files does also work with “old” metalink-files (v. 3.0 2nd edition) so it could be used to convert them to the new format

Having digital signatures support and a Bittorrent-TransferDataSource did not made it in though.

That’s a wrapup of what I did, I probably forgot a lot as it has been so much code: the diff has ~18.000 lines, more than 10.000 lines have been added a lot of comments, ui stuff and some code ;). [1] Interested people can look at the code here [2], an instruction can be found on the KGet-Ml in the “GSOC — Review”-thread. And yes, in fact it is planned to push all that to trunk, currently the code is being reviewed and changes are made to it.

What will the future bring? I am going to continue to work on KGet and plan to add some features, though the pace of changes will be a little slower now.

Thanks to my mentor Urs and to Lukas who helped me on my way along.

PS.: I’m a lazy blogger. I like writing code more than writing about writing code.

[1] Yeah, I’m proud of my work. πŸ™‚ I didn’t think that I would end up writing so much.


GSOC status

July 6, 2009

Downloads other than bittorrent can have multiple files now, metalink uses this, I did a file-model for that.

So now you can

  • rename files of a download while you download
  • change the download location while you download
  • add/remove mirrors and change the number of connections to the mirrors while you download (bittorrent not supported yet)
  • automatic search for supported checksums (e.g. md5, sha1 etc.) –> that is an own plugin
  • manually adding checksums to a file
  • automatic verification of the downloads (no gui yet)
  • I guess most work went into datasourcefactory to make all of the things mentioned above possible. Also much I added so far is only used by some plugins or is not shown in the gui, but I’ll change that in the future.

    The following screenshots (is there something more interesting? O_o) show some gui-parts I did, feedback on them is welcome and appreciated!


    The transfersettings showing all files of the download. NOTE: The checkboxes do nothing at the moment

    I used a metalink I manually created in the example above (the greyed out parts were there before) – most metalinks are for large files and I got sick of downloading a cd-image every time I test something πŸ˜€ – the metalink-plugin uses some of the xml-tags of the metalink-specification, though many are still missing.

    Btw. I renamed “Current Destination” to “Download Destination” to make it more clear. There you can move downloads to different place. The folder structure (if there was one created) is not removed though as the user might have added files to these folders. Maybe I’ll make make that more clever in the future (most likely after GSOC), like a check if there are any non-download files in these directories.


    You can manually add/remove mirrors or change the number of simultanous connections to mirror


    Rename files whenever you want

    after rename

    Choose a useful ending πŸ˜‰


    Manually add verification information or verify the file if you want

    In fact I did also lots of testing and found many errors in my code, depending on the changes some testing was useless a few commits later – but I like if everything is compileable and works mor or less. So I probably downloaded more than 20 GB just for testing. That does not mean that there are no bugs or problems though e.g. downloading is slow atm.

    PS.: git rules
    PPS.: git rebase -i rules even more
    PPPS.: Thanks for the help you gave me on the irc channels!

    Hi planet!

    July 6, 2009

    Hi there,

    my name is Matthias Fuchs and I am working on KGet as my GSOC-project. I know GSOC has started a “few” weeks ago but rest assured I started before GSOC started, I’m just the incarnation of a lazy blogger.

    Applying to GSOC was a very sponatous decision, on the one hand I came to know on short notice that I had time in the summer and on the other hand I was in search for a new challenge as I consider the comic applet — that introduced me to KDE developement — mostly feature complete (imo!) now. I know there are some bugs as well as feature requests — most of these are also on todo lists I did months ago — left and I’ll look into them when I have time.

    Good, so what will I work on, what did I work on? The most important part will be the implementation of a better metalink-support and thus implement adding additional URLs to downloads to speed them up, multi segment downloads, “repairing” (redownloading foul parts) of downloads via using md5 or sha1 etc. Some of these parts are already there, some need more work on them and others need to be implemented from scratch.

    As soon as I applied I started coding on some of the small ideas I had, these were mostly improvements to the GUI, shortcomings I experienced as a user.
    I improved the “Import Links” dialog — filtering works really nice now — activated the Nepomuk support [1] and improved it. Now you can not only tag and rate files (that was mostly there but not activated) but also add custom tags and tags to your download groups.
    The other things I changed were mostly minor things.

    PS.: This is my first blog entry ever (!), in general I do not like or at least I think that I do not like writing such kind of stuff.
    PPS.: The next blog entry will contain what I already did up to this point, parts of this entry have been written weeks ago and I simply was not motivated to publish it (see the PS above πŸ˜‰ )

    [1] Many parts were there thanks to boom1992 but it constantly crashed, I “tracked” that down to the CMakeList.txt file, though I have to admit that happened more on coincidence than on knowledge (still do not know why it did not work and I fear that some problems are still left)