I did some lazy-man parallel processing (we’re talking many seperate text files, wgets, data extraction/processing) years ago with bash, very bad practice but it it involved grep-ing the output of ps aux in a loop with a wait period, only firing off new tasks when the number of active tasks was below a certain threshold, and this number could be altered depending on how big a box it was running on The whole thing was dreadfully slow but it worked, and at the time it was only supposed to be used once, so I cared more about the result than the process I re-did the whole thing in python afterwards, and although that was multi-threaded it ran many times faster on much less powerful hardware
2 Comments
You have brought up a very great points, thanks for the post
I did some lazy-man parallel processing (we’re talking many seperate text files, wgets, data extraction/processing) years ago with bash, very bad practice but it it involved grep-ing the output of ps aux in a loop with a wait period, only firing off new tasks when the number of active tasks was below a certain threshold, and this number could be altered depending on how big a box it was running on The whole thing was dreadfully slow but it worked, and at the time it was only supposed to be used once, so I cared more about the result than the process I re-did the whole thing in python afterwards, and although that was multi-threaded it ran many times faster on much less powerful hardware
Leave A Comment