https://pinboard.in/u:rahuldave/public/ recent bookmarks from rahuldave 2012-05-22T17:04:50+00:00 http://python3porting.com/improving.html rahuldavepython programming https://pinboard.in/ https://pinboard.in/u:rahuldave/b:76a958c1a8a8/ 2012-05-03T14:44:49+00:00 http://www.johnmyleswhite.com/notebook/2012/05/03/cumplyr-extending-the-plyr-package-to-handle-cross-dependencies/ rahuldave= Value 1 AND Variable 2 >= Value 2, etc. This allows us to implement the backward-moving mean described earlier. Using Norm Balls Finally, we can consider a combination of upper and lower bounds. For simplicity, we'll assume that these bounds have a fixed tightness around the "center" of each subset of our split data. To articulate this tightness formally, we look at a specific hypothetical equality constraint like Variable 1 = Value 1 and then loosen it so that norm(Variable 1 - Value 1) <= r. When r = 0, this system gives the original equality constraint. But when r > 0, we produce a "ball" of data around the constraint whose tightness is r. This lets us estimate the local means from our third example. Implementation To demo these ideas in a usable fashion, I've created a draft package for R called cumplyr. Here is an extended example of its usage in solving simple variants of the problems described in this post: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 library('cumplyr')   data <- data.frame(Time = 1:5, Value = seq(1, 9, by = 2))   iddply(data, equality.variables = c('Time'), lower.bound.variables = c(), upper.bound.variables = c(), norm.ball.variables = list(), func = function (df) {with(df, mean(Value))})   iddply(data, equality.variables = c(), lower.bound.variables = c('Time'), upper.bound.variables = c(), norm.ball.variables = list(), func = function (df) {with(df, mean(Value))})   iddply(data, equality.variables = c(), lower.bound.variables = c(), upper.bound.variables = c('Time'), norm.ball.variables = list(), func = function (df) {with(df, mean(Value))})   iddply(data, equality.variables = c(), lower.bound.variables = c(), upper.bound.variables = c(), norm.ball.variables = list('Time' = 1), func = function (df) {with(df, mean(Value))})   iddply(data, equality.variables = c(), lower.bound.variables = c(), upper.bound.variables = c(), norm.ball.variables = list('Time' = 2), func = function (df) {with(df, mean(Value))})   iddply(data, equality.variables = c(), lower.bound.variables = c(), upper.bound.variables = c(), norm.ball.variables = list('Time' = 5), func = function (df) {with(df, mean(Value))}) You can download this package from GitHub and play with it to see whether it helps you. Please submit feedback using GitHub if you have any comments, complaints or patches. Comparing plyr with cumplyr In the long run, I'm hoping to make the functions in cumplyr robust enough to submit a patch to plyr. I see these tools as one logical extension of plyr to encompass more of the framework described in Hadley's paper on the Split-Apply-Combine strategy. For the time being, I would advise any users of cumplyr to make sure that you do not use cumplyr for anything that plyr could already do. cumplyr is very much demo software and I am certain that both its API and implementation will change. In contrast, plyr is fast and stable software that can be trusted to perform its job. But, if you have a problem that cumplyr will solve and plyr will not, I hope you'll try cumplyr out and submit patches when it breaks. Happy hacking! ]]> Programming Statistics https://pinboard.in/u:rahuldave/b:a5482cf69a97/ 2012-05-03T07:05:00+00:00 http://radar.oreilly.com/2012/05/functional-languages-functional-techniques.html rahuldave Programming clojure codepodcast concurrency d3 f functionalprogramming java javascript node rprogramming scala https://pinboard.in/u:rahuldave/b:eaa46ecde100/ 2012-04-09T14:00:19+00:00 http://www.r-bloggers.com/comparing-julia-and-r%e2%80%99s-vocabularies/ rahuldave > Basics Comparison >= >= Basics Comparison < < Basics Comparison <= <= Basics Comparison is.na Basics Comparison is.nan Basics Comparison is.finite Basics Comparison complete.cases Basics Comparison * * Basics Basic Math + + Basics Basic Math - - Basics Basic Math / / Basics Basic Math ^ ^ Basics Basic Math %% mod (%%) Basics Basic Math %/% div Basics Basic Math abs abs Basics Basic Math sign sign Basics Basic Math acos acos Basics Basic Math acosh acosh Basics Basic Math asin asin Basics Basic Math asinh asinh Basics Basic Math atan atan Basics Basic Math atan2 atan2 Basics Basic Math atanh atanh Basics Basic Math sin sin Basics Basic Math sinh sinh Basics Basic Math cos cos Basics Basic Math cosh cosh Basics Basic Math tan tan Basics Basic Math tanh tanh Basics Basic Math ceiling ceil Basics Basic Math floor floor Basics Basic Math round round Basics Basic Math trunc trunc Basics Basic Math signif Basics Basic Math exp exp Basics Basic Math log log Basics Basic Math log10 log10 Basics Basic Math log1p log1p Basics Basic Math log2 log2 Basics Basic Math logb Basics Basic Math sqrt sqrt Basics Basic Math cummax Basics Basic Math cummin Basics Basic Math cumprod cumprod Basics Basic Math cumsum cumsum Basics Basic Math diff diff Basics Basic Math max max Basics Basic Math min min Basics Basic Math prod prod Basics Basic Math sum sum Basics Basic Math range Basics Basic Math mean mean Basics Basic Math median median Basics Basic Math cor cor_pearson Basics Basic Math cov cov_pearson Basics Basic Math sd std Basics Basic Math var var Basics Basic Math pmax Basics Basic Math pmin Basics Basic Math rle Basics Basic Math function function Basics Functions missing Basics Functions on.exit Basics Functions return return Basics Functions invisible Basics Functions & & Basics Logical & Set Operations | | Basics Logical & Set Operations ! ! Basics Logical & Set Operations xor Basics Logical & Set Operations all all Basics Logical & Set Operations any any Basics Logical & Set Operations intersect intersect Basics Logical & Set Operations union union Basics Logical & Set Operations setdiff Basics Logical & Set Operations setequal Basics Logical & Set Operations which find Basics Logical & Set Operations c [] ({}) Basics Vectors and Matrices matrix [] ({}) Basics Vectors and Matrices length size (length) Basics Vectors and Matrices dim size Basics Vectors and Matrices ncol size(x, 1) Basics Vectors and Matrices nrow size(x, 2) Basics Vectors and Matrices cbind hcat Basics Vectors and Matrices rbind vcat Basics Vectors and Matrices names Basics Vectors and Matrices colnames Basics Vectors and Matrices rownames Basics Vectors and Matrices t ‘ Basics Vectors and Matrices diag eye Basics Vectors and Matrices sweep Basics Vectors and Matrices as.matrix Basics Vectors and Matrices data.matrix Basics Vectors and Matrices c [] ({}) Basics Making Vectors rep Basics Making Vectors seq [from:by:to] Basics Making Vectors seq_along Basics Making Vectors seq_len [1:len] Basics Making Vectors rev reverse Basics Making Vectors sample Basics Making Vectors choose factorial Basics Making Vectors factorial factorial Basics Making Vectors combn Basics Making Vectors (is/as).(character/numeric/logical) Basics Making Vectors list HashTable ([]) Basics Lists & Data Frames unlist Basics Lists & Data Frames data.frame Basics Lists & Data Frames as.data.frame Basics Lists & Data Frames split Basics Lists & Data Frames expand.grid Basics Lists & Data Frames if if Basics Control Flow && && Basics Control Flow || || Basics Control Flow for for Basics Control Flow while while Basics Control Flow next continue Basics Control Flow break break Basics Control Flow switch Basics Control Flow ifelse Basics Control Flow fitted Statistics Linear Models predict Statistics Linear Models resid Statistics Linear Models rstandard Statistics Linear Models lm Statistics Linear Models glm Statistics Linear Models hat Statistics Linear Models influence.measures Statistics Linear Models logLik Statistics Linear Models df Statistics Linear Models deviance Statistics Linear Models formula Statistics Linear Models ~ Statistics Linear Models I Statistics Linear Models anova Statistics Linear Models coef Statistics Linear Models confint Statistics Linear Models vcov Statistics Linear Models contrasts Statistics Linear Models apropos(‘\\.test$’) Statistics Miscellaneous Statistical Tests beta beta Statistics Random Numbers binom binom Statistics Random Numbers cauchy cauchy Statistics Random Numbers chisq chisq Statistics Random Numbers exp exp Statistics Random Numbers f f Statistics Random Numbers gamma gamma Statistics Random Numbers geom geom Statistics Random Numbers hyper hyper Statistics Random Numbers lnorm lnorm Statistics Random Numbers logis logis Statistics Random Numbers multinom multinom Statistics Random Numbers nbinom nbinom Statistics Random Numbers norm norm Statistics Random Numbers pois pois Statistics Random Numbers signrank signrank Statistics Random Numbers t t Statistics Random Numbers unif unif (rand) Statistics Random Numbers weibull weibull Statistics Random Numbers wilcox wilcox Statistics Random Numbers birthday birthday Statistics Random Numbers tukey tukey Statistics Random Numbers crossprod * Statistics Matrix Algebra tcrossprod * Statistics Matrix Algebra eigen eig Statistics Matrix Algebra qr qr Statistics Matrix Algebra svd svd Statistics Matrix Algebra %*% * Statistics Matrix Algebra %o% Statistics Matrix Algebra outer Statistics Matrix Algebra rcond Statistics Matrix Algebra solve \ Statistics Matrix Algebra duplicated Statistics Ordering and Tabulating unique Statistics Ordering and Tabulating merge Statistics Ordering and Tabulating order Statistics Ordering and Tabulating rank Statistics Ordering and Tabulating quantile quantile Statistics Ordering and Tabulating sort sort Statistics Ordering and Tabulating table Statistics Ordering and Tabulating ftable Statistics Ordering and Tabulating ls whos Working with R Workspace exists Working with R Workspace get Working with R Workspace rm Working with R Workspace getwd getcwd Working with R Workspace setwd setcwd Working with R Workspace q Ctrl-D Working with R Workspace source load Working with R Workspace install.packages Working with R Workspace library Working with R Workspace require Working with R Workspace help help Working with R Help ? help Working with R Help help.search Working with R Help apropos Working with R Help RSiteSearch Working with R Help citation Working with R Help demo Working with R Help example Working with R Help vignette Working with R Help traceback Working with R Debugging browser Working with R Debugging recover Working with R Debugging options(error =) Working with R Debugging stop Working with R Debugging warning Working with R Debugging message Working with R Debugging tryCatch try/catch Working with R Debugging try try Working with R Debugging print print (println) I/O Output cat I/O Output message I/O Output warning I/O Output dput I/O Output format I/O Output sink I/O Output data I/O Reading and Writing Data count.fields I/O Reading and Writing Data read.csv csvread I/O Reading and Writing Data read.delim dlmread I/O Reading and Writing Data read.fwf I/O Reading and Writing Data read.table I/O Reading and Writing Data library(foreign) I/O Reading and Writing Data write.table dlmwrite I/O Reading and Writing Data readLines readlines I/O Reading and Writing Data writeLines I/O Reading and Writing Data load I/O Reading and Writing Data save I/O Reading and Writing Data readRDS I/O Reading and Writing Data saveRDS I/O Reading and Writing Data dir I/O Files and Directories basename I/O Files and Directories dirname I/O Files and Directories file.path I/O Files and Directories path.expand I/O Files and Directories file.choose I/O Files and Directories file.copy I/O Files and Directories file.create I/O Files and Directories file.remove I/O Files and Directories path.rename I/O Files and Directories dir.create I/O Files and Directories file.exists I/O Files and Directories tempdir I/O Files and Directories tempfile I/O Files and Directories download.file I/O Files and Directories ISOdate Special Data Date / Time ISOdatetime Special Data Date / Time strftime Special Data Date / Time strptime Special Data Date / Time date Special Data Date / Time difftime Special Data Date / Time julian Special Data Date / Time months Special Data Date / Time quarters Special Data Date / Time weekdays Special Data Date / Time library(lubridate) Special Data Date / Time grep match Special Data Character Manipulation agrep Special Data Character Manipulation gsub Special Data Character Manipulation strsplit split Special Data Character Manipulation chartr Special Data Character Manipulation nchar strlen Special Data Character Manipulation tolower Special Data Character Manipulation toupper Special Data Character Manipulation substr Special Data Character Manipulation paste join Special Data Character Manipulation library(stringr) Special Data Character Manipulation factor Special Data Factors levels Special Data Factors nlevels Special Data Factors reorder Special Data Factors relevel Special Data Factors cut Special Data Factors findInterval Special Data Factors interaction Special Data Factors options(stringsAsFactors = FALSE) Special Data Factors array [] Special Data Array Manipulation dim size Special Data Array Manipulation dimnames Special Data Array Manipulation aperm Special Data Array Manipulation library(abind) Special Data Array Manipulation I’d like to note that holes in the list of Julia functions can exist for several reasons: The language does not yet have the relevant features. This is true of things like factor() or data.frame(). The language has draft implementations of the relevant features, but they are not yet ready to make their way into this list. This is true of Doug Bates’ GLM code, for example. I simply don’t know what the Julia equivalent is for an R function, but it may well exist. If you know of one, please fork the GitHub repository I’m using and revise the CSV file appropriately. I’ll integrate relevant pull requests as soon as I can find time. In addition to explaining the presence of the many holes you can see this in this list, I’d also like to note how quickly these holes are being filled in: Doug Bates already finished a wrapper for the Rmath library, which means that Julia now has tools for calculating the PDF’s, CDF’s, and inverse CDF’s of most statistical distributions as well as the ability to draw random samples from them. That means that almost any sort of MCMC you’d like to do is already possible in Julia. (I, for one, am really interested to see if someone will use Julia’s sparse matrix support and these new Rmath functions to build MCMC code that’s easy on the eyes while also running at an appropriately fast speed on complicated, big data problems like matrix factorizations.) On my end, I’ve been working on filling some of the missing entries in this list by adding in pieces that I think I understand well enough to implement from scratch, such as: Optimization algorithms (optim.jl): Simulated annealing Gradient descent Newton’s method Statistical hypothesis tests (stats.jl): t-Tests Utility functions (utils.jl): range keys cummax cummin To leave a comment for the author, please follow the link and comment on his blog: John Myles White » Statistics. R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series,ecdf, trading) and more... ]]> R_bloggers programming statistics https://pinboard.in/u:rahuldave/b:4a08330142b5/ 2012-03-02T20:53:01+00:00 http://feedproxy.google.com/~r/oreilly/radar/atom/~3/0RJmqXtrung/profile-of-the-data-journalist-2.html rahuldave Data Gov_2.0 Publishing dataconference datajournalism dataproduct datascience nicarinterview opensource programming https://pinboard.in/u:rahuldave/b:c4879a52616b/ 2012-02-23T03:48:00+00:00 http://www.johndcook.com/blog/2012/02/22/julia-random-number-generation/ rahuldave Software_development Julia Programming https://pinboard.in/u:rahuldave/b:d487208e6c8b/ 2012-02-02T18:07:10+00:00 http://code.google.com/edu/languages/google-python-class/introduction.html rahuldavegoogle python programming tutorial https://pinboard.in/ https://pinboard.in/u:rahuldave/b:073050eb4a0a/ 2012-01-03T15:13:13+00:00 http://www.25hoursaday.com/weblog/2012/01/03/WhatILearnedAfter3WeeksOfWritingMobileApps.aspx rahuldave Programming Web_Development https://pinboard.in/u:rahuldave/b:6eb08c4daf69/ 2012-01-02T20:24:35+00:00 http://eflorenzano.com/blog/2012/01/01/reducing-code-nesting/ rahuldave programming https://pinboard.in/u:rahuldave/b:52d9e585aa69/ 2011-12-28T11:00:00+00:00 http://radar.oreilly.com/2011/12/four-short-links-28-december-2-1.html rahuldave cloud javascript opensource programming search storage textanalysis web https://pinboard.in/u:rahuldave/b:f4b01c498a92/ 2011-12-28T11:00:00+00:00 http://feedproxy.google.com/~r/oreilly/radar/atom/~3/jLZhw5uJ1Ms/four-short-links-28-december-2-1.html rahuldave cloud javascript opensource programming search storage textanalysis web https://pinboard.in/u:rahuldave/b:8604fc8a64ab/ 2011-11-30T18:33:00+00:00 http://rss.slashdot.org/~r/slashdot/eqWf/~3/65pGuqnXP-o/why-was-hypercard-killed rahuldave programming https://pinboard.in/u:rahuldave/b:7b9cc06b0de3/ 2011-11-28T22:08:02+00:00 http://www.johndcook.com/blog/2011/11/28/fundamental-theorem-of-readability/ rahuldave Software_development Books Programming https://pinboard.in/u:rahuldave/b:fdd9ecb95dae/ 2011-11-14T18:47:29+00:00 http://www.johndcook.com/blog/2011/11/14/separating-presentation-from-content/ rahuldave Software_development LaTeX Programming https://pinboard.in/u:rahuldave/b:dac78de91c8a/ 2011-10-21T15:36:00+00:00 http://rss.slashdot.org/~r/slashdot/eqWf/~3/BAdyB9M-hGw/microsoft-roslyn-reinventing-the-compiler-as-we-know-it rahuldave programming https://pinboard.in/u:rahuldave/b:5042ae9c6746/ 2011-09-27T15:56:20+00:00 http://www.johndcook.com/blog/2011/09/27/sed-one-liners/ rahuldave Software_development Books Programming Sed https://pinboard.in/u:rahuldave/b:f6b7a7a6770b/ 2011-09-20T22:18:00+00:00 http://rss.slashdot.org/~r/slashdot/eqWf/~3/pHX0S2EU4ZU/Client-side-Web-REPL-For-15-Languages rahuldave programming https://pinboard.in/u:rahuldave/b:a619c38aa3d9/ 2011-04-19T12:03:15+00:00 http://www.johndcook.com/blog/2011/04/19/learn-one-sed-command/ rahuldave newfile.txt This will replace every instance of pattern1 with pattern2 in the file file.txt and will write the result to newfile.txt. The original file file.txt is unchanged. I used to think there was no reason to use sed when other languages like Python will do everything sed does and much more. Suppose you agree with that. Now suppose you find you often have to make global search-and-replace operations and so you write a script to do this, say a Python script. You’ve got to call your script something, remember what you called it, and put it in your path. How about calling it sed? Or better, don’t write your script, but pretend that you did. If you’re on Linux, it’s already in your path. One advantage of the real sed over your script named sed is that the former can do a lot more, should you ever need it to. Now for a few details regarding the sed command above. The “s” on the front stands for “substitute” and the “g” on the end stands for “global.” Without the “g” on the end, sed would only replace the first instance of the pattern on each line. If that’s what you want, then remove the “g.” The patterns inside a sed command are regular expressions, so it’s best to get in the habit of always quoting sed commands. This isn’t necessary for simple string substitutions, but regular expressions often contain characters that you’ll need to prevent the shell from interpreting. You may find the default regular expression support in sed odd or restrictive. If you’re used to regular expressions in Perl, Python, JavaScript, etc. and you’re using a Gnu implementation of sed, you can add the -r option for more familiar regular expression syntax. I got the idea for this post from Greg Grouthaus’ post Why you should learn just a little Awk. He makes a good case that you can benefit from learning just a few commands of a language like Awk with no intention to learn more of the language. Related posts: Good old regular expressions Tips for learning regular expressions A little awk ]]> Software_development Programming Regular_expressions https://pinboard.in/u:rahuldave/b:e2635f8f5b1a/ 2011-01-04T22:00:00+00:00 http://lifehacker.com/5724763/kod-is-a-free-text-editor-design-for-programmers rahuldave Downloads Mac_OS_X Mac_OS_X_Featured_Download Programming Text_Editors https://pinboard.in/u:rahuldave/b:cf495de5a594/ 2010-12-22T16:00:00+00:00 http://radar.oreilly.com/2010/12/how-will-the-elmcity-service-s.html rahuldave Programming blog calendar elmcity feed syndication https://pinboard.in/u:rahuldave/b:42648bc61fae/ 2010-05-02T15:34:00+00:00 http://rss.slashdot.org/~r/slashdot/eqWf/~3/LdJlH4NcNtI/What-Every-Programmer-Should-Know-About-Floating-Point-Arithmetic rahuldave programming https://pinboard.in/u:rahuldave/b:7c05893d0464/ 2010-04-26T06:03:19+00:00 http://www.mailund.dk/index.php/2010/04/26/is-r-an-epic-fail/ rahuldave Work programming R statistics https://pinboard.in/u:rahuldave/b:a98065d24020/ 2010-04-21T17:45:00+00:00 http://feeds.arstechnica.com/~r/arstechnica/index/~3/tGM5tqWsxfY/tutorial-use-twitters-new-real-time-stream-api-in-python.ars rahuldave Features Guides Open-source Web programming python tutorial twitter https://pinboard.in/u:rahuldave/b:58ebd66b4b7c/ 2010-04-21T08:03:18+00:00 http://www.mailund.dk/index.php/2010/04/21/on-code-and-comments/ rahuldave Rants Work programming https://pinboard.in/u:rahuldave/b:16bdf90ddc02/ 2010-04-15T13:56:41+00:00 http://www.johndcook.com/blog/2010/04/15/85-functional-language-purity/ rahuldave Software_development Functional_programming Programming https://pinboard.in/u:rahuldave/b:dc9134e871b5/ 2010-04-05T10:00:00+00:00 http://radar.oreilly.com/2010/04/four-short-links-5-april-2010.html rahuldave brains community hacks opensource programming ui https://pinboard.in/u:rahuldave/b:cf8d106a2e8f/ 2010-04-01T16:42:22+00:00 http://www.chrishowie.com/2010/04/01/git-svn-in-the-workplace/ rahuldave Git Programming https://pinboard.in/u:rahuldave/b:04d0bf3c5f17/ 2010-03-24T06:00:57+00:00 http://feedproxy.google.com/~r/catonmat/~3/sy5RTytKBuI/ rahuldave This one-liner opens the so-far typed command in your favorite text editor for further editing. This is handy if you are typing a lengthier shell command. After you have done editing the command, quit from your editor successfully to execute it. To cancel execution, just erase it. If you quit unsuccessfully, the command you had typed before diving into the editor will be executed. Actually, I have to educate you, it’s not a feature of the shell per se but a feature of the readline library that most shells use for command line processing. This particular binding CTRL-x CTRL-e only works in readline emacs editing mode. The other mode is readline vi editing mode, in which the same can be accomplished by pressing ESC and then v. The emacs editing mode is the default in all the shells that use the readline library. The usual command to change between the modes is set -o vi to change to vi editing mode and set -o emacs to change back to emacs editing mode. To change the editor, export the $EDITOR shell variable to your preference. For example, to set the default editor to pico, type export EDITOR=pico. Another way to edit commands in a text editor is to use fc shell builtin (at least bash has this builtin). The fc command opens the previous edited command in your favorite text editor. It’s easy to remember the fc command because it stands for “fix command.” Remember the ^foo^bar^ command from the first top ten one-liners? You can emulate this behavior by typing fc -s foo=bar. It will replace foo with bar in the previous command and execute it. #12. Empty a file or create a new file $ > file.txt This one-liner either wipes the file called file.txt empty or creates a new file called file.txt. The shell first checks if the file file.txt exists. If it does, the shell opens it and wipes it clean. If it doesn’t exist, the shell creates the file and opens it. Next the shell proceeds to redirecting standard output to the opened file descriptor. Since there is nothing on the standard output, the command succeeds, closes the file descriptor, leaving the file empty. Creating a new empty file is also called touching and can be done by $ touch file.txt command. The touch command can also be used for changing timestamps of the commands. Touch, however, won’t wipe the file clean, it will only change the access and modification timestamps to the current time. #13. Create a tunnel from localhost:2001 to somemachine:80 $ ssh -N -L2001:localhost:80 somemachine This one-liner creates a tunnel from your computer’s port 2001 to somemachine’s port 80. Each time you connect to port 2001 on your machine, your connection gets tunneled to somemachine:80. The -L option can be summarized as -L port:host:hostport. Whenever a connection is made to localhost:port, the connection is forwarded over the secure channel, and a connection is made to host:hostport from the remote machine. The -N option makes sure you don’t run shell as you connect to somemachine. To make things more concrete, here is another example: $ ssh -f -N -L2001:www.google.com:80 somemachine This one-liner creates a tunnel from your computer’s port 2001 to www.google.com:80 via somemachine. Each time you connect to localhost:2001, ssh tunnels your request via somemachine, where it tries to open a connection to www.google.com. Notice the additional -f flag - it makes ssh daemonize (go into background) so it didn’t consume a terminal. #14. Reset terminal $ reset This command resets the terminal. You know, when you have accidentally output binary data to the console, it becomes messed up. The reset command usually cleans it up. It does that by sending a bunch of special byte sequences to the terminal. The terminal interprets them as special commands and executes them. Here is what BusyBox’s reset command does: printf("\033c\033(K\033[J\033[0m\033[?25h"); It sends a bunch of escape codes and a bunch of CSI commands. Here is what they mean: \033c: “ESC c” - sends reset to the terminal. \033(K: “ESC ( K” - reloads the screen output mapping table. \033[J: “ESC [ J” - erases display. \033[0m: “ESC [ 0 m” - resets all display attributes to their defaults. \033[?25h: “ESC [ ? 25 h” - makes cursor visible. #15. Tweet from the shell $ curl -u user:pass -d status='Tweeting from the shell' http://twitter.com/statuses/update.xml This one-liner tweets your message from the terminal. It uses the curl program to HTTP POST your tweet via Twitter’s API. The -u user:pass argument sets the login and password to use for authentication. If you don’t wish your password to be saved in the shell history, omit the :pass part and curl will prompt you for the password as it tries to authenticate. Oh, and while we are at shell history, another way to omit password from being saved in the history is to start the command with a space! For example, curl ... won’t save the curl command to the shell history. The -d status='...' instructs curl to use the HTTP POST method for the request and send status=... as POST data. Finally, http://twitter.com/statuses/update.xml is the API URL to POST the data to. Talking about Twitter, I’d love if you followed me on Twitter! :) #16. Execute a command at midnight $ echo cmd | at midnight This one-liner sends the shell command cmd to the at-daemon (atd) for execution at midnight. The at command is light on the execution-time argument, you may write things like 4pm tomorrow to execute it at 4pm tomorrow, 9pm next year to run it on the same date at 9pm the next year, 6pm + 10 days to run it at 6pm after 10 days, or now +1minute to run it after a minute. Use atq command to list all the jobs that are scheduled for execution and atrm to remove a job from the queue. Compared to the universally known cron, at is suitable for one-time jobs. For example, you’d use cron to execute a job every day at midnight but you would use at to execute a job only today at midnight. Also be aware that if the load is greater than some number (for one processor systems the default is 0.8), then atd will not execute the command! That can be fixed by specifying a greater max load to atd via -l argument. #17. Output your microphone to other computer’s speaker $ dd if=/dev/dsp | ssh username@host dd of=/dev/dsp The default sound device on Linux is /dev/dsp. It can be both written to and read from. If it’s read from then the audio subsystem will read the data from the microphone. If it’s written to, it will send audio to your speaker. This one-liner reads audio from your microphone via the dd if=/dev/dsp command (if stands for input file) and pipes it as standard input to ssh. Ssh, in turn, opens a connection to a computer at host and runs the dd of=/dev/dsp (of stands for output file) on it. Dd of=/dev/dsp receives the standard input that ssh received from dd if=/dev/dsp. The result is that your microphone gets output on host computer’s speaker. Want to scare your colleague? Dump /dev/urandom to his speaker by dd if=/dev/urandom. #18. Create and mount a temporary RAM partition # mount -t tmpfs -o size=1024m tmpfs /mnt This command creates a temporary RAM filesystem of 1GB (1024m) and mounts it at /mnt. The -t flag to mount specifies the filesystem type and the -o size=1024m passes the size sets the filesystem size. If it doesn’t work, make sure your kernel was compiled to support the tmpfs. If tmpfs was compiled as a module, make sure to load it via modprobe tmpfs. If it still doesn’t work, you’ll have to recompile your kernel. To unmount the ram disk, use the umount /mnt command (as root). But remember that mounting at /mnt is not the best practice. Better mount your drive to /mnt/tmpfs or a similar path. If you wish your filesystem to grow dynamically, use ramfs filesystem type instead of tmpfs. Another note: tmpfs may use swap, while ramfs won’t. #19. Compare a remote file with a local file $ ssh user@host cat /path/to/remotefile | diff /path/to/localfile - This one-liner diffs the file /path/to/localfile on local machine with a file /path/to/remotefile on host machine. It first opens a connection via ssh to host and executes the cat /path/to/remotefile command there. The shell then takes the output and pipes it to diff /path/to/localfile - command. The second argument - to diff tells it to diff the file /path/to/localfile against standard input. That’s it. #20. Find out which programs listen on which TCP ports # netstat -tlnp This is an easy one. Netstat is the standard utility for listing information about Linux networking subsystem. In this particular one-liner it’s called with -tlnp arguments: -t causes netstat to only list information about TCP sockets. -l causes netstat to only list information about listening sockets. -n causes netstat not to do reverse lookups on the IPs. -p causes netstat to print the PID and name of the program to which the socket belongs (requires root). To find more detailed info about open sockets on your computer, use the lsof utility. See my article “A Unix Utility You Should Know About: lsof” for more information. That’s it for today. Tune in the next time for “Another Ten One-Liners from CommandLineFu Explained”. There are many more nifty commands to write about. But for now, have fun and see ya! PS. Follow me on twitter for updates! ]]> Programming at atd atq atrm audio commandlinefu cron csi_command curl daemon dd diff dsp echo editor emacs escape_code fc http if localhost microphone mount netstat of pico post ram ramfs readline redirect reset shell ssh standard_output tcp terminal tmpfs tunnel tweet twitter vi https://pinboard.in/u:rahuldave/b:78b0d0d1bd96/ 2010-03-23T23:13:45+00:00 http://rjlipton.wordpress.com/2010/03/23/its-ada-lovelace-day/ rahuldave History Ada_Lovelace computing programming women https://pinboard.in/u:rahuldave/b:f63c5389257b/ 2010-03-18T03:00:21+00:00 http://feedproxy.google.com/~r/catonmat/~3/GJRqxzmBW9c/ rahuldave> ~/.ssh/authorized_keys This one-liner saves a great deal of typing. Actually I just found out that there was a shorter way to do it: your-machine$ ssh remote-machine 'cat >> .ssh/authorized_keys' < .ssh/identity.pub #10. Capture video of a linux desktop $ ffmpeg -f x11grab -s wxga -r 25 -i :0.0 -sameq /tmp/out.mpg A pure coincidence, I have done so much video processing with ffmpeg that I know what most of this command does without looking much in the manual. The ffmpeg generally can be descibed as a command that takes a bunch of options and the last option is the output file. In this case the options are -f x11grab -s wxga -r 25 -i :0.0 -sameq and the output file is /tmp/out.mpg. Here is what the options mean: -f x11grab makes ffmpeg to set the input video format as x11grab. The X11 framebuffer has a specific format it presents data in and it makes ffmpeg to decode it correctly. -s wxga makes ffmpeg to set the size of the video to wxga which is shortcut for 1366×768. This is a strange resolution to use, I’d just write -s 800x600. -r 25 sets the framerate of the video to 25fps. -i :0.0 sets the video input file to X11 display 0.0 at localhost. -sameq preserves the quality of input stream. It’s best to preserve the quality and post-process it later. You can also specify ffmpeg to grab display from another x-server by changing the -i :0.0 to -i host:0.0. If you’re interested in ffmpeg, here are my other articles on ffmpeg that I wrote while ago: How to Extract Audio Tracks from YouTube Videos Converting YouTube Flash Videos to a Better Format with ffmpeg PS. This article was so fun to write, that I decided to write several more parts. Tune in the next time for “The Next Top Ten One-Liners from CommandLineFu Explained” :) Have fun. See ya! PSS. Follow me on twitter for updates. ]]> Programming authorized_keys bash cd combinatorics commandlinefu cp desktop display event_designators ffmpeg history identity.pub id_rsa.pub linux mtr oldpwd one_liners passwordless_authentication ping public_key_authentication python pythonpath root sets shell simplehttpserver ssh ssh_copy_id ssh_keygen sshv1 sshv2 sudo tee traceroute vim x11 https://pinboard.in/u:rahuldave/b:eb42c63da138/ 2010-03-17T02:03:00+00:00 http://rss.slashdot.org/~r/slashdot/eqWf/~3/0BqHlxmfoNk/Simpler-Hello-World-Demonstrated-In-C rahuldave programming https://pinboard.in/u:rahuldave/b:6abd35bb64d9/